rpms/nftables
5
0
Fork 0
Code Issues Pull Requests Releases Activity

nftables-1.0.4-2.el9

Browse Source 
* Fri Jun 24 2022 Phil Sutter <psutter@redhat.com> [1.0.4-2.el9]
- intervals: Do not sort cached set elements over and over again (Phil Sutter) [1917398]
- intervals: do not empty cache for maps (Phil Sutter) [1917398]
- intervals: do not report exact overlaps for new elements (Phil Sutter) [1917398]
- rule: collapse set element commands (Phil Sutter) [1917398]
- tests: shell: runtime set element automerge (Phil Sutter) [1917398]
Resolves: rhbz#1917398
This commit is contained in:
Phil Sutter 2022-06-24 16:56:40 +02:00
parent 98611e7b9d
commit 9a240b84de
6 changed files with 629 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

97
0001-tests-shell-runtime-set-element-automerge.patch Normal file
View File

@ -0,0 +1,97 @@
From 2a4f76ff5d5cc8d26a663ef8f8cd79c06560ca24 Mon Sep 17 00:00:00 2001
From: Phil Sutter <psutter@redhat.com>
Date: Fri, 24 Jun 2022 16:02:59 +0200
Subject: [PATCH] tests: shell: runtime set element automerge
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1917398
Upstream Status: nftables commit 8fafe4e6b5b30
commit 8fafe4e6b5b30f2539f16403da8d5c5f819e523b
Author: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Mon Jun 13 17:05:22 2022 +0200
tests: shell: runtime set element automerge
Add a test to cover runtime set element automerge.
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Signed-off-by: Phil Sutter <psutter@redhat.com>
---
tests/shell/testcases/sets/automerge_0 | 64 ++++++++++++++++++++++++++
1 file changed, 64 insertions(+)
create mode 100755 tests/shell/testcases/sets/automerge_0
diff --git a/tests/shell/testcases/sets/automerge_0 b/tests/shell/testcases/sets/automerge_0
new file mode 100755
index 0000000..c9fb609
--- /dev/null
+++ b/tests/shell/testcases/sets/automerge_0
@@ -0,0 +1,64 @@
+#!/bin/bash
+
+set -e
+
+RULESET="table inet x {
+ set y {
+ type inet_service
+ flags interval
+ auto-merge
+ }
+}"
+
+$NFT -f - <<< $RULESET
+
+tmpfile=$(mktemp)
+echo -n "add element inet x y { " > $tmpfile
+for ((i=0;i<65535;i+=2))
+do
+ echo -n "$i, " >> $tmpfile
+ if [ $i -eq 65534 ]
+ then
+ echo -n "$i" >> $tmpfile
+ fi
+done
+echo "}" >> $tmpfile
+
+$NFT -f $tmpfile
+
+tmpfile2=$(mktemp)
+for ((i=1;i<65535;i+=2))
+do
+ echo "$i" >> $tmpfile2
+done
+
+tmpfile3=$(mktemp)
+shuf $tmpfile2 > $tmpfile3
+i=0
+cat $tmpfile3 | while read line && [ $i -lt 10 ]
+do
+ $NFT add element inet x y { $line }
+ i=$((i+1))
+done
+
+for ((i=0;i<10;i++))
+do
+ from=$(($RANDOM%65535))
+ to=$(($from+100))
+ $NFT add element inet x y { $from-$to }
+ if [ $? -ne 0 ]
+ then
+ echo "failed to add $from-$to"
+ exit 1
+ fi
+ $NFT get element inet x y { $from-$to }
+ if [ $? -ne 0 ]
+ then
+ echo "failed to get $from-$to"
+ exit 1
+ fi
+done
+
+rm -f $tmpfile
+rm -f $tmpfile2
+rm -f $tmpfile3
--
2.36.1

236
0002-rule-collapse-set-element-commands.patch Normal file
View File

@ -0,0 +1,236 @@
From 0fb0e506d01f99548dbb9cabfef713bea7e447b5 Mon Sep 17 00:00:00 2001
From: Phil Sutter <psutter@redhat.com>
Date: Fri, 24 Jun 2022 16:02:59 +0200
Subject: [PATCH] rule: collapse set element commands
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1917398
Upstream Status: nftables commit 498a5f0c219d8
commit 498a5f0c219d8a118af4f172f248647d9b077101
Author: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Mon Jun 13 17:22:44 2022 +0200
rule: collapse set element commands
Robots might generate a long list of singleton element commands such as:
add element t s { 1.0.1.0/24 }
...
add element t s { 1.0.2.0/23 }
collapse them into one single command before the evaluation step, ie.
add element t s { 1.0.1.0/24, ..., 1.0.2.0/23 }
this speeds up overlap detection and set element automerge operations in
this worst case scenario.
Since 3da9643fb9ff9 ("intervals: add support to automerge with kernel
elements"), the new interval tracking relies on mergesort. The pattern
above triggers the set sorting for each element.
This patch adds a list to cmd objects that store collapsed commands.
Moreover, expressions also contain a reference to the original command,
to uncollapse the commands after the evaluation step.
These commands are uncollapsed after the evaluation step to ensure error
reporting works as expected (command and netlink message are mapped
1:1).
For the record:
- nftables versions <= 1.0.2 did not perform any kind of overlap
check for the described scenario above (because set cache only contained
elements in the kernel in this case). This is a problem for kernels < 5.7
which rely on userspace to detect overlaps.
- the overlap detection could be skipped for kernels >= 5.7.
- The extended netlink error reporting available for set elements
since 5.19-rc might allow to remove the uncollapse step, in this case,
error reporting does not rely on the netlink sequence to refer to the
command triggering the problem.
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Signed-off-by: Phil Sutter <psutter@redhat.com>
---
include/expression.h | 1 +
include/rule.h | 3 ++
src/libnftables.c | 17 ++++++++--
src/rule.c | 75 ++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 93 insertions(+), 3 deletions(-)
diff --git a/include/expression.h b/include/expression.h
index 2c3818e..53194c9 100644
--- a/include/expression.h
+++ b/include/expression.h
@@ -243,6 +243,7 @@ struct expr {
enum expr_types etype:8;
enum ops op:8;
unsigned int len;
+ struct cmd *cmd;
union {
struct {
diff --git a/include/rule.h b/include/rule.h
index e232b97..9081225 100644
--- a/include/rule.h
+++ b/include/rule.h
@@ -700,6 +700,7 @@ struct cmd {
enum cmd_obj obj;
struct handle handle;
uint32_t seqnum;
+ struct list_head collapse_list;
union {
void *data;
struct expr *expr;
@@ -728,6 +729,8 @@ extern struct cmd *cmd_alloc(enum cmd_ops op, enum cmd_obj obj,
const struct handle *h, const struct location *loc,
void *data);
extern void nft_cmd_expand(struct cmd *cmd);
+extern bool nft_cmd_collapse(struct list_head *cmds);
+extern void nft_cmd_uncollapse(struct list_head *cmds);
extern struct cmd *cmd_alloc_obj_ct(enum cmd_ops op, int type,
const struct handle *h,
const struct location *loc, struct obj *obj);
diff --git a/src/libnftables.c b/src/libnftables.c
index 6a22ea0..aac682b 100644
--- a/src/libnftables.c
+++ b/src/libnftables.c
@@ -501,7 +501,9 @@ static int nft_evaluate(struct nft_ctx *nft, struct list_head *msgs,
{
struct nft_cache_filter *filter;
struct cmd *cmd, *next;
+ bool collapsed = false;
unsigned int flags;
+ int err = 0;
filter = nft_cache_filter_init();
flags = nft_cache_evaluate(nft, cmds, filter);
@@ -512,17 +514,26 @@ static int nft_evaluate(struct nft_ctx *nft, struct list_head *msgs,
nft_cache_filter_fini(filter);
+ if (nft_cmd_collapse(cmds))
+ collapsed = true;
+
list_for_each_entry_safe(cmd, next, cmds, list) {
struct eval_ctx ectx = {
.nft = nft,
.msgs = msgs,
};
+
if (cmd_evaluate(&ectx, cmd) < 0 &&
- ++nft->state->nerrs == nft->parser_max_errors)
- return -1;
+ ++nft->state->nerrs == nft->parser_max_errors) {
+ err = -1;
+ break;
+ }
}
- if (nft->state->nerrs)
+ if (collapsed)
+ nft_cmd_uncollapse(cmds);
+
+ if (err < 0 || nft->state->nerrs)
return -1;
list_for_each_entry(cmd, cmds, list) {
diff --git a/src/rule.c b/src/rule.c
index 7f61bdc..0526a14 100644
--- a/src/rule.c
+++ b/src/rule.c
@@ -1279,6 +1279,8 @@ struct cmd *cmd_alloc(enum cmd_ops op, enum cmd_obj obj,
cmd->handle = *h;
cmd->location = *loc;
cmd->data = data;
+ init_list_head(&cmd->collapse_list);
+
return cmd;
}
@@ -1379,6 +1381,79 @@ void nft_cmd_expand(struct cmd *cmd)
}
}
+bool nft_cmd_collapse(struct list_head *cmds)
+{
+ struct cmd *cmd, *next, *elems = NULL;
+ struct expr *expr, *enext;
+ bool collapse = false;
+
+ list_for_each_entry_safe(cmd, next, cmds, list) {
+ if (cmd->op != CMD_ADD &&
+ cmd->op != CMD_CREATE) {
+ elems = NULL;
+ continue;
+ }
+
+ if (cmd->obj != CMD_OBJ_ELEMENTS) {
+ elems = NULL;
+ continue;
+ }
+
+ if (!elems) {
+ elems = cmd;
+ continue;
+ }
+
+ if (cmd->op != elems->op) {
+ elems = cmd;
+ continue;
+ }
+
+ if (strcmp(elems->handle.table.name, cmd->handle.table.name) ||
+ strcmp(elems->handle.set.name, cmd->handle.set.name)) {
+ elems = cmd;
+ continue;
+ }
+
+ collapse = true;
+ list_for_each_entry_safe(expr, enext, &cmd->expr->expressions, list) {
+ expr->cmd = cmd;
+ list_move_tail(&expr->list, &elems->expr->expressions);
+ }
+ elems->expr->size += cmd->expr->size;
+ list_move_tail(&cmd->list, &elems->collapse_list);
+ }
+
+ return collapse;
+}
+
+void nft_cmd_uncollapse(struct list_head *cmds)
+{
+ struct cmd *cmd, *cmd_next, *collapse_cmd, *collapse_cmd_next;
+ struct expr *expr, *next;
+
+ list_for_each_entry_safe(cmd, cmd_next, cmds, list) {
+ if (list_empty(&cmd->collapse_list))
+ continue;
+
+ assert(cmd->obj == CMD_OBJ_ELEMENTS);
+
+ list_for_each_entry_safe(expr, next, &cmd->expr->expressions, list) {
+ if (!expr->cmd)
+ continue;
+
+ list_move_tail(&expr->list, &expr->cmd->expr->expressions);
+ cmd->expr->size--;
+ expr->cmd = NULL;
+ }
+
+ list_for_each_entry_safe(collapse_cmd, collapse_cmd_next, &cmd->collapse_list, list) {
+ collapse_cmd->elem.set = set_get(cmd->elem.set);
+ list_add(&collapse_cmd->list, &cmd->list);
+ }
+ }
+}
+
struct markup *markup_alloc(uint32_t format)
{
struct markup *markup;
--
2.36.1

84
0003-intervals-do-not-report-exact-overlaps-for-new-eleme.patch Normal file
View File

@ -0,0 +1,84 @@
From afd566b56629bac4c8ca622413c8c001e2e7edfa Mon Sep 17 00:00:00 2001
From: Phil Sutter <psutter@redhat.com>
Date: Fri, 24 Jun 2022 16:02:59 +0200
Subject: [PATCH] intervals: do not report exact overlaps for new elements
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1917398
Upstream Status: nftables commit 87ba510fc704f
commit 87ba510fc704f766b5417d3bfc326e8ab9378c2a
Author: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Mon Jun 13 17:22:47 2022 +0200
intervals: do not report exact overlaps for new elements
Two new elements that represent an exact overlap should not trigger an error.
add table t
add set t s { type ipv4_addr; flags interval; }
add element t s { 1.0.1.0/24 }
...
add element t s { 1.0.1.0/24 }
result in a bogus error.
# nft -f set.nft
set.nft:1002:19-28: Error: conflicting intervals specified
add element t s { 1.0.1.0/24 }
^^^^^^^^^^
Fixes: 3da9643fb9ff ("intervals: add support to automerge with kernel elements")
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Signed-off-by: Phil Sutter <psutter@redhat.com>
---
src/intervals.c | 3 +--
tests/shell/testcases/sets/exact_overlap_0 | 22 ++++++++++++++++++++++
2 files changed, 23 insertions(+), 2 deletions(-)
create mode 100755 tests/shell/testcases/sets/exact_overlap_0
diff --git a/src/intervals.c b/src/intervals.c
index bc414d6..89f5c33 100644
--- a/src/intervals.c
+++ b/src/intervals.c
@@ -540,8 +540,7 @@ static int setelem_overlap(struct list_head *msgs, struct set *set,
}
if (mpz_cmp(prev_range.low, range.low) == 0 &&
- mpz_cmp(prev_range.high, range.high) == 0 &&
- (elem->flags & EXPR_F_KERNEL || prev->flags & EXPR_F_KERNEL))
+ mpz_cmp(prev_range.high, range.high) == 0)
goto next;
if (mpz_cmp(prev_range.low, range.low) <= 0 &&
diff --git a/tests/shell/testcases/sets/exact_overlap_0 b/tests/shell/testcases/sets/exact_overlap_0
new file mode 100755
index 0000000..1ce9304
--- /dev/null
+++ b/tests/shell/testcases/sets/exact_overlap_0
@@ -0,0 +1,22 @@
+#!/bin/bash
+
+RULESET="add table t
+add set t s { type ipv4_addr; flags interval; }
+add element t s { 1.0.1.0/24 }
+add element t s { 1.0.2.0/23 }
+add element t s { 1.0.8.0/21 }
+add element t s { 1.0.32.0/19 }
+add element t s { 1.1.0.0/24 }
+add element t s { 1.1.2.0/23 }
+add element t s { 1.1.4.0/22 }
+add element t s { 1.1.8.0/24 }
+add element t s { 1.1.9.0/24 }
+add element t s { 1.1.10.0/23 }
+add element t s { 1.1.12.0/22 }
+add element t s { 1.1.16.0/20 }
+add element t s { 1.1.32.0/19 }
+add element t s { 1.0.1.0/24 }"
+
+$NFT -f - <<< $RULESET || exit 1
+
+$NFT add element t s { 1.0.1.0/24 }
--
2.36.1

55
0004-intervals-do-not-empty-cache-for-maps.patch Normal file
View File

@ -0,0 +1,55 @@
From 5a1d00b37a06bdf37bf392af05236469b6636fb9 Mon Sep 17 00:00:00 2001
From: Phil Sutter <psutter@redhat.com>
Date: Fri, 24 Jun 2022 16:02:59 +0200
Subject: [PATCH] intervals: do not empty cache for maps
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1917398
Upstream Status: nftables commit d434de8b50dcf
commit d434de8b50dcf3f5f4ca027e122a7df9d4e5d8e1
Author: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Thu Jun 16 10:53:56 2022 +0200
intervals: do not empty cache for maps
Translate set element to range and sort in maps for the NFT_SET_MAP
case, which does not support for automerge yet.
Fixes: 81e36530fcac ("src: replace interval segment tree overlap and automerge")
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Signed-off-by: Phil Sutter <psutter@redhat.com>
---
src/intervals.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/src/intervals.c b/src/intervals.c
index 89f5c33..e203413 100644
--- a/src/intervals.c
+++ b/src/intervals.c
@@ -216,6 +216,12 @@ int set_automerge(struct list_head *msgs, struct cmd *cmd, struct set *set,
struct cmd *purge_cmd;
struct handle h = {};
+ if (set->flags & NFT_SET_MAP) {
+ set_to_range(init);
+ list_expr_sort(&init->expressions);
+ return 0;
+ }
+
if (existing_set) {
if (existing_set->init) {
list_splice_init(&existing_set->init->expressions,
@@ -229,9 +235,6 @@ int set_automerge(struct list_head *msgs, struct cmd *cmd, struct set *set,
set_to_range(init);
list_expr_sort(&init->expressions);
- if (set->flags & NFT_SET_MAP)
- return 0;
-
ctx.purge = set_expr_alloc(&internal_location, set);
setelem_automerge(&ctx);
--
2.36.1

139
0005-intervals-Do-not-sort-cached-set-elements-over-and-o.patch Normal file
View File

@ -0,0 +1,139 @@
From 013a3b226a0fa5f7a8469bae736150cbf2d092c4 Mon Sep 17 00:00:00 2001
From: Phil Sutter <psutter@redhat.com>
Date: Fri, 24 Jun 2022 16:02:59 +0200
Subject: [PATCH] intervals: Do not sort cached set elements over and over
again
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1917398
Upstream Status: nftables commit 59e3a59221fb8
commit 59e3a59221fb81c289a0868a85140dd452fb1c30
Author: Phil Sutter <phil@nwl.cc>
Date: Thu Jun 16 10:56:12 2022 +0200
intervals: Do not sort cached set elements over and over again
When adding element(s) to a non-empty set, code merged the two lists and
sorted the result. With many individual 'add element' commands this
causes substantial overhead. Make use of the fact that
existing_set->init is sorted already, sort only the list of new elements
and use list_splice_sorted() to merge the two sorted lists.
Add set_sort_splice() and use it for set element overlap detection and
automerge.
A test case adding ~25k elements in individual commands completes in
about 1/4th of the time with this patch applied.
Joint work with Pablo.
Fixes: 3da9643fb9ff9 ("intervals: add support to automerge with kernel elements")
Signed-off-by: Phil Sutter <phil@nwl.cc>
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Signed-off-by: Phil Sutter <psutter@redhat.com>
---
include/expression.h | 1 +
src/intervals.c | 46 +++++++++++++++++++++-----------------------
src/mergesort.c | 2 +-
3 files changed, 24 insertions(+), 25 deletions(-)
diff --git a/include/expression.h b/include/expression.h
index 53194c9..cf7319b 100644
--- a/include/expression.h
+++ b/include/expression.h
@@ -481,6 +481,7 @@ extern struct expr *compound_expr_alloc(const struct location *loc,
extern void compound_expr_add(struct expr *compound, struct expr *expr);
extern void compound_expr_remove(struct expr *compound, struct expr *expr);
extern void list_expr_sort(struct list_head *head);
+extern void list_splice_sorted(struct list_head *list, struct list_head *head);
extern struct expr *concat_expr_alloc(const struct location *loc);
diff --git a/src/intervals.c b/src/intervals.c
index e203413..dcc06d1 100644
--- a/src/intervals.c
+++ b/src/intervals.c
@@ -118,6 +118,26 @@ static bool merge_ranges(struct set_automerge_ctx *ctx,
return false;
}
+static void set_sort_splice(struct expr *init, struct set *set)
+{
+ struct set *existing_set = set->existing_set;
+
+ set_to_range(init);
+ list_expr_sort(&init->expressions);
+
+ if (!existing_set)
+ return;
+
+ if (existing_set->init) {
+ set_to_range(existing_set->init);
+ list_splice_sorted(&existing_set->init->expressions,
+ &init->expressions);
+ init_list_head(&existing_set->init->expressions);
+ } else {
+ existing_set->init = set_expr_alloc(&internal_location, set);
+ }
+}
+
static void setelem_automerge(struct set_automerge_ctx *ctx)
{
struct expr *i, *next, *prev = NULL;
@@ -222,18 +242,7 @@ int set_automerge(struct list_head *msgs, struct cmd *cmd, struct set *set,
return 0;
}
- if (existing_set) {
- if (existing_set->init) {
- list_splice_init(&existing_set->init->expressions,
- &init->expressions);
- } else {
- existing_set->init = set_expr_alloc(&internal_location,
- set);
- }
- }
-
- set_to_range(init);
- list_expr_sort(&init->expressions);
+ set_sort_splice(init, set);
ctx.purge = set_expr_alloc(&internal_location, set);
@@ -591,18 +600,7 @@ int set_overlap(struct list_head *msgs, struct set *set, struct expr *init)
struct expr *i, *n, *clone;
int err;
- if (existing_set) {
- if (existing_set->init) {
- list_splice_init(&existing_set->init->expressions,
- &init->expressions);
- } else {
- existing_set->init = set_expr_alloc(&internal_location,
- set);
- }
- }
-
- set_to_range(init);
- list_expr_sort(&init->expressions);
+ set_sort_splice(init, set);
err = setelem_overlap(msgs, set, init);
diff --git a/src/mergesort.c b/src/mergesort.c
index 8e6aac5..dca7142 100644
--- a/src/mergesort.c
+++ b/src/mergesort.c
@@ -70,7 +70,7 @@ static int expr_msort_cmp(const struct expr *e1, const struct expr *e2)
return ret;
}
-static void list_splice_sorted(struct list_head *list, struct list_head *head)
+void list_splice_sorted(struct list_head *list, struct list_head *head)
{
struct list_head *h = head->next;
struct list_head *l = list->next;
--
2.36.1

20
nftables.spec
View File

@ -1,6 +1,9 @@
%define rpmversion 1.0.4
%define specrelease 2
Name: nftables
Version: 1.0.4
Release: 1%{?dist}
Version: %{rpmversion}
Release: %{specrelease}%{?dist}%{?buildid}
# Upstream released a 0.100 version, then 0.4. Need Epoch to get back on track.
Epoch: 1
Summary: Netfilter Tables userspace utillites
@ -14,6 +17,12 @@ Source3: main.nft
Source4: router.nft
Source5: nat.nft
Patch1: 0001-tests-shell-runtime-set-element-automerge.patch
Patch2: 0002-rule-collapse-set-element-commands.patch
Patch3: 0003-intervals-do-not-report-exact-overlaps-for-new-eleme.patch
Patch4: 0004-intervals-do-not-empty-cache-for-maps.patch
Patch5: 0005-intervals-Do-not-sort-cached-set-elements-over-and-o.patch
BuildRequires: autoconf
BuildRequires: automake
BuildRequires: libtool
@ -122,6 +131,13 @@ sed -i -e 's/\(sofile=\)".*"/\1"'$sofile'"/' \
%{python3_sitelib}/nftables/
%changelog
* Fri Jun 24 2022 Phil Sutter <psutter@redhat.com> [1.0.4-2.el9]
- intervals: Do not sort cached set elements over and over again (Phil Sutter) [1917398]
- intervals: do not empty cache for maps (Phil Sutter) [1917398]
- intervals: do not report exact overlaps for new elements (Phil Sutter) [1917398]
- rule: collapse set element commands (Phil Sutter) [1917398]
- tests: shell: runtime set element automerge (Phil Sutter) [1917398]
* Thu Jun 09 2022 Phil Sutter <psutter@redhat.com> - 1:1.0.4-1
- Review package dependencies
- new version 1.0.4