Description of problem:

A customer in a Openshift virtualization environment reported that a VM is very slow to start when there are 25K nftables chains.

Version-Release number of selected component (if applicable):

RHEL 8.6

How reproducible:

We can reproduce a case where a nft command takes 48 seconds to complete.

Steps to Reproduce:
1.

Run this script a few times.

cat nft_add.sh
for a in

do
nft add table inet filter$a
nft add chain inet filter$a input

done

2. There should be ~36K chains:

1. nft list ruleset|wc -l
   36420

3. Then try and add a rule:

1. time nft add rule ip filter output ip daddr 192.168.1.0/24 counter
   Error: Could not process rule: No such file or directory
   add rule ip filter output ip daddr 192.168.1.0/24 counter
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

real 0m49.454s
user 0m8.915s
sys 0m40.407s

Actual results:

nft commands take 48 seconds to complete.

Expected results:

nft commands should return within a second.

Additional info:

It's spending a lot time in the netlink routines:

nft 45862 [003] 2940.260199: 844862 cycles:
ffffffffc1429105 nf_tables_dump_chains+0x65 (/lib/modules/4.18.0-372.26.1.el8_6.x86_64/kernel/net/netfilter/nf_tables.ko.xz)
ffffffffa4c5221a netlink_dump+0x18a (/usr/lib/debug/lib/modules/4.18.0-372.26.1.el8_6.x86_64/vmlinux)
ffffffffa4c52637 netlink_recvmsg+0x227 (/usr/lib/debug/lib/modules/4.18.0-372.26.1.el8_6.x86_64/vmlinux)
ffffffffa4bb70b1 ____sys_recvmsg+0x91 (/usr/lib/debug/lib/modules/4.18.0-372.26.1.el8_6.x86_64/vmlinux)
ffffffffa4bbaf7b ___sys_recvmsg+0x7b (/usr/lib/debug/lib/modules/4.18.0-372.26.1.el8_6.x86_64/vmlinux)
ffffffffa4bbb044 __sys_recvmsg+0x54 (/usr/lib/debug/lib/modules/4.18.0-372.26.1.el8_6.x86_64/vmlinux)
ffffffffa440430b do_syscall_64+0x5b (/usr/lib/debug/lib/modules/4.18.0-372.26.1.el8_6.x86_64/vmlinux)
ffffffffa4e000ad entry_SYSCALL_64_after_hwframe+0x65 (/usr/lib/debug/lib/modules/4.18.0-372.26.1.el8_6.x86_64/vmlinux)
7f0dfc28d198 __libc_recvmsg+0x18 (/usr/lib64/libc-2.28.so)