Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-5961

ListUnitFiles impacts systemd a lot, causing high CPU consumption and delay in operations

Details

    • Normal
    • sst_cs_plumbers
    • ssg_core_services
    • False
    • Hide

      None

      Show
      None
    • If docs needed, set a value

    Description

      Description of problem:

      This is a respin of BZ #1828759 ("systemctl list-unit-files" kills the system usability).

      With RHEL8.7 / systemd-239-68.el8 (didn't check with older ones yet), we can see that the ListUnitFiles DBus operation is very costly.
      When executing systemd under strace, we can observe that on a system having 145 units ("minimal installation" of RHEL8.7), the readlinkat/openat/getdents64 syscalls are called hundreds or even thousands of times on same files (see reproducer below), causing high CPU consumption.

      Version-Release number of selected component (if applicable):

      systemd-239-68.el8

      How reproducible:

      Always

      Steps to Reproduce:
      1. Execute the DBus ListUnitFiles operation while stracing systemd

      1. strace -CttTvyy -s 128 -o list-unit-files.strace -p 1 &
      2. time dbus-send --system --print-reply --reply-timeout=20000 --type=method_call --dest=org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager.ListUnitFiles
      3. kill %1

      2. Check number of syscalls

      1. tail -20 list-unit-files.strace
        % time seconds usecs/call calls errors syscall
                • ----------- ----------- --------- --------- ----------------
                  30.08 0.277966 2 119048 readlinkat
                  21.62 0.199787 2 79554 16869 openat
                  21.01 0.194170 2 89605 getdents64
                  10.02 0.092602 1 90426 fcntl
                  ...

      3. Check number of times readlinkat() is executed on a given path

      1. grep " readlinkat(" list-unit-files.strace | cut -f2 -d'"' | sort | uniq -c | sort -k1 -rn > list-unit-files.strace.readlinkat
      2. head list-unit-files.strace.readlinkat
        820 /etc/systemd/system/syslog.service
        819 /etc/systemd/system/multi-user.target.wants/remote-fs.target
        818 /etc/systemd/system/multi-user.target.wants/crond.service
        817 /etc/systemd/system/multi-user.target.wants/NetworkManager.service
        815 /etc/systemd/system/multi-user.target.wants/dnf-makecache.timer
        814 /etc/systemd/system/multi-user.target.wants/sssd.service
        ...
      1. grep "NetworkManager.service" list-unit-files.strace.readlinkat
        817 /etc/systemd/system/multi-user.target.wants/NetworkManager.service
        793 /etc/systemd/system/dbus-org.freedesktop.NetworkManager.service

      Actual results:

      Slowness, CPU consumption

      Expected results:

      Files/Symlinks resolved once.

      Attachments

        Activity

          People

            systemd-maint systemd maint mailing list
            rhn-support-rmetrich Renaud Metrich
            systemd maint mailing list systemd maint mailing list
            Frantisek Sumsal Frantisek Sumsal
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated: