[Pkg-utopia-maintainers] Bug#1025616: firewalld and cloud-init systemd unit files have ordering cycles

[Guillaume Knispel]

Package: firewalld
Version: 0.9.3-2
Severity: important
X-Debbugs-Cc: xilun at australdx.fr

Dear Maintainer,

firewalld and cloud-init have ordering cycles between their systemd unit
files, leading to more or less broken boot results when both are installed
and active, because at each boot systemd decides to skip a
non-deterministically choosen service (not necessarily cloud-init or
firewalld) to break the cycle.

I'm not sure if any of firewalld or cloud-init is more at fault (maybe
in not respecting some systemd rules?) so I'm also opening a duplicate
of this bug for the other package.

This can have various but potentially serious consequences, depending on
what should be, but is not, started.

Examples of boot traces of this issue happening:

* example 1:
sysinit.target: Found ordering cycle on cloud-init.service/start
sysinit.target: Found dependency on networking.service/start
sysinit.target: Found dependency on network-pre.target/start
sysinit.target: Found dependency on firewalld.service/start
sysinit.target: Found dependency on basic.target/start
sysinit.target: Found dependency on sockets.target/start
sysinit.target: Found dependency on uuidd.socket/start
sysinit.target: Found dependency on sysinit.target/start
sysinit.target: Job cloud-init.service/start deleted to break ordering cycle starting with sysinit.target/start

* example 2:
sysinit.target: Found ordering cycle on cloud-init.service/start
sysinit.target: Found dependency on networking.service/start
sysinit.target: Found dependency on network-pre.target/start
sysinit.target: Found dependency on firewalld.service/start
sysinit.target: Found dependency on dbus.service/start
sysinit.target: Found dependency on sysinit.target/start
sysinit.target: Job cloud-init.service/start deleted to break ordering cycle starting with sysinit.target/start

* example 3:
firewalld.service: Found ordering cycle on dbus.socket/start
firewalld.service: Found dependency on sysinit.target/start
firewalld.service: Found dependency on cloud-init.service/start
firewalld.service: Found dependency on networking.service/start
firewalld.service: Found dependency on network-pre.target/start
firewalld.service: Found dependency on firewalld.service/start
firewalld.service: Job dbus.socket/start deleted to break ordering cycle starting with firewalld.service/start

* example 4:
firewalld.service: Found ordering cycle on dbus.service/start
firewalld.service: Found dependency on sysinit.target/start
firewalld.service: Found dependency on cloud-init.service/start
firewalld.service: Found dependency on networking.service/start
firewalld.service: Found dependency on network-pre.target/start
firewalld.service: Found dependency on firewalld.service/start
firewalld.service: Job dbus.service/start deleted to break ordering cycle starting with firewalld.service/start
basic.target: Found ordering cycle on sysinit.target/start
basic.target: Found dependency on cloud-init.service/start
basic.target: Found dependency on networking.service/start
basic.target: Found dependency on network-pre.target/start
basic.target: Found dependency on firewalld.service/start
basic.target: Found dependency on basic.target/start
basic.target: Job cloud-init.service/start deleted to break ordering cycle starting with basic.target/start

* example 5:
basic.target: Found ordering cycle on sockets.target/start
basic.target: Found dependency on uuidd.socket/start
basic.target: Found dependency on sysinit.target/start
basic.target: Found dependency on cloud-init.service/start
basic.target: Found dependency on networking.service/start
basic.target: Found dependency on network-pre.target/start
basic.target: Found dependency on firewalld.service/start
basic.target: Found dependency on dbus.service/start
basic.target: Found dependency on basic.target/start
basic.target: Job sockets.target/start deleted to break ordering cycle starting with basic.target/start
firewalld.service: Found ordering cycle on dbus.socket/start
firewalld.service: Found dependency on sysinit.target/start
firewalld.service: Found dependency on cloud-init.service/start
firewalld.service: Found dependency on networking.service/start
firewalld.service: Found dependency on network-pre.target/start
firewalld.service: Found dependency on firewalld.service/start
firewalld.service: Job dbus.socket/start deleted to break ordering cycle starting with firewalld.service/start

* example 6:
networking.service: Found ordering cycle on network-pre.target/start
networking.service: Found dependency on firewalld.service/start
networking.service: Found dependency on dbus.service/start
networking.service: Found dependency on basic.target/start
networking.service: Found dependency on sockets.target/start
networking.service: Found dependency on uuidd.socket/start
networking.service: Found dependency on sysinit.target/start
networking.service: Found dependency on cloud-init.service/start
networking.service: Found dependency on networking.service/start
networking.service: Job network-pre.target/start deleted to break ordering cycle starting with networking.service/start


At first I experienced the issue on a Debian Stable (Bullseye), then
I was able to reproduce the problem on an up-to-date Bookworm.

Note that "systemd-analyze verify" seems to be unable to find the issue,
however, the following repro (tried on Bookworm) shows a way to detect
the cycles statically (plus, during a reboot you can observe it
directly):

$ sudo apt install firewalld cloud-init
$ echo "datasource_list: [ Fallback ]" | sudo tee /etc/cloud/cloud.cfg.d/99_fallback.cfg
$ sudo reboot
$ wget https://raw.githubusercontent.com/jantman/misc-scripts/4560db773f463101273539e625c9b48e9f53f87f/dot_find_cycles.py
$ #    ^ I found that script by reading https://github.com/systemd/systemd/issues/3829
$ sudo apt install 2to3 python3-pygraphviz python3-pydotplus python3-pydot python3-graphviz python3-networkx
$ 2to3 -w dot_find_cycles.py
$ chmod +x dot_find_cycles.py
$ sudo systemd-analyze dot --no-pager --order 2>/dev/null | python3 ./dot_find_cycles.py - | tee order-cycle_cloud-init-firewalld.txt
networking.service -> network-pre.target -> firewalld.service -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> basic.target -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> sysinit.target -> cloud-init.service -> networking.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service


Best regards,
Guillaume Knispel


-- System Information:
Debian Release: 11.5
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-19-amd64 (SMP w/64 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages firewalld depends on:
ii  dbus              1.12.24-0+deb11u1
ii  gir1.2-glib-2.0   1.66.1-1+b1
ii  gir1.2-nm-1.0     1.30.6-1+deb11u1
ii  iptables          1.8.7-1
ii  policykit-1       0.105-31+deb11u1
ii  python3           3.9.2-3
ii  python3-dbus      1.2.16-5
ii  python3-firewall  0.9.3-2
ii  python3-gi        3.38.0-2
ii  python3-nftables  0.9.8-3.1

Versions of packages firewalld recommends:
ii  ipset  7.10-1

firewalld suggests no packages.

-- Configuration Files:
/etc/firewalld/firewalld.conf [Errno 13] Permission denied: '/etc/firewalld/firewalld.conf'
/etc/firewalld/lockdown-whitelist.xml [Errno 13] Permission denied: '/etc/firewalld/lockdown-whitelist.xml'

-- no debconf information