[Pkg-systemd-maintainers] Bug#734951: Bug#734951: systemd: somehow starts LSB stuff in the wrong order

[Michael Stapelberg]

Hi Christoph,

Thanks for providing the log. I think I can shed some light on what
happens here.

Christoph Anton Mitterer <calestyo at scientia.net> writes:
> As you can see from there... systemd actually schedules
> iptables-persistend first... but it seems that this get's only executed
> much later (i.e. after fail2ban).
Nope:

$ grep -n '\(iptables-persistent\|fail2ban\)' log.txt
854:Jan 14 17:42:52 heisenberg systemd[1]: Installed new job iptables-persistent.service/start as 53
899:Jan 14 17:42:52 heisenberg systemd[1]: Installed new job
fail2ban.service/start as 101

The above two lines are from the original startup ordering and properly
represent the order.

1325:Jan 14 17:42:53 heisenberg systemd[1]: Forked /etc/init.d/iptables-persistent as 563
1326:Jan 14 17:42:53 heisenberg systemd[1]: iptables-persistent.service
changed dead -> start

Here systemd starts iptables-persistent, but as the configuration source
is a sysvinit script, all it can do is start it and hope for the
best. Specifically, it cannot know when iptables-persistent should be
considered ready (or “finished”, whatever you want to call it :-)).

For that, you’d need to write a native systemd service file. With
Type=oneshot you’ll easily get the expected behavior, see
systemd.service(5). In case you are new to systemd unit files, there are
good guides available on the internet, the manpages are thought of as a
reference, not a tutorial.

In general, the sysvinit configuration source works fairly well in
systemd, but there are good reasons why we want to migrate to systemd
entirely and not live with the compatibility layer forever. Edge cases
such as this one are such reasons :).

1473:Jan 14 17:42:53 heisenberg systemd[1]: Got D-Bus request:
org.freedesktop.DBus.Properties.GetAll() on
/org/freedesktop/systemd1/unit/fail2ban_2eservice
1477:Jan 14 17:42:53 heisenberg systemd[1]: Got D-Bus request: org.freedesktop.systemd1.Manager.StopUnit() on /org/freedesktop/systemd1
1478:Jan 14 17:42:53 heisenberg systemd[1]: Trying to enqueue job
fail2ban.service/stop/replace

Now this is interesting. Something is communicating with systemd and
tells it to stop fail2ban. This is the reason you first see the
“stopped” message. I don’t really know what the thing is that talks to
systemd here, but my guess is that it’s some sort of ifup.d-hook or
something like that. The iptables-persistent package doesn’t ship
anything like that, so maybe it’s a local system modification of yours?

1479:Jan 14 17:42:53 heisenberg systemd[1]: Job fail2ban.service/start finished, result=canceled
1480:Jan 14 17:42:53 heisenberg systemd[1]: Installed new job fail2ban.service/stop as 132
1481:Jan 14 17:42:53 heisenberg systemd[1]: Enqueued job fail2ban.service/stop as 132
1482:Jan 14 17:42:53 heisenberg systemd[1]: Job fail2ban.service/stop finished, result=done
1483:Jan 14 17:42:53 heisenberg systemd[1]: Stopped LSB: Start/stop fail2ban.
1485:Jan 14 17:42:53 heisenberg systemd[1]: Got D-Bus request: org.freedesktop.DBus.Properties.Get() on /org/freedesktop/systemd1/unit/fail2ban_2eservice
1486:Jan 14 17:42:53 heisenberg systemd[1]: Got D-Bus request: org.freedesktop.DBus.Properties.Get() on /org/freedesktop/systemd1/unit/fail2ban_2eservice
1487:Jan 14 17:42:53 heisenberg systemd[1]: Got D-Bus request: org.freedesktop.DBus.Properties.Get() on /org/freedesktop/systemd1/unit/fail2ban_2eservice
1522:Jan 14 17:42:53 heisenberg iptables-persistent[563]: Loading iptables rules... IPv4... IPv6...done.
1526:Jan 14 17:42:53 heisenberg systemd[1]: Child 563 belongs to iptables-persistent.service
1527:Jan 14 17:42:53 heisenberg systemd[1]: iptables-persistent.service: control process exited, code=exited status=0
1528:Jan 14 17:42:53 heisenberg systemd[1]: iptables-persistent.service got final SIGCHLD for state start
1529:Jan 14 17:42:53 heisenberg systemd[1]: iptables-persistent.service changed start -> exited
1530:Jan 14 17:42:53 heisenberg systemd[1]: Job iptables-persistent.service/start finished, result=done
1534:Jan 14 17:42:53 heisenberg systemd[1]: iptables-persistent.service: cgroup is empty


> Actually, I think, this points to a far bigger concern of mine.
>
> I'm really in favour of systemd (which is why I've adopted it early
> without any special need)... but undoubtedly, it's far more difficult to
> tell when which "service" is actually started - which was pretty clear
> by looking at /etc/rc*.d/ with sysvinit.
I don’t think that’s true, but I have worked with systemd far more than
with sysvinit, so I might be biased. Apart from the extensive logfile
that systemd can produce, there are tools to generate graphs out of the
boot order and other simulation tools (e.g. systemd --test). Obviously,
none of this can predict what actually happens _outside of systemd_,
such as the stop request systemd receives in your logfile.

> What I really fear are systems that heavily depend for their security on
> some service being _fully_ started _before_ some others are run.
> Typical example: iptables-presistent rules should be set up _before_
> interfaces are brought up, which can be easily security relevant if
> these e.g. enforce IPsec'ed traffic, which would go through plain
> otherwise.
How about using pre-up iptables-restore < /etc/network/iptables in
/etc/network/interfaces then? :)

I do agree that network config is a bit unsatisfactory in general,
though. An effective and mostly parallel boot only highlights this
problem, but doesn’t create it, IMO.

> Likewise, having the IPsec daemon run _before_ any other network
> services are started (i.e. directly after "networking" was brought up).
> All these are services where socket based activation is obviously
> useless.
Can’t really talk much about this, since it has been many years since I
last touched ipsec and I don’t know of anyone who uses it these days.

> How can we make sure that these things continue to work safely?
> And I mean this very bug shows, that they don't... fail2ban may not be
> strictly required for other serivces to work... but potentially even the
> short window until it gets started late could be used to do some evil
> things or trying to DoS attack a system.
With all due respect, if your machine gets DoSed due to the absence of
fail2ban, you’re doing something extremely wrong. I have survived for
many years with many servers with port 22 on the public internet and
never was that a problem. As far as I understand it, fail2ban is helpful
for keeping your logs clean, but I don’t believe in it being a security
measure to prevent DoS attacks.

More generally speaking, if there are services that you consider
absolutely critical to be up before your network is up, I’d recommend
doing precisely that: overwrite/amend the unit file responsible for
networking on your machine to contain an After=fail2ban.service etc.
That is only possible when the services (e.g. fail2ban) are written in
such a way that they don’t require the network interface to be up and
fully configured, obviously, but I don’t see how you could circumvent
that problem, ever.

Hope that helps.

-- 
Best regards,
Michael