Bug#798625: systemd-networkd: Runs arbitrary inappropriate scripts on network changes

[Martin Pitt]

Hello Josh,

josh at joshtriplett.org [2015-09-11 10:58 -0700]:
> > FTR, this was discussed a few months ago with Tom Gundersen (the
> > author of networkd) at the UOS discussion for this:
> > 
> >   https://blueprints.launchpad.net/ubuntu/+spec/foundations-w-networkd-vs-ifupdown
> 
> What was his response?  I'd be quite surprised to hear any upstream
> systemd or networkd developers favoring this approach.

He certainly wasn't too thrilled as he doesn't currently plan to add a
generic hook support upstream, but he understood that distros may
have this requirement and suggested various options how this can be
implemented.

> The proposal there also seems somewhat different than the one
> implemented here.

Yes, my first implementation attempt was a bit different (udev rule
and a helper .service), but this had various shortcomings. But the
details of the implementation don't matter so much I think, as you are
more interested in the actual effect.

> For that matter, it seems like there's a race condition here: if
> resolveconf is going to pull in systemd-resolved's resolv.conf, networkd
> hooks can't guarantee that systemd-resolved actually creates that
> *before* those hooks run.  Perhaps resolveconf could start using inotify
> (along with a list of places to look for generated resolv.conf files)?

If that's an actual race (i. e. an interface can go to state
"configured" while resolv.conf is not updated yet), we could add some
equivalent code to resolvconf (or networkd) to update resolvconf
again, similar to /etc/dhcp/dhclient-enter-hooks.d/resolvconf for
dhclient.

Does this really work so asynchronously? It seems to me that this
would also make networkd-wait-online much less useful.

> > The hooks are run asynchronously and don't block networkd.
> 
> Interesting; as far as I know, that's a change compared to how other as
> far as I know those hooks are normally run synchronously by other
> software.

What do you mean by "synchronous"? Certainly not for ifupdown itself
(different interfaces are being handled completely independent from
each other) and I don't believe NetworkManager blocks on them either
(it calls NM-dispatcher which runs the hooks, it does not block NM).

> I'd still be concerned about whether this adds any delay to the
> fast-path, though a bit less so.  Moving this to use networkd's
> existing event mechanism seems preferable to spawning a process from
> networkd.

I really want to avoid having to run a second daemon all the time
which just listens to rather scarce networkd messages. Having
network do a D-Bus call and activate some "networkd-dispatcher" was a
suggested alternative, but it's muuuuuuch more heavy-weight than the
simple suid wrapper.

> Hardly a zillion packages; here's a complete list of Debian packages:
> 
> This seems quite manageable, especially once you rule out the ones that
> make no sense at all with networkd (or with anything other than
> ifupdown).  I'd be happy to help evaluate these further.

That would certainly be nice. It's not that I like the hooks much, but
I believe that nothing at all is gained by tightly coupling
transitions that way, like "sorry, you can't use networkd without
resolved and without fixing those ~ 50 packages first". I like the
evolutionary approach more than the revolutionary one.

> I'd like to see networkd more widely used too, but not at the expense of
> diverging this drastically from upstream in an area where doing so
> creates an API and an expectation for future support of that API.  If we
> start doing this, trying to remove it in the future will *hurt*.  (As
> will dealing with portability between distributions.)

I don't understand this TBH. We aren't creating an API, the Debian
/etc/network/if-*.d/ API has existed for a long time. So dropping them
now doesn't hurt any less than dropping them in the future. Also,
fixing packages to remove the need for those hooks will benefit all
network setup solutions (ifupdown, NM, networkd, and whatnot), so your
quest for eliminating them is indeed a good one! (It's just not
something which I've got time for to do).

> For OpenSSH, and for any software that just re-listens when new
> interfaces come up, they should just listen to INADDR_ANY, rather than
> caring what specific interfaces are up or down.

Fully agreed.

> On any system networkd can run on, dynamically reloading OpenSSH and
> other listening daemons shouldn't be needed.)

networkd is not that "fire and forget at boot" -- there are hotplugged
devices, dynamically reconfiguring virtual interfaces, veths, etc.
Certainly less dynamic than wifis on desktops for sure.

> ntpd handles dynamic network configurations just fine, as does
> timesyncd; ntpdate *never* handled dynamic network configurations, and I
> don't think it makes sense to work around that with scripts when real
> NTP daemons do the right thing automatically.  ntpdate (and other
> software like tlsdate) makes sense as a standalone tool, but invoking it
> automatically seems less sensible.  (Plus, it makes more sense to
> continuously keep time up to date, rather than only at boot or network
> change.)

Also agreed - but again in Debian we don't really have a mechanism to
tell people something like "sorry, you use ntpdate, you can't run
network (or prevent package installation/service installation, etc.).
It's the same "coupling transitions and unrelated things" from above
again.

> As far as I know, avahi-autoipd is entirely obsoleted by networkd, which
> assigns link-local addresses itself; see the "LinkLocalAddressing"
> option in "man systemd.network".

*nod*

> networkd intentionally *doesn't* maintain such state.

That's not true -- devices have states just like NM, and it's also
being serialized to /run/systemd/netif/links, it's just that
networkd isn't as aggressive as NM when it comes to changing the
system state back to what it thinks the state should be.

> In particular, if networkd has no ongoing work to do, it can exit.
> And on systems that need networking in the initramfs (such as
> systems that need the network for the root filesystem), networkd can
> run in the initramfs, configure those interfaces, *exit*, and then
> re-run after switching to the root filesystem.

That's all true, but is unrelated to maintaining state. Pretty much
everything in systemd (systemd pid 1 itself, logind, networkd)
serializes state into /run and loads it back on startup/daemon-reexec.

> Any daemon or other software that has /etc/network/if-*.d/* hooks will
> also break if someone manually brings an interface up or down, yanks out
> a network cable, etc.

That's true, but that's totally outside the realm of ifupdown, NM, or
networkd -- if you chose not to use those, you can of course not
expect any distro integration of those to work either.

> And what happens if you then re-start networkd?  Will it call if-up.d/*
> ?

If networkd (re)configures an interface, it will (re-)call if-up.d/,
yes.

> How confident are you that every such script in Debian is idempotent?

Very confident, because this happens all the time with NetworkManager
too. WLANs and mobile networks are notoriously reconnecting.

> Will those scripts work if networkd brings up a bridge?  Or a bond?  Or
> a virtual network interface?  Or a network with no non-link-local IP?

Yes, just like ifupdown/NM these scripts are meant to be called on any
interface.

> > > , mountnfs's hooks shouldn't run (because they conflict with several
> > > other approaches to nfs handling that integrate properly with
> > > systemd)
> > 
> > That sounds like a real issue indeed, do you have some details? Again,
> > how is that not affecting NetworkManager?
> 
> See some of the comments at the top of mountnfs.  mountnfs calls into
> various /etc/init.d/* scripts, and mounts filesystems from /etc/fstab.
> systemd already handles mounting filesystems, and has its own logic for
> making sure network filesystems don't get mounted before bringing up the
> network.
> 
> Because of NetworkManager's target use case and system type, I doubt
> many people run NetworkManager on systems that have NFS filesystems
> listed in /etc/fstab, let alone critical ones. :)

I actually know quite a few :-) NFS-in-fstab-with-NM got broken during
Ubuntu vivid development when we switched to systemd, and I got quite
a number of bugs/IRC pings about it. Like you I thought "man, this is
crazy", but got told that this setup has worked for years for them. I
was quite surprised too! (This was of course not for an early-boot FS
like /var, but something like /home or /media).

> > > , avahi-daemon's hook is responsible for numerous problems and
> > > slowdowns even under ifupdown
> > 
> > This just disables avahi if there is a "real" .local domain. This
> > seems rather independent of the mechanism used to bring up networking?
> 
> It can cause *long* delays bringing up and configuring interfaces (including
> under NetworkManager).

What would cause the delay there? It's an async script that does a
check and pops up a notification, and if-up.d/ runs *after* an
interface has been configured. How can this possibly hold up
configuring an interface?

> > > What specific problem is this trying to solve?
> > 
> > See above -- making use of Debian's integration logic for a lot of
> > packages, to be on par with ifupdown and NetworkManager.
> 
> I'm asking to find out if there's some specific package or packages
> whose lack of such integration motivated the change, primarily to scope
> the problem of fixing those packages.

This came up when the Ubuntu server team looked at networkd and
stumbled upon  the missing distro integration.  For me personally,
openssh and postfix are the two most important use cases; other people
use different stuff of course.

> What I'm asking, though, is that before any packages start *relying* on
> networkd invoking these scripts (which thus becomes an API with
> potential backward-compatibility commitments), this change be reverted
> (or at the very least stuck in experimental or hidden behind a
> disabled-by-default configuration option), and we figure out how much
> work it would be to fix the critical packages that ship these scripts to
> do the right thing on *all* systems.

Again, I fully agree that this would be *nice*. But dynamic networking
has been a thing for like 10 years now and in all this time nobody got
around to fixing all packages to avoid the necessity of these hooks.
I agree that "someone" should fix all those, it's like always a
manpower problem, or perhaps sometimes upstreams which don't want to
adopt Linux-only functionality, etc.

Thanks,

Martin
-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)