Bug#864341: systemd-sysctl: failed to apply sysctl config at bootup

[Moritz Muehlenhoff]

Hi Arturo/Michael,

thanks for looping me in.

On Wed, Jun 07, 2017 at 01:13:12PM +0200, Arturo Borrero Gonzalez wrote:
> On 07/06/17 12:57, Michael Biebl wrote:
> >How do you load the conntrack kernel module?
> >Moritz brought this up on IRC already and from what we could conclude
> >this is an ordering issue / race between the kernel module being loaded
> >and the sysfs entries being available and the sysctl settings being applied.
> >
> >Moritz, did your investigations find something?

Indeed. In my day job I work for the Wikimedia Foundation and we ran into
the same bug as well:

The manpage of systemd-sysctl recommends to load kernel modules
creating a sysctl value via /etc/modules-load.d, but that's not sufficient.

While systemd-sysctl.service has a "After: systemd-modules-load", 
systemd-modules-load only initiates the loading of the kernel modules via 
kmod, but doesn't wait until the modules are fully loaded/the sysctl is
available. 

See https://phabricator.wikimedia.org/T136094#3298506 for an example run
with debug output added.

I could not track down why it's not sufficient in the case of conntrack,
we're using the same solution to apply some sysctl setting for Linux LVS,
which works race-free there. It's possibly related to the kernel modules
adding a sysctl itself.

I was planning to raise this upstream (if only as a documentation bug
since their current proposal with modules-load.d doesn't work), but
haven't found the time so far.

> What about running systemd-sysctl the last in the boot order chain?

Some of those settings might be needed earlier I guess.

A couple of possible solutions, but these are all rather something for
upstream development:
- sysctl.conf files could gain an additional parameter which specifies
  the kernel module creating the sysctl. systemd-sysctl could then
  trigger the loading of that module and wait until the specified sysctl
  is present (I think that's the cleanest solution)
- All sysctl settings should be entirely idempotent. systemd-sysctl could
  be re-run in multi-user.target (fairly ugly)
- Or maybe split off a separate systemd-sysctl-stage2.service, which reads
  a separate sysctl config (which can then be used for driver which are loaded
  late)

Cheers,
        Moritz