Bug#771122: restarting the journal breaks other services

[Michael Biebl]

Am 27.11.2014 um 00:17 schrieb Michael Biebl:
> There was a recent discussion on #systemd, where Lennart mentioned in a related
> context (full IRC log is attached), that restarting journald is a bad idea,
> as it is not properly supported atm:

For completeness sake, find attached the IRC log.

As Lennart also mentions, this makes it impossible to reconfigure
journald at runtime (since this requires a restart).
Imho this is quite a serious limitation and should probably be
documented more prominently.

Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
-------------- next part --------------
<JayF> If I'd like to reconfigure journald at runtime (using CoreOS and I'd like to set MaxLevelConsole=debug), is there a way other than editing the journald.conf directly? Does it take drop-ins like a unit file does?
<grawity> I think someone suggested a patch for that
<poettering> JayF: there was a patch, but it's not merged yet
<poettering> JayF: but note that restarting journald to actually make it count is not really that good an idea
<poettering> JayF: since we cannot push the stdout/stderr fds everywhere we will just close them
<poettering> that means as you restart journald you lose the stdout/stderr output of all daemons, and you cannot get it back
<poettering> JayF: it's a hard problem
<JayF> Yeah; I've noticed and had quite a bit of pain from that behavior
<poettering> JayF: we have some ideas, about nothing got implemented yet there
<poettering> JayF: but yeah, extending journald.conf via drop-ins is certainly a good idea, and patch is waiting for that
<JayF> Hmm. So essentially, there's no way to dynamically configure journald to debug log to console
<poettering> JayF: and making it restartable is on the TODO list but no patch yet
<JayF> I'm using systemd.journald.forward_to_console but I really need it to be debug
<poettering> JayF: yeah, it's a bit annoying
<JayF> I guess I'll have to figure how how to modify that config at ramdisk build time, rather than configuring it via cloud-config
<poettering> JayF: as soon as we have kdbus we'll open up a lot of settings for bus clients
<poettering> JayF: but as long as we have no kdbus we can't really, as journald runs in early-boot
<poettering> JayF: but dbus1 is not available in early-boot
<poettering> JayF: and the complexities of handling this madness are crazy, because it means journald would need to connect to dbus1 as soon as dbus-daemon is up, which is really hackisch
<JayF> Yeah; that makes sense. Honestly I'd handle the restart case above all else
<JayF> because that at least gives me the tooling to do what I need after the fact
<JayF> I'm kinda surprised journald holds the fds for stdout/err itself. I would've thought that would've gone through systemd somehow
<poettering> JayF: when systemd forks of a service it connects stdout/stderr to a stream fd of journald's
<poettering> JayF: we really don't want to have to bump off all messages from PID 1 there...
<JayF> Makes sense, then
<JayF> Any ideas for how to implement restartablity for the journal then?
<poettering> yes
<poettering> basically we want a new way how daemons can query the fds they shall listen on
<poettering> like a check-in API
<poettering> where daemons can ask at any time to query the fds
<poettering> and can actually push fds back into
<poettering> currently socket activation passes all fds accross exec()
<poettering> and the forked of process then makes sense of it
<poettering> we'd like to add a new per-service concept, where you can just ask PID 1 for the fds you are supposed to listen on via the bus
<poettering> and then you can also add fds to that set
<poettering> which means journald could just write all stream fds it accept()ed back to PID 1 and PID 1 would keep a reference
<JayF> So does that mean the restart would only not impact services that are cooperating with the journal then? (nbd for me, because all my software is running in an nspawn container and I suspect you'd update systemd-nspawn)
<poettering> and then, when journald is restarted, it would just get the full set fds back
<poettering> no, this would work for everything
<JayF> Yeah, that last line cleared it up for me
<poettering> it would allow journald to push the fds somewhere before restarting, and pull them out again after
<poettering> and continue where it left of
<JayF> that's awesome, and could be helpful for a bunch of different things as well
<phunyguy> Pantsu: sorry for the delay..  That's actually not a half-bad idea.  I will look into that when I finish this build.  For now it is just a wishlist item for me.
<JayF> Thanks for the insight, that's helpful and you saved me from going down a path that wouldn't work :)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/pkg-systemd-maintainers/attachments/20141127/1b1dd5ca/attachment-0002.sig>