Heads up: persistent journal has been enabled in systemd

[Guillem Jover]

Hi!

On Tue, 2020-02-11 at 08:19:46 +0100, Helmut Grohne wrote:
> I am an early adopter (at a time when you had to pass init= to use
> systemd) and I also enabled the persistent journal on practically all of
> my systems. I find myself liking the filtering that is enabled by
> journalctl, but it seems to come at a cost: performance.

> On multiple systems (embedded, VMs, desktops, servers) I observe that
> journalctl takes longer to display the initial batch than I am willing
> to wait. Unfortunately, this also affects systemctl status. I admit that
> my patience is quite limited here. Having to wait 3 seconds for
> systemctl status someservice is already more than I am willing to wait.
> As such, I find myself resorting to plaintext logs more often than not
> to avoid the annoying delay. It gets way worse on a busy system where
> you'd need the journal most to figure out what's wrong.
> 
> Do you happen to know more about the performance aspects?
>  * Known discussions?
>    -> https://github.com/systemd/systemd/issues/2460
>  * Workarounds / tricks?
>    -> I already apply a size limit.
> 
> Memory consumption by journald itself is also worth a mention. For
> servers, I usually don't care, but for embedded systems it is sometimes
> difficult to afford. rsyslog comes at around a fifth of what journald
> needs.
> 
> As such, I question whether the journal is ready for production while at
> the same time wanting it to be. I believe that the github issue above
> should be fixed before enabling the persistent journal by default.

Unfortunately this all matches our experience too. At work we also tried
to enable persistent journal, to potentially also remove rsyslogd, and
to get the enhanced logging combined with service status, etc. But had
not only to revert that as the run-time performance degraded significantly
(not just when using systemctl or journalctl, but during normal system
operation too), we even had to set:

  Storage=none
  RateLimitIntervalSec=0s
  RateLimitBurst=0

to cope. :/ The default does not affect us in any way, we'd really like
to switch fully, and we've not tried yet with newer systemd versions,
but just mentioning as a word of warning of what this might imply.

Thanks,
Guillem