Bug#1091636: journald: Error when rotating logs: "Failed to append tag when closing journal: Invalid argument"

[Luca Boccassi]

Control: tags -1 -moreinfo
Control: close -1 257-1

On Mon, 30 Dec 2024 at 13:17, Timo Sigurdsson
<public_timo.s at silentcreek.de> wrote:
>
> Hi,
>
> Luca Boccassi schrieb am 29.12.2024 12:28 (GMT +01:00):
>
> > Control: tags -1 moreinfo
> >
> > On Sat, 28 Dec 2024 23:31:18 +0100 (CET) "Timo Sigurdsson"
> > <public_timo.s at silentcreek.de> wrote:
> >> Package: systemd
> >> Version: 252.31-1~deb12u1
> >> Severity: normal
> >>
> >> Dear Maintainer,
> >>
> >> I use systemd-journald's sealing feature on all my Debian servers.
> >> Ever since upgrading to Bookworm, I frequently see the following error
> >> message after (automatically) rotating the logs:
> >> Failed to append tag when closing journal: Invalid argument
> >>
> >> This is on all my Debian installations. My configuration (across all)
> >> is not that special. I set up sealing keys and use the default
> >> configuration with the following drop-in configuration file
> >> /etc/systemd/journald.conf.d/local.conf:
> >> [Journal]
> >> Storage=persistent
> >> SplitMode=none
> >> SystemMaxUse=2G
> >> SystemMaxFileSize=128M
> >> MaxRetentionSec=3month
> >>
> >>
> >> Please note, that I also saw this error on Debian Bullseye, so it's
> >> not really new to Bookworm. But on Bullseye, the error occurred *far*
> >> less frequently (I think it was mostly when upgrading systemd packages
> >> after a new stable point release). Now I see it regularly, depending on
> >> the log volume almost daily or every other day. It seems to me that the
> >> first log rotation works without issues and then any subsequent
> >> rotation will exhibit the error (logs below). When I verify the logs
> >> with journalctl, the validation passes mostly, but every once in a
> >> while I do see a corrupted file (examples below). Other than that,
> >> journald seems to behave just fine despite the error message.
> >>
> >> What still puzzles me is that this error doesn't seem to be more
> >> common. I tried searching for it on the web and yielded no results (log
> >> message in quotes). Usually when I look up other error messages, I find
> >> other users with the same issue, but not in this case. I'm wondering if
> >> I'm the only Debian or systemd user actually using journald's sealing
> >> feature :P Because, as I mentioned, my setup is not that special, I
> >> think.
> >
> > Can you reproduce this on testing/unstable?
>
> No, I can't. I just set up two VMs, one with Debian testing and one with stable. I can easily reproduce the issue one stable, but not on testing.
>
> Here's what I did:
>
> 1) Install Debian in a VM using the netinstall image (debian-12.8.0-amd64-netinst.iso / debian-testing-amd64-netinst.iso) with btrfs as the root filesystem and only standard system utilities and SSH server
> 2) Add journald drop-in configuration file /etc/systemd/journald.conf.d/local.conf with the same contents as in my bug report:
> [Journal]
> Storage=persistent
> SplitMode=none
> SystemMaxUse=2G
> SystemMaxFileSize=128M
> MaxRetentionSec=3month
>
> 3) Run `journalctl --setup-keys'
> 4) Restart systemd-journald
> 5) Generate some log messages by executing `systemd-cat cat /proc/cpuinfo' a few times (5-10 times)
> 6) Rotate the journal manually with `journalctl --rotate'
> 7) Repeat staps 5) and 6) a few times (~5 times)
>
> On stable, this triggers the error and the error message is shown on the console:
> [  341.166029} systemd-journald[623]: Failed to append tag when closing journal: Invalid argument
>
> On testing, I do not see the error with this approach, even if I repeat steps 5) and 6) more often.

Ok, there have been a huge number of changes between stable and
testing, and unless you want to spend your time bisecting to find the
exact commit that fixed the issue (in which case it could be
backported), it's not going to be solved in bookworm.