Bug#1091636: journald: Error when rotating logs: "Failed to append tag when closing journal: Invalid argument"

[Timo Sigurdsson]

Hi,

Luca Boccassi schrieb am 29.12.2024 12:28 (GMT +01:00):

> Control: tags -1 moreinfo
> 
> On Sat, 28 Dec 2024 23:31:18 +0100 (CET) "Timo Sigurdsson"
> <public_timo.s at silentcreek.de> wrote:
>> Package: systemd
>> Version: 252.31-1~deb12u1
>> Severity: normal
>> 
>> Dear Maintainer,
>> 
>> I use systemd-journald's sealing feature on all my Debian servers.
>> Ever since upgrading to Bookworm, I frequently see the following error
>> message after (automatically) rotating the logs:
>> Failed to append tag when closing journal: Invalid argument
>> 
>> This is on all my Debian installations. My configuration (across all)
>> is not that special. I set up sealing keys and use the default
>> configuration with the following drop-in configuration file
>> /etc/systemd/journald.conf.d/local.conf:
>> [Journal]
>> Storage=persistent
>> SplitMode=none
>> SystemMaxUse=2G
>> SystemMaxFileSize=128M
>> MaxRetentionSec=3month
>> 
>> 
>> Please note, that I also saw this error on Debian Bullseye, so it's
>> not really new to Bookworm. But on Bullseye, the error occurred *far*
>> less frequently (I think it was mostly when upgrading systemd packages
>> after a new stable point release). Now I see it regularly, depending on
>> the log volume almost daily or every other day. It seems to me that the
>> first log rotation works without issues and then any subsequent
>> rotation will exhibit the error (logs below). When I verify the logs
>> with journalctl, the validation passes mostly, but every once in a
>> while I do see a corrupted file (examples below). Other than that,
>> journald seems to behave just fine despite the error message.
>> 
>> What still puzzles me is that this error doesn't seem to be more
>> common. I tried searching for it on the web and yielded no results (log
>> message in quotes). Usually when I look up other error messages, I find
>> other users with the same issue, but not in this case. I'm wondering if
>> I'm the only Debian or systemd user actually using journald's sealing
>> feature :P Because, as I mentioned, my setup is not that special, I
>> think.
> 
> Can you reproduce this on testing/unstable?

No, I can't. I just set up two VMs, one with Debian testing and one with stable. I can easily reproduce the issue one stable, but not on testing.

Here's what I did:

1) Install Debian in a VM using the netinstall image (debian-12.8.0-amd64-netinst.iso / debian-testing-amd64-netinst.iso) with btrfs as the root filesystem and only standard system utilities and SSH server
2) Add journald drop-in configuration file /etc/systemd/journald.conf.d/local.conf with the same contents as in my bug report:
[Journal]
Storage=persistent
SplitMode=none
SystemMaxUse=2G
SystemMaxFileSize=128M
MaxRetentionSec=3month

3) Run `journalctl --setup-keys'
4) Restart systemd-journald
5) Generate some log messages by executing `systemd-cat cat /proc/cpuinfo' a few times (5-10 times)
6) Rotate the journal manually with `journalctl --rotate'
7) Repeat staps 5) and 6) a few times (~5 times)

On stable, this triggers the error and the error message is shown on the console:
[  341.166029} systemd-journald[623]: Failed to append tag when closing journal: Invalid argument

On testing, I do not see the error with this approach, even if I repeat steps 5) and 6) more often.


Kind regards,

Timo