Bug#1133901: reboots do not issue warnings on console anymore since trixie upgrade

[Antoine Beaupre]

Package: systemd
Version: 257.9-1~deb13u1
Severity: normal

Hi folks!

It seems like we have a regression since our trixie fleet upgrade,
where delayed reboot notifications don't show up on SSH consoles
anymore.

For background, we manage about 150 Debian servers at the Tor project
and have some automation around reboots where we issue `shutdown -r
+10 "security upgrade` commands on a regular basis. We expect those to
give our shell users a heads up that stuff is going to happen,
particularly people logged into IRC bouncers or working over SSH
otherwise.

Since the trixie upgrade, this just stopped working altogether. We
don't see reboot notifications.

It's an open question for us whether we really need this anymore: more
and more of our deployments are done with containers and CI, and we
kind of hope to get rid of the old "SSH over and do things" manual
models. But we still have it, so, for now, I'm kind of assuming this
is supposed to work.

This is tracked as an issue on our end in:

https://gitlab.torproject.org/tpo/tpa/team/-/issues/42137

I understand that it might work under some circumstances. I've heard
reports like "this works fine for me", but this is real for us: across
the entire fleet, we never see, as far as I can tell, any of those
notifications anymore, and there's a clear correlation with the trixie
upgrade.

Specifically, this was previously discussed in #1087642, and #1075733,
and it seems both issues were closed as "works for me" and "tests
pass", but this just *doesn't* "work for me". This bug is real and
current.

It's unclear to me how to actually debug this either, and maintainers
have not provided clear guidance on how to actually start debugging
this problem.

I'm not very familiar with the systemd codebase, so I don't know
exactly where to start. I'm hoping Debian systemd folks could help! I
was told this could be related to upstream issue #36930
(sd_get_sessions doesn't report TTY information), and we do match the
symptoms in the sense that our logged in users do not have a TTY
attached:

anarcat at idle-dal-02:~$ loginctl 
SESSION  UID USER    SEAT LEADER CLASS         TTY IDLE SINCE
  22888    0 root    -    844925 user          -   no   -    
  22889    0 root    -    844931 manager-early -   no   -    
  22890 2128 anarcat -    844965 user          -   no   -    
  22891 2128 anarcat -    844970 manager       -   no   -    

4 sessions listed.

A fix for that (5af674214abbe16ee57352de24eaef5dc97afdd1) was shipped
in 258-rc1, while trixie has only 257.9-1~deb13u1.

I wonder if the "it works for me" reports are from users running
unstable that have the above fix.

I was hoping a backport of the patch could resolve the issue, but then
I was also told our systemd ships wihout utmp support, so all of the
above is essentially moot (or is that the root cause?).

Anyways, would love a little more guidance here to remove this small
irritant!

a.

-- Package-specific info:

-- System Information:
Debian Release: 13.4
  APT prefers stable-security
  APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'stable'), (1, 'experimental'), (1, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.12.74+deb13+1-amd64 (SMP w/16 CPU threads; PREEMPT)
Locale: LANG=en_CA.UTF-8, LC_CTYPE=en_CA.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages systemd depends on:
ii  libacl1            2.3.2-2+b1
ii  libapparmor1       4.1.0-1
ii  libc6              2.41-12+deb13u2
ii  libmount1          2.41-5
ii  libpam0g           1.7.0-5
ii  libseccomp2        2.6.0-2
ii  libselinux1        3.8.1-1
ii  libssl3t64         3.5.5-1~deb13u2
ii  libsystemd-shared  257.9-1~deb13u1
ii  libsystemd0        257.9-1~deb13u1
ii  mount              2.41-5

Versions of packages systemd recommends:
ii  dbus [default-dbus-system-bus]   1.16.2-2
ii  linux-sysctl-defaults            4.12.1
ii  systemd-cryptsetup               257.9-1~deb13u1
pn  systemd-timesyncd | time-daemon  <none>

Versions of packages systemd suggests:
ii  libtss2-tcti-device0t64 [libtss2-tcti-device0]  4.1.3-1.2
ii  polkitd                                         126-2
pn  systemd-boot                                    <none>
ii  systemd-container                               257.9-1~deb13u1
pn  systemd-homed                                   <none>
ii  systemd-repart                                  257.9-1~deb13u1
pn  systemd-resolved                                <none>
pn  systemd-userdbd                                 <none>

Versions of packages systemd is related to:
ii  dbus-user-session  1.16.2-2
pn  dracut             <none>
ii  initramfs-tools    0.148.3
ii  libnss-systemd     257.9-1~deb13u1
ii  libpam-systemd     257.9-1~deb13u1
ii  udev               257.9-1~deb13u1

-- no debconf information