Bug#1040622: systemd-sysv: reboot doesn't honor the grub-reboot settings; reboot -f does

[Theodore Y. Ts'o]

Package: systemd-sysv
Version: 252.6-1
Severity: normal

Dear Maintainer,

   * What led up to the situation?

I was updating the gce-xfstests[1] test appliance to Debian Bookworm from
Debian Bullseye.

[1] https://thunk.org/gce-xfstests

   * What exactly did you do (or not do) that was effective (or
     ineffective)?

Unfortunately kexec has not been reliable ever since sometime after the
5.4 kernel, at least on Google Compute Engine VM's.  (About 30-40% of
the time, the VM hangs after the kexec; about 10% of the time, the
machine is up, but it is very slow and limping, and /proc/interrupts
shows that some interrupt channel is going wild.  This is no doubt the
kernel bug interacting with some virtual hardware in the GCE VM, but
I've never been able to debug it.)

Because of issues with kexec, the primary way that I reboot into the
kernel that I want to test is to install the kernel as a dpkg package,
and then examine /boot/grub/grub.cfg to find out where it was inserted
into the grub's menu listing, and then run a command like "grub-reboot
1>4", where the number is found by examing the grub.cfg file.  An
example of this works can be found here[2].

[2] https://github.com/tytso/xfstests-bld/blob/9bae3253d57456987d995cf85379e9165e054381/test-appliance/files/usr/local/lib/gce-load-kernel#L169

This works *just* *fine* when using Debian Bullseye (which is using
systemd 247.3-7+deb11u2).

   * What was the outcome of this action?

Unfortunately, this no longer works in Debian Bookworm (with systemd
252.6-1).  In Debian Bookworm, the grub-reboot(8) setting is ignored
after triggering a reboot via /sbin/reboot.

Connecting to the serial console, it appears that "reboot" is going
through some code path that does NOT involve triggering a BIOS message
and going through grub, where (assuming the GRUB_TIMEOUT is set to some
non-zero value like 15) the grub menu would be displayed, and after 15
seconds, it would boot the kernel specified by grub-reboot, and then
clear the next_entry entry in /boot/grub/grubenv.

Instead, systemd appears to boot the default kernel, ignoring
/boot/grub/grubenv, so I don't enter the kernel which I had just
installed, and had selected via the grub-reboot(8) command.  Looking at
/boot/grub/grubenv, it still has the "next_entry=1>4" set by
grub-reboot(8), so it appears that /boot/grub/grubenv is being
completely ignored by reboot.

*However*, it is properly handled if I use reboot -f.  The "reboot -f"
command will briefly show the BIOS version information, and then the
Grub Menu, and then will boot the kernel selected by grub-reboot(8), and
afterwords the next_entry field is cleared from /boot/grub/grubenv.  So
this is not a "grub" problem, but in the reboot path selected by systemd
when doing a clean shutdown via the "reboot" command.   As near as I can
tell, grub is being bypassed, or at least grub is getting called in some
way whic causes it to ignore the /boog/grub/grubenv parameter.

In Debian Bookworm, "reboot" works like "reboot -f", in that we go
through the BIOS initialization step, printing the BIOS version, and
then grub is invoked in such a way that /boot/grub/grubenv is honored.

   * What outcome did you expect instead?

The "reboot" command should go through the normal, full reboot sequence,
such that grub-reboot(8) works correctly.  As a workaround, I've
replaced

   reboot
   sleep 60

with

   sync /boot
   sync
   sleep 1
   reboot -f
   sleep 60

However, it would be desirable to let the system go through a full,
clean shutdown, with file systems properly unmounted or remounted
read-only in the case of the root file system.


-- System Information:
Debian Release: 12.0
  APT prefers stable
  APT policy: (900, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-9-amd64 (SMP w/20 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages systemd-sysv depends on:
ii  systemd  252.6-1

Versions of packages systemd-sysv recommends:
ii  libnss-systemd  252.6-1
ii  libpam-systemd  252.6-1

systemd-sysv suggests no packages.

-- no debconf information