Bug#850074: systemd: "Freezing execution" mode breaks batch upgrades

[Zack Weinberg]

On Tue, Jan 3, 2017 at 3:55 PM, Michael Biebl <biebl at debian.org> wrote:
> Am 03.01.2017 um 21:41 schrieb Zack Weinberg:
>>
>> I don't expect I will be able to reproduce the situation until the
>> next time the systemd package is updated.  However, I recall this
>> happening the past several times the systemd package was updated.
>
> A "apt-get install --reinstall systemd" does not trigger the issue?

Oh, I didn't think of that.

> Hm, yes. It looks like the system (or systemd specifically) was already
> in a bad state before the daemon-reexec and the crash/freeze was a
> result of that.

Indeed.  The problem was /run being full.  Here is a reproduction recipe:

# cat /dev/zero > /run/eatspace
cat: write error: No space left on device
# df -h | grep /run
tmpfs           128M  128M     0 100% /run
# apt-get install --reinstall systemd
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 3 not upgraded.
Need to get 0 B/2,453 kB of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 269661 files and directories currently installed.)
Preparing to unpack .../systemd_232-8_amd64.deb ...
Unpacking systemd (232-8) over (232-8) ...
Setting up systemd (232-8) ...
addgroup: The group `systemd-journal' already exists as a system group. Exiting.

Broadcast message from systemd-journald at kenaz (Tue 2017-01-03 16:16:42 EST):

systemd[1]: Freezing execution.


Message from syslogd at kenaz at Jan  3 16:16:42 ...
 systemd[1]: Freezing execution.
Failed to try-restart systemd-networkd.service: Connection timed out
See system logs and 'systemctl status systemd-networkd.service' for details.
Failed to try-restart systemd-resolved.service: Connection timed out
See system logs and 'systemctl status systemd-resolved.service' for details.
Failed to try-restart systemd-timesyncd.service: Connection timed out
See system logs and 'systemctl status systemd-timesyncd.service' for details.
Processing triggers for man-db (2.7.6.1-2) ...
Processing triggers for dbus (1.10.14-1) ...
Error: Timeout was reached

And I get the same "Failed to serialize state: Input/output error" in
the logs.  There is no /core file produced, though.

I don't know *why* /run was full originally, since it gets cleared on
reboot, but at a minimum I suggest that systemd.preinst should abort
the upgrade if /run is full.

zw