Bug#850074: systemd: "Freezing execution" mode breaks batch upgrades

[Zack Weinberg]

On Tue, Jan 3, 2017 at 3:19 PM, Michael Biebl <biebl at debian.org> wrote:
>> An ideal fix would be for systemd to support seamless upgrades of itself,
>> i.e. pid 1 would re-exec itself using the new binary, without losing
>> any state.
>
> Well, that's actually what's already happening. In postinst we run
> systemctl daemon-reexec which will start the new binary and transfer the
> state from the old process.
>
> For some reason systemd crashed on your system before that happened.
>
> Can you reproduce the crash or do you at least have a core file from the
> crash? Do you have a complete journal log?

I don't expect I will be able to reproduce the situation until the
next time the systemd package is updated.  However, I recall this
happening the past several times the systemd package was updated.

I do not have any journal-format logs, having set Storage=none after
the incident some months ago when it filled up /run with journal
fragments.  The only things in traditional logfiles that appear to be
relevant are:

/var/log/dpkg.log ...
2017-01-03 13:58:59 upgrade systemd:amd64 232-7 232-8
2017-01-03 13:58:59 status half-configured systemd:amd64 232-7
2017-01-03 13:58:59 status unpacked systemd:amd64 232-7
2017-01-03 13:58:59 status half-installed systemd:amd64 232-7
2017-01-03 13:58:59 status triggers-pending dbus:amd64 1.10.14-1
2017-01-03 13:58:59 status triggers-pending dbus:amd64 1.10.14-1
2017-01-03 13:58:59 status half-installed systemd:amd64 232-7
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 startup packages configure
2017-01-03 13:59:00 configure systemd:amd64 232-8 <none>
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:00 status unpacked systemd:amd64 232-8
2017-01-03 13:59:01 status unpacked systemd:amd64 232-8
2017-01-03 13:59:01 status unpacked systemd:amd64 232-8
2017-01-03 13:59:01 status half-configured systemd:amd64 232-8
2017-01-03 14:00:16 status installed systemd:amd64 232-8

/var/log/messages ...
Jan  3 13:57:57 kenaz systemd[1]: Reloading.
Jan  3 13:57:57 kenaz systemd[1]: Failed to reload: Input/output error
Jan  3 13:58:59 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:58:59 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:58:59 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:58:59 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:58:59 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:58:59 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:58:59 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:58:59 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:58:59 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:00 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:00 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:00 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:00 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:00 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:00 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:00 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:00 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:00 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:00 kenaz dbus[512]: [system] Reloaded configuration
Jan  3 13:59:01 kenaz systemd[1]: Reloading.
Jan  3 13:59:01 kenaz systemd[1]: Failed to reload: Input/output error
Jan  3 13:59:01 kenaz systemd[1]: Reloading.
Jan  3 13:59:01 kenaz systemd[1]: Failed to reload: Input/output error
Jan  3 13:59:01 kenaz systemd[1]: Reloading.
Jan  3 13:59:01 kenaz systemd[1]: Failed to reload: Input/output error
Jan  3 13:59:01 kenaz systemd[1]: Reloading.
Jan  3 13:59:01 kenaz systemd[1]: Failed to reload: Input/output error
Jan  3 13:59:01 kenaz systemd[1]: Reloading.
Jan  3 13:59:01 kenaz systemd[1]: Failed to reload: Input/output error
Jan  3 13:59:01 kenaz systemd[1]: Failed to serialize state: Input/output error
Jan  3 13:59:01 kenaz systemd[1]: Freezing execution.

So you can see that the freeze was directly triggered by the upgrade,
but there seems to have been some pre-existing problem where unit
reloads were failing.  If you can think of reasons why that would
happen I can investigate further.  (There is no evidence pointing to a
disk failure or anything like that.)

I do not *believe* I have a core dump; where should I look for one?

> Which version of systemd was running (not installed) when you attempted
> the upgrade?

232-7.

zw