Bug#813879: systemd: Assertion 's->exec_command[SERVICE_EXEC_START]' failed service_enter_start()

[Yuriy M. Kaminskiy]

> Package: systemd
> Version: 215-17+deb8u3
> Severity: important
>
> Dear Maintainer,
>
> systemd crash badly while removing systemd-cron and installing cron. It does
> not respond anymore and reboot is not working.
>
> Installing systemd-cron
>
> Feb 05 20:45:48 debir systemd[1]: Reloading.
> Feb 05 20:45:49 debir systemd[1]: Reloading.
> Feb 05 20:45:49 debir systemd[1]: Starting [Cron] "0 * * * * root if [ -x /usr/sbin/backupninja ]; then /usr/sbin/backupninja; fi".
> Feb 05 20:45:49 debir systemd[1]: Started [Cron] "0 * * * * root if [ -x /usr/sbin/backupninja ]; then /usr/sbin/backupninja; fi".
> Feb 05 20:45:49 debir systemd[1]: Starting [Cron] "14 04 * * * root /usr/local/sbin/skcbackup 4".
> Feb 05 20:45:49 debir systemd[1]: Started [Cron] "14 04 * * * root /usr/local/sbin/skcbackup 4".
> Feb 05 20:45:49 debir systemd[1]: Starting systemd-cron path monitor.
> Feb 05 20:45:49 debir systemd[1]: Started systemd-cron path monitor.
> Feb 05 20:45:49 debir systemd[1]: Starting systemd-cron monthly timer.
> Feb 05 20:45:49 debir systemd[1]: Started systemd-cron monthly timer.
> Feb 05 20:45:49 debir systemd[1]: Starting systemd-cron weekly timer.
> Feb 05 20:45:49 debir systemd[1]: Started systemd-cron weekly timer.
> Feb 05 20:45:49 debir systemd[1]: Starting systemd-cron daily timer.
> Feb 05 20:45:49 debir systemd[1]: Started systemd-cron daily timer.
> Feb 05 20:45:49 debir systemd[1]: Starting systemd-cron hourly timer.
> Feb 05 20:45:49 debir systemd[1]: Started systemd-cron hourly timer.
> Feb 05 20:45:49 debir systemd[1]: Starting systemd-cron.
> Feb 05 20:45:49 debir systemd[1]: Reached target systemd-cron.
>
> Removing systemd-cron, installing cron
>
> Feb 05 20:51:02 debir systemd[1]: Stopping systemd-cron.
> Feb 05 20:51:02 debir systemd[1]: Stopped target systemd-cron.
> Feb 05 20:51:02 debir systemd[1]: Stopping systemd-cron hourly timer.
> Feb 05 20:51:02 debir systemd[1]: Stopped systemd-cron hourly timer.
> Feb 05 20:51:02 debir systemd[1]: Stopping systemd-cron daily timer.
> Feb 05 20:51:02 debir systemd[1]: Stopped systemd-cron daily timer.
> Feb 05 20:51:02 debir systemd[1]: Stopping systemd-cron weekly timer.
> Feb 05 20:51:02 debir systemd[1]: Stopped systemd-cron weekly timer.
> Feb 05 20:51:02 debir systemd[1]: Stopping systemd-cron monthly timer.
> Feb 05 20:51:02 debir systemd[1]: Stopped systemd-cron monthly timer.
> Feb 05 20:51:02 debir systemd[1]: Stopping [Cron] "14 04 * * * root /usr/local/sbin/skcbackup 4".
> Feb 05 20:51:02 debir systemd[1]: Stopped [Cron] "14 04 * * * root /usr/local/sbin/skcbackup 4".
> Feb 05 20:51:03 debir systemd[1]: Starting systemd-cron update units...
> Feb 05 20:51:03 debir systemd[1]: Reloading.
> Feb 05 20:51:03 debir systemd[1]: Assertion 's->exec_command[SERVICE_EXEC_START]' failed at ../src/core/service.c:1312, function service_enter_start(). Aborting.
> Feb 05 20:51:03 debir systemd[1]: Caught <ABRT>, dumped core as pid 30693.
> Feb 05 20:51:03 debir systemd[1]: Freezing execution.

Probably related:
systemd-cron contains cron.target
cron contains cron.service
Maybe systemd somehow mixes the two during reload? (Unlikely)

Probably related:
cron-update.service is triggered by some /etc/cron* directories change 
and invokes `systemctl daemon-reload` and `systemctl try-restart 
cron.target`. Maybe there are some racing when it is triggered right 
when cron.target is being stopped?

Probably related upstream commit: 96fb8242cc1ef6b0e28f6c86a4f57950095dd7f1
(aka v216-30-g96fb824), however, it likely fixes symptoms [assert() and 
abort], but not underlying issue [racing or whatever].

[...]
> Errors were encountered while processing:
>  cron
>  apticron
> E: Sub-process /usr/bin/dpkg returned an error code (1)
> root at debir:~#
>
> Can not reboot the system anymore:
>
> root at debir:~# reboot
> Failed to start reboot.target: Connection timed out
> Failed to open initctl FIFO: No such device or address
> Failed to talk to init daemon.
> root at debir:~#

Disclaimer: not an expert/maintainer/whatever, largely untested, feel 
free to correct.

Only 'init' process can reboot system normally, and after receiving any
fatal signal (SEGV, FPE, ABRT, etc), systemd switches to 'freezing'
state (for(;;) pause();), and then refuse to do anything.

Probably, only way out -
try to stop anything important by hand,
try to `umount ...` or `mount -o remount -r ...` all filesystems likely
not in use, then:
Ctrl-Alt-F1 (switch to console)
Alt-SysRq-e (this will kill all remaining processes with SIGTERM)
[wait a bit for everything to settle]
Alt-SysRq-i (SIGKILL remaining processes)
Alt-SysRq-s (sync filesystems)
Alt-SysRq-u (forcibly remount all filesystems readonly)
Alt-SysRq-b (reboot)

(see /usr/share/doc/linux-doc-3.16/Documentation/sysrq.txt.gz on details)

P.S. I wished there were backported systemd or something. There are a
lot of such issues silently fixed in upstream, that nearly 
impossible/impractical to backport into stable version :-\ (then again, 
there are a lot of incompatibilities with other packages, that would 
require more and more backports, to extent you'll end on 
sid/experimental :-|).

> root at debir:~# systemd-analyze dump
> Failed issue method call: Activation of org.freedesktop.systemd1 timed out
> root at debir:~# systemd-delta
> 0 overridden configuration files found.