Bug#768644: systemd: missing dependency on /var being mounted

[David Kozub]

Ok, thank you.

So here is the actual issue that triggered all this digging on my side 
(from systemctl --failed):

● debian-fixup.service loaded failed failed Various fixups to make 
systemd work better on Debian

I'm seeing the original issue on a Raspberry Pi with raspbian (i.e. not 
Debian) and I've been trying to reproduce the issue in Debian, to see if 
it's affected.

I eventually managed to reproduce this without any suspicious hacks in 
Debian configuration: I wrapped the qemu image with jessie in a dm delay, 
making all reads (and writes) delayed by 200 ms thus making the disk 
appear really slow for the jessie installation running inside the qemu.

I (still) think the cause is a race condition between mounting /var and 
executing the debian-fixup-service. It needs /var (as it accesses and 
potentially modifies /var/run and /var/lock). The error message (from 
jessie) is:

Nov 09 20:53:02 jessie debian-fixup[224]: ln: failed to create symbolic
link ‘/var/run’: Read-only file system

For the issue to trigger there must be two more things fulfilled:

1) the debian-fixup.service must be run; this means its conditions must be 
fulfilled

If the actual /var has not been mounted yet, the conditions check whatever 
is in /var on rootfs. In my jessie instalation where I tested this, the 
/var on rootfs contained something, including the run and lock symlinks, 
so the service would not be run even if the wrong /var was visible. (I 
wonder why the directory is not empty. I think it has been created that 
way by the debian installer but I'm not 100% sure after all the hacks...)

On the Raspberry Pi where I originally saw the issue, the /var on rootfs 
was empty so if the fixup service conditions were evaluated over /var on 
rootfs, they were fulfilled.

-> To fulfill the conditions, I just deleted the contents of /var on
rootfs so that it's empty.

2) rootfs has to be read only

To achieve this I just added "ro" to mount options of / in /etc/fstab.

If rootfs is rw then writing to /var on roofs (which is what I believe 
happens) will succeed and the issue will not be visible even if the 
debian-fixup.service accessed the wrong /var.

I'm attaching the output of "systemd-analyze dump" and "journalctl -alb" 
(with systemd.log_level=debug), as you requested in previous message.

Btw. in journalctl output I see:

...
Nov 09 20:53:02 jessie debian-fixup[224]: ln: failed to create symbolic 
link ‘/var/run’: Read-only file system
...
Nov 09 20:53:05 jessie systemd[1]: Mounting /var...
...

so this would corroborate that /var is mounted too late.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: journalctl-alb.log
URL: <http://alioth-lists.debian.net/pipermail/pkg-systemd-maintainers/attachments/20141109/e784730f/attachment-0004.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: systemd-analyze-dump.log
URL: <http://alioth-lists.debian.net/pipermail/pkg-systemd-maintainers/attachments/20141109/e784730f/attachment-0005.ksh>