Bug#872999: qemu image built by vmdebootstrap is unbootable due to bad root=/dev/mapper/loop0p1 kernel parameter

[Ben Hutchings]

On Thu, 2017-09-07 at 17:53 +0200, Raphael Hertzog wrote:
> On Thu, 07 Sep 2017, Ben Hutchings wrote:
> > On Thu, 2017-09-07 at 12:53 +0200, Raphael Hertzog wrote:
> > > On Thu, 07 Sep 2017, Raphael Hertzog wrote:
> > > > What informs udev about the new filesystem?
> > > 
> > > Somehow, it's the kernel apparently.
> > > 
> > > > Or is this a bug in the kernel really?
> > > 
> > > At least "udevadm monitor" shows that the kernel is sending
> > > less uevents when I run the command in the chroot.
> > 
> > Which command?
> 
> vmdebootstrap (for example as called by "make sid.raw" in that directory):
> https://anonscm.debian.org/cgit/cloud/debian-vm-templates.git/tree/vmdebootstrap-generic-qcow2
> 
> The command creates the disk image with commands similar to those shown in this log:
> 2017-09-07 16:11:54 DEBUG runcmd: ['qemu-img', 'create', '-f', 'raw', '/srv/autopkgtest-images/sid-amd64-new.img', '10000000000'] None {}
> 2017-09-07 16:11:54 INFO Creating partitions
> 2017-09-07 16:11:54 DEBUG runcmd: ['parted', '-s', '/srv/autopkgtest-images/sid-amd64-new.img', 'mklabel', 'msdos'] None {}
> 2017-09-07 16:11:54 DEBUG runcmd: ['parted', '-s', '/srv/autopkgtest-images/sid-amd64-new.img', 'mkpart', 'primary', '0%', '100%'] None {}
> 2017-09-07 16:11:54 DEBUG runcmd: ['parted', '-s', '/srv/autopkgtest-images/sid-amd64-new.img', 'set', '1', 'boot', 'on'] None {}
> 2017-09-07 16:11:54 DEBUG runcmd: ['kpartx', '-avs', '/srv/autopkgtest-images/sid-amd64-new.img'] None {}
> 2017-09-07 16:11:54 INFO Creating filesystem ext4
> 2017-09-07 16:11:54 DEBUG runcmd: ['mkfs', '-t', 'ext4', u'/dev/mapper/loop0p1'] None {}
> 2017-09-07 16:11:55 DEBUG mkdir /tmp/tmpSEWpRs
> 2017-09-07 16:11:55 INFO Mounting /dev/mapper/loop0p1 on /tmp/tmpSEWpRs
> 2017-09-07 16:11:55 DEBUG runcmd: ['mount', u'/dev/mapper/loop0p1', '/tmp/tmpSEWpRs'] None {}
> 
> Then goes on with debootstrap but at this point already you are past the point
> where the kernel should have emitted the uevent that triggers the by-uuid symlink
> creation.

It's the mkfs command that triggers it; specifically the close() call
on the block device.  But I can't see why the kernel would send device
events at that time for a device mapper device (or a loop device, which
I tested with because it's simpler to set up).

The answer seems to be that udev doesn't just listen for device events,
but also uses inotify to watch block devices.  But inotify operates on
inodes, not the underlying devices.  The chroot has a separate /dev
directory and inodes, so writing to a block device through one of those
inodes doesn't trigger the inotify watch.

If I bind-mount /dev into the chroot before running mkfs there, udev
does see the change events because mkfs opens the inodes that it's
watching.

So this is a limitation of udev and of the kernel's inotify interface,
but I don't think it's a bug in either of them.  vmdebootstrap should
not assume that udev will notice changes made by mkfs unless they are
operating on the same /dev filesystem (and even then, it should use
'udevadm settle' to avoid races).

Ben.

-- 
Ben Hutchings
All the simple programs have been written, and all the good names
taken.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://alioth-lists.debian.net/pipermail/pkg-systemd-maintainers/attachments/20170911/c38e848e/attachment-0002.sig>