Bug#872999: qemu image built by vmdebootstrap is unbootable due to bad root=/dev/mapper/loop0p1 kernel parameter

[Raphael Hertzog]

[ Ccing grub, udev, linux maintainers for the questions at the end,
  I want to understand why some /dev/disk/by-uuid/ entries are not
  properly created ]

Hello,

On Wed, 23 Aug 2017, Raphaël Hertzog wrote:
> Hello, I'm trying to update my qemu image used for autopkgtest, so I
> followed the instructions from man autopkgtest-virt-qemu to create the
> image but the resulting image does not boot and thus does not work
> for autopkgtest either.
> 
> I believe that the root cause is this error shown in the vmdebootstrap
> output:
> 
>     /usr/sbin/grub-probe: error: cannot find a device for / (is /dev mounted?).
>     WARNING: update-grub failed!
> 
> Maybe the behaviour of update-grub changed recently... vmdebootstrap

So this is not the root cause. This is just a problem in the customize
script which should mount /dev and /proc just like vmdebootstrap already
does when it calls update-grub. Or it should probably get rid of it
entirely as vmdebootstrap mostly overwrites the grub configuration anyway.

> When I inspect /boot/grub/grub.cfg it does indeed contain
> root=/dev/mapper/loop0p1 which we don't want.

After a discussion on #debian-cloud, I narrowed the problem a bit more
closely. In fact vmdebootstrap works properly when I run it unchrooted
on my system, but when I run it in "schroot" (with a default profile
which shares with the host the mount for /dev, /proc, /sys and more) then
I get this problem.

So it looks like that update-grub is behaving differently in each situation.

I have added "set -x" to the grub-mkconfig and it looks like grub is perfectly
able to find out the UUID associated to device:
$ virt-cat -a sid.raw /tmp/update-grub-log |grep -i uuid
+ /usr/sbin/grub-probe --device /dev/mapper/loop0p1 --target=fs_uuid
+ GRUB_DEVICE_UUID=600ae34d-dceb-4250-a832-f07fa0c932db
+ /usr/sbin/grub-probe --device /dev/mapper/loop0p1 --target=fs_uuid
+ GRUB_DEVICE_BOOT_UUID=600ae34d-dceb-4250-a832-f07fa0c932db
[...]

Then I added "set -x" to /etc/grub.d/10_linux to find out why this script
decided to not use the UUID even though it was available in the
environment. It boils down to
/dev/disk/by-uuid/600ae34d-dceb-4250-a832-f07fa0c932db does not exist.

The loop partition device is /dev/dm-4 and at the time when the customize
script runs, I have this output (it exists in by-partuuid, but not in by-uuid):

$ ls -al /dev/disk/by-uuid/
total 0
drwxr-xr-x 2 root root 140 sept.  5 15:42 .
drwxr-xr-x 7 root root 140 août  23 08:34 ..
lrwxrwxrwx 1 root root  10 sept.  1 08:54 51d0ffff-3c3e-49f5-adf6-6d53e7c0a8d6 -> ../../dm-0
lrwxrwxrwx 1 root root  10 sept.  1 08:54 5b668fa1-c4c5-44f7-bbec-ed5edfc33c0d -> ../../dm-3
lrwxrwxrwx 1 root root  10 sept.  1 08:54 92a990ff-52c2-4766-b865-a78cceb7e293 -> ../../dm-2
lrwxrwxrwx 1 root root  10 sept.  1 08:54 b3816590-a760-4815-9f43-0c2f4a362abc -> ../../sda2
lrwxrwxrwx 1 root root  10 sept.  1 08:54 BF87-23E5 -> ../../sda1

$ ls -al /dev/disk/by-partuuid/
total 0
drwxr-xr-x 2 root root 200 sept.  7 09:04 .
drwxr-xr-x 7 root root 140 août  23 08:34 ..
lrwxrwxrwx 1 root root  10 sept.  1 08:54 01f46df3-eb66-47b9-b0ea-a667e4a02020 -> ../../sda5
lrwxrwxrwx 1 root root  10 sept.  1 08:54 1dd56def-bfb8-423e-8ea5-d03ca1f2ff75 -> ../../sda6
lrwxrwxrwx 1 root root  10 sept.  7 09:04 3dfece35-01 -> ../../dm-4
lrwxrwxrwx 1 root root  10 sept.  1 08:54 426ccd8e-0ccd-4864-9152-31fc831ae98a -> ../../sda4
lrwxrwxrwx 1 root root  10 sept.  1 08:54 525345f4-e509-4283-8489-a6c1e7625975 -> ../../sda3
lrwxrwxrwx 1 root root  10 sept.  1 08:54 6ce3844f-39e8-4a7f-8e37-05b6bb9efe2f -> ../../sda1
lrwxrwxrwx 1 root root  10 sept.  1 08:54 caff8cf5-b264-498e-b6be-e6160fcf2085 -> ../../sda7
lrwxrwxrwx 1 root root  10 sept.  1 08:54 d67d853a-592f-4e21-9ed7-5992359aaa7d -> ../../sda2

$ ls -al /dev/mapper/loop0p1
brw-rw---- 1 root disk 254, 4 sept.  7 09:04 /dev/mapper/loop0p1


And when I run it outside from my schroot, then I have the entry
in /dev/disk/by-uuid/. What is responsible of creating that entry?

I guess it's udev. udev is not running in the chroot, obviously.
But the underlying device (and the mount point) is seen by the host so I wonder
why it's not created.

What informs udev about the new filesystem?

Is there anything that can be done by vmdebootstrap to force-trigger the
creation of that entry even when chrooted?

Should update-grub not try to be too smart and just trust the value
obtained by grub-probe instead even if /dev/disk/by-uuid/$foo does not
exist?

Or is this a bug in the kernel really?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/