Bug#432154: LVM not working when *any* md array degraded

[Trent Buck]

Package: grub-pc
Version: 1.95+20070626-1
Severity: normal

I have the following disk setup:

    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=35350c38:74a95022:4eeb7f3f:b5bc804c devices=/dev/sdc1,/dev/sdd1
    ARRAY /dev/md1 level=raid1 num-devices=2 UUID=3c0b822d:7ad0dd9a:997fa798:3fa99da6 devices=/dev/sdc2,/dev/sdd2
    ARRAY /dev/md2 level=raid1 num-devices=2 UUID=e70dddb8:ff6cc34d:9cc0d25e:24d7f956 devices=/dev/sda1,/dev/sdb1
    ARRAY /dev/md3 level=raid1 num-devices=2 UUID=c3425a67:34302754:8254f148:ae65912a devices=/dev/sda4,/dev/sdb4

The md0 array is a LVM PV used by the "ares" VG:

    ares-root    /
    ares-home    /home
    ares-srv     /srv
    ares-swap

All filesystems are ext3, except for md3, which is reiserfs.

Note that / and /boot are on LVM, on RAID1.  While grub-probe does not
correctly detect the necessary modules (pc lvm raid ext2), manually
creating core.img and installing it allows grub to boot directly from
(ares-root).

However... I simulated the death of /dev/sdb (or it may have been sda,
I forget).  When I booted, instead of the grub terminal I was
presented with

    Welcome to GRUB!

    Entering into rescue mode...
    grub rescue> _

lsmod reported that the raid and lvm modules were definitely loaded.
Attempting to do "insmod normal" resulted in the "error: unknown
device."  Running the command "ls" printed a number of (mdX), (hdX)
and (hdX,Y) devices, but no LVM devices -- that is, (ares-X).

I COULD NOT WORK OUT HOW TO GET GRUB PAST THE RESCUE STAGE.

Fortunately, lilo was on another MBR, and it successfully booted from
/dev/mapper/ares-root.  After fiddling with all sorts of knobs, I
eventually gave up trying to get grub to work again, and waited for an
hour while /dev/md3 rebuilt.  (md2 is quite small and was rebuilt
before I started fiddling.)  Once all the raid arrays were all fully
functional, GRUB suddenly started working correctly again.

Since md3 had absolutely nothing to do with md0 and ares-root -- it
wasn't even on the same disks -- my initial suspicion is that grub
cannot currently boot from an LVM-on-RAID1 /boot if there are *any*
degraded arrays on the system.  This strikes me as a VERY VERY BAD
thing -- you need an OS to rebuild the array, and you need the array
rebuilt for grub to boot an OS.  Without an external unwedge like LILO
or an OS on a non-LVM partition, it seems impossible to get the
machine back up.

Additional notes:

While the array was degraded, "grub-probe /" gave the error "unknown
device" (IIRC), and update-grub terminates unsuccessfully.  When all
arrays working, "grub-probe /" gives the error "Unknown partition
map", and update-grub terminates successfully.  In both cases
'grub-install (hd0)' terminates unsuccessfully.

Finally, a very big thanks to the maintainer for holding my hand in
#grub on freenode for a couple of hours while I raged at rescue mode,
etc.

-- Package-specific info:

*********************** BEGIN /boot/grub/device.map
(hd0)   /dev/sdc
(hd1)   /dev/sdd
(hd2)   /dev/sda
(hd3)   /dev/sdb
*********************** END /boot/grub/device.map

*********************** BEGIN /boot/grub/grub.cfg
set default=0
set timeout=5
set root=(ares-root)

font (ares-root)/usr/share/grub/unifont.pff
set gfxmode=1024x768
insmod gfxterm
insmod vbe
terminal gfxterm

menuentry "Debian GNU/Linux" {
        linux   (ares-root)/vmlinuz root=/dev/mapper/ares-root ro vga=790
        initrd  (ares-root)/initrd.img
}
menuentry "Debian GNU/Linux (single-user mode)" {
        linux   (ares-root)/vmlinuz root=/dev/mapper/ares-root ro vga=790 singl e
        initrd  (ares-root)/initrd.img
}
menuentry "Debian GNU/Linux (rescue partition)" {
        linux   (md1)/vmlinuz root=/dev/md1 ro vga=790
        initrd  (md1)/initrd.img
}
menuentry "Debian GNU/Linux (old partition)" {
        linux   (md2)/vmlinuz root=/dev/md2 ro vga=790
        initrd  (md2)/initrd.img
}
*********************** END /boot/grub/grub.cfg

-- System Information:
Debian Release: 4.0
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-4-686
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8)

Versions of packages grub-pc depends on:
ii  libc6                       2.3.6.ds1-13 GNU C Library: Shared libraries
ii  liblzo2-2                   2.02-2       data compression library

grub-pc recommends no packages.

-- no debconf information