Bug#764798: grub2: Grub rescue shell with RAID 6 mdadm over 8 disks

[Mike]

Hi Colin,

I sent this yesterday but not sure it was processed:

OK so after spending many more hours, I found that 10 disk RAID 6 members
are bootable (mdadm assembles in grub) while 11 disk and above does not.

This is from very simple VMWare test using the latest Jessie ISO build, and
a clean install.  Once you hit 11 members in RAID 6, grub drops to rescue
complaining it can't find the uuid.

This aligns with the 8-disk limit.  In RAID 6, you will need 8 usable disks
to assemble a 10-disk member array (two parity).  So grub can see only the
first 8 disks, which is enough to get us started.

So I believe there is still an 8-disk limit in grub and this is a bug.

As for the VMWare setup, I created (14) HDD's @ 10GB in size each and
assigned them to an LSI Logic SAS bus controller.  Installed Debian,
assigned each 14 disk to RAID, then tried various mdadm configurations
randing from 8 disks to 14 disks to determine that grub is stopping at the 8
disk limit with mdadm.  I also tried RAID 0 to make sure it wasn't a RAID 6
issue and had the same result.

I am happy to post my entire VMWare file if you want (~2GB).  I'll email you
the link.

Regarding previous versions of grub, I know only that the stable 1.99 has
the same problem because I upgraded to 2.x in an effort to fix it.

-----Original Message-----
From: Colin Watson [mailto:cjwatson at debian.org] 
Sent: Sunday, October 12, 2014 2:36 AM
To: Mike B; 764798 at bugs.debian.org
Subject: Re: Bug#764798: grub2: Grub rescue shell with RAID 6 mdadm over 8
disks

On Sat, Oct 11, 2014 at 03:00:11AM -0500, Mike B wrote:
> Package: grub2-common
> Version: 2.02~beta2-11
> Severity: critical
> File: grub2
> Justification: breaks the whole system

Do you know if there was a previous version of GRUB 2 where this worked?
For example, how does the version in testing behave?  (This is for
practical reasons: if this has just always been broken in GRUB 2, then
it shouldn't block migration of 2.02~beta2 to testing.)

> I initially was running an 8-disk RAID 6 mdadm.  Everything worked
> fine.  The system was upgraded to 14 disks RAID 6 online.  Eventually
> the system was rebooted and now grub drops to rescue complaining of
> invalid UUID.
> 
> Upon further inspection, grub is only seeing 8 disks (ls) instead of
> 14 and cannot assemble the root mdadm device.

OK, so that implies that this doesn't have much directly to do with RAID
handling: the problem is probably simply that GRUB can't see all 14
disks.

> I've tested using VMWare and confirmed that 8-disk RAID 6 mdadm arrays
> work fine with grub for root, but 9 disk and above break the system
> and drop to rescue shell.  

A quick test in qemu shows that GRUB can see more than eight disks
there.  Could you please share details of your VMware setup so that I
might stand a chance of reproducing this?

Thanks,

-- 
Colin Watson                                       [cjwatson at debian.org]