Bug#760148: grub-common: grub-mount hangs forever, trying to mount ext4 partition, causing 100% CPU load

[Andreas Born]

Hi,

since this issue appeared here too with an Arch Linux installation,
I'd like to provide some more details collected by me, which might
eventually help to get this fixed. Given the issue I experiences
matches this bug report, I decided to add to this report albeit it
being old. Also note that neither os-prober nor grub development
repositories fix this issue as of today nor am I aware of pending
fixes.

Here's my observations:
- The `ls /mnt/lib*/ld*.so*' call is causative while e.g. `ls /mnt' is
unproblematic.
- Eventually after many minutes with high cpu and memory usage by
grub-mount the ls call returns successfully, i.e. grub-mount/ls do not
hang per se. The whole of os-prober was something like 20-40min for
me. Don't be too impatient, if you want OS detection. ;D
- Reproducible with a large directory (Arch's /lib contains >3900
entries here). Example:
$ mkdir blub
$ for i in {0000..2000}; do echo $i >blub/test_$i; done
- But not really reproducible, i.e. significantly faster, for a
small/empty filesystem.
- Not reproducible in a grub-shell during boot, i.e. ls blub or ls lib
return almost instantly.
- Not reproducible for a single file in such a directory:  `ls /lib/ld-2.22.so'

Below I've attached a shell transcript, which shows the behaviour with
time. It also shows how the reaction time significantly improves with
an empty filesystem and how the same applies to xfs.
Additionaly I'm attaching a file containing part of the strace ouput
of grub-mount, while it "hangs". Looking closely one can see that it's
not just dead looping a lot of reads and seeks, but progressing.
Overall it seems to be a problem with (GRUB's) FUSE implementation.

Should you still need an archive of the lib tree to reproduce let me
know. My Arch /lib* is unfortunately 4GB...

Cheers,
Andreas

*********************** BEGIN TRANSCRIPT
# grub-mount -V
grub-mount (GRUB) 2.02~beta2

# mount /dev/<ARCH-ROOT> /normal-mnt
# mkdir /normal-mnt/blub
# for i in {0000..2000}; do echo $i >/normal-mnt/blub/test_$i; done
# sync
# grub-mount /dev/<ARCH-ROOT> /grub-mnt

# time ls -1f /grub-mnt/blub | wc -l <--- Not as slow as lib but still slow
2003

real 1m3.576s
user 0m0.007s
sys 0m0.000s

# time ls -1f /normal-mnt/blub | wc -l
2003

real 0m0.004s
user 0m0.003s
sys 0m0.000s

# time ls -1f /grub-mnt/lib | wc -l  <--- Here it's even without
asteriks really slow
3918

real 6m9.033s
user 0m0.003s
sys 0m0.003s

# time ls -1f /normal-mnt/lib | wc -l
3915

real 0m0.161s
user 0m0.003s

# truncate -s 1G blub
# mkfs -t ext4 blub
[...]
# mount -o loop blub /mnt
# mkdir mnt/blub
# for i in {0000..2000}; do echo $i>mnt/blub/test_$i; done
# umount mnt
# grub-mount blub /mnt

# time ls -1f mnt/blub | wc -l  <-- for empty fs much better performance
2003

real 0m5.137s
user 0m0.003s
sys 0m0.000s

# rm blub
# truncate -s 1G blub
# mkfs -t xfs blub
[...]
# mount -o loop blub mnt
# mkdir mnt/blub
# for i in {0000..2000}; do echo $i>mnt/blub/test_$i; done
# umount mnt
# ./grub-mount blub /mnt  <-- only grub-mount from latest git master
handles xfs properly

# time ls -1f mnt/blub/ | wc -l  <-- not just limited to ext*
2003

real 0m9.701s
user 0m0.000s
sys 0m0.000s
*********************** END SHELL SESSION
-------------- next part --------------
A non-text attachment was scrubbed...
Name: grub-lib.log.gz
Type: application/x-gzip
Size: 23856 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-grub-devel/attachments/20160130/145ec453/attachment-0001.bin>