[pkg-cryptsetup-devel] Bug#1032734: OOM when unlocking encrypted root in initramfs
Guilhem Moulin
guilhem at debian.org
Sat Mar 11 18:59:09 GMT 2023
Control: found -1 2:2.1.0-5+deb10u2
Control: tag -1 moreinfo
Hi kibi,
On Sat, 11 Mar 2023 at 15:16:01 +0100, Cyril Brulebois wrote:
> Guilhem Moulin <guilhem at debian.org> (2023-03-11):
>> On Sat, 11 Mar 2023 at 08:26:27 -0500, Jérôme Charaoui wrote:
>>> Today I upgraded a small KVM machine with a LUKS2 encrypted root and 1GiB of
>>> RAM to bookworm, and was very surprised to be confronted with an OOM
>>> immediately upon entering my LUKS password in the initramfs prompt:
>>> […]
>>> The problem appears to be perhaps related to #924560, but in this instance,
>>> the issue causing an unbootable system post-upgrade.
>>
>> No, this is related to #1028250 and https://gitlab.com/cryptsetup/cryptsetup/-/issues/802#note_1287298872 .
>> Don't think we can do anything in src:cryptsetup for existing volumes
>> unfortunately. You might need to manually lower the parameters of your
>> PBKDF.
>
> Existing systems failing to boot after an upgrade doesn't seem to be
> “only” important to me…
I fail to see how that's different from an existing resource-constrained
system (sarge recommends for 64MiB RAM for desktop i386) being unusable
after dist-upgrading to a more recent Debian release. Granted I haven't
tried it, but I very much doubt GNOME would still work with that little
memory :-)
Anyway, while it definitely isn't ideal (not very future proof) that the
key slots were created with a memory cost close to the whole available
memory, there is actually not that much difference in resident set size
before and after the upgrade. The test environment is a VM with 1024M
RAM, and initialized with d-i's debian-10.12.0-amd64-netinst.iso and
“Encrypted LVM” partition scheme. In my case the PBKDF benchmark chose
the following parameters (close to half the amount of physical memory
indeed):
~# cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:
PBKDF: argon2i
Time cost: 4
Memory: 504962
Threads: 2
## buster (cryptsetup 2:2.1.0-5+deb10u2, linux 4.19+105+deb10u18)
~# command time cryptsetup luksOpen --test-passphrase /dev/vda5 <<<test
1.98user 0.09system 0:01.13elapsed 182%CPU (0avgtext+0avgdata 534408maxresident)k
566inputs+0outputs (0major+2212minor)pagefaults 0swaps
## dist-upgrade to bullseye (cryptsetup 2:2.3.7-1+deb11u1, linux 5.10.162-1)
~# command time cryptsetup luksOpen --test-passphrase /dev/vda5 <<<test
1.93user 0.11system 0:01.14elapsed 179%CPU (0avgtext+0avgdata 534012maxresident)k
566inputs+0outputs (0major+2185minor)pagefaults 0swaps
## dist-upgrade to bookworm (cryptsetup 2:2.6.1-1, linux 6.1.12-1)
~# command time cryptsetup luksOpen --test-passphrase /dev/vda5 <<<test
2.05user 0.08system 0:01.17elapsed 183%CPU (0avgtext+0avgdata 512940maxresident)k
6102inputs+0outputs (37major+1649minor)pagefaults 0swaps
So one could argue it's not a cryptsetup regression, at least not since
memory-hard KDFs are used by default (i.e. since buster). The kernel
happens to kill cryptsetup because it's the main memory consumer at that
point (again due to unfortunate KDF parameters), but the reason why it
ran out of memory in the first place, and didn't with earlier releases,
appears to be independent from cryptsetup.
>> Lowering the severity, because this shouldn't block the transition of -2
>> into bookworm (which fixes an unrelated and arguably much more severe RC
>> bug).
>
> That's not really how RC bugs work: bugs aren't less RC because it makes
> sense for a specific version to migrate…
>
> Either the bug appeared specifically in the version it was filed against,
> and it makes sense to block the migration since that's a new RC bug in
> that particular version, and the RC-ness stays.
>
> Or the bug was already there in the version currently in testing, and that
> means that's not a regression, and the RC-ness stays. You only need to
> record the bug as also being found in the previous version (possibly
> plural) to make sure britney knows it's not a regression.
Ah cool didn't know that, thanks for the information. Marking this as
found all the way back to buster then. I'm indeed able to trigger the
OOM-killer on a buster system when I artificially fill the memory so the
memory cost exceeds what remains. Furthermore I don't see what can be
done about existing keyslots, and that includes everything created since
buster.
> Sure, we can discuss the severity of the bug I filed. But #1032734 really
> can't be “just” important.
Tagging ‘moreinfo’ then. I can definitely see how one can reproduce
this theoretically (and possibly in the future when the kernel's memory
requirement increase high enough), and mentioned that in the upstream
bug, but I'm unable to find a reproducer after dist-upgrading bullseye
systems to bookworm (all created from d-i's debian-11.6.0-amd64-netinst.iso,
and “Encrypted LVM” partition scheme, on VMs with 1024M RAM).
Jérôme, what memory cost is the keyslot using? (Paste the output of
`cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:`.) Would also be
interested to see by how much the amount of memory available to
cryptsetup has changed before and after the uprade. Please edit
/usr/share/initramfs-tools/scripts/local-top/cryptroot and add `free` at
the begining of the setup_mapping() function (patch attached). My own
findings are as follows (again on a minimal netinst system without
changing any default). cryptsetup isn't even close to memory
exhaustion.
## bullseye (cryptsetup 2:2.3.7-1+deb11u1, linux 5.10.162-1)
~# cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:
PBKDF: argon2i
Time cost: 4
Memory: 499892
Threads: 2
~# systemctl reboot
Loading Linux 5.10.0-21-amd64 ...
Loading initial ramdisk ...
total used free shared buff/cache available
Mem: 999776 39276 843280 52 117220 777124
Swap: 0 0 0
Please unlock disk vda5_crypt:
1.90user 0.18system 0:04.19elapsed 49%CPU (0avgtext+0avgdata 525900maxresident)k
566inputs+0outputs (0major+2400minor)pagefaults 0swaps
cryptsetup: vda5_crypt: set up successfully
## pin src:cryptsetup to 2:2.3.7-1+deb11u1 and dist-upgrade everything else to bookworm
Loading Linux 6.1.0-5-amd64 ...
Loading initial ramdisk ...
total used free shared buff/cache available
Mem: 993508 48892 748520 56 196096 683768
Swap: 0 0 0
Please unlock disk vda5_crypt:
1.94user 0.21system 0:18.21elapsed 11%CPU (0avgtext+0avgdata 525592maxresident)k
566inputs+0outputs (0major+2402minor)pagefaults 0swaps
cryptsetup: vda5_crypt: set up successfully
## remove the pin and finalize the dist-upgrade (cryptsetup 2:2.6.1-1, linux 6.1.12-1)
Loading Linux 6.1.0-5-amd64 ...
Loading initial ramdisk ...
total used free shared buff/cache available
Mem: 993508 40672 759596 52 193240 694828
Swap: 0 0 0
Please unlock disk vda5_crypt:
2.04user 0.12system 0:04.57elapsed 47%CPU (0avgtext+0avgdata 507772maxresident)k
566inputs+0outputs (0major+1382minor)pagefaults 0swaps
cryptsetup: vda5_crypt: set up successfully
--
Guilhem.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/pkg-cryptsetup-devel/attachments/20230311/2751739a/attachment.sig>
More information about the pkg-cryptsetup-devel
mailing list