[pkg-cryptsetup-devel] Bug#1032734: OOM when unlocking encrypted root in initramfs

[Jérôme Charaoui]

> Tagging ‘moreinfo’ then.  I can definitely see how one can reproduce
> this theoretically (and possibly in the future when the kernel's memory
> requirement increase high enough), and mentioned that in the upstream
> bug, but I'm unable to find a reproducer after dist-upgrading bullseye
> systems to bookworm (all created from d-i's debian-11.6.0-amd64-netinst.iso,
> and “Encrypted LVM” partition scheme, on VMs with 1024M RAM).
> 
> Jérôme, what memory cost is the keyslot using?  (Paste the output of
> `cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:`.)  Would also be
> interested to see by how much the amount of memory available to
> cryptsetup has changed before and after the uprade.  Please edit
> /usr/share/initramfs-tools/scripts/local-top/cryptroot and add `free` at
> the begining of the setup_mapping() function (patch attached).  My own
> findings are as follows (again on a minimal netinst system without
> changing any default).  cryptsetup isn't even close to memory
> exhaustion.

Memory cost:

----- 8< -----
# cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:
         PBKDF:      argon2i
         Time cost:  4
         Memory:     787907
         Threads:    1
----- >8 -----

Free memory:

----- 8< -----
Loading Linux 6.1.0-5-amd64 ...
Loading initial ramdisk ...
               total        used        free      shared  buff/cache 
available
Mem:         993756       74720      725012          60      194024 
660412
Swap:             0           0           0
----- >8 -----

I've also tested on another of my virtual machines that has 1GiB of RAM 
and got another result, closer to what you got in your experimentation:

----- 8< -----
# cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:
         PBKDF:      argon2i
         Time cost:  6
         Memory:     499912
         Threads:    1
----- >8 -----

So, I think what's happening is that the first VM may have been created 
with a different (larger) memory configuration, and was reduced at a 
later point in time. I don't have absolute certainty of this, but it 
would very well explain the discrepancy in memory cost.

Also, I think I agree with your assessment that in the memory usage 
increase of the kernel may be involved: between the two releases, 
according to your numbers it appears to have increased nearly 25% (!). 
So it could also explain why it (probably very nearly) worked under 
bullseye.

I there any way we could make the cryptsetup-initramfs hook aware of 
this, and emit a warning if it finds that the encrypted root lacks a 
keyslot with appropriate (low-enough) memory cost?


Thanks,

-- Jérôme