[pkg-cryptsetup-devel] Spooky cryptsetup-initramfs experience

[Siim Põder]

Hi

tl;dr: my remote server using cryptsetup-initramfs (with 
dropbear-initramfs) failed to boot, causing excitement! I don't think 
there was an obvious bug anywhere, but I don't think I did anything 
tremendously wrong either. So maybe take this as an usability FYI in 
case there is ever an opportunity to tweak something. If you think there 
might be a bug, LMK and I can file some part of it as a bug report, too.

In more words:

I have a server with encrypted root that I boot every couple of years, 
usually when disks fail and are replaced. There is some kind of trouble 
with booting pretty much every time so I was prepared for things to get 
messy and was not disappointed. I prepared by making sure dropbear 
authorized_keys looked sane, regenerated initramfs and ran grub-install 
on the remaining disks. When the disk was replaced and the machine 
rebooted I could SSH in, however was greeted with the message:

Error: Timeout reached while waiting for askpass

Internet hinted this is related to askpass and a fifo. Even with 
hindsight, I'm not sure why this happened because AFAIK I didn't do 
anything specific to fix it. I did boot a rescue image, cleaned up 
crypttab and fstab, unpacked the initramfs to ensure askpass was present 
(it was) and then regenerated initramfs just to be sure. This time, 
initramfs generation informed me:

warning: the initramfs image may not contain cryptsetup binaries (and so 
forth)

Seems like I now needed to set CRYPTSETUP=Y in 
/etc/cryptsetup-initramfs/conf-hook, doing so made that warning go away. 
Reboot!

This time the initramfs boots up but with no LVM VG present. I manually 
modprobe dm-crypt but then the cryptsetup-unlock just hangs there. Seems 
/cryptroot/crypttab is empty! It's probably because I did something 
strange in the rescue image while mounting disks, but sucks that there 
was no warning about this at initramfs generation time as there was for 
missing CRYPTSETUP=Y.

Rescue image again - generate initramfs again, look inside, this time 
crypttab is there. I may have tweaked something silly, like changed some 
spaces in crypttab to tabs - I don't suppose that should change 
anything? Or maybe just set things up cleaner because of I had gotten 
better from repetition. Not exactly sure what changed though.

This time after reboot I was able to cleanly start up the server!

Overall, this episode is indicative of my experience with cryptroot. I 
used to be really scared of it being broken but by now I've gotten 
decent enough at debugging/dealing with this so it's just a mild time 
sink every couple of years. But either I'm doing something weird or my 
setup is somehow weird or this use case really does require a good 
helping of technical expertise to be able to use it?

Also - thanks for keeping this working in the first place! Even though 
I've had scary experiences and sunk some time into it, at the end of the 
day I'm quite pleased with knowing that my data will not leak without 
someone actively subverting my control channel to the server! Cheers!