Bug#403075: [Pkg-cryptsetup-devel] Bug#403075: cryptsetup luksOpen can kill unrelated processes (out of memory killer)

Rob Walker rob at tenfoot.org.uk
Wed Dec 20 10:32:19 UTC 2006


> >>
> >> Are you able to reproduce the error with the version of cryptsetup that
> >> is currently in unstable?
> >
> >I can reproduce the bug using the cryptsetup from unstable.  I've also
> > tried it on another machine with the same results.
>
> Ok, could you then provide me with the exact steps that you took to
> reproduce it cause so far I haven't managed to do so.


I think I've found the problem.  In lib/utils.c, the sector_size function is 
this:

static int sector_size(int fd) 
{
	int bsize;
	ioctl(fd,BLKSSZGET, &bsize);
	return bsize;
}


For a file, the ioctl will fail.  Since bsize is not initialized, it's value 
will be random (when I ran it under gdb, I got several million) and as this 
value will later be used to allocate a buffer, this explains the runaway 
memory allocation I saw, but you didn't.

The attached patch checks the return code from ioctl and returns -EINVAL if it 
fails.  The users of sector_size are also changed to abort if sector_size 
returns a negative value.   

With this patch applied, I get the appropriate error messages from cryptsetup 
if I give it a file instead of a block device.

Regards

Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cryptsetup_sector_size_error.patch
Type: text/x-diff
Size: 983 bytes
Desc: not available
Url : http://lists.alioth.debian.org/pipermail/pkg-cryptsetup-devel/attachments/20061220/b507dbde/cryptsetup_sector_size_error.bin


More information about the pkg-cryptsetup-devel mailing list