Bug#543941: OpenVPN with LDAP+TLS authentication runs into file exhaustion -- Analysis and suggested workaround.

[Lars Ellenberg]

attached is a simple program to reproduce,
and workaround the issue.

libgcrypt standard behaviour is, at least on linux,
to open /dev/urandom once,
save that file descriptor in some static variable,
and re-use it wherever appropriate.
and never ever close that file descriptor,
but on exit or fork.

problem is:
pam_start() via various indirections may dlopen()s libgcrypt,
pam_stop() will dlclose() it again.

which means the libgcrypt will be unloaded,
and its static urandom fd with it.

but there is no destructor to close the FD.

on the next iteration,
a new instance of libgcrypt will be loaded,
with freshly initialized data segment,
resulting in an additional open of urandom.

that is the leak.

Workaround:
grab an additional reference on libgcrypt.
these workarounds seem to have precedence, see the
void nasty_pthread_hack (void) __attribute__ ((constructor));
void nasty_ssl_hack (void) __attribute__ ((constructor));
in libpam_ldap: pam_ldap.c

This should only be done as a short term workaround, though.

Real fix would be for libgcrypt to properly clean up on unload,
i.e. to provide proper destructor functions.

try.c is attached.
example session:

~/src/try$ gcc -o try try.c -lpam -ldl -pthread
~/src/try$ strace -e open ./try 2>&1 | grep urandom
open("/dev/urandom", O_RDONLY)          = 4
open("/dev/urandom", O_RDONLY)          = 6
open("/dev/urandom", O_RDONLY)          = 7

~/src/try$ gcc -DFIXIT -o try try.c -lpam -ldl -pthread
~/src/try$ strace -e open ./try 2>&1 | grep urandom
open("/dev/urandom", O_RDONLY)          = 4


Cheers,

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: try.c
Type: text/x-csrc
Size: 1655 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-gnutls-maint/attachments/20091025/32403a1a/attachment.c>