[Pkg-openssl-devel] Re: OpenSSL bug in libcrypto.so:RAND_poll()
crashes apache2 @ startup
vito at hostway.com
vito at hostway.com
Tue Apr 4 14:14:56 UTC 2006
Hello Christoph,
I have not heard anything from the openssl-bugs list, my submission seems to
have been ignored there.
The patch has fixed the problem for us here, previously apache2 would either
deadlock in a futex_wait shortly after the RAND_poll() select on high fd's
or simply segfault, with around 400 virtual hosts. Same server now has 800
virtual hosts and the problem is gone with patched libcrypto.
It's a costly bug too, because the automation adds virtual hosts without any
human involvement, and issues just an apache2 reload to realize the new
configurations - avoiding the bug. So generally a week or more goes by of
normal operation with new hosts getting added, then some maint. cron job issues
a restart and the monitoring screams http/https is down. The sysadmins will not
figure it out, and the end result is lots of down time.
Cheers,
Vito Caputo
On Tue, Apr 04, 2006 at 10:36:53AM +0200, Christoph Martin wrote:
> Hi Vito,
>
> vito at hostway.com schrieb:
> > Hello Christoph,
> >
> > Included below is an email I've already submitted to openssl-bugs at openssl.org,
> > you may want to get the debian package updated to fix this bug for debian users.
>
> Thanks for your report and the patch. I did not see any feedback on your
> mail to openssl-bugs. Did you get any reply on your patch proposal?
>
> Christoph
>
> > -------------------------------------------------------------------------
> >
> > Hello,
> >
> > I have found a bug in libcrypto.so which causes Apache2 to crash or
> > deadlock when a few hundred virtual hosts are configured in a
> > SSL-enabled Apache2 instance.
> >
> > The problem is Apache2 opens a number of files per virtual host before
> > initializing libcrypto.so's random seed, given enough virtual hosts the
> > file descriptor the RAND_poll() open() gets when it opens its entropy
> > source will exceed FD_SETSIZE. This is a serious problem because RAND_poll()
> > internally uses select() to watch for data ready to be read from the
> > entropy source. When the fd exceeds FD_SETSIZE (1024 on modern linux
> > systems) this will cause deadlocks / segfaults.
> >
> > I have created a patch to convert this to use poll() which does not have
> > this limitation and is much better suited for watching a single file
> > descriptor. The patch is included below. I'm not sure if you guys need
> > to make this check if poll() is available in the system to keep things
> > portable, but if the select() call needs to be kept when poll() is
> > unavailable, it has to deal with fd >= FD_SETSIZE and not give it to
> > select().
> >
> > This was using 0.9.7e-3sarge1 from debian stable as the original source tree, I
> > checked the 0.9.7i source on openssl.org and the related code is the same.
> >
> > It's a relatively high priority because it makes apache2 flip out when it gets
> > hit, might wanna get a fixed version out soon.
> >
> > Thanks for the OpenSSL project,
> > Vito Caputo
> > Hostway Linux Systems Developer
> >
> >
> >
> >
> >
> > --- rand_unix.c.orig 2006-03-21 17:01:24.000000000 -0600
> > +++ rand_unix.c 2006-03-21 21:58:12.000000000 -0600
> > @@ -114,6 +114,7 @@
> > #include "cryptlib.h"
> > #include <openssl/rand.h>
> > #include "rand_lcl.h"
> > +#include <sys/poll.h>
> >
> > #if !(defined(OPENSSL_SYS_WINDOWS) || defined(OPENSSL_SYS_WIN32) || defined(OPENSSL_SYS_VMS) || defined(OPENSSL_SYS_OS2) || defined(OPENSSL_SYS_VXWORKS))
> >
> > @@ -124,6 +125,7 @@
> > #include <unistd.h>
> > #include <time.h>
> >
> > +
> > #ifdef __OpenBSD__
> > int RAND_poll(void)
> > {
> > @@ -165,53 +167,37 @@ int RAND_poll(void)
> > * have this. Use /dev/urandom if you can as /dev/random may block
> > * if it runs out of random entries. */
> >
> > - for (randomfile = randomfiles; *randomfile && n < ENTROPY_NEEDED; randomfile++)
> > - {
> > + for (randomfile = randomfiles; *randomfile && n < ENTROPY_NEEDED; randomfile++) {
> > if ((fd = open(*randomfile, O_RDONLY|O_NONBLOCK
> > #ifdef O_NOCTTY /* If it happens to be a TTY (god forbid), do not make it
> > our controlling tty */
> > - |O_NOCTTY
> > + |O_NOCTTY
> > #endif
> > #ifdef O_NOFOLLOW /* Fail if the file is a symbolic link */
> > - |O_NOFOLLOW
> > + |O_NOFOLLOW
> > #endif
> > - )) >= 0)
> > - {
> > - struct timeval t = { 0, 10*1000 }; /* Spend 10ms on
> > - each file. */
> > + )) >= 0) {
> > int r;
> > - fd_set fset;
> > + struct pollfd rfd = {fd, POLLIN, 0};
> > + int t = 10; /* Spend 10ms on each file. */
> > +
> > + /* Patched to use poll() instead of select(), sometimes this gets called
> > + * with >= FD_SETSIZE files opened already (apache2).
> > + * When fd is >= FD_SETSIZE the behavior is undefined (likely a buffer
> > + * overflow...), I observed segfaults & deadlocks.
> > + * 3/21/2006 - Vito Caputo - <vito at hostway.com> */
> >
> > - do
> > - {
> > - FD_ZERO(&fset);
> > - FD_SET(fd, &fset);
> > + do {
> > r = -1;
> >
> > - if (select(fd+1,&fset,NULL,NULL,&t) < 0)
> > - t.tv_usec=0;
> > - else if (FD_ISSET(fd, &fset))
> > - {
> > - r=read(fd,(unsigned char *)tmpbuf+n,
> > - ENTROPY_NEEDED-n);
> > - if (r > 0)
> > - n += r;
> > - }
> > -
> > - /* Some Unixen will update t, some
> > - won't. For those who won't, give
> > - up here, otherwise, we will do
> > - this once again for the remaining
> > - time. */
> > - if (t.tv_usec == 10*1000)
> > - t.tv_usec=0;
> > + if((poll(&rfd, 1, t) > 0) && (rfd.revents & POLLIN)) {
> > + r = read(fd, (unsigned char *)tmpbuf + n, ENTROPY_NEEDED - n);
> > + if(r > 0) n += r;
> > }
> > - while ((r > 0 || (errno == EINTR || errno == EAGAIN))
> > - && t.tv_usec != 0 && n < ENTROPY_NEEDED);
> > -
> > + } while((r > 0 || (errno == EINTR || errno == EAGAIN)) && n < ENTROPY_NEEDED);
> > close(fd);
> > - }
> > }
> > + }
> > #endif
> >
> > #ifdef DEVRANDOM_EGD
> >
>
> --
> ============================================================================
> Christoph Martin, EDV der Verwaltung, Uni-Mainz, Germany
> Internet-Mail: Christoph.Martin at Verwaltung.Uni-Mainz.DE
> Telefon: +49-6131-3926337
> Fax: +49-6131-3922856
>
More information about the Pkg-openssl-devel
mailing list