Bug#858417: libapache2-mod-shib2: Lots of apache workers in "Closing connection" state. Endless sleeping of apache workers.

[Ferenc Wágner]

Pavel Polacek <pavel.polacek at ujep.cz> writes:

> (gdb) bt
> #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1  0x00007f032831c479 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
> #2  0x00007f032831c2a0 in __GI___pthread_mutex_lock (mutex=0x7f032ab42420) at ../nptl/pthread_mutex_lock.c:79
> #3  0x00007f031fa7ab8f in log4shib::Category::removeAllAppenders() () from /usr/lib/x86_64-linux-gnu/liblog4shib.so.1
> #4  0x00007f031fa7b9da in log4shib::HierarchyMaintainer::shutdown() () from /usr/lib/x86_64-linux-gnu/liblog4shib.so.1
> #5  0x00007f031fa7bc4c in log4shib::HierarchyMaintainer::~HierarchyMaintainer() () from /usr/lib/x86_64-linux-gnu/liblog4shib.so.1
> #6  0x00007f0327f9eb29 in __run_exit_handlers (status=0, listp=0x7f032830c5a8 <__exit_funcs>, run_list_atexit=run_list_atexit at entry=true)
>      at exit.c:82
> #7  0x00007f0327f9eb75 in __GI_exit (status=<optimized out>) at exit.c:104
> #8  0x00007f0325281aea in itk_fork_process () from /usr/lib/apache2/modules/mpm_itk.so
> #9  0x00007f0328e81f40 in ap_run_process_connection ()
> #10 0x00007f0324c6f7ba in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
> #11 0x00007f0324c6fa01 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
> #12 0x00007f0324c70667 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
> #13 0x00007f0328e5c7ee in ap_run_mpm ()
> #14 0x00007f0328e555f3 in main ()
>
> Apache mod_shib module is in suspicion.
> So I set logging off in /etc/shibboleth/native.logger, all line comment out.
> [...]
> Now I have 50 workers instead of 1000 after 10 hours of running apache.

Hi,

Do you mean that disabling logging fixed the problem for you and there
are no hung workers in that case?

Also, please provide the output of the "thread apply all bt" gdb command
on a hung process.

This looks like a log4shib threading problem, probably inherited from
log4cpp.
-- 
Thanks,
Feri