Bug#858417: libapache2-mod-shib2: Lots of apache workers in "Closing connection" state. Endless sleeping of apache workers.

[Pavel Polacek]

Package: libapache2-mod-shib2
Version: 2.5.3+dfsg-2
Severity: normal

Dear Maintainer,

LAMP server: apache2-mpm-itk + mod_php + mod_shib. Apache workers hangs in "Closing connection" state.

Apache in "C" state, wait for another apache worker (process 754):
strace -p 754 
Process 754 attached
wait4(771,

Apache process 771:
gdb --pid 771
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f032831c479 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f032831c2a0 in __GI___pthread_mutex_lock (mutex=0x7f032ab42420) at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f031fa7ab8f in log4shib::Category::removeAllAppenders() () from /usr/lib/x86_64-linux-gnu/liblog4shib.so.1
#4  0x00007f031fa7b9da in log4shib::HierarchyMaintainer::shutdown() () from /usr/lib/x86_64-linux-gnu/liblog4shib.so.1
#5  0x00007f031fa7bc4c in log4shib::HierarchyMaintainer::~HierarchyMaintainer() () from /usr/lib/x86_64-linux-gnu/liblog4shib.so.1
#6  0x00007f0327f9eb29 in __run_exit_handlers (status=0, listp=0x7f032830c5a8 <__exit_funcs>, run_list_atexit=run_list_atexit at entry=true)
     at exit.c:82
#7  0x00007f0327f9eb75 in __GI_exit (status=<optimized out>) at exit.c:104
#8  0x00007f0325281aea in itk_fork_process () from /usr/lib/apache2/modules/mpm_itk.so
#9  0x00007f0328e81f40 in ap_run_process_connection ()
#10 0x00007f0324c6f7ba in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#11 0x00007f0324c6fa01 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#12 0x00007f0324c70667 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#13 0x00007f0328e5c7ee in ap_run_mpm ()
#14 0x00007f0328e555f3 in main ()

Apache mod_shib module is in suspicion.
So I set logging off in /etc/shibboleth/native.logger, all line comment out.

native.logger:
log4j.rootCategory=INFO, native_log

# This is a Debian-specific change.
log4j.appender.native_log=org.apache.log4j.LocalSyslogAppender
log4j.appender.native_log.syslogName=shibboleth-sp
log4j.appender.native_log.facility=3
log4j.appender.native_log.layout=org.apache.log4j.BasicLayout


Logging off is only workaround. 

Now I have 50 workers instead of 1000 after 10 hours of running apache.

    Thank you  Pavel Polacek


-- System Information:
Debian Release: 8.7
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages libapache2-mod-shib2 depends on:
ii  apache2-bin [apache2-api-20120211]  2.4.10-10+deb8u8
ii  libc6                               2.19-18+deb8u7
ii  libgcc1                             1:4.9.2-10
ii  libgssapi-krb5-2                    1.12.1+dfsg-19+deb8u2
ii  libshibsp-plugins                   2.5.3+dfsg-2
ii  libshibsp6                          2.5.3+dfsg-2
ii  libstdc++6                          4.9.2-10
ii  libxerces-c3.1                      3.1.1-5.1+deb8u3
ii  libxmltooling6                      1.5.3-2+deb8u1
ii  shibboleth-sp2-utils                2.5.3+dfsg-2

libapache2-mod-shib2 recommends no packages.

libapache2-mod-shib2 suggests no packages.

-- no debconf information