[Pkg-openldap-devel] Bug#827135: slapd won't stop (shutdown) on multi-core system under stress

[Zvika Ferentz]

Package: slapd
Version: 2.4.31-2+deb7u2
+X-Debbugs-CC: ryan at openldap.org


Greetings,

I've noticed that sometimes the "slapd" fails to stop. I can reproduce the problem only on machines with multiple cores  (more than 2) when there are many
concurrent connections and the server is under a heavy "stress" .
Note that I tried the current standard version (2.4.31-deb7u2) as well as the wheezy-backports -  2.4.31+really2.4.40+dfsg-1+deb8u1~bpo70+1 .

In the logs i was able to identify the first "shutdown" log:
   daemon: shutdown requested and initiated
However the next expected shutdown logs are missing ("waiting for ...operations/tasks to finish" and "slapd stopped") .
Hence, i believe that it's related to closing the sockets/listeners (this is my guess based on the source code, however i never debugged/proved it).
Please note that i'm using virtual environment so i was able to reproduce it only when i used more than 4 cores.
Also please note that my system is a customized (hardened) Debian wheezy (7.9) and I built the slapd from the Debian sources (apt-get source ...) .
I REALLY (!!) hope that this is valid bug since I'm surprised that I'm the only one who encountered this situation.

How to reproduced it:
-------------------------------
I guess that there are a few ways to reproduce it , I managed to easily reproduce it with two terminates - one producing "ldapsearch" stress and the other restarting slapd :
- Open two terminals.
- On terminal #1 i'm just manually running  "slapd restart" commands:
   # /etc/init.d/slapd status ; /etc/init.d/slapd restart
- On terminal #2 i'm running a infinite loops of simple "ldapsearch" (100
concurrent processes running loops of ldapsearch). Terminal #2 is trying to
simulate many concurrent read operations. see "more information" later for the
exact scripts that i used.

Incorrect behavior:
-------------------------
The "slapd restart" works a few times, and then the "stop" operation fails.
The stop continues to fail even if i stop all "stress" and terminate all
ldapsearch/connections  (CPU is 99% idle !)

Expected Behavior:
-------------------------
All slapd stop/restart operations complete successfully


More Information (optional - my exact scripts):
----------------------------------------------------------------
On terminal#2 i used a very simple script to generate a "read only" stress:
# cat > ldaploop.sh << EOF
#!/bin/sh
while true ; do  ldapsearch -x -Z ; done
EOF

# cat > manyloops.sh << "EOF"
#!/bin/sh
for i in `seq 1 100` ; do ( ./ldaploop.sh &) ; done
EOF

As previously mentioned, i ran the "manyloops.sh" to generate 100 running
processes where each one simply runs "ldapsearch" (locally).

thanks in advance,
Zvika Ferentz

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-openldap-devel/attachments/20160612/5379002e/attachment.html>