[Pkg-openldap-devel] Bug#673038: slapd: slapcat output truncated every now and then

Axel Beckert beckert at phys.ethz.ch
Tue May 15 16:33:57 UTC 2012 

Package: slapd
Severity: grave
Version: 2.4.23-7.2
Justification: Causes data loss

Every now and then slapcat's output does not contain the whole content
of the LDAP but is truncated at some LDIF entry border (i.e. all printed
LDIF records seem complete).

To reproduce run "while sleep 1; do slapcat | wc; done" on a moderately
busy LDAP server (writes every few seconds to minutes; about 10000
entries) while slapd is running and notice the occasionally occurring
huge change in wc's printed values, e.g.:

# while sleep 1; do slapcat | wc; done
 471698 1015498 15336677
 471698 1015498 15336677
 471698 1015498 15336677
 471698 1015498 15336677
 471698 1015498 15336677
 471698 1015497 15336630
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015498 15336677
 471698 1015497 15336635
 471698 1015497 15336635
 471698 1015497 15336635
 281829  606820 8632165  <--
 471698 1015497 15336635
 471698 1015497 15336635
 471698 1015497 15336635
 308627  664573 9476751  <--
 471698 1015497 15336635
 471698 1015497 15336635
 471698 1015497 15336637
 471698 1015497 15336637
 471698 1015497 15336637
 471698 1015497 15336637
 471698 1015497 15336631
 471698 1015497 15336631
 471698 1015498 15336673
 471698 1015498 15336673
 471698 1015498 15336725
 471698 1015498 15336725
 471698 1015497 15336682
 471698 1015498 15336725
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015498 15336719
 471698 1015497 15336680
 471698 1015497 15336682
 471698 1015497 15336682
 471698 1015497 15336742
 471698 1015497 15336742
 471698 1015497 15336742
 471698 1015497 15336742
 471698 1015497 15336742
 471698 1015498 15336781
 471698 1015498 15336781
 471698 1015498 15336781
 471698 1015498 15336781
 471698 1015498 15336781
 471698 1015498 15336779
 471698 1015498 15336779
 471698 1015497 15336740
 471698 1015497 15336740
 471698 1015497 15336740
 471698 1015497 15336740
 471698 1015496 15336724
 471698 1015496 15336724
 471698 1015496 15336724
 471698 1015496 15336724
 471698 1015496 15336724
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015497 15336763
 471698 1015497 15336763

According to the slapcat man page it should be "always safe to run
slapcat with the slapd-bdb(5) ... backends" even if slapd runs. We do
use a BDB backend.

Using "slapcat -c" instead of just "slapcat" seems only to lower the
error rate a little bit, but it may also be that we just haven't tested
an significant amount of slapcat calls.

Even running "slapcat -d -1" gives no error message and it always exits
with exit code zero, so except checking the output's length there seems
no chance of catching a truncated output.

As slapcat is used to make backups of LDAP database, having an
unreliable slapcat means to have unreliable backups, too. Seems to have
happened with Lenny back then, too:

[...]
-rw-r--r-- 1 root root  1397053 Jul 17  2010 ldif.2010-07-17.gz
-rw-r--r-- 1 root root  1397255 Jul 18  2010 ldif.2010-07-18.gz
-rw-r--r-- 1 root root  1397523 Jul 19  2010 ldif.2010-07-19.gz
-rw-r--r-- 1 root root    89419 Jul 20  2010 ldif.2010-07-20.gz
-rw-r--r-- 1 root root  1398508 Jul 21  2010 ldif.2010-07-21.gz
-rw-r--r-- 1 root root  1397746 Jul 22  2010 ldif.2010-07-22.gz
-rw-r--r-- 1 root root  1398243 Jul 23  2010 ldif.2010-07-23.gz
[...]

Same counts for cases where the slapcat output is used to export the
LDAP content to other database formats like NIS.

-- System Information:
Debian Release: 6.0.5
  APT prefers stable
  APT policy: (990, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.26-2-xen-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages slapd depends on:
ii  adduser                3.112+nmu2        add and remove users and groups
ii  coreutils              8.5-1             GNU core utilities
ii  debconf [debconf-2.0]  1.5.36.1          Debian configuration management sy
ii  libc6                  2.11.3-3          Embedded GNU C Library: Shared lib
ii  libdb4.8               4.8.30-2          Berkeley v4.8 Database Libraries [
ii  libgnutls26            2.8.6-1+squeeze2  the GNU TLS library - runtime libr
ii  libldap-2.4-2          2.4.23-7.2        OpenLDAP libraries
ii  libltdl7               2.2.6b-2          A system independent dlopen wrappe
ii  libperl5.10            5.10.1-17squeeze3 shared Perl library
ii  libsasl2-2             2.1.23.dfsg1-7    Cyrus SASL - authentication abstra
ii  libslp1                1.2.1-7.8         OpenSLP libraries
ii  libwrap0               7.6.q-19          Wietse Venema's TCP wrappers libra
ii  lsb-base               3.2-23.2squeeze1  Linux Standard Base 3.2 init scrip
ii  perl [libmime-base64-p 5.10.1-17squeeze3 Larry Wall's Practical Extraction 
ii  psmisc                 22.11-1           utilities that use the proc file s
ii  unixodbc               2.2.14p2-1        ODBC tools libraries

Versions of packages slapd recommends:
pn  libsasl2-modules              <none>     (no description available)

Versions of packages slapd suggests:
ii  ldap-utils                    2.4.23-7.2 OpenLDAP utilities

-- Configuration Files:
/etc/default/slapd changed:
SLAPD_USER="openldap"
SLAPD_GROUP="openldap"
SLAPD_PIDFILE=
SLURPD_START=auto
SLAPD_SERVICES="ldap:// ldap://<someip>:<someport>/ ldaps:///"
SLAPD_OPTIONS=""
SLURPD_OPTIONS=""


-- debconf information:
  slapd/allow_ldap_v2: false
  slapd/password_mismatch:
  slapd/tlsciphersuite:
  slapd/fix_directory: true
  slapd/invalid_config: true
  shared/organization: ethz.ch
  slapd/upgrade_slapcat_failure:
  slapd/slurpd_obsolete:
  slapd/no_configuration: false
  slapd/migrate_ldbm_to_bdb: false
  slapd/move_old_database: true
  slapd/upgrade_slapadd_failure:
  slapd/suffix_change: false
  slapd/slave_databases_require_updateref:
  slapd/dump_database_destdir: /var/backups/slapd-VERSION
  slapd/autoconf_modules: true
  slapd/purge_database: false
  slapd/domain: ethz.ch
  slapd/backend: BDB
  slapd/dump_database: when needed

More information about the Pkg-openldap-devel mailing list