Bug#1008951: openldap FTBFS on musl-linux-any: conflicting declaration of calloc

[Helmut Grohne]

Source: openldap,musl
Tags: upstream
User: helmutg at debian.org
Usertags: rebootstrap

Hi,

openldap fails to build from source on musl-linux-any. The issue is not
Debian-specific and has been reported for gentoo already
(https://bugs.gentoo.org/546556). The visible issue is this:

|   In file included from ../../../../libraries/librewrite/context.c:22:
|   ../../../../libraries/librewrite/rewrite-int.h:44:25: error: conflicting types for ‘ber_memcalloc’; have ‘void *(size_t,  size_t)’ {aka ‘void *(unsigned int,  unsigned int)’}
|      44 | #define calloc(x,y)     ber_memcalloc(x,y)
|         |                         ^~~~~~~~~~~~~
|   In file included from ../../../../libraries/librewrite/rewrite-int.h:34:
|   ../../../../include/lber.h:608:1: note: previous declaration of ‘ber_memcalloc’ with type ‘void *(ber_len_t,  ber_len_t)’ {aka ‘void *(long unsigned int,  long unsigned int)’}
|     608 | ber_memcalloc LDAP_P((
|         | ^~~~~~~~~~~~~

The gentoo bug explains this quite well. LBER_LEN_T is statically
defined as "long" in configure.ac. Then include/lber_types.h uses that:

| typedef unsigned LBER_LEN_T ber_len_t;

And then, it declares ber_memcalloc using ber_len_t in include/lber.h.
Finally, it #defines calloc (as be seen above) to point to
ber_memcalloc. That #define lives in libraries/librewrite/rewrite-int.h,
which happens to be #included before some musl header that happens to
pull the definition of calloc. Thus musl's declaration is renamed to
ber_memcalloc and since it uses size_t, which happens to be defined as
unsigned int rather than unsigned long, we get a conflicting
declaration.

Looking deeper things get a little surprising. rewrite-int.h is careful
to #include <ac/stdlib.h> early. Presumably, it wants to avoid such
conflicting declarations. Indeed, that's not the place the conflicting
ber_memcalloc declaration originates. After the #define calloc, there is
an #include <ldap_pvt_thread.h>, which happens to indirectly #include
<pthread.h>, which happens to #include <sched.h>, which happens to
declare calloc, which was defined as ber_memcalloc earlier.

This is sounds doubly wrong. Why would openldap re#define calloc and
then #include system headers? That's risky and it makes the build fail.
Why would musl declare calloc in a header different from stdlib.h? I
understand neither and fixing either fixes the build.

The gentoo bug has a few more details and proposed two distinct
workarounds:
 a. Drop the _GNU_SOURCE define. Doing so makes musl's sched.h not emit
    a calloc declaration. I don't know what other consequences this
    would have.
 b. Change LBER_LEN_T to size_t. I suppose doing so would break ABI on
    64bit architectures.

Any clue how we can move forward here?

Helmut