[Pkg-openldap-devel] Bug#327585: embedding perl, libltdl and RTLD_GLOBAL

[Niko Tyni]

Hi,

I've been looking at the "libltdl and RTLD_GLOBAL" issue with embedding
perl in a dlopen'd plugin.

An instance of this with freeradius is #416266 (recently reassigned to
perl), and I see #327585 against openldap is another one.

To recap, the problem is that lt_dlopen() from the Debian system libltdl
has called dlopen(3) with RTLD_LOCAL instead of RTLD_GLOBAL ever since
#195821 was fixed. As the compiled XS modules aren't linked against
libperl, its symbols aren't exposed to them, resulting in errors like
'/usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so: undefined symbol: Perl_sv_cmp'.

Observations:

- this problem isn't specific to perl and can easily be triggered with the
  freeradius rlm_python module too [1]

- it's clearly possible to dlopen() compiled Perl modules from a dlopen'd
  module if you don't use libltdl, see apache2+libapache2-mod-perl2 for
  an example

- the XS modules are actually plugins in a private directory, not generic
  shared libraries. Having unresolved symbols in a plugin without
  a corresponding NEEDED entry seems to be very common, see for
  example /usr/lib/apache2/modules, /usr/lib/python2.5/lib-dynload/,
  /usr/lib/cdebconf etc.

- as noted in #327585, linking the XS shared objects against libperl
  is potentially a problem on *i386, where /usr/bin/perl is statically
  linked with libperl.a for performance reasons. (I don't have any data
  about these performance reasons myself, I'm relying on hearsay and
  /usr/share/doc/perl/README.Debian.gz here.)

  While this does seem to work in a quick and limited test of mine,
  it would bring in both libperl.a and libperl.so for all uses of
  /usr/bin/perl that need XS modules, and I'm not sure which version of
  the functions would get used later. If the PIC versions win, we'd be
  giving away the performance benefit we got from static linking in the
  first place.

  At the very least, it would add 1.5M to the size of the perl-base
  package on i386 AFAICS. I'm not sure how much the memory footprint of
  the /usr/bin/perl invocations would increase.

  Also note that we currently ship /usr/lib/libperl.a on all the
  architectures, so everything that applies to the i386 /usr/bin/perl
  case applies to anybody using the static library on the other archs too.

  Given that i386 is still our most popular architecture, the other
  proposed options don't seem very appealing either:
    * only link the modules against libperl.so on the other architectures
      (no fix for i386)
    * link /usr/bin/perl dynamically on i386 too
      (reduced performance in the very common case
       for the benefit of a very uncommon case)

- it turns out libltdl nowadays does have an interface where you can
  specify RTLD_GLOBAL. From the libtool Changelog.2007:

    2007-05-08  Gary V. Vaughan  <gary at gnu.org>

        Without this patch, lt_dlopen always opens modules with symbol
        visibility set according to the underlying implementation.
        Here, we add lt_dlopenadvise() to allow callers to request,
        among other things, local or global symbol visibility from the
        underlying dlloader:

  Indeed, the attached proof of concept makes the freeradius problem go away
  for me, and I expect openldap could work with something similar. (FWIW,
  note that the trivial my_dlopenextglobal() function was adapted from the
  libtool documentation, so it might be considered to be under the GFDL.)

Josip: based on the above, I think #416266 should be fixed in freeradius
and not in perl. If you agree, please reassign back yourself.

[1]: add python to the instantiate{} block in radiusd.conf
and something like

# cat /etc/freeradius/modules/python
python {
                mod_instantiate = radiusd_test
                func_instantiate = instantiate
}
# cat /usr/local/lib/python2.5/site-packages/radiusd_test.py 
import sys
import socket

def instantiate(test):
    sys.stderr.write("hello, world!")

and you get 

rlm_python:EXCEPT:<type 'exceptions.ImportError'>: /usr/lib/python2.5/lib-dynload/_socket.so: undefined symbol: PyExc_ValueError
rlm_python:python_load_function: failed to import python function 'radiusd_test.instantiate'
-- 
Niko Tyni   ntyni at debian.org