Bug#774213: zope2.13: import zope.security._proxy -> SystemError: dynamic module not initialized properly

[Kirill Smelkov]

On Sun, Jan 18, 2015 at 05:41:10PM +0000, Jonathan Wiltshire wrote:
> Control: severity -1 normal
> Control: tag -1 - unreproducible
> 
> Hi,
> 
> On Tue, Dec 30, 2014 at 02:32:17PM +0400, Kirill Smelkov wrote:
> > With zope2.13 I've tried to create a (user) instance and start it, but a
> > `SystemError: dynamic module not initialized properly` is raised while
> > zopectl tries to import zope.security._proxy .
> > 
> > The bug could be demonstrated in essence this way:
> > 
> >     $ /usr/lib/zope2.13/bin/python 
> >     
> >     >>> import zope.security._proxy
> >     Traceback (most recent call last):
> >       File "<console>", line 1, in <module>
> >       File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/__init__.py", line 19, in <module>
> >         from zope.security.management import checkPermission
> >       File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/management.py", line 23, in <module>
> >         from zope.security.checker import CheckerPublic
> >       File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/checker.py", line 46, in <module>
> >         from zope.security._proxy import _Proxy as Proxy, getChecker
> >     SystemError: dynamic module not initialized properly
> 
> We have been working on this as part of a BSP and have some findings.
> 
> The problem appears to be when zope2.13, and python-zope.proxy are
> installed but NOT python-zope.security. In that situation:
> 
> import zope.security
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
>   File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/__init__.py", line 19, in <module>
>     from zope.security.management import checkPermission
>   File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/management.py", line 23, in <module>
>     from zope.security.checker import CheckerPublic
>   File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/checker.py", line 46, in <module>
>     from zope.security._proxy import _Proxy as Proxy, getChecker
> SystemError: dynamic module not initialized properly
> 
> It seems that zope.security gets loaded from the bundled copy, and
> zope.proxy from the system. This explodes loudly.
> 
> If both or neither external packages are installed, everything is fine.

Thanks for looking into this. I've also digged a bit previously and my
findings were posted in the bugtracker:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774213#34

In short (quoting from the detailing mail):

"""
Conclusion: not enough isolation leads to either flaky behaviour (i.e.
zope.security imports, but version is  4.0.1-1 instead of expected by
zope2.13 3.7.4) or import failing at all.

And there is some hidden state which depends on packages installation
order etc...
"""

> The reporter got into this situation by installing python-zope.security,
> which has a dependency on python-zope.proxy, and then removing
> python-zope.security. python-zope.proxy therefore stays around, causing the
> SystemError.

_No_, I did _not_ install python-zope.security at all - initially I've
just installed zope2.13 and also python-zodb which I needed for other
tasks and python-zodb depends on python-zope.proxy.

So I did _not_ remove python-zope.security or did anything lower than
apt-get install or remove and I did _not_ used anything like
--force-depends.

> Downgrading since it all works fine if both extensions are bundled or both
> extensions are external, and 'apt install python-zope.security' does the
> right thing.
  ^^^^^^^^^^^

> I notice that the external packages are not the same versions as their
                                          ^^^^^^^^^^^^^^^^^^^^^

> bundled counterparts, so perhaps an interface changed at some point or a
> rebuild is needed or something.

exactly.

Is it really the "right thing" that different version of packages get
imported? To me this whole "let's bundle everything" approach is
only justified because each package then could specify which version of
dependencies to use _exactly_. And if not we should just drop that
shipped-libs and use system-wide packages right away.

And again, as noted in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774213#34
there is not enough isolation between system-wide and shipped packages
which leads to `what gets imported is not deterministic`; and that "not
deterministic" also depends on packages installation / removal order.

I strongly suspect the effect is not limited only to zope.proxy and
zope.security and affects the whole import mechanism...

Thanks,
Kirill