[Python-modules-team] Bug#625784: python-virtualenv: 'virtualenv -p python3 xx' should work but doesn't

[Barry Warsaw]

Package: python-virtualenv
Version: 1.4.9-3ubuntu1
Severity: important


On Wheezy, which has virtualenv 1.6, the following should work:

$ virtualenv -p python3 /tmp/xx

but it fails with this traceback:

-----snip snip-----
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/virtualenv.py", line 1892, in <module>
    main()
  File "/usr/lib/python2.6/dist-packages/virtualenv.py", line 753, in main
    prompt=options.prompt)
  File "/usr/lib/python2.6/dist-packages/virtualenv.py", line 851, in create_environment
    install_distribute(py_executable, unzip=unzip_setuptools)
  File "/usr/lib/python2.6/dist-packages/virtualenv.py", line 575, in install_distribute
    _install_req(py_executable, unzip, distribute=True)
  File "/usr/lib/python2.6/dist-packages/virtualenv.py", line 474, in _install_req
    import pkg_resources
  File "/usr/share/pyshared/pkg_resources.py", line 45
    def _bypass_ensure_directory(name, mode=0777):
                                               ^
SyntaxError: invalid token
-----snip snip-----

I've analyzed why this happens, but I haven't figured out a good fix for it
yet.  Here's my analysis.

First, virtualenv supports Python 3 as of 1.6 (and PyPy 1.5 as of 1.6.1, but
that's for a different bug report).  This is true even without specifically
building python-virtualenv for Python 3.  In other words, in a pristine
environment, you build Python 2.7 from source --prefix=/tmp/py27 and then use
'/tmp/py27/bin/python setup.py install' from the upstream virtualenv 1.6.1
tarball, then run '/tmp/py27/bin/virtualenv -p python3 /tmp/xx' it works just
fine.

Why does it fail in Debian then?  I'm glad you asked!

Notice that the traceback is actually coming from
/usr/share/pyshared/pkg_resources.py, and indeed the token 0777 is not legal
in Python 3.2.  This traceback will happen even if you've installed the
python3-pkg-resource package.  The traceback happens because virtualenv is
picking up the pkg_resources modules from Python 2 and *not* from Python3.

virtualenv works by re-invoking the Python interpreter you gave to the -p
option in a subprocess.  The arguments to that subprocess are the absolute
path back to the virtualenv.py file and the target directory you invoked on
the command line.  It also sets an environment variable so that it knows it's
running recursively.  So, in our test case, this is what the subprocess runs:

$ VIRTUALENV_INTERPRETER_RUNNING=true python3 /usr/lib/python2.7/dist-packages/virtualenv.py

That *should* be perfectly fine because virtualenv.py is Python3 compatible.
Try it yourself and you'll see that that fails with the above traceback.

The reason is because of Debian's symlink farm.  When virtualenv.py tries to
import pkg_resources, it picks up the one from /usr/share/pyshared instead of
the one from /usr/lib/python3/dist-packages/pkg_resources.py as you would
expect.  Why is that?  Well, it's because
/usr/lib/python2.7/dist-packages/virtualenv.py is a symlink to
/usr/share/pyshared/virtualenv.py and this tricks Python's default sys.path
setup.

When Python is handed a script in argv[1], it puts that directory at the front
of sys.path, but before it does so, it chases the symlink of argv[1].  So,
Python 3 starts up, sees /usr/lib/python2.7/dist-packages/virtualenv.py in
argv[1], chases that to /usr/share/pyshared/virtualenv.py, and sticks
/usr/share/pyshared at the front of sys.path.  Our python-pkg-resources
package happens to drop a pkg_resources.py file in that directory, but that
file is only appropriate for Python 2.  So when Python 3 tries to import
pkg_resources, it gets the one from /usr/share/pyshared and boom!

Why doesn't the same thing happen in a from-source build?  Well, it's because
in a from-source build, pkg_resources doesn't live in site-packages, it lives
in site-packages/distribute-0.6.16-py2.7.egg so when Python 3 runs
.../python2.7/site-packages/virtualenv.py there is no pkg_resources.py file
right next to it, even though .../python2.7/site-packages is the first thing
on sys.path.

I don't know what the proper fix for this is though.  I'll bring it up on
debian-python and see if anybody has any ideas.


-- System Information:
Debian Release: squeeze/sid
  APT prefers natty-updates
  APT policy: (500, 'natty-updates'), (500, 'natty-security'), (500, 'natty')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.38-8-generic (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages python-virtualenv depends on:
ii  python                   2.7.1-0ubuntu5  interactive high-level object-orie
ii  python-pkg-resources     0.6.15-1ubuntu1 Package Discovery and Resource Acc
ii  python-setuptools        0.6.15-1ubuntu1 Python Distutils Enhancements (set
ii  python-support           1.0.10ubuntu3   automated rebuilding support for P

Versions of packages python-virtualenv recommends:
ii  python-pip                0.8.2-0ubuntu1 alternative Python package install

python-virtualenv suggests no packages.

-- no debconf information