[python-daemon-Bugs][315147] Bad file descriptor using tempfile/os.urandom on Python 3.4

Sat Aug 22 05:05:47 UTC 2015

python-daemon-Bugs item #315147 was changed at 2015-08-22 05:05 by Torfus Polymorphus
You can respond by visiting: 
https://alioth.debian.org/tracker/?func=detail&atid=413098&aid=315147&group_id=100328

Status: Open
Priority: 3
Submitted By: Torfus Polymorphus (torf-guest)
Assigned to: Nobody (None)
Summary: Bad file descriptor using tempfile/os.urandom on Python 3.4 

Initial Comment:
I'm trying to use Python's tempfile module inside DaemonContext. That works
fine in Python 2.7, however on Python 3.4 I'm running into problems related
to os.urandom.

Code to reproduce:

    #!/usr/bin/env python

    import sys
    import tempfile
    import traceback

    from daemon import DaemonContext

    with DaemonContext(stderr=sys.stderr):
        try:
            tempfile.TemporaryFile()
        except Exception as e:
            traceback.print_exc()

On Python 3.4, this prints the following traceback for me:

    Traceback (most recent call last):
      File "./sandbox.py", line 13, in <module>
        tempfile.TemporaryFile()
      File "/usr/lib/python3.4/tempfile.py", line 632, in TemporaryFile
        dir = gettempdir()
      File "/usr/lib/python3.4/tempfile.py", line 370, in gettempdir
        tempdir = _get_default_tempdir()
      File "/usr/lib/python3.4/tempfile.py", line 293, in _get_default_tempdir
        name = next(namer)
      File "/usr/lib/python3.4/tempfile.py", line 247, in __next__
        choose = self.rng.choice
      File "/usr/lib/python3.4/tempfile.py", line 238, in rng
        self._rng = _Random()
      File "/usr/lib/python3.4/random.py", line 90, in __init__
        self.seed(x)
      File "/usr/lib/python3.4/random.py", line 108, in seed
        a = int.from_bytes(_urandom(32), 'big')
    OSError: [Errno 9] Bad file descriptor

On Python 2.7 the code works without problems.

The problem seems to be that importing tempfile somehow automatically opens a
file descriptor that is used by os.urandom (the _urandom from the traceback
is just os.urandom). When the daemon context is entered, that file descriptor
goes bad. This can be seen by listing the open file descriptors before and
after importing tempfile, for example using the following code:

    #!/usr/bin/env python

    import os
    import os.path

    def print_open_fds():
        for root, dirnames, filenames in os.walk('/proc/self/fd'):
            for filename in filenames:
                fullname = os.path.join(root, filename)
                if not os.path.exists(fullname):
                    continue
                print(os.readlink(fullname))

    print("Open fds before importing tempfile:")
    print_open_fds()

    import tempfile

    print("Open fds before after tempfile:")
    print_open_fds()

On Python 3.4 this shows that a file descriptor for /dev/urandom is opened
when tempfile is imported (on Python 2.7 this does not happen). The same
behavior on both 3.4 and 2.7 can be seen if you replace "import tempfile"
in the code above with "os.urandom(32)". The problem also does not occur
if tempfile is only imported after the daemon context has been entered.

According to Python issue 18756 (https://bugs.python.org/issue18756) the
behavior was introduced on purpose to avoid performance problems with the
old code (which openend and closed the file descriptor for every call to
os.urandom). That change was also back-ported to Python 2.7, however it seems
that it's not in Python 2.7.6 (that's the 2.7 version I'm running).

The problem is that that file descriptor is neither documented nor publicly
accessible, as far as I can see. This means that it's hard to tell
DaemonContext to keep the file descriptor open via files_preserve. One
possibility is to manually find the file descriptor:

    import os
    import os.path
    import sys
    import tempfile
    import traceback

    from daemon import DaemonContext

    def get_fd(path):
        """
        Get an already open file descriptor for a file.
        """
        path = os.path.realpath(path)
        for root, dirnames, filenames in os.walk('/proc/self/fd'):
            for filename in filenames:
                fullname = os.path.join(root, filename)
                if not os.path.exists(fullname):
                    continue
                if os.path.realpath(fullname) == path:
                    return int(filename)
        raise RuntimeError('No open file descriptor for %r.' % path)

    fds = []
    try:
        fds.append(get_fd('/dev/urandom'))
    except RuntimeError:
        # No file descriptor for /dev/urandom, probably Python 2.7
        pass

    with DaemonContext(stderr=sys.stderr, files_preserve=fds):
        try:
            tempfile.TemporaryFile()
        except Exception as e:
            traceback.print_exc()

That works, but feels hackish.

I'd like to hear your opinion on this issue. I don't really think that this
is a problem with python-daemon, but it's an issue that users of
python-daemon are quite likely to encounter when they are working with
temporary files.

----------------------------------------------------------------------

Comment By: Torfus Polymorphus (torf-guest)
Date: 2015-08-22 05:05

Message:
It seems that others have had similar problems, cf. https://github.com/Pylons/pyramid/issues/1554. That thread mentions that this issue should be fixed in newer versions of Python 3.4 and in the upcoming Python 3.5, cf. Python issue 21207 (https://bugs.python.org/issue21207).

----------------------------------------------------------------------

Comment By: Torfus Polymorphus (torf-guest)
Date: 2015-08-22 04:55

Message:
Argh, is there any way to keep code formatted in a bug report?

----------------------------------------------------------------------

You can respond by visiting: 
https://alioth.debian.org/tracker/?func=detail&atid=413098&aid=315147&group_id=100328