[Python-modules-team] Bug#932188: libapache2-mod-wsgi-py3: sys.getfilesystemencoding() regression

[Sebastian Rittau]

Package: libapache2-mod-wsgi-py3
Version: 4.6.5-1
Severity: normal

Dear Maintainer,

since upgrading from Debian 9 to Debian 10, our WSGI app starts
reporting UnicodeEncodeErrors when trying to access file names containing
umlauts. Consider the following file:

--------------

import os
import sys

def application(env, sr):
    print(sys.getfilesystemencoding())
    print(os.environ)
    with open("/tmp/föo", "w"):
        pass
    sr("200 OK", [("Content-Type", "text/plain")])
    return [b"Ok"]

--------------

This will log the following (snipping Apache log stuff):

--------------

ascii
environ({'APACHE_RUN_DIR': '/var/run/apache2', 'APACHE_PID_FILE': '/var/run/apache2/apache2.pid', 'JOURNAL_STREAM': '9:3328729', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'INVOCATION_ID': '9002aec454134616ab90af90af7c3c79', 'APACHE_LOCK_DIR': '/var/lock/apache2', 'LANG': 'de_DE.UTF-8', 'APACHE_RUN_USER': 'www-data', 'APACHE_RUN_GROUP': 'www-data', 'APACHE_LOG_DIR': '/var/log/apache2', 'PWD': '/', 'LC_ALL': 'de_DE.UTF-8', 'HOME': '/var/www'})
mod_wsgi (pid=851): Exception occurred processing WSGI script '/srv/intranet.example.com/foo.wsgi'.
Traceback (most recent call last):
  File "/srv/intranet.example.com/foo.wsgi", line 7, in application
    with open("/tmp/f\xc3\xb6o", "w"):
UnicodeEncodeError: 'ascii' codec can't encode character '\\xf6' in position 6: ordinal not in range(128)

--------------

As can be seen from the first line, sys.getfilesystemencoding() returns
ASCII, despite the environment variables LANG and LC_ALL being set
correctly. (de_DE.UTF-8 also exists as a locale.) This is the corresponding
config:

--------------

        WSGIApplicationGroup intranet.example.com
        WSGIDaemonProcess intranet.example.com display-name=%{GROUP} lang='de_DE.UTF-8' locale='de_DE.UTF-8' python-path=/usr/local/virtualenv/intranet/lib/python3.7/site-packages/ processes=8 threads=50
        WSGIProcessGroup intranet.example.com

--------------

Potentially related to #872887.

-- System Information:
Debian Release: 10.0
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-5-amd64 (SMP w/8 CPU cores)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages libapache2-mod-wsgi-py3 depends on:
ii  apache2-bin [apache2-api-20120211]  2.4.38-3
ii  libc6                               2.28-10
ii  libpython3.7                        3.7.3-2
ii  python3                             3.7.3-1

libapache2-mod-wsgi-py3 recommends no packages.

libapache2-mod-wsgi-py3 suggests no packages.

-- no debconf information