[Python-modules-team] Bug#932183: libapache2-mod-wsgi-py3: sys.getfilesystemencoding() regression

[Sebastian Rittau]

Package: libapache2-mod-wsgi-py3
Version: 4.6.5-1
Severity: normal

Dear Maintainer,

apologies in advance if this gets reported twice.


Since upgrading from Debian 9 to Debian 10, our WSGI app starts
reporting UnicodeEncodeErrors when trying to access file names containing
umlauts. Consider the following file:

--------------

import os
import sys

def application(env, sr):
     print(sys.getfilesystemencoding())
     print(os.environ)
     with open("/tmp/föo", "w"):
         pass
     sr("200 OK", [("Content-Type", "text/plain")])
     return [b"Ok"]

--------------

This will log the following (snipping Apache log stuff):

--------------

ascii
environ({'APACHE_RUN_DIR': '/var/run/apache2', 'APACHE_PID_FILE': 
'/var/run/apache2/apache2.pid', 'JOURNAL_STREAM': '9:3328729', 'PATH': 
'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 
'INVOCATION_ID': '9002aec454134616ab90af90af7c3c79', 'APACHE_LOCK_DIR': 
'/var/lock/apache2', 'LANG': 'de_DE.UTF-8', 'APACHE_RUN_USER': 
'www-data', 'APACHE_RUN_GROUP': 'www-data', 'APACHE_LOG_DIR': 
'/var/log/apache2', 'PWD': '/', 'LC_ALL': 'de_DE.UTF-8', 'HOME': 
'/var/www'})
mod_wsgi (pid=851): Exception occurred processing WSGI script 
'/srv/intranet.example.com/foo.wsgi'.
Traceback (most recent call last):
   File "/srv/intranet.example.com/foo.wsgi", line 7, in application
     with open("/tmp/f\xc3\xb6o", "w"):
UnicodeEncodeError: 'ascii' codec can't encode character '\\xf6' in 
position 6: ordinal not in range(128)

--------------

As can be seen from the first line, sys.getfilesystemencoding() returns
ASCII, despite the environment variables LANG and LC_ALL being set
correctly. (de_DE.UTF-8 also exists as a locale.) This is the corresponding
config:

--------------

         WSGIApplicationGroup intranet.example.com
         WSGIDaemonProcess intranet.example.com display-name=%{GROUP} 
lang='de_DE.UTF-8' locale='de_DE.UTF-8' 
python-path=/usr/local/virtualenv/intranet/lib/python3.7/site-packages/ 
processes=8 threads=50
         WSGIProcessGroup intranet.example.com

--------------

Potentially related to #872887.

-- System Information:
Debian Release: 10.0
   APT prefers stable
   APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-5-amd64 (SMP w/8 CPU cores)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), 
LANGUAGE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages libapache2-mod-wsgi-py3 depends on:
ii  apache2-bin [apache2-api-20120211]  2.4.38-3
ii  libc6                               2.28-10
ii  libpython3.7                        3.7.3-2
ii  python3                             3.7.3-1

libapache2-mod-wsgi-py3 recommends no packages.

libapache2-mod-wsgi-py3 suggests no packages.

-- no debconf information