[Pkg-mailman-hackers] Bug#878597: mailman: messages shunted due to uncaught runner exception (UnicodeDecodeError in pipermail.py)

[Benoit Panizzon]

Package: mailman
Version: 1:2.1.23-1+deb9u4
Followup-For: Bug #878597

Dear Maintainer,

I'm experiencing exactly the same problem after upgrading to 2.1.23.

I run several mailinglists with different languages, so many emails are sent in utf8.

After upgrade a couple of lists stopped working and all messages are shunted with messages like:

Mar 02 07:18:00 2019 (29638) SHUNTING: 1551496266.198737+c6746c07f14b963960db9eaf05fcebbfa7b49439
Mar 02 07:18:00 2019 (29638) Uncaught runner exception: 'utf8' codec can't decode byte 0xaa in position 26: invalid start byte
Mar 02 07:18:00 2019 (29638) Traceback (most recent call last):
  File "/var/lib/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop
    self._onefile(msg, msgdata)
  File "/var/lib/mailman/Mailman/Queue/Runner.py", line 190, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/var/lib/mailman/Mailman/Queue/ArchRunner.py", line 77, in _dispose
    mlist.ArchiveMail(msg)
  File "/var/lib/mailman/Mailman/Archiver/Archiver.py", line 214, in ArchiveMail
    h.processUnixMailbox(f)
  File "/var/lib/mailman/Mailman/Archiver/pipermail.py", line 596, in processUnixMailbox
    self.add_article(a)
  File "/var/lib/mailman/Mailman/Archiver/pipermail.py", line 640, in add_article
    author = fixAuthor(article.decoded['author'])
  File "/var/lib/mailman/Mailman/Archiver/pipermail.py", line 63, in fixAuthor
    while i>0 and (L[i-1][0] in lowercase or
UnicodeDecodeError: 'utf8' codec can't decode byte 0xaa in position 26: invalid start byte

I have re-installed mailman etc, unshunted the messages, looked at the mbox files (I don't see anything wrong)
but the problem persists. U have no clue how I can get those lists back working.

Also deleting the shunted messages is not the solution, as new messages equaly get shunted.

What I have to mention, during the upgrade, something went wrong with the locales and I had to re-install them.
Could that be that maybe mailman compiled language files or similar while the locales were gone and this
caused the problem?

I would have guessed that re-installing mailman after fixing the locales would solve such a problem.

Any help greatly appriciated.

-Benoit-

-- System Information:
Debian Release: 9.8
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 3.2.0-4-686-pae (SMP w/1 CPU core)
Locale: LANG=de_CH.UTF-8, LC_CTYPE=de_CH.UTF-8 (charmap=UTF-8), LANGUAGE=de_CH.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages mailman depends on:
ii  apache-ssl [httpd]           1.3.34-4.1+etch1
ii  apache2 [httpd]              2.4.10-10+deb8u8
ii  apache2-mpm-prefork [httpd]  2.4.10-10+deb8u8
ii  cron [cron-daemon]           3.0pl1-128+deb9u1
ii  debconf [debconf-2.0]        1.5.61
ii  libc6                        2.24-11+deb9u4
ii  logrotate                    3.11.0-0.1
ii  lsb-base                     9.20161125
ii  python                       2.7.13-2
ii  python-dnspython             1.15.0-1+deb9u1
ii  ucf                          3.0036

Versions of packages mailman recommends:
ii  sendmail-bin [mail-transport-agent]  8.15.2-8

Versions of packages mailman suggests:
pn  listadmin     <none>
ii  lynx          2.8.9dev11-1
pn  spamassassin  <none>

-- Configuration Files:
/etc/cron.d/mailman changed:
0 8 * * * list [ -x /usr/lib/mailman/cron/checkdbs ] && /usr/lib/mailman/cron/checkdbs
0 9 * * * list [ -x /usr/lib/mailman/cron/disabled ] && /usr/lib/mailman/cron/disabled
0 12 * * * list [ -x /usr/lib/mailman/cron/senddigests ] && /usr/lib/mailman/cron/senddigests
0 5 1 * * list [ -x /usr/lib/mailman/cron/mailpasswds ] && /usr/lib/mailman/cron/mailpasswds
*/5 * * * * list [ -x /usr/lib/mailman/cron/gate_news ] && /usr/lib/mailman/cron/gate_news
27 3 * * * list [ -x /usr/lib/mailman/cron/nightly_gzip ] && /usr/lib/mailman/cron/nightly_gzip
30 4 * * * list [ -x /usr/lib/mailman/cron/cull_bad_shunt ] && /usr/lib/mailman/cron/cull_bad_shunt


-- debconf information:
* mailman/used_languages: ca de en es eu fi fr hu ia it nl no pl pt pt_BR ro sr sv tr
* mailman/site_languages: sr, tr, sv, ro, pt_BR, pt, pl, no, nl, it, ia, hu, fr, fi, eu, es, de, ca, en
* mailman/create_site_list:
  mailman/gate_news: true
* mailman/default_server_language: en
* mailman/queue_files_present: continue regardless