[Pkg-mailman-hackers] Bug#919975: mailman3-web: whoosh python2 and python3 databases are not compatible

[Sampo Sorsa]

Package: mailman3-web
Version: 0+20180916-2~bpo9+1

Dear Maintainer,

After upgrading from earlier python2-based packages, I ran into this problem of Whoosh databases between python2 and python3 being incompatible:

$ sudo -u www-data /usr/share/mailman3-web/manage.py update_index_one_list example at example.com

Indexing XXX emails
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3/dist-packages/whoosh/writing.py", line 1015, in run
    writer.commit(*self.commitargs, **self.commitkwargs)
  File "/usr/lib/python3/dist-packages/whoosh/writing.py", line 922, in commit
    finalsegments = self._merge_segments(mergetype, optimize, merge)
  File "/usr/lib/python3/dist-packages/whoosh/writing.py", line 827, in _merge_segments
    return mergetype(self, self.segments)
  File "/usr/lib/python3/dist-packages/whoosh/writing.py", line 101, in MERGE_SMALL
    writer.add_reader(reader)
  File "/usr/lib/python3/dist-packages/whoosh/writing.py", line 710, in add_reader
    self.add_postings_to_pool(reader, basedoc, docmap)
  File "/usr/lib/python3/dist-packages/whoosh/writing.py", line 647, in add_postings_to_pool
    for item in items:
  File "/usr/lib/python3/dist-packages/whoosh/writing.py", line 583, in _process_posts
    for fieldname, text, docnum, weight, vbytes in items:
  File "/usr/lib/python3/dist-packages/whoosh/reading.py", line 426, in iter_postings
    m = self.postings(fieldname, btext)
  File "/usr/lib/python3/dist-packages/whoosh/reading.py", line 820, in postings
    matcher = FilterMatcher(matcher, deleted, exclude=True)
  File "/usr/lib/python3/dist-packages/whoosh/matching/wrappers.py", line 277, in __init__
    self._find_next()
  File "/usr/lib/python3/dist-packages/whoosh/matching/wrappers.py", line 302, in _find_next
    while child.is_active() and child.id() in ids:
  File "/usr/lib/python3/dist-packages/whoosh/codec/whoosh3.py", line 970, in id
    self._read_ids()
  File "/usr/lib/python3/dist-packages/whoosh/codec/whoosh3.py", line 1072, in _read_ids
    self._read_data()
  File "/usr/lib/python3/dist-packages/whoosh/codec/whoosh3.py", line 1067, in _read_data
    self._data = loads(b)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 4: ordinal not in range(128)

The solution is to remove /var/lib/mailman3/web/fulltext_index/ and run:

$ sudo -u www-data /usr/share/mailman3-web/manage.py rebuild_index

This is mostly FYI for other stretch-backports users. As mailman3-web is not in stretch, it's probably not worth it to bother handling this when upgrading packages.

--
Sampo Sorsa