[Pkg-mailman-hackers] Bug#942540: sends an email a minute after server crash

[Antoine Beaupre]

Package: mailman3-web
Version: 0+20180916-8
Severity: normal

My server ran out of power today and totally crashed. This seems to have
corrupted the Whoosh search database, because I would receive a
traceback a minute by email after rebooting:


Subject: Cron <www-data at marcos>     [ -f /usr/bin/django-admin ] && flock -n /var/run/mailman3-web/cron.minutely /usr/share/mailman3-web/manage.py runjobs minutely
To: www-data at anarc.at
Date: Thu, 17 Oct 2019 14:43:34 -0400

[ERROR/MainProcess] Failed indexing 1 - 11 (retry 5/5): ord() expected a character, but string of length 0 found (pid 32248): ord() expected a character, but string of length 0 found
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/haystack/management/commands/update_index.py", line 97, in do_update
    backend.update(index, current_qs, commit=commit)
  File "/usr/lib/python3/dist-packages/haystack/backends/whoosh_backend.py", line 178, in update
    self.setup()
  File "/usr/lib/python3/dist-packages/haystack/backends/whoosh_backend.py", line 126, in setup
    self.index = self.storage.open_index(schema=self.schema)
  File "/usr/lib/python3/dist-packages/whoosh/filedb/filestore.py", line 176, in open_index
    return indexclass(self, schema=schema, indexname=indexname)
  File "/usr/lib/python3/dist-packages/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/usr/lib/python3/dist-packages/whoosh/index.py", line 632, in read
    check_size("int", _INT_SIZE)
  File "/usr/lib/python3/dist-packages/whoosh/index.py", line 626, in check_size
    sz = stream.read_varint()
  File "/usr/lib/python3/dist-packages/whoosh/filedb/structfile.py", line 191, in read_varint
    return read_varint(self.read)
  File "/usr/lib/python3/dist-packages/whoosh/util/varints.py", line 102, in read_varint
    b = ord(readfn(1))
TypeError: ord() expected a character, but string of length 0 found

There are a few problems here:

 1. it's very hard to act upon that error message: it doesn't say which
    file is corrupted or where
 2. it sends this *every minute* which is a lot
 3. it shouldn't happen in the first place: if the database is
    corrupted, it can regenerate it from scratch (maybe?)

This, for the record, is the "corrupt" database:

anarcat at marcos:~(master)$ sudo ls -al /var/lib/mailman3/web/fulltext_index.orig
total 8
drwxr-xr-x 2 www-data www-data 4096 oct 17 10:46 .
drwxr-x--- 5 www-data www-data 4096 oct 17 15:19 ..
-rw-r--r-- 1 www-data www-data    0 oct 17 10:46 _MAIN_367516.toc
-rw-r--r-- 1 www-data www-data    0 oct 17 10:46 MAIN_rv0glpgztufuptbi.seg
-rwxr-xr-x 1 www-data www-data    0 fév  3  2019 MAIN_WRITELOCK

ie. a bunch of empty files. That specific corruption could be easy to
recover from...

-- System Information:
Debian Release: 10.1
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.19.0-6-amd64 (SMP w/2 CPU cores)
Locale: LANG=fr_CA.UTF-8, LC_CTYPE=fr_CA.UTF-8 (charmap=UTF-8), LANGUAGE=fr_CA.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages mailman3-web depends on:
ii  dbconfig-sqlite3           2.0.11+deb10u1
ii  debconf [debconf-2.0]      1.5.71
ii  lsb-base                   10.2019051400
ii  node-less                  1.6.3~dfsg-3
ii  python3                    3.7.3-1
ii  python3-django-hyperkitty  1.2.2-1
ii  python3-django-postorius   1.2.4-1
ii  python3-mysqldb            1.3.10-2
ii  python3-psycopg2           2.7.7-1
ii  python3-whoosh             2.7.4+git6-g9134ad92-4
ii  sassc                      3.5.0-1
ii  ucf                        3.0038+nmu1
ii  uwsgi                      2.0.18-1
ii  uwsgi-plugin-python3       2.0.18-1

Versions of packages mailman3-web recommends:
ii  nginx               1.14.2-2+deb10u1
ii  nginx-full [nginx]  1.14.2-2+deb10u1

Versions of packages mailman3-web suggests:
ii  mariadb-server-10.3 [virtual-mysql-server]  1:10.3.17-0+deb10u1

-- debconf information:
* mailman3-web/db/basepath: /var/lib/dbconfig-common/sqlite3/mailman3-web
  mailman3-web/remote/port: 3306
* mailman3-web/superuser-mail: root at localhost
  mailman3-web/pgsql/authmethod-user: password
  mailman3-web/mysql/method: Unix socket
  mailman3-web/internal/skip-preseed: false
* mailman3-web/database-type: sqlite3
  mailman3-web/passwords-do-not-match:
  mailman3-web/pgsql/manualconf:
  mailman3-web/pgsql/authmethod-admin: ident
* mailman3-web/db/dbname: mailman3web.db
  mailman3-web/dbconfig-upgrade: true
* mailman3-web/dbconfig-reinstall: false
  mailman3-web/purge: false
* mailman3-web/mysql/admin-user: debian-sys-maint
  mailman3-web/upgrade-error: abort
  mailman3-web/install-error: abort
* mailman3-web/django-site: lists.anarc.at
  mailman3-web/remote/newhost:
  mailman3-web/remove-error: abort
  mailman3-web/pgsql/admin-user: postgres
* mailman3-web/dbconfig-install: true
* mailman3-web/emailname: anarc.at
  mailman3-web/missing-db-package-error: abort
* mailman3-web/configure-webserver: none
  mailman3-web/upgrade-backup: true
  mailman3-web/pgsql/method: TCP/IP
  mailman3-web/remote/host: localhost
  mailman3-web/nginx-choice:
  mailman3-web/pgsql/changeconf: false
* mailman3-web/restart-webserver: false
  mailman3-web/internal/reconfiguring: false
  mailman3-web/dbconfig-remove: true
  mailman3-web/pgsql/no-empty-passwords:
  mailman3-web/db/app-user: mailman3-web at localhost
* mailman3-web/superuser-name: admin