[Pkg-mailman-hackers] Bug#1043098: python3-django-hyperkitty: tracebacks with update_index_one_list on mailman2 migration

[Steve Langasek]

Package: python3-django-hyperkitty
Version: 1.3.4-4
Severity: important

Dear Maintainer,

Having just upgraded a system to bullseye, I am in the process of migrating
my mailing lists from the obsolete mailman 2 to mailman3.  Following guides
such as https://docs.mailman3.org/en/latest/migration.html leads me to run,
as list:

$ django-admin hyperkitty_import --settings mailman-web \
    --pythonpath /var/lib/mailman3/hyperkitty/ -l $list_email $mbox_path
$

which succeeds (/var/lib/mailman3/hyperkitty/mailman-web.py is a forked copy
of /etc/mailman3/mailman-web.py with appropriate permissions).

I am then supposed to run update_index_one_list, which fails with a 
confusing error about missing templates:

$ django-admin update_index_one_list --settings mailman-web \
    --pythonpath /var/lib/mailman3/hyperkitty/ -l $list_email
Indexing 421 emails
[ERROR/MainProcess] Failed indexing 1 - 421 (retry 5/5): search/indexes/hyperkitty/email_text.txt (pid 6818): search/indexes/hyperkitty/email_text.txt
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/haystack/management/commands/update_index.py", line 111, in do_update
    backend.update(index, current_qs, commit=commit)
  File "/usr/lib/python3/dist-packages/haystack/backends/whoosh_backend.py", line 269, in update
    doc = index.full_prepare(obj)
  File "/usr/lib/python3/dist-packages/haystack/indexes.py", line 236, in full_prepare
    self.prepared_data = self.prepare(obj)
  File "/usr/lib/python3/dist-packages/haystack/indexes.py", line 227, in prepare 
    self.prepared_data[field.index_fieldname] = field.prepare(obj)
  File "/usr/lib/python3/dist-packages/haystack/fields.py", line 235, in prepare
    return self.convert(super(CharField, self).prepare(obj))
  File "/usr/lib/python3/dist-packages/haystack/fields.py", line 99, in prepare
    return self.prepare_template(obj)
  File "/usr/lib/python3/dist-packages/haystack/fields.py", line 212, in prepare_template
    t = loader.select_template(template_names)
  File "/usr/lib/python3/dist-packages/django/template/loader.py", line 47, in select_template
    raise TemplateDoesNotExist(', '.join(template_name_list), chain=chain)
django.template.exceptions.TemplateDoesNotExist: search/indexes/hyperkitty/email_text.txt
Traceback (most recent call last):
  File "/usr/bin/django-admin", line 5, in <module>
    management.execute_from_command_line()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/hyperkitty/management/commands/update_index_one_list.py", line 41, in handle
    update_index(listname=options.get("listname")[0],
  File "/usr/lib/python3/dist-packages/hyperkitty/search_indexes.py", line 117, in update_index
    update_cmd.update_backend("hyperkitty", "default")
  File "/usr/lib/python3/dist-packages/haystack/management/commands/update_index.py", line 319, in update_backend
    max_pk = do_update(
  File "/usr/lib/python3/dist-packages/haystack/management/commands/update_index.py", line 111, in do_update
    backend.update(index, current_qs, commit=commit)
  File "/usr/lib/python3/dist-packages/haystack/backends/whoosh_backend.py", line 269, in update
    doc = index.full_prepare(obj)
  File "/usr/lib/python3/dist-packages/haystack/indexes.py", line 236, in full_prepare
    self.prepared_data = self.prepare(obj)
  File "/usr/lib/python3/dist-packages/haystack/indexes.py", line 227, in prepare   
    self.prepared_data[field.index_fieldname] = field.prepare(obj)
  File "/usr/lib/python3/dist-packages/haystack/fields.py", line 235, in prepare
    return self.convert(super(CharField, self).prepare(obj))
  File "/usr/lib/python3/dist-packages/haystack/fields.py", line 99, in prepare
    return self.prepare_template(obj)
  File "/usr/lib/python3/dist-packages/haystack/fields.py", line 212, in prepare_template
    t = loader.select_template(template_names)
  File "/usr/lib/python3/dist-packages/django/template/loader.py", line 47, in select_template
    raise TemplateDoesNotExist(', '.join(template_name_list), chain=chain)
django.template.exceptions.TemplateDoesNotExist: search/indexes/hyperkitty/email_text.txt
$

This is quite confusing, as 
/usr/lib/python3/dist-packages/hyperkitty/templates/search/indexes/hyperkitty/email_text.txt
certainly exists.

It appears the only impact is on searchability of the mailing list 
archive, which isn't terribly important to me, but I'd like to know why 
it throws a backtrace.

-- System Information:
Debian Release: 11.7
  APT prefers oldstable-updates
  APT policy: (500, 'oldstable-updates'), (500, 'oldstable')
Architecture: i386 (x86_64)
Foreign Architectures: amd64

Kernel: Linux 4.19.0-24-amd64 (SMP w/1 CPU thread)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/bash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages python3-django-hyperkitty depends on:
ii  fonts-glyphicons-halflings   1.009~3.4.1+dfsg-2
ii  libjs-bootstrap4             4.5.2+dfsg1-8~deb11u1
ii  libjs-sphinxdoc              3.4.3-2
ii  python3                      3.9.2-3
ii  python3-dateutil             2.8.1-6
ii  python3-django               2:2.2.28-1~deb11u1
ii  python3-django-compressor    2.4-2
ii  python3-django-extensions    3.0.3-3
ii  python3-django-gravatar2     1.4.4-2
ii  python3-django-haystack      3.0-1
ii  python3-django-mailman3      1.3.5-2
ii  python3-django-q             1.2.1-1
ii  python3-djangorestframework  3.12.1-1
ii  python3-elasticsearch        7.1.0-3
ii  python3-flufl.lock           5.0.1-1
ii  python3-mailmanclient        3.3.2-1
ii  python3-networkx             2.5+ds-2
ii  python3-robot-detection      0.4.0-2
ii  python3-tz                   2021.1-1

Versions of packages python3-django-hyperkitty recommends:
ii  mailman3-web  0+20200530-2

python3-django-hyperkitty suggests no packages.

-- no debconf information