[Pkg-mailman-hackers] Bug#1014037: mailman3-web: Possible memory leak: uwsgi OOMs after a few weeks

[Michael Tremer]

Hello,

Apologies for my late reply.

> On 7 Feb 2025, at 03:12, Antoine Beaupré <anarcat at debian.org> wrote:
> 
> On 2025-01-16 09:40:05, Antoine Beaupré wrote:
>>> I would be interested to hear what you are making of this all.
>> 
>> Clearly there's a memory leak in this implementation as well, but we'll
>> know better whether it's specific to apache/uwsgi when we test with
>> gunicorn.
>> 
>> Stay tuned!
> 
> At last, we have news!
> 
> I *think* I have identified the culprit. While handling an unrelated
> issue (GDPR anyone?) we had to rebuild the search indexes and, while
> testing *that*, we found that we could pretty reliably crash mailman-web
> by... well, just searching all lists crashes it.
> 
> Boom. It's search?

Hmm, I don’t want to bring everybody down, but I think I cannot confirm this.

> I'm in the process of switching to Xapian now. This brings a whole lot
> of other issues (it uses more disk space and there's a bug in the
> xapian-haystack library that crashes indexing, see #), but so far, we've
> completely cleared out any OOM errors we were previously getting.
> 
> Check out this beauty:
> 
> <Screenshot 2025-02-06 at 22-08-59 View panel - Memory usage - system - Dashboards - Grafana.png>
> In this graph, the vertical line at 21:26 is when xapian was
> deployed. You can see there's precisely zero errors after that.
> 
> So I would advise everyone with this bug to try to switch off the search
> engine (not sure how) or switch to Xapian or another search
> engine. Doing the switch is a little obscure because upstream docs are
> quite ... sparse, but i've written a bit about it here:
> 
> https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/lists#search-engine
> 
> We used this in mailman's mailman-web.py:
> 
> # Haystack is the Django search engine plugin which is used by
> # Hyperkitty to index mail.
> HAYSTACK_CONNECTIONS = {
>    'default': {
>        #'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
>        #'PATH': os.path.join(BASE_DIR, "fulltext_index"),
>        # setup Xapian search engine instead, see:
>        # https://django-haystack.readthedocs.io/en/master/tutorial.html#xapian
>        # requires the python3-xapian-haystack package
>        # other options include SolR and ElasticSearch
>        'ENGINE': 'xapian_backend.XapianEngine',
>        'PATH': "/var/lib/mailman3/web/xapian_index",
>    },
> }

This is my configuration which my instance has been on all along:

# FTS
HAYSTACK_CONNECTIONS = {
    'default' : {
        'PATH'   : "/var/lib/mailman3/web/fulltext_index",
        'ENGINE' : 'xapian_backend.XapianEngine'
    },
}

Looks pretty much the same to me.

Xapian has not been giving us great results in the rest of our infrastructure. We used it in dovecot and it is creating HUGE indexes which were about half the size of the original inboxes and therefore was even very slow to search in it. We migrated to Solr there, but that was not an option for Mailman.

Whoosh was expectedly worse.

So, has this solved it all for good for you guys? What release of xapian are you on?

# apt-cache show python3-xapian-haystack
Package: python3-xapian-haystack
Source: python-xapian-haystack
Version: 2.1.1-1+deb12u1
Installed-Size: 91
Maintainer: Debian Python Team <team+python at tracker.debian.org>
Architecture: all
Depends: python3-django-haystack, python3-xapian, python3-django, python3:any
Enhances: python3-django-haystack
Description-en: Xapian backend for Django-Haystack (Python3 version)
 Xapian-haystack is a backend of Django-Haystack for the Xapian search engine.
 It provides all the standard features of Haystack:
  * Weighting
  * Faceted search (date, query, etc.)
  * Sorting
  * Spelling suggestions
  * EdgeNGram and Ngram (for autocomplete)
 The endswith search operation is not supported.
 .
 This package contains the Python 3 version of the library.
Description-md5: 5e43ae0149e2df6b3df16ddcf87f3b13
Homepage: https://github.com/notanumber/xapian-haystack/
Section: python
Priority: optional
Filename: pool/main/p/python-xapian-haystack/python3-xapian-haystack_2.1.1-1+deb12u1_all.deb
Size: 21412
MD5sum: c203fd6ef9a992ad418f0685a528a45e
SHA256: 9b70209f36b9bccbfda0346b048d024c48fd4c168e8a0bfe811a3c770eb18287

> and ran update-index, basically.
> 
> the indexing is *much* faster. indexing a single list used to take a
> a long time, sometimes almost a day, now we're about to complete a full
> index (if we don't run out of disk space) within less than two hours.
> 
> a.
> 
> -- 
> L'adversaire d'une vraie liberté est un désir excessif de sécurité.
>                        - Jean de la Fontaine