[Pkg-mailman-hackers] Bug#1014037: mailman3-web: Possible memory leak: uwsgi OOMs after a few weeks

[Antoine Beaupré]

On 2025-01-16 09:40:05, Antoine Beaupré wrote:
>> I would be interested to hear what you are making of this all.
>
> Clearly there's a memory leak in this implementation as well, but we'll
> know better whether it's specific to apache/uwsgi when we test with
> gunicorn.
>
> Stay tuned!

At last, we have news!

I *think* I have identified the culprit. While handling an unrelated
issue (GDPR anyone?) we had to rebuild the search indexes and, while
testing *that*, we found that we could pretty reliably crash mailman-web
by... well, just searching all lists crashes it.

Boom. It's search?

I'm in the process of switching to Xapian now. This brings a whole lot
of other issues (it uses more disk space and there's a bug in the
xapian-haystack library that crashes indexing, see #), but so far, we've
completely cleared out any OOM errors we were previously getting.

Check out this beauty:

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot 2025-02-06 at 22-08-59 View panel - Memory usage - system - Dashboards - Grafana.png
Type: image/png
Size: 745624 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/pkg-mailman-hackers/attachments/20250206/f60e9332/attachment-0001.png>
-------------- next part --------------

In this graph, the vertical line at 21:26 is when xapian was
deployed. You can see there's precisely zero errors after that.

So I would advise everyone with this bug to try to switch off the search
engine (not sure how) or switch to Xapian or another search
engine. Doing the switch is a little obscure because upstream docs are
quite ... sparse, but i've written a bit about it here:

https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/lists#search-engine

We used this in mailman's mailman-web.py:

# Haystack is the Django search engine plugin which is used by
# Hyperkitty to index mail.
HAYSTACK_CONNECTIONS = {
    'default': {
        #'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
        #'PATH': os.path.join(BASE_DIR, "fulltext_index"),
        # setup Xapian search engine instead, see:
        # https://django-haystack.readthedocs.io/en/master/tutorial.html#xapian
        # requires the python3-xapian-haystack package
        # other options include SolR and ElasticSearch
        'ENGINE': 'xapian_backend.XapianEngine',
        'PATH': "/var/lib/mailman3/web/xapian_index",
    },
}


and ran update-index, basically.

the indexing is *much* faster. indexing a single list used to take a
a long time, sometimes almost a day, now we're about to complete a full
index (if we don't run out of disk space) within less than two hours.

a.

-- 
L'adversaire d'une vraie liberté est un désir excessif de sécurité.
                        - Jean de la Fontaine