[Pkg-mailman-hackers] Bug#1014037: mailman3-web: Possible memory leak: uwsgi OOMs after a few weeks

[Michael Tremer]

Hello Antoine,

> On 24 Feb 2025, at 16:51, Antoine Beaupré <anarcat at debian.org> wrote:
> 
> On 2025-02-24 15:56:57, Michael Tremer wrote:
> 
> [...]
> 
>> So, has this solved it all for good for you guys? What release of xapian are you on?
> 
> In the end, no, not really. Things have *improved*: we used to have 4-5
> OOM/day, with peaks at 15, 120 when reindexing, and this is down to 1-5
> a day, depending on the day. Kind of hard to track discrete events like
> this...
> 
> We had a single OOM in the last 48h. That's "nice".

Okay, this might still be a slight step in the right direction.

> We also have stupidly large Xapian indexes now, it's ridiculous. Clearly
> something wrong either with the haystack or the hyperkitty
> implementation. So far I've filed it in the latter:
> 
> https://gitlab.com/mailman/hyperkitty/-/issues/533
> 
> So, TL;DR: improved, but not fixed. I suspect we had a multi-dimensional
> issue, of which search/whoosh *was* a part of, because we would see a
> huge increase in OOMs when rebuilding the indexes. But we're still
> having an issue, so perhaps there's something else.

This is my experience with Xapian and I have found confirmation that this is supposed to be normal. My mailbox indexes were massive and there was no point having them any more. So I can confirm that this looked very similar in Dovecot, too.

> We tried to hookup a memory profiler (austin) but it failed because it
> didn't work with Python 3.11... so maybe that's something we'll try to
> revisit after our trixie upgrades (hopefully soon!).

I tried that but I was struggling with a missing sssd and some other things. Not sure I am ready to try again. It would also not help us to find out where exactly this went wrong if it were fixed :(

> 
> a.
> 
> -- 
> On reconnait la grandeur et la valeur d'une nation à la façon dont
> celle-ci traite ses animaux.
>                        - Mahatma Gandhi