Bug#609845: huge memory leak when syncing large mailboxes

[Phil Pearl (Lobbes)]

Hi Michael and Gilles,

Sorry for the delay in responding.

> >> I've been trying to sync a larger IMAP mailbox (~6Gb including all
> >> folders) and imapsync constantly gets terminated by the kernel
> >> OOM-killer because it takes up all available memory (1024 mb).

Michael, do you know what the size is of largest message that is being
sync'd?  I saw you mentioned maybe 60MB so I can try that out as a test
case I guess.

> > This maybe a well known problem coming directly from
> > libmail-imapclient-perl.  The problem doesn't come with large
> > mailbox but with large messages.  imapsync uses memory nearly 17
> > times the largest message size.  There is also an old memory issue
> > on freebsd systems.

Mail::IMAPClient has some major inefficiencies and I suppose imapsync
ends up doubling those because of the "from" and "to" objects each
creating copies of the messages (and because methods used keep data in
memory instead of moving them to files... it's a trade-off for sure).

I'll try to look a bit more at this problem over the next few weeks.
I've had a few complaints about memory usage.  I am not convinced we
have a leak but we definitely waste resources.  I'll have to do some leg
work to find something that helps expose the root of the problem(s) if
there are any outside of the obvious inefficiencies.

  > Or can imapsync workaround this limitation in Mail:IMAPClient
  > somehow?

Gilles, you said...

  I try but I partially succeeded. I went to decrease 17 to 4 times but
  I got errors and I gave up. I need to read carefully the IMAP RFC
  before going on again.

Any chance you want to share some details on changes you made and/or
attempted?  Or were they simply changing of which methods you used and
not changes working around Mail::IMAPClient behavior? I know you've
chosen to not share code anymore, but I hope you will reciprocate
if/when appropriate.

In theory append_file() should be more (memory) efficient than
append()/append_string() on large messages.  Likewise for
message_string() vs message_to_file().  Last I checked, imapsync still
uses append_string() which causes the entire message contents to get
stored in memory (several times over perhaps).  It might be worth while
to use append_string() for messages up to size X and then switch to
append_file()/message_file() when the messages are > size X if those
cases can easily be identified by imapsync.  Otherwise, imapsync will
always be straining system memory for large files.

For everyone's reference, the bug filed on CPAN for Mail::IMAPClient is:

  Bug #61292: Memory consumption with message_string() and append()
  https://rt.cpan.org/Public/Bug/Display.html?id=61292

Phil