Large messages causing mail loops

[chris coleman]

To fix this bug, I recommend the offlineimap team implement storing each message's Message-ID in the SQLite database, and use that Message-ID as an additional piece of information to help decide whether a message is already uploaded or downloaded, and possibly not upload or download it.  


IETF email standards have matured a lot since offlineimap was first designed and built.  Message-ID is now a standard unique identifier string, required to be placed in all email message headers by mail server software MTA's.  This fix would help offlineimap to correctly identify unique individual email messages, prevent duplicate downloads via infinte mail loops, avoid filling up storage with many gigabytes of incorrectly repeated identical copies of email messages, and allow smooth running of user's email communications.



On Wednesday, July 16, 2014 9:23 AM, T. Joseph Carter <tjcarter at spiritsubstance.com> wrote:
 


Hi,

I've run into an issue with OfflineIMAP that has caused some MASSIVE 
duplication with dreamhost.  This is somewhat difficult for me to 
debug as my immediate reaction to finding gigabytes upon gigabytes of 
duplicated mail is to immediately shut things down and start purging.

(There's also the little problem that this command did not terminate 
and was counting down to its next refresh…

tjcarter at amaya:~$ offlineimap -d imap -1 -o -a tjcarter

… because the account in question has

idlefolders = [ 'INBOX' ]

specified.  But that's a fairly minor niggle since idle is still 
indicated to be somewhat experimental.)


Anyway, how it tends to happen:

I will send a fairly large file via email (significantly > 5 MiB) 
from a client that doesn't easily tell you how big the thing you're 
sending happens to be.  Like a phone, for example.

The email may or may not deliver, depending on what the destination 
server does with zomg-huge emails.  It may deliver, it may bounce, or 
it may be silently deleted.  I've seen 35 MiB files send just fine to 
some, and emails (after encoding) as small as 1000000 total bytes 
silently discarded. Regardless, the client stores its copy in 
INBOX/Sent on the server, as expected.

Then offlineimap downloads this massive email.  Then it becomes 
convinced that this email is new and uploads the email.  Then it 
downloads the "new" email.  Then it determines the email is new and 
uploads it…

If the remote bounces the message back as too large with the full 
message attached, this too starts the mail duplication cycle.  And 
whatever you do, DO NOT delete all these duplicate emails into your 
trash folder!  That will start the duplicate email cycle on EACH of 
the already massively duplicated emails—so yes, it can be triggered 
without another MUA putting massively large messages into folders.  
Oh and when cleaning up this way, there may still be one or more 
copies on the server that will be downloaded and continue the massive 
mail duplication you just tried to stop.  Without direct access to 
the message directory on the server, your best bet for cleanup is to 
ctrl-c offlineimap, then kill -9 it because it doesn't terminate 
(*sigh*), then nuke the messages by hand in your local folders, then 
log into squirrelmail and do the same on the remote folders, and then 
hopefully probably it won't do that again.

You can work around the problem by telling offlineimap to skip large 
messages, but this really isn't desired.  It's what I'm doing at the 
moment, but … I am not sure how big the message has to be before 
things start getting out of hand.

I'm unable to realistically test this further because having it 
happen while I was away for a week brought me too near CenturyLink's 
usage cap.  And when you cross the threshold, they start throwing 
packets on the floor randomly.  Incidentally, I will be canceling 
their service next week, and I intend to tell them exactly why.  If 
they give me any crap, the cancel-my-service call will be published.

Other clients (mutt, Mail.app, Thunderbird, etc.) do not exhibit this 
behavior.

Joseph



_______________________________________________
OfflineIMAP-project mailing list
OfflineIMAP-project at lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/offlineimap-project

OfflineIMAP homepage: http://software.complete.org/offlineimap
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/offlineimap-project/attachments/20140716/c893cfe0/attachment.html>