<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:12pt"><div><span><br></span></div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span>To fix this bug, I recommend the offlineimap team implement storing each message's Message-ID in the SQLite database, and use that Message-ID as an additional piece of information to help decide whether a message is already uploaded or downloaded, and possibly not upload or download it. <br></span></div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span><br></span></div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family:
HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span>IETF email standards have matured a lot since offlineimap was first designed and built. Message-ID is now a standard unique identifier string, required to be placed in all email message headers by mail server software MTA's. This fix would help offlineimap to correctly identify unique individual email messages, prevent duplicate downloads via infinte mail loops, avoid filling up storage with many gigabytes of incorrectly repeated identical copies of email messages, and allow smooth running of user's email communications.</span></div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span><br></span></div> <div class="qtdSeparateBR"><br><br></div><div style="display: block;"
class="yahoo_quoted"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> On Wednesday, July 16, 2014 9:23 AM, T. Joseph Carter <tjcarter@spiritsubstance.com> wrote:<br> </font> </div> <br><br> <div class="y_msg_container">Hi,<br><br>I've run into an issue with OfflineIMAP that has caused some MASSIVE <br>duplication with dreamhost. This is somewhat difficult for me to <br>debug as my immediate reaction to finding gigabytes upon gigabytes of <br>duplicated mail is to immediately shut things down and start purging.<br><br>(There's also the little problem that this command did not terminate <br>and was counting down to its next refresh…<br><br><a ymailto="mailto:tjcarter@amaya"
href="mailto:tjcarter@amaya">tjcarter@amaya</a>:~$ offlineimap -d imap -1 -o -a tjcarter<br><br>… because the account in question has<br><br>idlefolders = [ 'INBOX' ]<br><br>specified. But that's a fairly minor niggle since idle is still <br>indicated to be somewhat experimental.)<br><br><br>Anyway, how it tends to happen:<br><br>I will send a fairly large file via email (significantly > 5 MiB) <br>from a client that doesn't easily tell you how big the thing you're <br>sending happens to be. Like a phone, for example.<br><br>The email may or may not deliver, depending on what the destination <br>server does with zomg-huge emails. It may deliver, it may bounce, or <br>it may be silently deleted. I've seen 35 MiB files send just fine to <br>some, and emails (after encoding) as small as 1000000 total bytes <br>silently discarded. Regardless, the client stores its copy in <br>INBOX/Sent on the server, as expected.<br><br>Then
offlineimap downloads this massive email. Then it becomes <br>convinced that this email is new and uploads the email. Then it <br>downloads the "new" email. Then it determines the email is new and <br>uploads it…<br><br>If the remote bounces the message back as too large with the full <br>message attached, this too starts the mail duplication cycle. And <br>whatever you do, DO NOT delete all these duplicate emails into your <br>trash folder! That will start the duplicate email cycle on EACH of <br>the already massively duplicated emails—so yes, it can be triggered <br>without another MUA putting massively large messages into folders. <br>Oh and when cleaning up this way, there may still be one or more <br>copies on the server that will be downloaded and continue the massive <br>mail duplication you just tried to stop. Without direct access to <br>the message directory on the server, your best bet for cleanup
is to <br>ctrl-c offlineimap, then kill -9 it because it doesn't terminate <br>(*sigh*), then nuke the messages by hand in your local folders, then <br>log into squirrelmail and do the same on the remote folders, and then <br>hopefully probably it won't do that again.<br><br>You can work around the problem by telling offlineimap to skip large <br>messages, but this really isn't desired. It's what I'm doing at the <br>moment, but … I am not sure how big the message has to be before <br>things start getting out of hand.<br><br>I'm unable to realistically test this further because having it <br>happen while I was away for a week brought me too near CenturyLink's <br>usage cap. And when you cross the threshold, they start throwing <br>packets on the floor randomly. Incidentally, I will be canceling <br>their service next week, and I intend to tell them exactly why. If <br>they give me any crap, the cancel-my-service call will be
published.<br><br>Other clients (mutt, Mail.app, Thunderbird, etc.) do not exhibit this <br>behavior.<br><br>Joseph<br><br><br><br>_______________________________________________<br>OfflineIMAP-project mailing list<br><a ymailto="mailto:OfflineIMAP-project@lists.alioth.debian.org" href="mailto:OfflineIMAP-project@lists.alioth.debian.org">OfflineIMAP-project@lists.alioth.debian.org</a><br><a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/offlineimap-project" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/offlineimap-project</a><br><br>OfflineIMAP homepage: <a href="http://software.complete.org/offlineimap" target="_blank">http://software.complete.org/offlineimap</a><br><br></div> </div> </div> </div> </div></body></html>