Offlineimap duplicating email redux

[Edward Z. Yang]

Six years ago, a user wrote in to report that OfflineIMAP was
downloading then reuploading files to the server:

    http://www.mail-archive.com/debian-user@lists.debian.org/msg131430.html

I would like to state that I've seen this behavior, and I did
some more investigative work.  I think the following code in OfflineImap
is wrong:

        for uid in self.getmessagelist().keys():
            if uid < 0:                 # Ignore messages that pass 1 missed.
                continue
            if not uid in dest_messagelist:
                if self.suggeststhreads():
                    self.waitforthread()
                    thread = InstanceLimitedThread(\
                        self.getcopyinstancelimit(),
                        target = self.copymessageto,
                        name = "Copy message %d from %s" % (uid,
                                                            self.getvisiblename()),
                        args = (uid, applyto))
                    thread.setDaemon(1)
                    thread.start()
                    threads.append(thread)
                else:
                    self.copymessageto(uid, applyto, register = 0)

In particular, notice that we only test dest_messagelist (presumably
LocalStatus) when determining whether or not to copymessageto, which
then unconditionally resaves the message.  So this is consistent with
the behavior I (and the other user) saw that if LocalStatus is wrong,
you are so out of luck: OfflineIMAP will duplicate your messages into
kingdom come.  Experimentally, this seems to be the case:

    Copy message 187524 Maildir[INBOX] -> IMAP[INBOX], LocalStatus[INBOX]
    Copy message 187525 Maildir[INBOX] -> IMAP[INBOX], LocalStatus[INBOX]
    Copy message 187526 Maildir[INBOX] -> IMAP[INBOX], LocalStatus[INBOX]

and checking my inbox...

    >>> imapobj.uid('fetch','187526','(BODY.PEEK[])')
    ('OK', [('139083 (UID 187526 BODY[] {7448}', ' ...

(Note that it would have been impossible to have actually managed to copy
the message to that UID, so it must have been there prior.  I terminated
offlineimap before it got to 187527, but I checked, and sure enough:

    >>> imapobj.uid('fetch','187527','(BODY.PEEK[])')
    ('OK', [('139084 (UID 187527 BODY[] {5218}', 
)

This is particularly annoying for me, because whenever a message is
resaved to IMAP, the Maildir file must change (since we have no control
over the UIDs that the IMAP server assigns), which I think confuses the fuck
out of my mail client (Sup).

What's more concerning, and what I believe has caused me to lose some email
permanently, is that the deletion algorithm appears to do about the same thing.
So I frequently notice OfflineIMAP deleting /huge/ amounts of mail from my
Maildir that I *know* I did not delete (I actually wouldn't have noticed if
Sup wasn't realizing the files were going missing and let me know about it.)

Right now, I've disabled deletion from Maildir and uploading to IMAP, which
seems to be working OK as a stopgap. But please, let's fix these algorithms!

Cheers,
Edward