KeyError during IMAP to IMAP sync of deleted msgs

[Dominic LoBue]

On Tue, Dec 8, 2009 at 7:36 AM, Dan Christensen <jdc at uwo.ca> wrote:
> Dan Christensen <jdc at uwo.ca> writes:
>
>> Dan Christensen <jdc at uwo.ca> writes:
>>
>>> Unfortunately, the problem is more serious than having to restart
>>> offlineimap.  After the sync finally completed, one message was missing
>>> from the local IMAP folder, and 55 messages were duplicated!
>>
>> Tried from scratch again, and one message went missing and 29 were
>> duplicated.  Also lost a read mark.  The folder I'm trying to sync
>> has about 6000 new messages at the local end, and 0 at the remote
>> end.
>
> Sorry, I take it back:  the missing message was due to a problem in
> populating the imap group in the first place, before initiating the
> sync.  But the duplicated messages (and the keyerrors during sync)
> appear to be bugs in offlineimap.
>
> I tried the whole process another time, first populating the remote imap
> server with the 6000 messages, and using offlineimap to copy them to the
> local server.  This time there were no problems at all.  Unfortunately,
> this method causes double the network traffic.
>
> Summary:  it appears that there's a bug involving copying many messages
> from a local imap server to a remote one.
>
> This is now Issue #193 on trac:
>
>  http://software.complete.org/software/issues/show/193
>
> where I have provided logs and also figured out that the problem occurs
> more often with maxconnections set higher.
>
> Dan
>
>
> _______________________________________________
> OfflineIMAP-project mailing list
> OfflineIMAP-project at lists.alioth.debian.org
> http://lists.alioth.debian.org/mailman/listinfo/offlineimap-project
>
> OfflineIMAP homepage: http://software.complete.org/offlineimap
>

Dan,

The cause of the problem you are seeing is a result of how the
multithreading was implemented, and good old Murphy.

I won't bore you with the details, but basically the way that
OfflineIMAP speeds up syncs is by utilizing several connections at
once, one folder sync operation per connection.

What I believe to be happening is that while a folder sync operation
is in progress on one of/the massive folder(s), another sync operation
begins on that same directory. IMAP weirdness and Murphy step in, and
all of a sudden OfflineIMAP crashes, taking the rest of the
connections with it.

The cause of the lost flags is as I described above: one of the sync
operations had not yet gotten around to setting flags before the
entire application crashed. When you restart the sync operation,
OfflineIMAP sees there is no flag on the remote end, and gives that
side precedence.

I'm in the process of porting OfflineIMAP to a different backend which
will probably solve your problem as a byproduct. This will take me a
while however.

Running OfflineIMAP in single-threaded mode should resolve your
problem in the meantime.

-- 
Dominic LoBue