imaplib2 eating data :-(

[Sebastian Spaeth]

Here is a bug of recent offlineimap falsly trying to delete around 8k
emails as it thinks they are gone from the IMAP server.
https://bugzilla.redhat.com/show_bug.cgi?id=708898

 Fortunately, we got a full IMAP trace in:
https://bugzilla.redhat.com/attachment.cgi?id=501695 and I have now
analyzed it, concluding that IMAPLIB2 is suppressing data in its
response which is bad...

Piers, any idea?

Let me give you the summary:
To get the list of uids we send FETCH 1:25166 (FLAGS UID)\r\n to the
server and it responds with a whole bunch of responses. And one bunch of
responses between "reader rcvd 1024" and "reader poll =>" which the
server definitely sends goes missing. This comes from the server:

who-t.net reader: DEBUG[imap]:   26:24.48 who-t.net reader rcvd 1024
who-t.net reader: DEBUG[imap]:   26:24.48 who-t.net reader < * 16241 FETCH (FLAGS (Old NonJunk) UID 32768)\r\n
...
who-t.net reader: DEBUG[imap]:   26:24.47 who-t.net reader < * 16240 FETCH (FLAGS (Old NonJunk) UID 32767)\r\n
who-t.net reader: DEBUG[imap]:   26:24.47 who-t.net reader poll => [(5,1)]

...

But then the summary response (the _get_untagged_response(FETCH))
includes all UIDS, but not those 8401 ones...

....
Folder sync [who-t.net]: DEBUG[imap]:   26:36.59 Folder sync [who-t.net]
_get_untagged_response(FETCH) => ['1 (FLAGS (\\Seen Old) UID 4807)', '2
(FLAGS (Old) UID 5400)',...


And so, we assume they have been deleted on the server, and we try to
delete them from the local Maildir too:
...
Folder sync [who-t.net]: Deleting 8401 messages (32768, 32769, 32771,
32772, 32775, 32776, 32777, 32778, 32779, 32780, 32781, 32782, 32783,
32784, 32785, 32786, 32787, 32788....


This is really bad, and the log seems pretty clearly to indicate the
imaplib2 is not handing back all data that it receives. Which is an
outch, because it leads to mails being deleted. We got at least 2
examples now where recent offlineimap erronously tries to delete emails.

Can we pretty please get this debugged and working before we send
offlineimap to the people? I *really* feel uncomfortable with this
situation.

Sebastian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110606/90a099c5/attachment.sig>