Gmail to "gmail apps" sync: ValueError: Backend could not find uid for message

Mon Jun 27 05:24:19 BST 2011

1) IMO imaplib2 should be updated to handle the case where Gmail is sending a new CAPABILITIES response after login, that corresponds to the particular imap server the user's Gmail account is hosted on.  Because Brandon is right, IMPA4rev1 protocol says the client must be able to take any response at any time. It's strange to see a second CAPABILITIES response a moment later, with different capabilities, but it's legit, and should be easy to support.

The same kind of bug happened recently with an off-topic (but still totally legitimate) response from the IMAP server coming in the middle of a message list operation.  imaplib2 incorrectly interpreted this off-topic response as the end of the message list and stopped processing the list right there (but should've continued).  So imaplib2 threw an exception, and caused offlineimap to lose all the rest of the messages in the list !!  Very bad critical data loss bug.

2)  If Google's IMAP implementation is not returning the UID with the APPEND command's OK response, then it's not implementing RFC 4315 ("IMAP - UIDPLUS Extension," December 2005).

Brandon, I feel Google should implement UIDPLUS according to the RFC4315 on the Gmail servers that support UIDPLUS. Then, offlineimap will be able to communicate with Gmail and get expected UID responses while doing APPENDs, etc.

http://tools.ietf.org/html/rfc4315

RFC 4315 Abstract:

"The UIDPLUS extension of the Internet Message Access Protocol (IMAP) provides a set of features intended to reduce the amount of time and resources used by some client operations."

RFC 4315 Section 1:

"...this document recommends new status response codes in IMAP that SHOULD be returned by all server implementations, regardless of whether or not the UIDPLUS extension is implemented. The added facilities of the features in UIDPLUS are optimizations; clients can provide equivalent functionality, albeit less efficiently, by using facilities in the base protocol."

RFC 4315 Section 3:

"In this example, A003 and A004 demonstrate successful appending and copying to a mailbox that returns the UIDs assigned to the messages."

"Example: 

C: A003 APPEND saved-messages (\Seen) {297} 

C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 

C: From: Fred Foobar <foobar at example.com> 

C: Subject: afternoon meeting 

C: To: mooch at example.com 

C: Message-Id: <B27397-0100000 at example.com> 

C: MIME-Version: 1.0 

C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 

C: 

C: Hello Joe, do you think we can meet at 3:30 tomorrow? 

C: 

S: A003 OK [APPENDUID 38505 3955] APPEND completed"

Best,
Chris Coleman

________________________________
From: Brandon Long <blong at google.com>
To: offlineimap-project at lists.alioth.debian.org
Cc: Sebastian at sspaeth.de; Marc MERLIN <merlin at google.com>
Sent: Saturday, June 25, 2011 6:06 PM
Subject: Re: Gmail to "gmail apps" sync: ValueError: Backend could not find uid for message

Another Googler pointed me to this thread, I think I can clear some things up.

Sebastian wrote:
> Hi,
>
> Alexander has sent me the log files and I analyzed them (the 'backend
> could not assign uid' problem). I am a bit puzzled and at the moment, I
> blame it entirely on the gmail imap implementation:
>
> Alexander has 2 problems:
>
> 1) His gmail server is not advertising the UIDPLUS extension all the
> time which directly allows to get the UID of an APPENDed message. His
> gmail imap capabilities reply looked like this:
>
> 'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA XLIST CHILDREN XYZZY SASL-IR AUTH=XOAUTH'

This is the pre-auth CAPABILITY response.  There is a different
response after login, and we also automatically send an untagged
CAPABILITY response after login, ie:

* OK Gimap ready for requests from 71.202.122.239 k6if6834428icw.34
a CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH
a OK Thats all she wrote! k6if6834428icw.34
a login login at gmail.com password
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE
a OK login at gmail.com User authenticated (Success)

The reason for this is that we roll out new features on a per-user
basis, so we don't know before login what features are available to
the user.  When we're certain the feature is stable and doesn't need
to be rolled back, we can start advertising it pre-login.

> while mine look like this:
>
> 'IMAP4REV1', 'UIDPLUS', 'CHILDREN', 'NAMESPACE',
> 'THREAD=ORDEREDSUBJECT', 'THREAD=REFERENCES', 'SORT', 'QUOTA', 'IDLE',
> 'STARTTLS'

This isn't a Gmail IMAP response, though maybe you aren't implying it
is.  We don't currently implement any THREAD= or SORT, nor do we
advertise STARTTLS because we only provide TLS-wrapped IMAP.

> both are from imap.gmail.com, so their servers seem very different under
> whatever mysterious conditions.
>
> 2)
> Without UIDPLUS, we munge our mail header to insert a custom header
> like this: X-OfflineIMAP: 3132605382-7478425151
> and APPEND the message.
>
> (despite not announcing UIDPLUS, gmail DOES return the new UID ('APPENDUID
> 648448589') which we could have used in the first place, so that is
> definitely a gmail bug on their side.)
>
> We then try to search gmail for the newly appended message:
> UID SEARCH HEADER X-OfflineIMAP "3132605382-7478425151"
>
> which leads to "OK SEARCH completed (Success)" returning an empty set of
> messages.

Hmm, I don't have access to the source depot at the moment to see when
we added support for substring header search, but I think it was last
year, so this should definitely work.  I don't recommend it, however,
as it requires us to load the headers for every message in the folder
in order to find the matching messages.

> The message we just uploaded was not found, although it must be there
> (we just uploaded it), so we fail to identify the new UID.  Which leads
> to the exception as we cannot return the UID of the message in question.

This probably depends on the exact order of operations.  Ie, if you do
a SELECT, APPEND, UID SEARCH, I'm not sure if it should be visible,
you may need to do a NOOP or CHECK to make sure that the new message
is visible.  I guess we could instead generate an untagged EXISTS
response after the APPEND to tell the client about the new message,
but we currently don't do this, nor do we currently update the folder
state when appending to a selected folder.

From my reading of RFC 3501 5.2 & 5.5, I'm still not sure if this is
invalid or not.  Its possible we should just update the mailbox state
after the APPEND and issue the relevant EXISTS response.  Certainly
easy enough for us to make that change, though it will definitely make
APPENDs more expensive.

> So 2 things going on: 1) Gmail fails to advertize UIDPLUS (in some cases)
> although they definitely support it, which is bad.
> 2) Gmail fails to find the message header in a message we have just
> uploaded. This is doubly bad on the gmail side.
>
> I know that gmail is popular and often unavoidable. But it is not proper
> IMAP :-(. The one workaround that we could do is to hardcode UIDPLUS
> support in the gmail case, since we know that gmail supports it (but who
> knows if they really support it in all cases?). I would be rather
> hesitant to do that hard coding.

I don't believe that either of these things is invalid.  We are
automatically issuing the new CAPABILITY response after login, and
clients are required to accept any response at any time, so we handle
#1 just fine.  #2 may be an edge case in the protocol, or maybe its
not and we should fix it, but I can't tell which from the RFC.

Brandon
-- 
 Brandon Long <blong at google.com>
 Staff Engineer
 Gmail Delivery TLM
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110626/1d89564d/attachment-0003.html>