Gmail to "gmail apps" sync: ValueError: Backend could not find uid for message

[Brandon Long]

On Sun, Jun 26, 2011 at 9:24 PM, chris coleman <christocoleman at yahoo.com> wrote:
> 1) IMO imaplib2 should be updated to handle the case where Gmail is sending
> a new CAPABILITIES response after login, that corresponds to the particular
> imap server the user's Gmail account is hosted on.  Because Brandon is
> right, IMPA4rev1 protocol says the client must be able to take any response
> at any time. It's strange to see a second CAPABILITIES response a moment
> later, with different capabilities, but it's legit, and should be easy to
> support.
>
> The same kind of bug happened recently with an off-topic (but still totally
> legitimate) response from the IMAP server coming in the middle of a message
> list operation.  imaplib2 incorrectly interpreted this off-topic response as
> the end of the message list and stopped processing the list right there (but
> should've continued).  So imaplib2 threw an exception, and caused
> offlineimap to lose all the rest of the messages in the list !!  Very bad
> critical data loss bug.
>
> 2)  If Google's IMAP implementation is not returning the UID with the APPEND
> command's OK response, then it's not implementing RFC 4315 ("IMAP - UIDPLUS
> Extension," December 2005).
> Brandon, I feel Google should implement UIDPLUS according to the RFC4315 on
> the Gmail servers that support UIDPLUS. Then, offlineimap will be able to
> communicate with Gmail and get expected UID responses while doing APPENDs,
> etc.

We do implement this and we do return the UID on APPEND.  If someone
has an example where we don't, I'll be happy to investigate.  The rest
of my discussion was actually on whether or not you can do a UID
SEARCH immediately after an APPEND if you've already selected the
folder, which is the fallback if the server doesn't support UIDPLUS,
but Gmail's server definitely support UIDPLUS.

Brandon

> http://tools.ietf.org/html/rfc4315
> RFC 4315 Abstract:
> "The UIDPLUS extension of the Internet Message Access Protocol (IMAP)
> provides a set of features intended to reduce the amount of time and
> resources used by some client operations."
> RFC 4315 Section 1:
> "...this document recommends new status response codes in IMAP that SHOULD
> be returned by all server implementations, regardless of whether or not the
> UIDPLUS extension is implemented. The added facilities of the features in
> UIDPLUS are optimizations; clients can provide equivalent functionality,
> albeit less efficiently, by using facilities in the base protocol."
> RFC 4315 Section 3:
> "In this example, A003 and A004 demonstrate successful appending and copying
> to a mailbox that returns the UIDs assigned to the messages."
> "Example:
> C: A003 APPEND saved-messages (\Seen) {297}
> C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
> C: From: Fred Foobar <foobar at example.com>
> C: Subject: afternoon meeting
> C: To: mooch at example.com
> C: Message-Id: <B27397-0100000 at example.com>
> C: MIME-Version: 1.0
> C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
> C:
> C: Hello Joe, do you think we can meet at 3:30 tomorrow?
> C:
> S: A003 OK [APPENDUID 38505 3955] APPEND completed"
>
> Best,
> Chris Coleman
>
>
> ________________________________
> From: Brandon Long <blong at google.com>
> To: offlineimap-project at lists.alioth.debian.org
> Cc: Sebastian at sspaeth.de; Marc MERLIN <merlin at google.com>
> Sent: Saturday, June 25, 2011 6:06 PM
> Subject: Re: Gmail to "gmail apps" sync: ValueError: Backend could not find
> uid for message
>
> Another Googler pointed me to this thread, I think I can clear some things
> up.
>
> Sebastian wrote:
>> Hi,
>>
>> Alexander has sent me the log files and I analyzed them (the 'backend
>> could not assign uid' problem). I am a bit puzzled and at the moment, I
>> blame it entirely on the gmail imap implementation:
>>
>> Alexander has 2 problems:
>>
>> 1) His gmail server is not advertising the UIDPLUS extension all the
>> time which directly allows to get the UID of an APPENDed message. His
>> gmail imap capabilities reply looked like this:
>>
>> 'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA XLIST CHILDREN XYZZY SASL-IR
>> AUTH=XOAUTH'
>
> This is the pre-auth CAPABILITY response.  There is a different
> response after login, and we also automatically send an untagged
> CAPABILITY response after login, ie:
>
> * OK Gimap ready for requests from 71.202.122.239 k6if6834428icw.34
> a CAPABILITY
> * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
> X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH
> a OK Thats all she wrote! k6if6834428icw.34
> a login login at gmail.com password
> * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
> X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE
> a OK login at gmail.com User authenticated (Success)
>
> The reason for this is that we roll out new features on a per-user
> basis, so we don't know before login what features are available to
> the user.  When we're certain the feature is stable and doesn't need
> to be rolled back, we can start advertising it pre-login.
>
>> while mine look like this:
>>
>> 'IMAP4REV1', 'UIDPLUS', 'CHILDREN', 'NAMESPACE',
>> 'THREAD=ORDEREDSUBJECT', 'THREAD=REFERENCES', 'SORT', 'QUOTA', 'IDLE',
>> 'STARTTLS'
>
> This isn't a Gmail IMAP response, though maybe you aren't implying it
> is.  We don't currently implement any THREAD= or SORT, nor do we
> advertise STARTTLS because we only provide TLS-wrapped IMAP.
>
>> both are from imap.gmail.com, so their servers seem very different under
>> whatever mysterious conditions.
>>
>> 2)
>> Without UIDPLUS, we munge our mail header to insert a custom header
>> like this: X-OfflineIMAP: 3132605382-7478425151
>> and APPEND the message.
>>
>> (despite not announcing UIDPLUS, gmail DOES return the new UID ('APPENDUID
>> 648448589') which we could have used in the first place, so that is
>> definitely a gmail bug on their side.)
>>
>> We then try to search gmail for the newly appended message:
>> UID SEARCH HEADER X-OfflineIMAP "3132605382-7478425151"
>>
>> which leads to "OK SEARCH completed (Success)" returning an empty set of
>> messages.
>
> Hmm, I don't have access to the source depot at the moment to see when
> we added support for substring header search, but I think it was last
> year, so this should definitely work.  I don't recommend it, however,
> as it requires us to load the headers for every message in the folder
> in order to find the matching messages.
>
>> The message we just uploaded was not found, although it must be there
>> (we just uploaded it), so we fail to identify the new UID.  Which leads
>> to the exception as we cannot return the UID of the message in question.
>
> This probably depends on the exact order of operations.  Ie, if you do
> a SELECT, APPEND, UID SEARCH, I'm not sure if it should be visible,
> you may need to do a NOOP or CHECK to make sure that the new message
> is visible.  I guess we could instead generate an untagged EXISTS
> response after the APPEND to tell the client about the new message,
> but we currently don't do this, nor do we currently update the folder
> state when appending to a selected folder.
>
> From my reading of RFC 3501 5.2 & 5.5, I'm still not sure if this is
> invalid or not.  Its possible we should just update the mailbox state
> after the APPEND and issue the relevant EXISTS response.  Certainly
> easy enough for us to make that change, though it will definitely make
> APPENDs more expensive.
>
>> So 2 things going on: 1) Gmail fails to advertize UIDPLUS (in some cases)
>> although they definitely support it, which is bad.
>> 2) Gmail fails to find the message header in a message we have just
>> uploaded. This is doubly bad on the gmail side.
>>
>> I know that gmail is popular and often unavoidable. But it is not proper
>> IMAP :-(. The one workaround that we could do is to hardcode UIDPLUS
>> support in the gmail case, since we know that gmail supports it (but who
>> knows if they really support it in all cases?). I would be rather
>> hesitant to do that hard coding.
>
> I don't believe that either of these things is invalid.  We are
> automatically issuing the new CAPABILITY response after login, and
> clients are required to accept any response at any time, so we handle
> #1 just fine.  #2 may be an edge case in the protocol, or maybe its
> not and we should fix it, but I can't tell which from the RFC.
>
> Brandon
> --
>  Brandon Long <blong at google.com>
>  Staff Engineer
>  Gmail Delivery TLM
>
>



-- 
 Brandon Long <blong at google.com>
 Staff Engineer
 Gmail Delivery TLM