Gmail to "gmail apps" sync: ValueError: Backend could not find uid for message

[Brandon Long]

Another Googler pointed me to this thread, I think I can clear some things up.

Sebastian wrote:
> Hi,
>
> Alexander has sent me the log files and I analyzed them (the 'backend
> could not assign uid' problem). I am a bit puzzled and at the moment, I
> blame it entirely on the gmail imap implementation:
>
> Alexander has 2 problems:
>
> 1) His gmail server is not advertising the UIDPLUS extension all the
> time which directly allows to get the UID of an APPENDed message. His
> gmail imap capabilities reply looked like this:
>
> 'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA XLIST CHILDREN XYZZY SASL-IR AUTH=XOAUTH'

This is the pre-auth CAPABILITY response.  There is a different
response after login, and we also automatically send an untagged
CAPABILITY response after login, ie:

* OK Gimap ready for requests from 71.202.122.239 k6if6834428icw.34
a CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH
a OK Thats all she wrote! k6if6834428icw.34
a login login at gmail.com password
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE
a OK login at gmail.com User authenticated (Success)

The reason for this is that we roll out new features on a per-user
basis, so we don't know before login what features are available to
the user.  When we're certain the feature is stable and doesn't need
to be rolled back, we can start advertising it pre-login.

> while mine look like this:
>
> 'IMAP4REV1', 'UIDPLUS', 'CHILDREN', 'NAMESPACE',
> 'THREAD=ORDEREDSUBJECT', 'THREAD=REFERENCES', 'SORT', 'QUOTA', 'IDLE',
> 'STARTTLS'

This isn't a Gmail IMAP response, though maybe you aren't implying it
is.  We don't currently implement any THREAD= or SORT, nor do we
advertise STARTTLS because we only provide TLS-wrapped IMAP.

> both are from imap.gmail.com, so their servers seem very different under
> whatever mysterious conditions.
>
> 2)
> Without UIDPLUS, we munge our mail header to insert a custom header
> like this: X-OfflineIMAP: 3132605382-7478425151
> and APPEND the message.
>
> (despite not announcing UIDPLUS, gmail DOES return the new UID ('APPENDUID
> 648448589') which we could have used in the first place, so that is
> definitely a gmail bug on their side.)
>
> We then try to search gmail for the newly appended message:
> UID SEARCH HEADER X-OfflineIMAP "3132605382-7478425151"
>
> which leads to "OK SEARCH completed (Success)" returning an empty set of
> messages.

Hmm, I don't have access to the source depot at the moment to see when
we added support for substring header search, but I think it was last
year, so this should definitely work.  I don't recommend it, however,
as it requires us to load the headers for every message in the folder
in order to find the matching messages.

> The message we just uploaded was not found, although it must be there
> (we just uploaded it), so we fail to identify the new UID.  Which leads
> to the exception as we cannot return the UID of the message in question.

This probably depends on the exact order of operations.  Ie, if you do
a SELECT, APPEND, UID SEARCH, I'm not sure if it should be visible,
you may need to do a NOOP or CHECK to make sure that the new message
is visible.  I guess we could instead generate an untagged EXISTS
response after the APPEND to tell the client about the new message,
but we currently don't do this, nor do we currently update the folder
state when appending to a selected folder.

>From my reading of RFC 3501 5.2 & 5.5, I'm still not sure if this is
invalid or not.  Its possible we should just update the mailbox state
after the APPEND and issue the relevant EXISTS response.  Certainly
easy enough for us to make that change, though it will definitely make
APPENDs more expensive.

> So 2 things going on: 1) Gmail fails to advertize UIDPLUS (in some cases)
> although they definitely support it, which is bad.
> 2) Gmail fails to find the message header in a message we have just
> uploaded. This is doubly bad on the gmail side.
>
> I know that gmail is popular and often unavoidable. But it is not proper
> IMAP :-(. The one workaround that we could do is to hardcode UIDPLUS
> support in the gmail case, since we know that gmail supports it (but who
> knows if they really support it in all cases?). I would be rather
> hesitant to do that hard coding.

I don't believe that either of these things is invalid.  We are
automatically issuing the new CAPABILITY response after login, and
clients are required to accept any response at any time, so we handle
#1 just fine.  #2 may be an edge case in the protocol, or maybe its
not and we should fix it, but I can't tell which from the RFC.

Brandon
-- 
 Brandon Long <blong at google.com>
 Staff Engineer
 Gmail Delivery TLM