Gmail to "gmail apps" sync: ValueError: Backend could not find uid for message

[Sebastian Spaeth]

Hi Brandon,

@Piers, please read to see if this should be fixed in imaplib2 or
offlineimap. The issue is that Gmail sends different CAPABILITIES pre-
and post-auth and imaplib2 queries them only pre-auth (automatically).

On Sat, 25 Jun 2011 15:06:25 -0700, Brandon Long <blong at google.com> wrote:
> Another Googler pointed me to this thread, I think I can clear some things up.
> > 'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA XLIST CHILDREN XYZZY SASL-IR AUTH=XOAUTH'
> 
> This is the pre-auth CAPABILITY response.  There is a different
> response after login, and we also automatically send an untagged
> CAPABILITY response after login, ie:

Thanks, that is good to know, and clears things up a bit. Our imaplib
does query CAPABILITIES explicitely after connecting, but not after
authentication. We'll see if we can fix this either in the underlying
imaplib or if offlineimap should be doing that itself after
authentication.

> The reason for this is that we roll out new features on a per-user
> basis, so we don't know before login what features are available to
> the user.  When we're certain the feature is stable and doesn't need
> to be rolled back, we can start advertising it pre-login.

Right, gotcha.

> > 'IMAP4REV1', 'UIDPLUS', 'CHILDREN', 'NAMESPACE',
> > 'THREAD=ORDEREDSUBJECT', 'THREAD=REFERENCES', 'SORT', 'QUOTA', 'IDLE',
> > 'STARTTLS'

> This isn't a Gmail IMAP response, though maybe you aren't implying it
> is.  We don't currently implement any THREAD= or SORT, nor do we
> advertise STARTTLS because we only provide TLS-wrapped IMAP.

No, this isn't an Gmailr response, I was merely comparing my ISPs IMAP
response with one from Gmail, would have to reread the original email to
see context.
 
> > 2)
> > Without UIDPLUS, we munge our mail header to insert a custom header
> > like this: X-OfflineIMAP: 3132605382-7478425151
> > and APPEND the message.
> > ...
> > We then try to search gmail for the newly appended message:
> > UID SEARCH HEADER X-OfflineIMAP "3132605382-7478425151"
> >
> > which leads to "OK SEARCH completed (Success)" returning an empty set of
> > messages.
> 
> Hmm, I don't have access to the source depot at the moment to see when
> we added support for substring header search, but I think it was last
> year, so this should definitely work.  I don't recommend it, however,
> as it requires us to load the headers for every message in the folder
> in order to find the matching messages.

Yes, that is definitely not the recommended way, and is what offlineimap
falls back to to get the UID of APPENDed messages. It does use
UIDPLUS/APPENDUID whenever possible and we have worked around the
"missing" UIDPLUS capability reply in offlineimap already. I am glad
that we'll be able to remove that workaround, now that we know that
GMail isn't as buggy as I thought :-).

The search was definitely failing though, and it was not a substring
header search but a search for the full header (which should have
worked), so *something* was fishy back then.

> > The message we just uploaded was not found, although it must be there
> > (we just uploaded it), so we fail to identify the new UID.  Which leads
> > to the exception as we cannot return the UID of the message in question.
> 
> This probably depends on the exact order of operations.  Ie, if you do
> a SELECT, APPEND, UID SEARCH, I'm not sure if it should be visible,
> you may need to do a NOOP or CHECK to make sure that the new message
> is visible.

Yes, we were already doing a 'CHECK' after APPENDING, so that can't have
been the issue.

> From my reading of RFC 3501 5.2 & 5.5, I'm still not sure if this is
> invalid or not.  Its possible we should just update the mailbox state
> after the APPEND and issue the relevant EXISTS response.  Certainly
> easy enough for us to make that change, though it will definitely make
> APPENDs more expensive.

If searches work after APPEND and CHECK that should be fine and all we
need.  (As UIDPLUS works now, we don't even need the search at all.)

> > So 2 things going on: 1) Gmail fails to advertize UIDPLUS (in some cases)
> > although they definitely support it, which is bad.
> > 2) Gmail fails to find the message header in a message we have just
> > uploaded. This is doubly bad on the gmail side.
> >
> > I know that gmail is popular and often unavoidable. But it is not proper
> > IMAP :-(. The one workaround that we could do is to hardcode UIDPLUS
> > support in the gmail case, since we know that gmail supports it (but who
> > knows if they really support it in all cases?). I would be rather
> > hesitant to do that hard coding.
> 
> I don't believe that either of these things is invalid.  We are
> automatically issuing the new CAPABILITY response after login, and
> clients are required to accept any response at any time, so we handle
> #1 just fine.  #2 may be an edge case in the protocol, or maybe its
> not and we should fix it, but I can't tell which from the RFC.

1) CAPABILITY after login is fine, and we'll be using that.
2) APPEND/CHECK/SEARCH *should* be working and it hadn't, according to
the logs. We don't need it anymore, but Gmail shouldn't be too broken :)

Thanks for the feedback, much appreciated. We'll be making sure to
contact you next time we believe Gmail to be buggy.

@Piers, does imaplib2 want to query capabilities post-auth or should be
be doin it from offlineimap after login?

Sebastian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110627/bf4bc12d/attachment-0001.sig>