Gmail to "gmail apps" sync: ValueError: Backend could not find uid for message

[Brandon Long]

On Tue, Jun 28, 2011 at 1:20 AM, Sebastian Spaeth <Sebastian at sspaeth.de> wrote:
> On Mon, 27 Jun 2011 20:05:54 -0700 (PDT), chris coleman wrote:
>> @Sebastian : Wouldn't a better way to solve this bug (caused by failing to process legitimate, untagged messages from the IMAP server), and solve it (any any other unforeseen bugs) for any future scenario:  for offlineimap to delay querying imaplib2 for UIDPLUS capability until JUST BEFORE the various places in the offlineimap code where a UID might be received as part of a response.  Then, depending on the response, the code would branch between different code paths: the UID case and the no-UID case.
>>
>
>> It sounds like this solution is possible without any upgrade to
>> imaplib2, which updates its cached CAPABILITIES array every time it
>> receives new capabilities from the imap server, from what you've said.
>
> Nah, it isn't possible, and it doesn't cache updated CAPABILITIES every
> time it receveives them (well, it stores it just as it stores any
> untagged response, so we could query for that response after each
> operation, to be precise). It currently excplicitly asks once right
> after connecting. We need to explicitly query capabilities every time we
> want to get them.
>
>> The bigger point is: it seems this bug was caused by offlineimap not fully complying to the IMAP4 client spec according to the requirement to accept and process any response from the server at any time.
>
> AFAIK, we need to be prepared to receive a server response at every
> point in time. I am not sure we need to be prepared to dynamically adapt
> to changing CAPABILITIES over time. What if the server says it accepts
> UTF8 and then claims to not support it anymore while we are in a data
> transmission? IMAP is ripe with enough edge cases already. I think it
> makes sense to accept a different CAPABILITY set post-login so that it
> can be per-user. But parsing every server response to see if it might
> contain a CAPABILITY string sounds like it could kill off performance
> quite severely.

I wouldn't think it would be that bad, but its probably a different
model than either imaplib or imaplib2 implement.  You'd want to
implement a callback based model where every untagged response would
be looked up in a table and the callback invoked if one is present, or
something like that.

For instance, are you watching for EXISTS & EXPUNGE responses on any
command?  You could probably ignore them and just wait for the next
cycle to handle any changes, probably easier that way.

> Having to query the server capabilities 20k times (forcing a
> server/response roundtrip), just because we want to synchronize 20k
> emails doesn't sound very appealing to me (and we'd probably even find
> ourselves locked out of eg Gmail quite quickly) in terms of
> performance, when it is pretty reasonable to assume that capabilities
> won't change over the life-time of an IMAP session. Also again, do NOTE
> that we correctly detect an APPENDUID server reply already now, even if
> we are not aware of server capabilities, so we *are already dynamically*
> taking advantage of the feature on the offlineimap side.

Yeah, that's definitely overkill.

> One reason is because the IMAP specs are ambiguous at best, so it is
> hard to follow them precisely and often the best way to follow them is
> to see how servers interpret/implement them. Second reason is that
> various servers complying to the specs with varying success, so even if
> we clung 100% correctly to the specs, things wouldn't work out
> correctly.
>
> The specs eg don't even specify the character encoding that one should
> be using for folder names etc, they just recommend a weird one (which
> hopefully dies very soon, being replaced with UTF8).

Weird, I'd never noticed that it was specified as "by convention"
instead of an actual requirement.

> Again, assume the CAPABILITIES suddenly start saying they don't support
> STARTTLS when we logged in via STARTTLS. Should we immediately drop the
> connection? (or more realistically, my UTF8 example from before) I don't
> think this is something I want to get into :).

Actually, from the spec you're supposed to query for CAPABILITIES
again after STARTTLS, and STARTTLS is supposed to go away because you
shouldn't call it again.

In general, though, yes, they shouldn't go away.

Brandon
-- 
 Brandon Long <blong at google.com>
 Staff Engineer
 Gmail Delivery TLM