Gmail to "gmail apps" sync: ValueError: Backend could not find uid for message

[chris coleman]

>On Tue, Jun 28, 2011 at 1:20 AM, Sebastian Spaeth wrote:
>> On Mon, 27 Jun
 2011 20:05:54 -0700 (PDT), chris coleman wrote:
>>> @Sebastian : Wouldn't a better way to solve this bug (caused by failing to process legitimate, untagged messages from the IMAP server), and solve it (any any other unforeseen bugs) for any future scenario:  for offlineimap to delay querying imaplib2 for UIDPLUS capability until
 JUST BEFORE the various places in the offlineimap code where a UID might be received as part of a response.  Then, depending on the response, the code would branch between different code paths: the UID case and the no-UID case.
>>>
>>
>>> It sounds like this solution is possible without any upgrade to
>>> imaplib2, which updates its cached CAPABILITIES array every time it
>>> receives new capabilities from the imap server, from what you've said.
>>
>> Nah, it isn't possible, and it doesn't cache updated CAPABILITIES every
>> time it receveives them (well, it stores it just as it stores any
>> untagged response, so we could query for that response after each
>> operation, to be precise). It currently excplicitly asks once right
>> after connecting. We need to explicitly query capabilities every time we
>> want to get
 them.
>>
>>> The bigger point is: it seems this bug was caused by offlineimap not fully complying to the IMAP4 client spec according to the requirement to accept and process any response from the server at any time.
>>
>> AFAIK, we need to be prepared to receive a server response at every
>> point in time. I am not sure we need to be prepared to dynamically adapt
>> to changing CAPABILITIES over time. What if the server says it accepts
>> UTF8 and then claims to not support it anymore while we are in a data
>> transmission? IMAP is ripe with enough edge cases already. I think it
>> makes sense to accept a different CAPABILITY set post-login so that it
>> can be per-user. But parsing every server response to see if it might
>> contain a CAPABILITY string sounds like it could kill off performance
>> quite severely.
>
>I wouldn't think
 it would be that bad, but its probably a different
>model than either imaplib or imaplib2 implement.  You'd want to
>implement a callback based model where every untagged response would
>be looked up in a table and the callback invoked if one is present, or
>something like that.
>
>For instance, are you watching for EXISTS & EXPUNGE responses on any
>command?  You could probably ignore them and just wait for the next
>cycle to handle any changes, probably easier that way.


Callback based model is one way to call it.

I was going to say, Event-driven model, or Message-loop.  

From windows/mac desktop apps.

According to protocol, you've got to respond to IMAP server responses, more or less as they come in (sometimes defer, like when in the middle of a big message or whatever).

This is why the Gmail issue happened, with us missing the UIDPLUS
 capabilities response completely - it actually violates the protocol to look ONLY at the responses that arrive when you assume that they should come in, and ignore the untagged responses.




>> Having to query the server capabilities 20k times (forcing a
>> server/response roundtrip), just because we want to synchronize 20k
>> emails doesn't sound very appealing to me (and we'd probably even find
>> ourselves locked out of eg Gmail quite quickly) in terms of
>> performance, when it is pretty reasonable to assume that
 capabilities
>> won't change over the life-time of an IMAP session. Also again, do NOTE
>> that we correctly detect an APPENDUID server reply already now, even if
>> we are not aware of server capabilities, so we *are already dynamically*
>> taking advantage of the feature on the offlineimap side.
>
>Yeah, that's definitely overkill.

Right, that would be ridiculous.  

And I didn't' mean that either.  

What I meant was, we (offlineimap and imaplib2) shouldn't be working off cached copies of these capabilities values - because IMAP protocol implies that they can and DO go stale.  See example where we crashed on Gmail changed capabilities. 

The worst part is, we don't even know how many other untagged responses are getting missed.

The offlineimap code should be working off the TRUE values, which are subject to change, and should be stored/cached ONLY in imaplib2.  

For example , we should get UIDPLUS via a GetUIDPLUS() getter method only, every time the offlineimap code logic needs to check it, and never cache that value in the offlineimap data space,
 except maybe at a "current single message loop" level of granularity.



>> One reason is because the IMAP specs are ambiguous at best, so it is
>> hard to follow them precisely and often the best way to follow them is
>> to see how servers interpret/implement them. Second reason is that
>> various servers complying to the specs with varying success, so even if
>> we clung 100% correctly to the specs, things wouldn't work out
>> correctly.
>>
>> The specs eg don't even specify the character encoding that one should
>> be using for folder names etc, they just
 recommend a weird one (which
>> hopefully dies very soon, being replaced with UTF8).
>
>Weird, I'd never noticed that it was specified as "by convention"
>instead of an actual requirement.


Character sets that causes crashes/lost data/logic failures...  this has bitten us in the a** before, and will probably do it again many more times in the future, unless we get a firm grip on it once and for all...


>> Again, assume the CAPABILITIES suddenly start saying they don't support
>> STARTTLS when we logged in via STARTTLS. Should we immediately drop the
>> connection? (or more realistically, my UTF8 example from before) I don't
>> think this is something I want to get into :).
>
>Actually, from the spec you're supposed to query for CAPABILITIES
>again after STARTTLS, and STARTTLS is supposed to go away because you
>shouldn't call it again.
>
>In general, though, yes, they shouldn't go away.
>
>Brandon


Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110629/669d913f/attachment-0003.html>