Gmail to "gmail apps" sync: ValueError: Backend could not find uid for message

Tue Jun 28 08:20:03 UTC 2011

On Mon, 27 Jun 2011 20:05:54 -0700 (PDT), chris coleman wrote:
> @Sebastian : Wouldn't a better way to solve this bug (caused by failing to process legitimate, untagged messages from the IMAP server), and solve it (any any other unforeseen bugs) for any future scenario:  for offlineimap to delay querying imaplib2 for UIDPLUS capability until JUST BEFORE the various places in the offlineimap code where a UID might be received as part of a response.  Then, depending on the response, the code would branch between different code paths: the UID case and the no-UID case.
> 

> It sounds like this solution is possible without any upgrade to
> imaplib2, which updates its cached CAPABILITIES array every time it
> receives new capabilities from the imap server, from what you've said.

Nah, it isn't possible, and it doesn't cache updated CAPABILITIES every
time it receveives them (well, it stores it just as it stores any
untagged response, so we could query for that response after each
operation, to be precise). It currently excplicitly asks once right
after connecting. We need to explicitly query capabilities every time we
want to get them.

> The bigger point is: it seems this bug was caused by offlineimap not fully complying to the IMAP4 client spec according to the requirement to accept and process any response from the server at any time.

AFAIK, we need to be prepared to receive a server response at every
point in time. I am not sure we need to be prepared to dynamically adapt
to changing CAPABILITIES over time. What if the server says it accepts
UTF8 and then claims to not support it anymore while we are in a data
transmission? IMAP is ripe with enough edge cases already. I think it
makes sense to accept a different CAPABILITY set post-login so that it
can be per-user. But parsing every server response to see if it might
contain a CAPABILITY string sounds like it could kill off performance
quite severely.

Having to query the server capabilities 20k times (forcing a
server/response roundtrip), just because we want to synchronize 20k
emails doesn't sound very appealing to me (and we'd probably even find
ourselves locked out of eg Gmail quite quickly) in terms of
performance, when it is pretty reasonable to assume that capabilities
won't change over the life-time of an IMAP session. Also again, do NOTE
that we correctly detect an APPENDUID server reply already now, even if
we are not aware of server capabilities, so we *are already dynamically*
taking advantage of the feature on the offlineimap side.

But then, it is not me to decide all this anyway. This is all done by
imaplib2, which we are merely a user off. Perhaps one of the biggest and
most vocal users, but a user none the less. Piers is the person to
decide on the strategies and I've copied this mail to the imaplib2
mailing list (which one needs to be subscribed to, to send emails).

Chris, if you feel strongly about this, I recommend to negotiate with
Piers directly :).

> Taking a step back, why shouldn't offlineimap/imaplib2 follow the IMAP
> spec completely?

One reason is because the IMAP specs are ambiguous at best, so it is
hard to follow them precisely and often the best way to follow them is
to see how servers interpret/implement them. Second reason is that
various servers complying to the specs with varying success, so even if
we clung 100% correctly to the specs, things wouldn't work out
correctly.

The specs eg don't even specify the character encoding that one should
be using for folder names etc, they just recommend a weird one (which
hopefully dies very soon, being replaced with UTF8).

Again, assume the CAPABILITIES suddenly start saying they don't support
STARTTLS when we logged in via STARTTLS. Should we immediately drop the
connection? (or more realistically, my UTF8 example from before) I don't
think this is something I want to get into :).

> Obviously it's not an easy client spec to implement 100%, because the
> client must behave more like a hybrid client/server.

No, we are definitely an IMAP client and behave as such all the time. We
just happen to use both directions of transferring messages that are
offered by the IMAP API. *I* don't want to get into the IMAP server
coding business. :-)

> A simple dumb client (such as POP3) can be very simple, and way easier
> to implement, because it just makes requests, and receives only
> deterministic answers stemming directly from those requests.

Actually, pop3 is nowadays often used in ways that IMAP was supposed to
do, ie leave mails on server, download selectively, move mails between
server folders etc.

> It's a much greater challenge to make offlineimap as utterly bulletproof as fetchmail/pop3.

Definitely yes. :) (although I am sure the authors of fetchmail will
disagree with me)

Sebastian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/offlineimap-project/attachments/20110628/32285354/attachment.pgp>