<DKIM> Re:OfflineIMAP-project Digest, Vol 67, Issue 21

[Nicolas Sebrecht]

On Tue, Apr 21, 2015 at 07:33:43PM -0700, chris coleman wrote:
> 
>     Yes Sebastian, this is exactly what I meant.
>     The UID is not *really* static , this is admitted in the IMAP RFC 2060,
>     UIDVALIDITY must be saved and compared between IMAP sessions, when it
>     changes, our entire local cache is invalid and must be burned and
>     re-downloaded.

RFC 2060 is obsolete, take the RFC 3501.

>     This is a huge drawback to the original design by John Goerzen, in my
>     humble opinion. Myself and many other users of OLI have had to slowly
>     redownload entire mailboxes over slow 1-2 mbps links to remote
>     servers...

Does it happen often? When was the last time a UIDVALIDITY changed for
you?

>     The only truly static, universally unique ID is the "message-ID"
>     header.

Not true as explained by Sabastian. But I guess your point was more
about mixing message-ID and the hash of the headers with UID.

Notice that with more information involved, we have to handle more
complex cases:

  What changed ?
 +-----+-------------+------------+--------------------+
 | UID | HEADER HASH | MESSAGE-ID | OfflineIMAP action |
 +-----+-------------+------------+--------------------+
 | yes | no          | no         | affect new UID     |
 +-----+-------------+------------+--------------------+
 | yes | yes         | no         | ?                  |
 +-----+-------------+------------+--------------------+
 | yes | yes         | yes        | re-download        |
 +-----+-------------+------------+--------------------+
 | yes | no          | yes        | <invalid>          |
 +-----+-------------+------------+--------------------+
 | no  | yes         | yes        | ?                  |
 +-----+-------------+------------+--------------------+
 | no  | no          | yes        | <invalid>          |
 +-----+-------------+------------+--------------------+
 | no  | yes         | no         | ?                  |
 +-----+-------------+------------+--------------------+

Also, remember that any change can happen on EITHER side.

I guess lines 1&3 are the easy cases we can think of. It could be
possible to decide to re-download whatever information changed.

>     I recommend we cache the Message-ID in our SQLite database, plus UID,
>     plus sha256 hash of the message, so that "remote move from Inbox to
>     Archive-2015-Q2" will replace "download from Inbox, then re-upload to
>     Archive-2015-Q2", saving often 1-20MB when that message has attached
>     documents.
>     This is what I meant by improving the algo.

So, in fact you're not talking about the algo but how to handle
UIDVALIDITY changes. I agree there are possible ways of improvement in
this area. One could be to do a 2 steps merge:

- fetch the UID and headers;
- fetch the mails when either header hash or Message-ID don't match;
  re-assign new UID when both match.

OfflineIMAP would ONLY work with additional information on UIDVALIDITY
changes. There's no need to complicate the day-to-day algo for just
this.

Also, changing the day-to-day algo would NOT be as reliable as we can
guess. Not to talk about speed issues because it's a lot more cases to
handle (think about any local change).

Another issue is that IMAP does NOT provide any way to get the hash of
a header/mail. This can only be computed locally with a FETCH command.
So, I'm against maintaining hashes of headers and Message-ID in the
cache for the merge algo.

But computing mixed information might make sense on UIDVALIDITY changes
for big mailboxes on slow links, yes.

The other area where there are ways of improvement is for mails changed
in a local _maildir_. In this case, maintaining additional information
might make sense to detect local copies and moves. This would enable to
request the IMAP server with COPY and UID MOVE commands (as per RFC
6851).

-- 
Nicolas Sebrecht