PARTIALLY REMOVING MAXAGE (was: [PATCH v4] make maxage use UIDs to avoid timezone issues)

[Nicolas Sebrecht]

 In-Reply-To: <20150327063801.GB14329 at perseid>

[ I'm breaking the thread and changing the title because I'd like to
  catch attention of the subscribed members whose are away from the
  discussion while being concerned of a removal. ]

On Fri, Mar 27, 2015 at 02:38:02AM -0400, Janna Martl wrote:

> There is another caveat though (*sigh*): suppose remote is empty, and
> local has messages with UID's L_1 < ... < L_n. Then they get copied to
> remote in a random order, so, if R_1 < ... < R_n are the remote UID's in
> order, then maybe R5 is the message that was copied from L22. This is
> bad: if min_uid = L_k, and this corresponds to R_l, the local
> messagelist [L_k, ...] has nothing to do with the remote messagelist
> [R_l, ...], even after correcting for UID mapping.

Right.

> I figured that this problem would go away if you make sure the copylist
> is in order, but (1) this probably isn't the nicest solution;

It is the nicest solution. At least, in respect to how OfflineIMAP works
today.

And today, I can't find a sightly different approach that might fit.
The UID thing is what IMAP allows us to work with to uniquely identify a
mail remotely.

>                                                               (2) I
> think I implemented it kind of awkwardly;

*joker*

>                                           (3) it doesn't even always
> work -- even after sorting, I still found a pair of local messages whose
> remote counterparts weren't in the same order (?).

And here I continue about my point above.

I'm not surprised. As already discussed before, IMAP servers are
supposed to assign ascending UIDs to upcoming mails. Keeping the order
is not a pre-requisite nor is it defined in the RFCs. So, we would be
wrong to assume it's a valid design.

Since the order of UIDs won't match the order of the sending, we are
wrong to try to apply the min(UID) logic to the IMAP/IMAP and IMAP/Gmail
case. I didn't know that. Actually, I was expecting UIDs to be assigned
in a more predictive way.

IOW, it appears that:

1. Strictly relying on internal date is not realiable.

2. Strictly relying on UID ranges is not reliable.

3. Relying on both internal date and UID ranges as given by the remote
   servers, whatever the logic we apply and how we mix both information, is
   simply not reliable.

4. IMAP protocol does not provide anything else that could help for the
   purpose.

As a consequence to all of these, it appears that applying maxage in the
IMAP/IMAP or IMAP/Gmail cases is WRONG. We are far beyond implementation
issues. It's BROKEN BY DESIGN. We should NOT have supported this thing at
all in the first place.


Unless someone PROVES me that I'm wrong, I'm in favour to forbid the
maxage feature outside the case of a local Maildir.

Frankly, I'd rather remove this feature as soon as possible rather than
waiting for someone to prove I'm mistaken and implement it. If this
occurs, I'd be glad to re-introduce it with a reliable design and a
fixed implementation.


Ok, I'm a bit lying around.

If we wanted to get it right, I guess it could still be possible. We'd
have to keep the dates and times of each synced mail in our local cache.
Then, we'd have to build the min(UID) for each side regarding to the
cached dates and times values we stored locally. IOW, apply the current
Maildir/IMAP way of working to the IMAP/Gmail and IMAP/IMAP cases.

With the above premise in mind of how this MIGHT be proved reliable, my
guess is that this would require to:

- augment the data stored in the cache;
- update its format with backward compatibility issues in mind;
- make double sides computations to determine the correct ranges of
  matching/relevant UIDs to sync;
- probably more edge cases handling.

This might even require caching medadata of the WHOLE mailboxes
(folders) for both sides at a first invokation to allow working with the
cached values rather than relying on what IMAP servers provide on
successive requests.

This is out of the scope of fixing maxage and the work you're currently
doing. Hence, the removal.

-- 
Nicolas Sebrecht