maxage causes loss of local email

[Janna Martl]

On Sun, Mar 22, 2015 at 12:21:15PM +0100, Nicolas Sebrecht wrote:
> On Sun, Mar 22, 2015 at 11:25:24AM +0100, Nicolas Sebrecht wrote:
> 
> > Good catch!
> 
> Now that we talked about the L back in time case, there is another
> tricky case we won't fix it might worth to be aware of.
> 
> If message M is like L, UID unkown but newer time T, it might temporary
> put put the scan into a bad state, returning a wrong list.
> 
> 1. M is downloaded behind OfflineIMAP's back. UID unkown, time T within
>    maxage range. Sync, M get assigned a new UID = V on remote.
> 
> 2. Second sync, M is rewritten with new UID V, time T.
> 
> 3. Third sync at t3, if we are unlucky, M is taken as reference for
>    uid_min while the UID V is bigger than it should be regarding other
>    messages with time within maxage and UID < V.
>
> 4. Another sync at t4, M is no longer the UID cursor. Messages wrongly
>    excluded at t3 are now synced, back to normal. BUT messages wrongly
>    excluded at t3 AND not anymore within the current maxage range gets
>    never synced.
 
If V is the reference point in step 3, for messages to be wrongly
excluded then, they would have to be within maxage, but have UID < V.
But V only gets chosen as reference if it is the smallest UID among
messages within maxage. With Fix #1, messages might get synced when
they're too old, but messages of acceptable age are always synced.

These edge cases are amplified if someone has a pre-existing
Maildir and wants to start using offlineimap to sync it to a new
remote. Maildir.py's _scanfolder() uses os.listdir(), which gives a
list of filenames [F_1,...,F_n] with times [T_1,...,T_n] which are not
in any kind of order. As with the examples we've been discussing,
these are copied to the server and given UID's U_1, ..., U_n, where
now U_i < U_{i+1}. Restrict the local messagelist based on maxage, and
you get a seemingly random subset of U_i's. The cutoff point is the
min of these, and if Fix #1 is applied, tons of messages with UID's >
this but times earlier than this will be synced, causing the user to
think something is wrong. I suppose this should be documented
somewhere...

>The fix #1 you suggested before is the correct fix. But don't call the
>lists "old" and "new", it's confusing. Something like "strict_maxage"
>and "excluded_from_strict_maxage".

OK, I'll do this tonight or tomorrow.


-- J.M.