maxage causes loss of local email

[Nicolas Sebrecht]

On Thu, Mar 19, 2015 at 12:32:40PM +0100, Nicolas Sebrecht wrote:
> On Wed, Mar 18, 2015 at 07:01:11PM -0400, Janna Martl wrote:

> > > About what we care, since we are asking for message up to (maxage + n)
> > > with n in days, few divergences in the UIDs order regarding the "real
> > > delivery date order" should have no impact to our algorithm. When the
> > > IMAP infra consider a new mail, it is assigned a new ascending UID.
> > >
> > > I wonder if there's something else that could hurt.
> > 
> > I only noticed this because I had a few messages where the gap was 5
> > days. That does mess things up: you get situations like this
> > 
> >                               A           E
> >     (local)          |--------------|--------------->
> >                    -48              0
> >                               A  B*    C*      D
> >     (remote)  |--------------|--------------->
> >               -48            0
> > 
> > where A has UID higher than it should, and so our procedure picks C*
> > as having the smallest acceptable UID, but A doesn't get excluded because
> > UID(A) > UID(C*).
> > 
> > I really want to believe that these anomalous messages are somehow the
> > fault of my own experimentation (even though they go back further in
> > time than I've been messing around with this stuff).
> 
> I don't know. I don't get this sample because C* (why '*'?) isn't even
> there locally.

Got the '*' explanation if the patch.

I'm not much satisfied with it. Not that it's wrong but I find it to add
too much complexity in the code for an edge-case about timezones.

> Do you think you could redo some tests on a new account/folder to check
> again with the same offending mails?

If the culprit appears to be the IMAP server internals, its way of
working with UIDs and INTERNALDATE, we could do something else.

Stop me if I'm wrong but we are currently requesting another SEARCH
(maxage + Y) with a range greater than (maxage +n) messages and manually
exclude messages from this second set.

Let's revamp the strategy, again (I know, I'm a pain).

Instead, we could request the remote side ONCE for all for UIDs in
(maxage +n), take the lowest UID from this set and request the other
side for all the UIDs bigger than lowest_UID and sync, period.

1. Request remote with (maxage + n); remote.messageslist won't change.
2. Find the lowest_UID in remote.messageslist.
3. Request local for UIDs greater than lowest_UID; we have local.messageslist.
4. Sync remote.messageslist against local.messageslist.

We would avoid some "problems" like:
- what to do when there's no common UID;
- having to compute the lowest_common_UID;
- require two SEARCH requests against the same IMAP server before a sync
  (while acceptable for edge-cases, it's still not the most effective).


-- 
Nicolas Sebrecht