maxage causes loss of local email

[Nicolas Sebrecht]

On Tue, Mar 17, 2015 at 08:45:42PM -0400, Janna Martl wrote:

> So I implemented this, tested it, and found that it was still (rarely)
> erroneously deleting my mail.

Could you share your WIP?

>                               Culprit: the IMAP server (gmail)
> is actually occasionally assigning UIDs out of order. I tried doing
> IMAP by hand, and got the following transcript:
> 
> tag fetch 24774 (UID INTERNALDATE)
> * 24774 FETCH (UID 45894 INTERNALDATE "15-Feb-2015 15:48:09 +0000")
> tag OK Success
> 
> tag fetch 24775 (UID INTERNALDATE)
> * 24775 FETCH (UID 45895 INTERNALDATE "15-Feb-2015 15:47:41 +0000")
> tag OK Success
> 
> Am I doing something wrong here, or can we really not trust UIDs to
> come in order?

This would be serious breakage fo the RFC. See

  https://tools.ietf.org/html/rfc3501#section-2.3.1.1

The INTERNALDATE is more subject to approximations:

  https://tools.ietf.org/html/rfc3501#section-2.3.3

Delivering the same mail at the same time by either IMAP or SMTP can
give different INTERNALDATE, for example. Also, big infra tend to have
decoupled SMTP/IMAP/DATABASE/etc.

So, I'm not surprised about the result you have. Take this (bigdata)
example:

0. The mail is received by SMTP at t0.
1. It is sent to the DATABASE at t1.
2. It is received by the DATABASE at t2.
3. It is persistently stored by the DATABASE at t3.
4. It is considered from the IMAP infra at t4.
N. It is given an INTERNALDATE at some point in time, we don't known
when exactly (at t0, t1, t2, somwhere else), what's the value of it (the
reference time taken) and we don't know who is in charge of assigning
it.

I'd say we shouldn't rely on INTERNALDATE at all. Comparing the UIDs
order with the INTERNALDATE order is not posssible. I think that
that's the INTERNALDATE that is assigned a date out of delivery order.

Well, even the terms "delivery order" is subject to interpretation. When
is a mail considered "delivered"? At t0, t1, t2, t3, t4, whatever?

Gmail is known to take liberties with the standards but I'm pretty sure
the UID ascending thing is honored.

Because we highly rely on UIDs in OfflineIMAP, we should prefer working
with UIDs for maxage too to avoid having to workaround unexpected
results like the one above.


About what we care, since we are asking for message up to (maxage + n)
with n in days, few divergences in the UIDs order regarding the "real
delivery date order" should have no impact to our algorithm. When the
IMAP infra consider a new mail, it is assigned a new ascending UID.

I wonder if there's something else that could hurt. Apart from this, I'm
curious about the code. ,-)

-- 
Nicolas Sebrecht