<DKIM> maxage causes loss of local email

[Janna Martl]

On Tue, Mar 10, 2015 at 10:49:07AM +0100, Nicolas Sebrecht wrote:
>On Mon, Mar 09, 2015 at 10:18:17PM -0400, Janna Martl wrote:
>> Also, even though I think messages are no longer being deleted when
>> they shouldn't be, I can still cause messages to get copied when they
>> already exist in the destination... I'll try to look into that some
>> more.

I figured this out; it was sort of a false positive. (I'd been testing
this stuff by changing the maxage configuration setting; if you
increase maxage from N to N+1, then messages whose age fall in the
range (N, N+1) get copied from the server, because they weren't in the
statusfolder, which had only been tracking messages age > N, so this
is expected behavior.)

>This is because it is not reducing the messagelist on both sides.  Here
>is the offending case (letters are messages, time +N define the hours of
>the message around maxage):
>
>    - N+0 == time.time() + maxage
>
>    - Local has                   [A+2, B+1, Y-1     ]
>    - IMAP has                    [A+2, B+1, Y-1, Z-2]
>
>    - Local (maxage)     returns  [          Y-1     ]
>    - Local (maxage + 1) returns  [A+2, B+1, Y-1     ]
>    - SINCE (maxage)     returns  [               Z-2]
>    - SINCE (maxage + 1) returns  [A+2, B+1, Y-1, Z-2]
>
>Before the timezone fix:
>    - Local (maxage)     returns  [          Y-1     ]
>    - SINCE (maxage)     returns  [               Z-2]
>=> FAILED: Y-1 is deleted on IMAP, .
>
>With the current fix:
>  In order to not delete B+1, we request with SINCE (maxage + 1):
>    - Local (maxage)     returns  [          Y-1     ]
>    - SINCE (maxage + 1) returns  [A+2, B+1, Y-1, Z-2]
>  If after exclusion (still based on maxage computations), we get:
>    - Local (maxage)     returns  [          Y-1     ]
>    - SINCE (maxage    ) returns  [     B+1, Y-1, Z-2]
>=> FAILED: B+1 is duplicated on local.

B+1 should have been excluded here, though, because its internaldate
is older than maxage (no timezone issues, because internaldate comes
with a timestamp, including explicit timezone, that we convert to UTC).

>Relying on delicate computations (you've seen how it works...), is sad
>because we have the UIDs.
> ...

On Tue, Mar 10, 2015 at 11:19:52AM +0100, Nicolas Sebrecht wrote:
>Request both sides with (maxage + 1):
>  - Local (maxage + 1) returns  [A+2, B+1, Y-1     ]
>  - SINCE (maxage + 1) returns  [A+2, B+1, Y-1, Z-2]
>First, compare UIDs and exclude messages on both sides within the
>range [(maxage)..(maxage + 1)]:
>  - Local (maxage + 1) returns  [                  ]
>  - SINCE (maxage + 1) returns  [               Z-2]

(I'm not sure why in these examples Y-1 is going away in the >
maxage check; isn't this supposed to be a message that's one hour
younger than maxage?)

OK, so I think I'm understanding this UID check idea: we have two
timezones going on; e.g. the top one represents the local messagelist,
containing X, Y, and W, measured in UTC, and the bottom one represents
the remote messagelist, containing X, Z, and W, measured in UTC -0900
(so "0" in the bottom timeline means "00:00 -0900" = "09:00 +0000").

                 X  Y      W
 |--------------|---------------|
-24             0             24

                    Y  Z   W
         |--------------|---------------|
        -24             0             24

If we don't try to correct for timezones, each messagelist will
contain things > 0 (with respect to its native timezone), so
messagelist1 = [X,Y,W] and messagelist2 = [W]. This is bad because Y
will get deleted from messagelist1. However, this is the only thing
that can go wrong. Create a (maxage + 1) messagelist for the bottom
that contains time > -24, and delete anything from messagelist1 that
is on this expanded list but not in messagelist2. Do the same in the
opposite direction, because we don't know a priori which direction the
shift is in. I really like this -- it's so much less messy than what I
was trying to do.


>THEN, exclude whatever ABOVE Y-1, fallback on (maxage) if no identic
>UIDs were found:
>  - Local (maxage + 1) returns  [                  ]
>  - SINCE (maxage + 1) returns  [               Z-2]

The above made sense to me; I don't see why you have to do any more
here. (And it doesn't seem to have changed anything in your example?)
If there are no duplicates (e.g. Y/Y in my example) then I think that
means nothing has to be corrected...

-- J.M.