maxage causes loss of local email

[Nicolas Sebrecht]

On Sat, Mar 21, 2015 at 09:23:29PM -0400, Janna Martl wrote:

> bugfix for 428349e3: don't copy messages if not necessary

Applied, thank you.

> To be more precise: there's a message R on the IMAP server with UID =
> U, that was recently copied from a local message L which now also has
> UID = U.
>  
> > >           (Note: in this case, internaldate = filename date, and
> > > that's actually bad.) 
> > 
> > There is not real concept of INTERNALDATE locally, neither from Maildir
> > format nor from the format of the filenames we use.
> > 
> > Pretending INTERNALDATE == filename does not make sense.
>  
> I mean that R's internaldate is L's original date (a long time ago),
> and L's filename reflects this same original date.

Ok, I see.

> > This is where you left me behind. :-)
> > 
> > >                       So it's not in the local messagelist
> > > (the one that's calculated first) because it's not within maxage, but
> > > it *is* in the remote messagelist, because its UID is higher than
> > > whatever cutoff we're using. So it gets deleted remotely.
> > 
> > Let's put the UID restriction process outside of the scan method, then.
> >
> > I think you're currently more screwed by the lack of proper OOP in the
> > code than the algorithm logic which is good. ,-)
> > 
> > Since local is the first being involved:
> > 
> > - scan for all previously-synced filenames (UID is known) within maxage
> >   *and* filenames with unknown valid UID/datetimes.
> > 
> > - caller extract min_uid from localmessagelist,
> >   request the remote for min_uid:*,
> >   sync
> > 
> >   - if no min_uid found,
> >     the list is empty of known UID,
> >     apply maxage against remote,
> >     request remote for mails within (maxage + 1),
> >     sync
> 
> I'm not sure if I'm understanding you correctly, but I think this is
> what is already happening. In your first point, I'm assuming that
> "unknown valid UID" means negative UID (messages, like L above, that
> don't come with a valid UID because they haven't been synced before,
> but might have a valid time, get assigned a negative UID). So min_uid
> has to be the min of the non-negative UID's (i.e. the "known ones") in
> the local messagelist. The third point seems right but doesn't apply
> to my scenario.

Yes.

> In my scenario, L has a valid time T but no valid UID, so it gets
> assigned a negative UID and uploaded; call the new remote incarnation
> R. The server assigns R a real UID U, and L's local filename is
> changed to contain T and U. That's the first sync. On the second sync,
> L has valid UID, time, etc. and doesn't make it into the local
> messagelist because T is too early, but U is recent, so R makes it
> into the remote messagelist. This is bad.

Ok.

>                                           To fix this, sync 1 should
> change L's filename to now_U_... instead of T_U_....

Or correctly scan the folder. ,-)

The fix #1 you suggested before is the correct fix. But don't call the
lists "old" and "new", it's confusing. Something like "strict_maxage"
and "excluded_from_strict_maxage".

This would require good comments to prevent further developers to
"optimize" the scan process back to what we had previously and
introduce this issue. The above R/L explanations is quite good, BTW.

Good catch!

-- 
Nicolas Sebrecht