[PATCH 3/3] Revamp Syncing strategy.

[Sebastian Spaeth]

On Thu, 20 Jan 2011 15:49:16 +0100, Johannes Stezenbach <js at sig21.net> wrote:
> Hi,

Hi Johannes,

these 4 passes are the same as before, so there is nothing new that I
introduced here (besides small behavioral tweaks). I tried to come up
with a smarter method, but I always ended up with a similar design as
John's :-).

Some passes make more sense if the SRC/DEST repos are of a specific type
and we could make it a bit cleverer based on that information.

> > A folder sync strategy is mostly like it was before:
> >         Pass1: Transfer new local messages
> >                Upload msg with negative/no UIDs to dstfolder. dstfolder
> >                should assign that message a new UID. Update statusfolder.
> >         Pass2: Copy existing messages
> >                Copy messages in self, but not statusfolder to dstfolder
> >                if not already in dstfolder. Update statusfolder.
> 
> If dest is IMAP, then copying will create a new UID, so how is
> pass 2 different from pass 1?

Pass1 will only look at messages with no/invalid UID, so that would be
either:
a) a new mail that I put on the maildir or
b) a mail that I copy from a maildir folder to another one, thereby invalidating
the folder MD5 making the UID invalid.

Therefore, it really only ever makes sense from a Maildir type repository to an
IMAP type (which will assign a UID).

It will be a Noop on an IMAP as source (as all mails should always
have a UID there). But it shouldn't really hurt running through it either.

Pass2 will only look at messages with a positive/valid UID, so this will
catch most of the new mail in an IMAP repository. When syncing *to* an
IMAP server, this will mean that we get can verify if the IMAP server
already has a mail of that UID and ignore it. This is what happens in my
patches when we blow away the LocalStatus files. We have valid UIDs on
our mails, and won't upload them if the server already has a mail with
that UID.

*Most* of the time (in the regular, non-corrupt case), this pass will
not catch anything when syncing from Maildir to IMAP though. It doesn't
hurt running it though.

Of course there is also the IMAP<->IMAP case, where IMAP1 mails will
already have valid UIDs and we maintain a translation table between
IMAP1 UIDs and IMAP2 UIDs. All that should fall under pass2.


Could Pass1 and Pass2 be unified? Perhaps yes. But the patch was so big
that I did not want to change the logic more than needed. Having them as
2 passes won't harm and as there should be no invalid UIDs except in the
case of local modifications on a MAILDIR, pass1 will seldom do something.

> >         Pass3: Remove deleted messages
> >                Get all UIDS in statusfolder but not self. These are
> >                messages that we have locally deleted. Delete those from
> >                dstfolder and statusfolder.
 
> What about deleting mails from local folder (Maildir) which
> have been deleted in remote (IMAP) folder?

Happens in pass3 when syncing from REMOTE IMAP to LOCAL MAILDIR. In that
case the mail will be in LOCALSTATUS and LOCAL, but not in REMOTE
anymore. Pass3 in the REMOTE->LOCAL direction will notice that the UID
is in statusfolder but not on REMOTE. It will then attempt to 1) delete
the mail from LOCAL and 2) from LOCALSTATUS.

Sebastian