Bugreport: maxage deleting old email from destination

Mon Aug 1 23:03:39 BST 2011

> > I don't understand that. What is a 'LIVE' mailbox and live sync?
> I mean I am trying on a production server, where I am not allowed to
> break everything to test.

I see.

> > You have local and remote mailboxes. In the local mailbox you deleted old
> > mails, but you don't want to delete them from remote mailbox. So you run
> > offlineimap with maxage setting. But you still see old emails being
> > deleted.
> Well, this is a bit confusing, because offlineimap syncs both ways, so
> I don't see much difference between 'remote' and 'local'. In my case
> local is local dovecot and remote is gmail.

ok. Still 'local' and 'remote' is descriptive.

> I am actually interested in syncing mail FROM gmail (because nothing
> arrives at the local server) TO local server. I even tried readonly=1
> for gmail repository. That was for a different purpose - for mail
> backup. I delete old messages from gmail but don't want them to delete
> from my local (backup) server.

> Anyway, I've done what you suggested and now I'm even more confused :).

I think I'm starting to understand now, but maybe you'll prove me wrong
:)

Let me now do a bit of guessing. You set maxage for your remote(gmail)
account, right? And there is no limit for your local one.

I'll speculate a bit more.
So you start with both folders fully synced. Then you set maxage for
remote, which _hides_ older mail there from OfflineIMAP. OfflineIMAP
suddenly can't see the older mails at remote site, as if you deleted
them. So during sync it deletes the corresponding mails locally.

[...]

> So we have 133 emails. There are still emails older than 180 days ago
> (02-Feb-2011) - we'll check if they are deleted in the next step.
> 
> 3. I set maxage=180 and rerun the sync.
> 
> Syncing 1A_invoices: Gmail -> MappedIMAP
> Deleting 26 messages (49, 50, 51, 134, 135, 136, 137, 138, 139, 140,
> 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154,
> 155, 156) in MappedIMAP[1A_invoices]

It's deleting them locally and not on remote as I thought before.

> 4. Trying a local imap FETCH again, still 133 emails

This bit confuses me. If OfflineIMAP deleted 26 messages, should not you
have 107 of them locally?

> 5. Trying a FETCH on a remote imap (gmail)
> 
> . fetch 1:* internaldate
...
> * 133 FETCH (INTERNALDATE "04-Dec-2010 03:51:45 +0000")
> . OK Success

No mails deleted on remote. We have 133 of them.  (!!!)

> 6. Just to prove I'm not a complete idiot

That sounds like my life motto :)

> complaining about the deleted mail which is not being deleted, I
> remove maxage and run the sync again.

So now OfflineIMAP can see all the mail again on remote (the mail
sort of suddenly appeared), so it's time to sync it back.

> Syncing 1A_invoices: Gmail -> MappedIMAP
> Copy message 49 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 50 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 51 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 134 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 135 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 136 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 137 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 138 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 139 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 140 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 141 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 142 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 143 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 144 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 145 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 146 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 147 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 148 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 149 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 150 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 151 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 152 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 153 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 154 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 155 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]
> Copy message 156 Gmail[1A_invoices] -> MappedIMAP[1A_invoices]

26 of them copied from remote to local. I would expect that. It's the 26
mails deleted just a minute ago.

But I don't understand the rest. It seems that we are adding some emails
to remote (Gmail). But just a moment ago we saw we have 133 of them
(I marked that by !!!)

> Copy message -11 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -26 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -25 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -24 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -23 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -22 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -21 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -20 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> 
> at this point it hung somehow (gmail is unstable today, this may be
> the reason), so I killed it and started again:
> 
> Syncing 1A_invoices: Gmail -> MappedIMAP
> Deleting 8 messages (141, 150, 151, 152, 153, 154, 155, 156) in
> MappedIMAP[1A_invoices]
> Copy message -11 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -18 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -17 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -16 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -15 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -14 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -13 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -12 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -2 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -10 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -9 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -8 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -7 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -6 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -5 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -4 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -3 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]
> Copy message -1 MappedIMAP[1A_invoices] -> Gmail[1A_invoices]

That is 26+ for gmail and 8 - for local imap. 

> Again, logging in to local imap - 133 emails. Remote gmail - 133 emails.

Which gives me 159 gmail, 125 local imap. Where did I go wrong?

> I don't understand:
> - what are negative UIDs?

I'm not sure, but I see negative UIDs when I create mail in my Maildir,
so that it has no UID.

> - what did it delete? The IDs 153, 154, 155, 156 are the most
> confusing, because there were just 133 emails in the folder. Are those
> different IDs?

Every mail is given UID. The UID can't be reused. If you start from
scratch, have 100 mails, delete 99 and add one, you will have UIDs 100
and 101.

To see UIDs, run:
. uid fetch 1:* ...

> - what did it copy? It actually downloaded something, it wasn't just instant!

I'm confused about that too, since my math according to your logs does
not end up with 133 mails on both sides.

> Now I suspect it's just the debug output which is wrong, but that
> needs an explaination anyway. It actually deleted something when I add
> maxage and actually downloads something when I remove it. If I just
> rerun the sync with the same parameters - nothing happens.

So I guess you actually want maxage on your _local_ imap. That way you
hide old mail from _local_ storage, so it gets deleted on gmail.

Am I anywhere close to real situation?
-- 
	Vlad