maxage causes loss of local email

[Janna Martl]

On Sun, Mar 22, 2015 at 12:55:57AM +0100, Nicolas Sebrecht wrote:
> On Sat, Mar 21, 2015 at 06:12:50PM -0400, Janna Martl wrote:
> 
> > So I did something really stupid in my (supposedly) "trivial things"
> > patch: in __syncmessagesto_copy(), instead of
> > 	copylist.remove(uid)
> > it should be
> > 	continue
> > (line 773?).
> 
> A bit hard to follow from start to end. Could you please talk with
> patches?

bugfix for 428349e3: don't copy messages if not necessary

Fix bug preventing messages with UID's already in the destination folder
from getting excluded from the copy list.

Signed-off-by: Janna Martl <janna.martl109 at gmail.com>
---
 offlineimap/folder/Base.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/offlineimap/folder/Base.py b/offlineimap/folder/Base.py
index e033ab2..e3c80b7 100644
--- a/offlineimap/folder/Base.py
+++ b/offlineimap/folder/Base.py
@@ -770,7 +770,7 @@ class BaseFolder(object):
                     content = self.getmessage(uid)
                     rtime = emailutil.get_message_date(content, 'Date')
                 statusfolder.savemessage(uid, None, flags, rtime)
-                copylist.remove(uid)
+                continue
 
             self.ui.copyingmessage(uid, num+1, num_to_copy, self, dstfolder)
             # exceptions are caught in copymessageto()
-- 
2.3.3


> > So that takes care of the maxage --> maxage + 1 "issue". However, I do
> > claim that my point still holds, with a different edge case: the user
> > copies (e.g. manually, with mutt's "save to mailbox" feature, etc.) an
> > old message on the server to their local folder. It's "new", so it
> > gets copied to the remote, whereupon it gets assigned a new UID.
> > localfolder's change_message_uid() then updates the UID and changes
> > the filename.
> 
> It's actually the behaviour since the beginning of OfflineIMAP. Whatever
> coming in (copied/paste/etc) from outside of OfflineIMAP is considered
> new and synced back again.
>
> > Now sync again: the message was recently assigned a UID, but has an
> > old date.
> 
> So, you're talking about the IMAP server.

To be more precise: there's a message R on the IMAP server with UID =
U, that was recently copied from a local message L which now also has
UID = U.
 
> >           (Note: in this case, internaldate = filename date, and
> > that's actually bad.) 
> 
> There is not real concept of INTERNALDATE locally, neither from Maildir
> format nor from the format of the filenames we use.
> 
> Pretending INTERNALDATE == filename does not make sense.
 
I mean that R's internaldate is L's original date (a long time ago),
and L's filename reflects this same original date.

> This is where you left me behind. :-)
> 
> >                       So it's not in the local messagelist
> > (the one that's calculated first) because it's not within maxage, but
> > it *is* in the remote messagelist, because its UID is higher than
> > whatever cutoff we're using. So it gets deleted remotely.
> 
> Let's put the UID restriction process outside of the scan method, then.
>
> I think you're currently more screwed by the lack of proper OOP in the
> code than the algorithm logic which is good. ,-)
> 
> Since local is the first being involved:
> 
> - scan for all previously-synced filenames (UID is known) within maxage
>   *and* filenames with unknown valid UID/datetimes.
> 
> - caller extract min_uid from localmessagelist,
>   request the remote for min_uid:*,
>   sync
> 
>   - if no min_uid found,
>     the list is empty of known UID,
>     apply maxage against remote,
>     request remote for mails within (maxage + 1),
>     sync

I'm not sure if I'm understanding you correctly, but I think this is
what is already happening. In your first point, I'm assuming that
"unknown valid UID" means negative UID (messages, like L above, that
don't come with a valid UID because they haven't been synced before,
but might have a valid time, get assigned a negative UID). So min_uid
has to be the min of the non-negative UID's (i.e. the "known ones") in
the local messagelist. The third point seems right but doesn't apply
to my scenario.

In my scenario, L has a valid time T but no valid UID, so it gets
assigned a negative UID and uploaded; call the new remote incarnation
R. The server assigns R a real UID U, and L's local filename is
changed to contain T and U. That's the first sync. On the second sync,
L has valid UID, time, etc. and doesn't make it into the local
messagelist because T is too early, but U is recent, so R makes it
into the remote messagelist. This is bad. To fix this, sync 1 should
change L's filename to now_U_... instead of T_U_....


-- J.M.