maxage causes loss of local email

[Janna Martl]

I use the maxage setting, and for a long time I've noticed that every
day, a random subset of mail of age maxage + 1 gets deleted locally
(but not on the remote server). I think I've found the cause:

Normally, offlineimap deletes mail that exists locally and in the
statusfolder, but not in the remote folder. When maxage comes into
play, this means it deletes UIDs whose local incarnation is younger
than maxage, but whose remote incarnation is older than maxage. Age of
the local copy is determined by the first string of numbers in the
filename, and it seems this is just the current time when the mail is
retrieved. So the trouble happens when something is retrieved locally
much later than its remote timestamp.

(As another consequence of this problem, if I have very old mail on
the remote server that isn't in my local folder for some reason, I
can't save it locally, because the new local file gets marked with the
current timestamp, and then gets deleted on the next pass of
offlineimap.)

One solution would be to check a local message's internal timestamp in
addition to the filename, when doing the maxage check. But this seems
slow. Another solution would be to include the email's internal time
in the filename. I wrote a (probably extremely clumsy!) patch that
makes new Maildir filenames
    (internal date)_(retrieval date)_(uniqueness string)...
which, at least, fixes this problem for me.

Thoughts?

-- J.M.
-------------- next part --------------
--- /usr/lib/python2.7/site-packages/offlineimap/folder/Maildir.py	2015-02-28 14:12:37.959293351 -0500
+++ Maildir.py	2015-02-28 14:10:40.768698235 -0500
@@ -22,6 +22,7 @@
 import tempfile
 from .Base import BaseFolder
 from threading import Lock
+from offlineimap import emailutil
 
 try:
     from hashlib import md5
@@ -235,16 +236,21 @@
         filepath = os.path.join(self.getfullname(), filename)
         return os.path.getmtime(filepath)
 
-    def new_message_filename(self, uid, flags=set()):
+    def new_message_filename(self, uid, flags=set(), rtime=None):
         """Creates a new unique Maildir filename
 
         :param uid: The UID`None`, or a set of maildir flags
         :param flags: A set of maildir flags
         :returns: String containing unique message filename"""
         timeval, timeseq = _gettimeseq()
-        return '%d_%d.%d.%s,U=%d,FMD5=%s%s2,%s' % \
-            (timeval, timeseq, os.getpid(), socket.gethostname(),
-             uid, self._foldermd5, self.infosep, ''.join(sorted(flags)))
+        if rtime is None:
+            return '%d_%d.%d.%s,U=%d,FMD5=%s%s2,%s' % \
+                (timeval, timeseq, os.getpid(), socket.gethostname(),
+                 uid, self._foldermd5, self.infosep, ''.join(sorted(flags)))
+        else:
+            return '%d_%d_%d.%d.%s,U=%d,FMD5=%s%s2,%s' % \
+                (rtime, timeval, timeseq, os.getpid(), socket.gethostname(),
+                 uid, self._foldermd5, self.infosep, ''.join(sorted(flags)))
 
 
     def save_to_tmp_file(self, filename, content):
@@ -314,7 +320,9 @@
         # Otherwise, save the message in tmp/ and then call savemessageflags()
         # to give it a permanent home.
         tmpdir = os.path.join(self.getfullname(), 'tmp')
-        messagename = self.new_message_filename(uid, flags)
+        if rtime is None:
+            rtime = emailutil.get_message_date(content)
+        messagename = self.new_message_filename(uid, flags, rtime=rtime)
         tmpname = self.save_to_tmp_file(messagename, content)
         if rtime != None:
             os.utime(os.path.join(self.getfullname(), tmpname), (rtime, rtime))
@@ -382,8 +390,10 @@
         oldfilename = self.messagelist[uid]['filename']
         dir_prefix, filename = os.path.split(oldfilename)
         flags = self.getmessageflags(uid)
+        content = self.getmessage(uid)
+        rtime = emailutil.get_message_date(content)
         newfilename = os.path.join(dir_prefix,
-          self.new_message_filename(new_uid, flags))
+          self.new_message_filename(new_uid, flags, rtime=rtime))
         os.rename(os.path.join(self.getfullname(), oldfilename),
                   os.path.join(self.getfullname(), newfilename))
         self.messagelist[new_uid] = self.messagelist[uid]