[PATCH] Let OfflineIMAP sync all flags between IMAP servers, not just Maildir-compatible flags

Erik Quaeghebeur offlineimap at equaeghe.nospammail.net
Sun Mar 10 16:37:31 GMT 2013


Dear maintainers & other interested subscribers,


As an intermediate step towards my goal of being able to use OfflineIMAP 
to sync all IMAP flags (i.e., including custom keywords) with, e.g., 
notmuch, via an xattrs-enhanced maildir, I have made an attempt at 
making OfflineIMAP sync all IMAP flags between IMAP servers while 
preserving the current behavior when syncing with a maildir.

Probably due to its historical development, most of the internal 'flags' 
variables were actually maildir flags (as defined in imaputil.flagmap) 
and the corresponding IMAP flags were generated as needed when syncing 
with an IMAP server (cf. imaputil.flagsimap2maildir). What I did was 
make the internal 'flags' variables use the IMAP flags (also in a set, 
not as a flagstring; so I also had to modify imaputil.flagsimap2maildir 
and imaputil.flagsmaildir2imap) and generate the maildir flags as needed.

Because flagstrings are also used, I also had to add two new conversion 
functions imaputil.flagstring2flagset and imaputil.flagset2flagstring.

This patch [based on the next branch, commit 611f6e89c07b] was 
(successfully) tested in a very limited way: I synced a single message 
including all of its custom flags between two distinct IMAP servers and 
synced the same message to a maildir to observe that the old behavior 
was preserved.

I would like you to test and have a critical look at the patch and tell 
me what further changes are needed to eventually let it be included. I 
am not only thinking about formatting issues, but also issues I have not 
spotted because I am not really familiar with the OfflineIMAP code. If a 
maintainer wishes to take the code from here and modify it (him|her)self 
, then that is also fine by me. (Actually, I'd think that would be 
great, because I have limited time to spend and the next few hours to 
work on this might be soon, or a month away.)


Best,

Erik

N.B.: I may have introduced linebreaks while copy-pasting the patch here 
(sorry little experience with that), so I have also attached it as a 
text file. If needed, I think I could for on Github, and create a pull 
request there; let me know if you'd prefer this.

---
  offlineimap/folder/IMAP.py    | 12 ++++++------
  offlineimap/folder/Maildir.py | 28 ++++++++++++++++------------
  offlineimap/imaputil.py       | 28 ++++++++++++++++++----------
  3 files changed, 40 insertions(+), 28 deletions(-)

diff --git a/offlineimap/folder/IMAP.py b/offlineimap/folder/IMAP.py
index ec753c7..18d7a8f 100644
--- a/offlineimap/folder/IMAP.py
+++ b/offlineimap/folder/IMAP.py
@@ -196,7 +196,7 @@ class IMAPFolder(BaseFolder):
                                            minor = 1)
              else:
                  uid = long(options['UID'])
-                flags = imaputil.flagsimap2maildir(options['FLAGS'])
+                flags = imaputil.flagstring2flagset(options['FLAGS'])
                  rtime = imaplibutil.Internaldate2epoch(messagestr)
                  self.messagelist[uid] = {'uid': uid, 'flags': flags, 
'time': rtime}

@@ -546,7 +546,7 @@ class IMAPFolder(BaseFolder):
                  #Do the APPEND
                  try:
                      (typ, dat) = imapobj.append(self.getfullname(),
-                                       imaputil.flagsmaildir2imap(flags),
+                                       imaputil.flagset2flagstring(flags),
                                         date, content)
                      retry_left = 0                # Mark as success
                  except imapobj.abort as e:
@@ -630,7 +630,7 @@ class IMAPFolder(BaseFolder):
                  self.ui.flagstoreadonly(self, [uid], flags)
                  return
              result = imapobj.uid('store', '%d' % uid, 'FLAGS',
-                                 imaputil.flagsmaildir2imap(flags))
+                                 imaputil.flagset2flagstring(flags))
              assert result[0] == 'OK', 'Error with store: ' + '. 
'.join(result[1])
          finally:
              self.imapserver.releaseconnection(imapobj)
@@ -639,7 +639,7 @@ class IMAPFolder(BaseFolder):
              self.messagelist[uid]['flags'] = flags
          else:
              flags = 
imaputil.flags2hash(imaputil.imapsplit(result)[1])['FLAGS']
-            self.messagelist[uid]['flags'] = 
imaputil.flagsimap2maildir(flags)
+            self.messagelist[uid]['flags'] = 
imaputil.flagstring2flagset(flags)

      def addmessageflags(self, uid, flags):
          self.addmessagesflags([uid], flags)
@@ -676,7 +676,7 @@ class IMAPFolder(BaseFolder):
              r = imapobj.uid('store',
                              imaputil.uid_sequence(uidlist),
                              operation + 'FLAGS',
-                            imaputil.flagsmaildir2imap(flags))
+                            imaputil.flagset2flagstring(flags))
              assert r[0] == 'OK', 'Error with store: ' + '. '.join(r[1])
              r = r[1]
          finally:
@@ -696,7 +696,7 @@ class IMAPFolder(BaseFolder):
                  continue
              flagstr = attributehash['FLAGS']
              uid = long(attributehash['UID'])
-            self.messagelist[uid]['flags'] = 
imaputil.flagsimap2maildir(flagstr)
+            self.messagelist[uid]['flags'] = 
imaputil.flagstring2flagset(flagstr)
              try:
                  needupdate.remove(uid)
              except ValueError:          # Let it slide if it's not in 
the list
diff --git a/offlineimap/folder/Maildir.py b/offlineimap/folder/Maildir.py
index 24d943c..5d30a94 100644
--- a/offlineimap/folder/Maildir.py
+++ b/offlineimap/folder/Maildir.py
@@ -21,6 +21,7 @@ import re
  import os
  from .Base import BaseFolder
  from threading import Lock
+from offlineimap import imaputil

  try:
      from hashlib import md5
@@ -124,9 +125,9 @@ class MaildirFolder(BaseFolder):
          for the respective element.  If flags are empty or cannot be
          detected, we return an empty flags list.

-        :returns: (prefix, UID, FMD5, flags). UID is a numeric "long"
-            type. flags is a set() of Maildir flags"""
-        prefix, uid, fmd5, flags = None, None, None, set()
+        :returns: (prefix, UID, FMD5, maildirflags). UID is a numeric 
"long"
+            type. maildirflags is a set() of Maildir flags"""
+        prefix, uid, fmd5, maildirflags = None, None, None, set()
          prefixmatch = self.re_prefixmatch.match(filename)
          if prefixmatch:
              prefix = prefixmatch.group(1)
@@ -142,8 +143,8 @@ class MaildirFolder(BaseFolder):
          if flagmatch:
              # Filter out all lowercase (custom maildir) flags. We don't
              # handle them yet.
-            flags = set((c for c in flagmatch.group(1) if not c.islower()))
-        return prefix, uid, fmd5, flags
+            maildirflags = set((c for c in flagmatch.group(1) if not 
c.islower()))
+        return prefix, uid, fmd5, maildirflags

      def _scanfolder(self):
          """Cache the message list from a Maildir.
@@ -173,7 +174,7 @@ class MaildirFolder(BaseFolder):
                          self.getfullname(), filepath)) > maxsize):
                  continue

-            (prefix, uid, fmd5, flags) = self._parse_filename(filename)
+            (prefix, uid, fmd5, maildirflags) = 
self._parse_filename(filename)
              if uid is None: # assign negative uid to upload it.
                  uid = nouidcounter
                  nouidcounter -= 1
@@ -186,7 +187,8 @@ class MaildirFolder(BaseFolder):
                  else:
                      uid = long(uidmatch.group(1))
              # 'filename' is 'dirannex/filename', e.g. 
cur/123,U=1,FMD5=1:2,S
-            retval[uid] = {'flags': flags, 'filename': filepath}
+            retval[uid] = {'flags': 
imaputil.flagsmaildir2imap(maildirflags),
+                           'filename': filepath}
          return retval

      def quickchanged(self, statusfolder):
@@ -198,7 +200,7 @@ class MaildirFolder(BaseFolder):
              return True
          # Also check for flag changes, it's quick on a Maildir
          for (uid, message) in self.getmessagelist().iteritems():
-            if message['flags'] != statusfolder.getmessageflags(uid):
+            if not(message['flags'] <= statusfolder.getmessageflags(uid)):
                  return True
          return False  #Nope, nothing changed

@@ -234,8 +236,9 @@ class MaildirFolder(BaseFolder):
          timeval, timeseq = gettimeseq()
          return '%d_%d.%d.%s,U=%d,FMD5=%s%s2,%s' % \
              (timeval, timeseq, os.getpid(), socket.gethostname(),
-             uid, self._foldermd5, self.infosep, ''.join(sorted(flags)))
-
+             uid, self._foldermd5, self.infosep,
+             ''.join(sorted(imaputil.flagsimap2maildir(flags))))
+
      def savemessage(self, uid, content, flags, rtime):
          """Writes a new message, with the specified uid.

@@ -304,7 +307,7 @@ class MaildirFolder(BaseFolder):
          oldfilename = self.messagelist[uid]['filename']
          dir_prefix, filename = os.path.split(oldfilename)
          # If a message has been seen, it goes into 'cur'
-        dir_prefix = 'cur' if 'S' in flags else 'new'
+        dir_prefix = 'cur' if '\\Seen' in flags else 'new'

          if flags != self.messagelist[uid]['flags']:
              # Flags have actually changed, construct new filename Strip
@@ -313,7 +316,8 @@ class MaildirFolder(BaseFolder):
              infomatch = self.re_flagmatch.search(filename)
              if infomatch:
                  filename = filename[:-len(infomatch.group())] #strip off
-            infostr = '%s2,%s' % (self.infosep, ''.join(sorted(flags)))
+            infostr = '%s2,%s' % (self.infosep,
+ 
''.join(sorted(imaputil.flagsimap2maildir(flags))))
              filename += infostr

          newfilename = os.path.join(dir_prefix, filename)
diff --git a/offlineimap/imaputil.py b/offlineimap/imaputil.py
index fe69b7a..4f4e194 100644
--- a/offlineimap/imaputil.py
+++ b/offlineimap/imaputil.py
@@ -166,28 +166,36 @@ def imapsplit(imapstring):
                  break
      return retval

+def flagstring2flagset(flagstring):
+    """Convert string '(\\Draft Old)' into a flags set(['\\Draft', 
'Old'])"""
+    imapflaglist = flagstring[1:-1].split()
+    return set(imapflaglist)
+
+def flagset2flagstring(flagset):
+    """Convert flags set(['\\Draft', 'Old']) into a string '(\\Draft 
Old)'"""
+    return '(' + ' '.join(flagset) + ')'
+
  flagmap = [('\\Seen', 'S'),
             ('\\Answered', 'R'),
             ('\\Flagged', 'F'),
             ('\\Deleted', 'T'),
             ('\\Draft', 'D')]

-def flagsimap2maildir(flagstring):
-    """Convert string '(\\Draft \\Deleted)' into a flags set(DR)"""
+def flagsimap2maildir(flags):
+    """Convert set([\\Draft, \\Deleted]) into a flags set(DR)"""
      retval = set()
-    imapflaglist = flagstring[1:-1].split()
      for imapflag, maildirflag in flagmap:
-        if imapflag in imapflaglist:
+        if imapflag in flags:
              retval.add(maildirflag)
      return retval

-def flagsmaildir2imap(maildirflaglist):
-    """Convert set of flags ([DR]) into a string '(\\Deleted \\Draft)'"""
-    retval = []
+def flagsmaildir2imap(maildirflags):
+    """Convert set of flags ([DR]) into a set([\\Draft, \\Deleted])"""
+    retval = set()
      for imapflag, maildirflag in flagmap:
-        if maildirflag in maildirflaglist:
-            retval.append(imapflag)
-    return '(' + ' '.join(sorted(retval)) + ')'
+        if maildirflag in maildirflags:
+            retval.add(imapflag)
+    return retval

  def uid_sequence(uidlist):
      """Collapse UID lists into shorter sequence sets
-- 
1.8.1.5
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Let-OfflineIMAP-sync-all-flags-between-IMAP-servers-.patch
Type: text/x-patch
Size: 9443 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20130310/deb29f69/attachment-0002.bin>


More information about the OfflineIMAP-project mailing list