OfflineIMAP-project Digest, Vol 23, Issue 54
Théodore Biadala
theodore at biadala.net
Mon Aug 22 12:10:23 BST 2011
offlineimap-project-request at lists.alioth.debian.org a écrit :
>Send OfflineIMAP-project mailing list submissions to
> offlineimap-project at lists.alioth.debian.org
>
>To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/offlineimap-project
>
>or, via email, send a message with subject or body 'help' to
> offlineimap-project-request at lists.alioth.debian.org
>
>You can reach the person managing the list at
> offlineimap-project-owner at lists.alioth.debian.org
>
>When replying, please edit your Subject line so it is more specific
>than "Re: Contents of OfflineIMAP-project digest..."
>
>
>Today's Topics:
>
> 1. [PATCH] Fix standard port for SSL/TLS (Sebastian Spaeth)
> 2. Re: Better error throwing and ID sequences (Sebastian Spaeth)
> 3. [PATCH v2] Maildir relative paths change was not complete
> (Sebastian Spaeth)
> 4. Re: [PATCH 2/2] Re: Temporarily delete old-style lock files
> (Sebastian Spaeth)
> 5. [PATCH 1/3] Per-account locking (Sebastian Spaeth)
> 6. [PATCH 2/3] Perform legacy global lock in addition to new
> per-account lock (Sebastian Spaeth)
>
>
>----------------------------------------------------------------------
>
>Message: 1
>Date: Mon, 22 Aug 2011 10:33:55 +0200
>From: Sebastian Spaeth <Sebastian at SSpaeth.de>
>To: offlineimap-project at lists.alioth.debian.org
>Cc: Sebastian Spaeth <Sebastian at SSpaeth.de>
>Subject: [PATCH] Fix standard port for SSL/TLS
>Message-ID: <1314002035-18476-1-git-send-email-Sebastian at SSpaeth.de>
>
>443 is of course the https and not the IMAPS standard port. Fix.
>
>Thanks to Daniel Shahaf <d.s at daniel.shahaf.name> for the heads up.
>
>Signed-off-by: Sebastian Spaeth <Sebastian at SSpaeth.de>
>---
> offlineimap/imapserver.py | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
>diff --git a/offlineimap/imapserver.py b/offlineimap/imapserver.py
>index 707464d..3ce751c 100644
>--- a/offlineimap/imapserver.py
>+++ b/offlineimap/imapserver.py
>@@ -311,7 +311,7 @@ class IMAPServer:
> elif SSLError and isinstance(e, SSLError) and e.errno == 1:
> # SSL unknown protocol error
> # happens e.g. when connecting via SSL to a non-SSL service
>- if self.port != 443:
>+ if self.port != 993:
> reason = "Could not connect via SSL to host '%s' and non-s"\
> "tandard ssl port %d configured. Make sure you connect"\
> " to the correct port." % (self.hostname, self.port)
>--
>1.7.4.1
>
>
>
>
>------------------------------
>
>Message: 2
>Date: Mon, 22 Aug 2011 10:41:15 +0200
>From: Sebastian Spaeth <Sebastian at SSpaeth.de>
>To: Nicolas Sebrecht <nicolas.s-dev at laposte.net>
>Cc: Nicolas Sebrecht <nicolas.s-dev at laposte.net>, offlineimap-project
> <offlineimap-project at lists.alioth.debian.org>
>Subject: Re: Better error throwing and ID sequences
>Message-ID: <878vqlvnx0.fsf at SSpaeth.de>
>Content-Type: text/plain; charset="us-ascii"
>
>On Fri, 19 Aug 2011 18:47:08 +0200, Nicolas Sebrecht <nicolas.s-dev at laposte.net> wrote:
>> It introduces a bug:
>>
>> ERROR: Exceptions occurred during the run!
>> ERROR: Aborting folder sync 'INBOX.ABUL' [acc: 'nis']
>> FETCHING UIDs in folder [Remotenis]INBOX.ABUL failed. Server responded
>> '[NO] ['No matching messages']'
>> ERROR: ERROR in syncfolder for dev.gmx folder Drafts: Traceback (most
>> recent call last):
>> File "/usr/lib64/python2.7/site-packages/offlineimap/accounts.py",
>> line 343, in syncfolder
>> remotefolder.cachemessagelist()
>> File "/usr/lib64/python2.7/site-packages/offlineimap/folder/IMAP.py",
>> line 174, in cachemessagelist
>> messagestr = messagestr.split(' ', 1)[1]
>> AttributeError: 'NoneType' object has no attribute 'split'
>
>Uhh, it had worked for me, that is weird. IMHO I haven't changed how the
>server response is treated. Interestingly enough we would have failed
>with your IMAP server response:
>
>[NO] ['No matching messages']
>
>without this topic branch too, so I am not sure that things had gotten
>any worse than before.
>
>We want to robustify handling the server response, and we need to check
>what we had been sending the server (to see if it was any invalid
>command, so you have an IMAP log of that run by any chance?).
>
>Sebastian
>-------------- next part --------------
>A non-text attachment was scrubbed...
>Name: not available
>Type: application/pgp-signature
>Size: 197 bytes
>Desc: not available
>URL: <http://lists.alioth.debian.org/pipermail/offlineimap-project/attachments/20110822/88bc722d/attachment-0001.pgp>
>
>------------------------------
>
>Message: 3
>Date: Mon, 22 Aug 2011 11:09:07 +0200
>From: Sebastian Spaeth <Sebastian at SSpaeth.de>
>To: offlineimap-project at lists.alioth.debian.org
>Cc: Vladimir Marek <vlmarek at volny.cz>,
> 1313584239-20451-1-git-send-email-Sebastian at SSpaeth.de, Sebastian
> Spaeth <Sebastian at SSpaeth.de>
>Subject: [PATCH v2] Maildir relative paths change was not complete
>Message-ID: <1314004147-19182-1-git-send-email-Sebastian at SSpaeth.de>
>
>From: Vladimir Marek <vlmarek at volny.cz>
>
>Commit e023f190b0eed84fefc10e28bfe5e4e0467ff257 changed the storing of
>file paths in the messagelist variable to be relative paths, but we were
>using the full absolute path anyway as we missed one spot.
>
>Adapt this and construct the full file path in the one place where we
>need it.
>
>Signed-off-by: Sebastian Spaeth <Sebastian at SSpaeth.de>
>---
>The first version is was mising out some of the changes that should be
>included, so please ignore. This one is fresh on the next branch and
>contains the changes that Vladimir agreed would be needed on top of his
>origin patch.
>
> offlineimap/folder/Maildir.py | 8 ++++----
> 1 files changed, 4 insertions(+), 4 deletions(-)
>
>diff --git a/offlineimap/folder/Maildir.py b/offlineimap/folder/Maildir.py
>index f72f4ea..cedcb5d 100644
>--- a/offlineimap/folder/Maildir.py
>+++ b/offlineimap/folder/Maildir.py
>@@ -133,7 +133,7 @@ class MaildirFolder(BaseFolder):
> folderstr = ',FMD5=' + foldermd5
> for dirannex in ['new', 'cur']:
> fulldirname = os.path.join(self.getfullname(), dirannex)
>- files.extend(os.path.join(fulldirname, filename) for
>+ files.extend(os.path.join(dirannex, filename) for
> filename in os.listdir(fulldirname))
> for file in files:
> messagename = os.path.basename(file)
>@@ -151,10 +151,9 @@ class MaildirFolder(BaseFolder):
>
> #Check and see if the message is too big if the maxsize for this account is set
> if(maxsize != -1):
>- filesize = os.path.getsize(file)
>- if(filesize > maxsize):
>+ size = os.path.getsize(os.path.join(self.getfullname(), file))
>+ if(size > maxsize):
> continue
>-
>
> foldermatch = messagename.find(folderstr) != -1
> if not foldermatch:
>@@ -177,6 +176,7 @@ class MaildirFolder(BaseFolder):
> flags = set(flagmatch.group(1))
> else:
> flags = set()
>+ # 'filename' is 'dirannex/filename', e.g. cur/123_U=1_FMD5=1:2,S
> retval[uid] = {'uid': uid,
> 'flags': flags,
> 'filename': file}
>--
>1.7.4.1
>
>
>
>
>------------------------------
>
>Message: 4
>Date: Mon, 22 Aug 2011 12:09:13 +0200
>From: Sebastian Spaeth <Sebastian at SSpaeth.de>
>To: Nicolas Sebrecht <nicolas.s-dev at laposte.net>
>Cc: Nicolas Sebrecht <nicolas.s-dev at laposte.net>,
> offlineimap-project at lists.alioth.debian.org
>Subject: Re: [PATCH 2/2] Re: Temporarily delete old-style lock files
>Message-ID: <8762lpvjue.fsf at SSpaeth.de>
>Content-Type: text/plain; charset="us-ascii"
>
>On Fri, 19 Aug 2011 18:35:23 +0200, Nicolas Sebrecht <nicolas.s-dev at laposte.net> wrote:
>> > Doing all this without race conditions is something that will not be
>> > trivial. The exploitable window will be quite small, but it will exist
>> > unless we do really smart stuff.
>>
>> I'm not seeing possible race conditions in the above.
>
>1st we check for new-style locks. None there, next, we check for
>old-style lock. But in the mean time, we could have another new-style
>lock created, etc. Just locks of little places between testing for and
>creating the various lock files which makes me a little nervous.
>
>> > How about: 1) We start writing out the new per-account lock but we still
>> > honor the old global lock.
>> > We keep this dual system for 1-2 stable releases before ditching the old
>> > lock system. This way we can still keep subsequent stable releases
>> > running without danger. At some point we can then say: "offlineimap 6.3.10
>> > will not honor the locking system of 6.3.5 anymore". This sounds like a
>> > more backward-compatible approach without too much ballast.
>>
>> I'm fine with that, too. I have in mind to move to 6.4 or 7.0 (maybe
>> changing releases for X.Y). I still wait because I'd like a test suites.
>> It would be also the time to do such lock migration.
>
>This strategy is much easier to implement and I propose a patch series
>that does:
>#1 Implement new-per account lock, ripping out the old lock system
>#2 Re-implement old-style lock with comments, in a few versions this
> patch needs to be reverted to simply ditch the old lock system and
> just have the new one. We also throw an OfflineImap Error when we
> detect such a lock.
>#3 Improve the error logging when we detect an OfflineImapError in the
> main thread by using our new ui.error() infrastructure.
>
>We can leave that in for 1-2 releases and then release a 6.4 or 7.0 or
>whatever and ditch the legacy locking support.
>
>Sebastian
>-------------- next part --------------
>A non-text attachment was scrubbed...
>Name: not available
>Type: application/pgp-signature
>Size: 197 bytes
>Desc: not available
>URL: <http://lists.alioth.debian.org/pipermail/offlineimap-project/attachments/20110822/2941c180/attachment-0001.pgp>
>
>------------------------------
>
>Message: 5
>Date: Mon, 22 Aug 2011 12:12:05 +0200
>From: Sebastian Spaeth <Sebastian at SSpaeth.de>
>To: offlineimap-project at lists.alioth.debian.org
>Cc: Nicolas Sebrecht <nicolas.s-dev at laposte.net>, Sebastian Spaeth
> <Sebastian at SSpaeth.de>
>Subject: [PATCH 1/3] Per-account locking
>Message-ID: <1314007927-21637-1-git-send-email-Sebastian at SSpaeth.de>
>
>Previously, we were simply locking offlineimap whenever it was
>running. Howver there is no reason why we shouldn't be able to invoke it
>in parallel, e.g. to synchronize several accounts in one offlineimap
>each.
>
>This patch implements the locking per-account, so that it is possible to
>sync different accounts at the same time. If in refresh mode, we will
>attempt to loop three times before giving up.
>
>This also fixes Debian bug #586655
>
>Signed-off-by: Sebastian Spaeth <Sebastian at SSpaeth.de>
>---
> Changelog.draft.rst | 7 +++++++
> offlineimap/accounts.py | 36 ++++++++++++++++++++++++++++++++++++
> offlineimap/init.py | 20 --------------------
> 3 files changed, 43 insertions(+), 20 deletions(-)
>
>diff --git a/Changelog.draft.rst b/Changelog.draft.rst
>index 74b6368..3da9689 100644
>--- a/Changelog.draft.rst
>+++ b/Changelog.draft.rst
>@@ -17,6 +17,13 @@ New Features
> synchronization, but only skip that message, informing the user at the
> end of the sync run.
>
>+* Implement per-account locking, so that it will possible to sync
>+ different accounts at the same time. The old global lock is still in
>+ place for backward compatibility reasons (to be able to run old and
>+ new versions of OfflineImap concurrently) and will be removed in the
>+ future. Starting with this version, OfflineImap will be
>+ forward-compatible with the per-account locking style.
>+
> Changes
> -------
>
>diff --git a/offlineimap/accounts.py b/offlineimap/accounts.py
>index 31ea2b7..e3dad0e 100644
>--- a/offlineimap/accounts.py
>+++ b/offlineimap/accounts.py
>@@ -25,6 +25,11 @@ import os
> from sys import exc_info
> import traceback
>
>+try:
>+ import fcntl
>+except:
>+ pass # ok if this fails, we can do without
>+
> def getaccountlist(customconfig):
> return customconfig.getsectionlist('Account')
>
>@@ -159,6 +164,35 @@ class SyncableAccount(Account):
> functions :meth:`syncrunner`, :meth:`sync`, :meth:`syncfolders`,
> used for syncing."""
>
>+ def __init__(self, *args, **kwargs):
>+ Account.__init__(self, *args, **kwargs)
>+ self._lockfd = None
>+ self._lockfilepath = os.path.join(self.config.getmetadatadir(),
>+ "%s.lock" % self)
>+
>+ def lock(self):
>+ """Lock the account, throwing an exception if it is locked already"""
>+ self._lockfd = open(self._lockfilepath, 'w')
>+ try:
>+ fcntl.lockf(self._lockfd, fcntl.LOCK_EX|fcntl.LOCK_NB)
>+ except NameError:
>+ #fcntl not available (Windows), disable file locking... :(
>+ pass
>+ except IOError:
>+ self._lockfd.close()
>+ raise OfflineImapError("Could not lock account %s." % self,
>+ OfflineImapError.ERROR.REPO)
>+
>+ def unlock(self):
>+ """Unlock the account, deleting the lock file"""
>+ #If we own the lock file, delete it
>+ if self._lockfd and not self._lockfd.closed:
>+ self._lockfd.close()
>+ try:
>+ os.unlink(self._lockfilepath)
>+ except OSError:
>+ pass #Failed to delete for some reason.
>+
> def syncrunner(self):
> self.ui.registerthread(self.name)
> self.ui.acct(self.name)
>@@ -175,6 +209,7 @@ class SyncableAccount(Account):
> while looping:
> try:
> try:
>+ self.lock()
> self.sync()
> except (KeyboardInterrupt, SystemExit):
> raise
>@@ -194,6 +229,7 @@ class SyncableAccount(Account):
> if self.refreshperiod:
> looping = 3
> finally:
>+ self.unlock()
> if looping and self.sleeper() >= 2:
> looping = 0
> self.ui.acctdone(self.name)
>diff --git a/offlineimap/init.py b/offlineimap/init.py
>index 93b7224..f7b2eef 100644
>--- a/offlineimap/init.py
>+++ b/offlineimap/init.py
>@@ -30,14 +30,6 @@ from offlineimap.ui import UI_LIST, setglobalui, getglobalui
> from offlineimap.CustomConfig import CustomConfigParser
>
>
>-try:
>- import fcntl
>- hasfcntl = 1
>-except:
>- hasfcntl = 0
>-
>-lockfd = None
>-
> class OfflineImap:
> """The main class that encapsulates the high level use of OfflineImap.
>
>@@ -46,17 +38,6 @@ class OfflineImap:
> oi = OfflineImap()
> oi.run()
> """
>- def lock(self, config, ui):
>- global lockfd, hasfcntl
>- if not hasfcntl:
>- return
>- lockfd = open(config.getmetadatadir() + "/lock", "w")
>- try:
>- fcntl.flock(lockfd, fcntl.LOCK_EX | fcntl.LOCK_NB)
>- except IOError:
>- ui.locked()
>- ui.terminate(1)
>-
> def run(self):
> """Parse the commandline and invoke everything"""
>
>@@ -253,7 +234,6 @@ class OfflineImap:
> config.set(section, "folderfilter", folderfilter)
> config.set(section, "folderincludes", folderincludes)
>
>- self.lock(config, ui)
> self.config = config
>
> def sigterm_handler(signum, frame):
>--
>1.7.4.1
>
>
>
>
>------------------------------
>
>Message: 6
>Date: Mon, 22 Aug 2011 12:12:06 +0200
>From: Sebastian Spaeth <Sebastian at SSpaeth.de>
>To: offlineimap-project at lists.alioth.debian.org
>Cc: Nicolas Sebrecht <nicolas.s-dev at laposte.net>, Sebastian Spaeth
> <Sebastian at SSpaeth.de>
>Subject: [PATCH 2/3] Perform legacy global lock in addition to new
> per-account lock
>Message-ID: <1314007927-21637-2-git-send-email-Sebastian at SSpaeth.de>
>
>We simply lock OfflineImap the same global way that we have always done
>in addition to the previously implemented per-account lock. We can keep
>both systems in parallel and then after a few stable releases, drop the
>old-style global lock. by reverting this patch
>
>Signed-off-by: Sebastian Spaeth <Sebastian at SSpaeth.de>
>---
> offlineimap/accounts.py | 1 +
> offlineimap/init.py | 19 ++++++++++++++++++-
> 2 files changed, 19 insertions(+), 1 deletions(-)
>
>diff --git a/offlineimap/accounts.py b/offlineimap/accounts.py
>index e3dad0e..67194f4 100644
>--- a/offlineimap/accounts.py
>+++ b/offlineimap/accounts.py
>@@ -172,6 +172,7 @@ class SyncableAccount(Account):
>
> def lock(self):
> """Lock the account, throwing an exception if it is locked already"""
>+ # Take a new-style per-account lock
> self._lockfd = open(self._lockfilepath, 'w')
> try:
> fcntl.lockf(self._lockfd, fcntl.LOCK_EX|fcntl.LOCK_NB)
>diff --git a/offlineimap/init.py b/offlineimap/init.py
>index f7b2eef..e5560a9 100644
>--- a/offlineimap/init.py
>+++ b/offlineimap/init.py
>@@ -24,8 +24,13 @@ import signal
> import socket
> import logging
> from optparse import OptionParser
>+try:
>+ import fcntl
>+except ImportError:
>+ pass #it's OK
> import offlineimap
> from offlineimap import accounts, threadutil, syncmaster
>+from offlineimap.error import OfflineImapError
> from offlineimap.ui import UI_LIST, setglobalui, getglobalui
> from offlineimap.CustomConfig import CustomConfigParser
>
>@@ -310,6 +315,18 @@ class OfflineImap:
> #various initializations that need to be performed:
> offlineimap.mbnames.init(config, syncaccounts)
>
>+ #TODO: keep legacy lock for a few versions, then remove.
>+ self._legacy_lock = open(self.config.getmetadatadir() + "/lock",
>+ 'w')
>+ try:
>+ fcntl.lockf(self._legacy_lock, fcntl.LOCK_EX|fcntl.LOCK_NB)
>+ except NameError:
>+ #fcntl not available (Windows), disable file locking... :(
>+ pass
>+ except IOError:
>+ raise OfflineImapError("Could not take global lock.",
>+ OfflineImapError.ERROR.REPO)
>+
> if options.singlethreading:
> #singlethreaded
> self.sync_singlethreaded(syncaccounts, config)
>@@ -329,7 +346,7 @@ class OfflineImap:
> return
> except (SystemExit):
> raise
>- except:
>+ except Exception, e:
> ui.mainException()
>
> def sync_singlethreaded(self, accs, config):
>--
>1.7.4.1
>
>
>
>
>------------------------------
>
>_______________________________________________
>OfflineIMAP-project mailing list
>OfflineIMAP-project at lists.alioth.debian.org
>http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/offlineimap-project
>
>End of OfflineIMAP-project Digest, Vol 23, Issue 54
>***************************************************
More information about the OfflineIMAP-project
mailing list