[PATCH/RFC] New feature: Update folder structure to new config parameters.
Hubert Pineault
hpineault at riseup.net
Sat Mar 2 09:22:21 GMT 2019
>From 38a9e7de795fff5733a8df5c4fa3b3ea8cf8a602 Mon Sep 17 00:00:00 2001
From: Hubert Pineault <hpineault at riseup.net> Date: Thu, 28 Feb 2019
01:20:00 -0500 Subject: [PATCH 1/7] New feature: Update folder structure
to new config parameters.
Motivation:
If you change some of the config file paremeters, like nametrans and
utf8 decode, it will completly mess your maildir folder
structure. So, to make changes in config file, you need to
re-download the whole imap account. If it weights a few gigs, it's a
bit annoying. With the coming of features like remote folder
creation and utf8 decoding of folder names, a converting tool could
be handy for some of us.
Introduction:
The objectif is to convert an actual maildir folder structure to a
new one. There is still a farely good amount of work to do in order
to make it ready to merge with main branch. I have already
identified some changes that should be made in the update
process. Those points are raised in details related commits'
messages. I will need some help with cleaning the code and, notably,
exceptions handling.
The three main points that will need a complete rewritting are:
- The invoking method (actually, we load two independent config
files)
- The way different config parameters are handled by objects
- (actually, we use two accounts, so two remote, local and status
repos)
- Instead of altering existing class, it could be a better idea to
create a child class specific for updating.
Patch content (7 commits):
- The one your reading. Only a commit message. Load up new config
- file to which we're uploading Invoke updating process Various new
- methods invoked by the update-process Alterations to existing
- methods needed by the update process Main update method. Loop
- through each account, and prepare for update. Get content from
- old folder structure and copy it to new structure
Signed-off-by: Hubert Pineault <hpineault at riseup.net>
--
2.11.0
>From 8ee4fe52c6b142323dd0ba9f51c545b0eaaa826f Mon Sep 17 00:00:00 2001
From: Hubert Pineault <hpineault at riseup.net> Date: Wed, 30 Jan 2019
00:46:25 -0500 Subject: [PATCH 2/7] Load up new config file to which
we're uploading
Adds three command options:
--update-conf [new config file] move-content : simpy move the files
--instead of copying message restore-update [aborted update dir]
Logic: Loads the new config file the same way it's done for regular
config. If --restore-update is provided, check that the dir exists.
Set the same command options for the new config file. Initialize some
vars that will be used in the update process.
Discussion: I'm thinking of completly changing the way the update
process takes its input. Instead of loading a new config file, this
could be done in a single config files with parameters for updating to
new nametrans, folderfilter and encoding. This could could open new
possibilities to improve performence of the updating process. It could
also make the
--restore-update cmd option easier to use, since we can record its dir
in the config file.
Signed-off-by: Hubert Pineault <hpineault at riseup.net>
---
offlineimap/init.py | 63
+++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed,
63 insertions(+)
diff --git a/offlineimap/init.py b/offlineimap/init.py index
80158bd..b36db25 100644
--- a/offlineimap/init.py
+++ b/offlineimap/init.py @@ -97,6 +97,8 @@ class OfflineImap(object):
mbnames.write() elif options.deletefolder:
return self.__deletefolder(options) + elif
options.newconfigfile: + return self.__sync(options)
else:
return self.__sync(options)
@@ -160,6 +162,26 @@ class OfflineImap(object):
metavar="[section:]option=value", help="override
configuration file option")
+ parser.add_option("--update-conf", dest="newconfigfile", +
metavar="NEWCONFIGFILE", + default=None, + help="(EXPERIMENTAL: do not
support changes in account names)" + "convert actual config to new
config from source file," + "and replace configfile with sourcefile.")
+ + parser.add_option("--restore-update", dest="failedupdaterestore", +
metavar="FAILEDUPDATERESTORE", + default=None, + help="in conjonction
with --update-conf, " + "restore a failed update process.") + +
parser.add_option("--move-content", + action="store_true", +
dest="movecontent", + default=False, + help="in conjonction with
--update-conf, " + "move content of folder instead of copying.") +
parser.add_option("-o",
action="store_true", dest="runonce", default=False,
@@ -246,6 +268,47 @@ class OfflineImap(object):
section = "general" config.set(section, key, value)
+ # Update and convert config file. + if options.newconfigfile: +
newconfigfilename = os.path.expanduser(options.newconfigfile) + + # Read
new configfile + newconfigfile = CustomConfigParser() + if not
os.path.exists(newconfigfilename): + # TODO, initialize and make use of
chosen ui for logging + logging.error(" *** New config file '%s' does
not exist; aborting!"% + newconfigfilename) + sys.exit(1) +
newconfigfile.read(newconfigfilename) + + if options.dryrun: + dryrun =
newconfigfile.set('general', 'dry-run', 'True') +
newconfigfile.set_if_not_exists('general', 'dry-run', 'False') + + if
options.failedupdaterestore: + failedupdaterestore =
os.path.expanduser(options.failedupdaterestore) + if not
os.path.exists(failedupdaterestore): + # TODO, initialize and make use
of chosen ui for logging + logging.error(" *** Failed update '%s' does
not exist; aborting!"% + failedupdaterestore) + sys.exit(1) + else: +
failedupdaterestore = '' + + # Set update options in both config files +
# and initialize some vars. + self.config_filename = configfilename +
self.newconfig_filename = newconfigfilename +
newconfigfile.set('general', 'update-conf', 'True') +
newconfigfile.set('general', 'is-new-config-source', 'True') +
newconfigfile.set('general', 'movecontent', str(options.movecontent)) +
newconfigfile.set('general', 'failedupdaterestore', failedupdaterestore)
+ config.set('general', 'update-conf', 'True') + config.set('general',
'movecontent', str(options.movecontent)) + self.newconfig =
newconfigfile + config.set('general', 'is-new-config-source', 'False') +
config.set_if_not_exists('general', 'update-conf', 'False') +
# Which ui to use? CLI option overrides config file.
ui_type = config.getdefault('general', 'ui', 'ttyui') if
options.interface != None:
--
2.11.0
>From d0199b402f1d4f2d2e2220e08a11d0c287756dbd Mon Sep 17 00:00:00 2001
From: Hubert Pineault <hpineault at riseup.net> Date: Wed, 30 Jan 2019
01:06:36 -0500 Subject: [PATCH 3/7] Invoke updating process
The update process is invoked through init.__sync() method.
Logic: The idea is not to duplicate sig_handler and UI initializations.
Although, I think it should be rewritten in a distinct methode that
would called by run(). The sync process and the update process shouldn't
mixed.
Discussion: I'm thinking of two ways to improve. Either write a new
method that would deal with sig_handler and be called by both sync and
update processes. Or, simply copy __sync() code and adapt it for the
update process.
New method:
_newactiveaccounts()
The method is almost a copy/paste of _get_activeaccounts(). We first
make the list of accounts in the new config file. Then we check that
they exist in the old config file. It returns only accounts found in
both config files. I think this code should be included in another
method so that we use _get_activeaccounts() and then test config's files
accounts correspondance. (Note that the whole war the update process
takes its input through files should probably be re-thought)
Signed-off-by: Hubert Pineault <hpineault at riseup.net>
---
offlineimap/init.py | 51
++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 50
insertions(+), 1 deletion(-)
diff --git a/offlineimap/init.py b/offlineimap/init.py index
b36db25..e82d808 100644
--- a/offlineimap/init.py
+++ b/offlineimap/init.py @@ -480,6 +480,46 @@ class
OfflineImap(object):
return activeaccounts
+ def _get_newactiveaccounts(self, activeaccounts, options): + """Check
that each accounts in new config file exists in old config + file. +
Assumes that self.newconfig is defined. + Based on
_get_activeaccounts() + """ + + oldactiveaccounts = activeaccounts +
newactiveaccounts = [] + errormsg = None + + # Load accounts in new
config file source + newactiveaccountnames =
self.newconfig.get("general", "accounts") + if options.accounts: +
newactiveaccountnames = options.accounts + newactiveaccountnames =
[x.lstrip() + for x in newactiveaccountnames.split(",")] + +
allnewaccounts = accounts.getaccountlist(self.newconfig) + + # Check for
new config file accounts integrety + # (not sure if it does) and if
check if they exists + # in old config file + for accountname in
newactiveaccountnames: + if accountname in allnewaccounts \ + and
accountname in oldactiveaccounts: +
newactiveaccounts.append(accountname) + else: + errormsg = "Valid
accounts are: %s"% ( + ", ".join(oldactiveaccounts)) +
self.ui.error("The account '%s' does not exist"% accountname) + + if
len(activeaccounts) < 1: + errormsg = "No accounts are defined!" + + if
errormsg is not None: + self.ui.terminate(1, errormsg=errormsg) + +
return newactiveaccounts +
def __sync(self, options):
"""Invoke the correct single/multithread syncing
@@ -526,7 +566,16 @@ class OfflineImap(object):
activeaccounts = self._get_activeaccounts(options)
mbnames.init(self.config, self.ui, options.dryrun)
- if options.singlethreading:
+ if options.newconfigfile: + # Update directory structure and folder
names + # instead of syncing. + newactiveaccounts =
self._get_newactiveaccounts(activeaccounts, + options) +
mbnames.init(self.newconfig, self.ui, options.dryrun) +
self.__updateconf(activeaccounts, newactiveaccounts, +
options.profiledir) + + elif options.singlethreading:
# Singlethreaded.
self.__sync_singlethreaded(activeaccounts,
options.profiledir) else:
--
2.11.0
>From 647c6c215cc2b8f9f211eec881ff93501a796557 Mon Sep 17 00:00:00 2001
From: Hubert Pineault <hpineault at riseup.net> Date: Sun, 3 Feb 2019
05:54:57 -0500 Subject: [PATCH 4/7] Various new methods invoked by the
update-process
The main update process is not in this commit and all alteration of
existing methods will be subtmitted in distinct commits. The methods
found in this commit are of minor importance. Here is the list and
explanation.
folder.IMAP.IMAPFolder.getimapname():
Simply returns 'self.imap', which is a proprety containing the
original imap response before decoding in utf8.
repository.MailDir.moveroot(newroot, update_mbnames):
Move the local maildir root folder to another location in the
filesystem and set self.root accordingly. The method is used to move a
maildir root folder to a backup of temporary folder.
repository.MailDir.getmessagefilename(uid):
return filename...
repository.Base.BaseRepository.getmetadata():
Add the new property self.metadatadir. Return a tuple of
self.metadatadir, self.mapdir and self.uiddir. Initialize properties
if it's needed. This method is implemented in order to be called by
account.movemetadata). Since mapdir and uiddir initilization are done
in this method, BaseRepository.__init__() will be modified in another
commit.
repository.LocalStatus.LocalStatusRepository.getmetadata():
Same idea as BaseRepository.getmetadata(), but we need a specific
method for status repos.
ui.UIBase: there's 10 new methods for printing output.
account.movemetadatadir(newmetadatadir, update_mbnames):
Move the metadata root folder to another location in the filesystem
and call account.getmetadata() to set account proprieties acconrdingly
. The method is used to move a metadata to a backup of temporary
folder. Note that this method implies some code alteration in
accout.init() and account.getmetadata()
update_mbnames: I still need to figure out how mbnames work. Not Sure
if this will be usefull. Some help please?
account.get_folderlist():
Initialize remote and local repos.
To make the link between local folders and remote folders, old and new
config files need to have their own remoterepos. Although, it
shouldn't have to send two request to imap server. I haven't found a
way to skip the imap list cmd for the new config file while keeping it
from pointing to the old config file remote repo. One way to could be
to write a whole new logic of initialysing repositories. This methods
would need to get imap result and pass it to the new remoterepos.
Signed-off-by: Hubert Pineault <hpineault at riseup.net>
---
offlineimap/accounts.py | 123 +++++++++++++++++++++++++++++++---
offlineimap/folder/IMAP.py | 3 + offlineimap/folder/Maildir.py | 4 +-
offlineimap/repository/Base.py | 18 +++++
offlineimap/repository/LocalStatus.py | 24 +++++++
offlineimap/repository/Maildir.py | 24 +++++++ offlineimap/ui/UIBase.py
| 85 +++++++++++++++++++++++ 7 files changed, 272 insertions(+), 9
deletions(-)
diff --git a/offlineimap/accounts.py b/offlineimap/accounts.py index
4aca7c4..0ae8b0d 100644
--- a/offlineimap/accounts.py
+++ b/offlineimap/accounts.py @@ -257,7 +257,8 @@ class
SyncableAccount(Account):
pass # Failed to delete for some reason.
def syncrunner(self):
- """The target for both single and multi-threaded modes."""
+ """The target for three mode: + single mode, multi-threaded mode and
update-conf mode."""
self.ui.registerthread(self) try:
@@ -279,8 +280,13 @@ class SyncableAccount(Account):
while looping:
self.ui.acct(self) try:
+ # if not self.config.getboolean('general', 'is-new-config-source'):
self.__lock()
- self.__sync()
+ if self.config.getboolean('general', 'update-conf'): +
self.get_folderlist() + else: + self.__sync() +
except (KeyboardInterrupt, SystemExit):
raise except OfflineImapError as e:
@@ -292,17 +298,26 @@ class SyncableAccount(Account):
raise self.ui.error(e, exc_info()[2])
except Exception as e:
- self.ui.error(e, exc_info()[2], msg=
- "While attempting to sync account '%s'"% self)
+ if self.config.newconfigsource: + self.ui.error(e, exc_info()[2], msg=
+ "While attempting to update account '%s'"% self) + else: +
self.ui.error(e, exc_info()[2], msg= + "While attempting to sync account
'%s'"% self)
else:
# After success sync, reset the looping counter to 3.
if self.refreshperiod:
looping = 3 finally:
self.ui.acctdone(self)
- self._unlock() if looping and self._sleeper() >= 2:
+ # If in update-conf mode, keep original account locked + # and don't
run sleeper + if self.config.getboolean('general', 'update-conf'):
looping = 0 + else: + self._unlock() + if looping
and self._sleeper() >= 2: + looping = 0
def get_local_folder(self, remotefolder):
"""Return the corresponding local folder for a given
remotefolder.""" @@ -312,8 +327,8 @@ class SyncableAccount(Account):
replace(self.remoterepos.getsep(),
self.localrepos.getsep()))
- # The syncrunner will loop on this method. This means it is called
- # more than once during the run.
+ # The syncrunner will loop on this method (unless in update-conf
mode). + # This means it is called more than once during the run.
def __sync(self):
"""Synchronize the account once, then return.
@@ -452,6 +467,98 @@ class SyncableAccount(Account):
except Exception as e:
self.ui.error(e, exc_info()[2], msg="Calling hook")
+ ######## + ### Config update methodes: + # + def movemetadatadir(self,
newmetadatadir, update_mbnames=False): + '''Move metadatadir to a new
path. + + This function is called by the config updating process. + +
TODO: update mbnames according to new root''' + +
self.ui.movemetadatadir(self.name, newmetadatadir) + if update_mbnames:
+ self.ui.warn("Updating mbname for moving metadatadir NOT YET
IMPLEMENTED") + # raise + #XXX TODO + if self.dryrun: + return + + try:
+ from shutil import move + # Get list of metadata objects to move to
backup folder + movables = [] + movables.append(self.getaccountmeta()) +
if self._lockfilepath: + movables.append(self._lockfilepath) +
movables.append(self.localrepos.metadatadir) +
movables.append(self.remoterepos.metadatadir) + + # Set path for new
meta + self.config.set('general', 'metadata', newmetadatadir) +
self.metadatadir = self.config.getmetadatadir() + + # Move metadata +
for m in movables: + basename = os.path.basename(m) + move(m,
os.path.join(newmetadatadir, basename)) + + self._lockfilepath =
os.path.join( + newmetadatadir, + os.path.basename(self._lockfilepath))
+ self.metadatadir = newmetadatadir + self.localrepos.metadatadir = None
+ self.localrepos.getmetadata() + self.remoterepos.metadatadir = None +
self.remoterepos.getmetadata() + self.statusrepos.metadatadir = None +
self.statusrepos.getmetadata() + + except OSError or OfflineImapError as
e: + self.ui.error(e, exc_info()[2], + "Moving metadatadir to '%s'"% +
(newmetadatadir)) + raise + + # The syncrunner will loop on this
method. This means it is called more than + # once during the run. +
def get_folderlist(self): + """Get the account folderlist once, then
return. + + Assumes that `self.remoterepos`, `self.localrepos`, and +
`self.statusrepos` has already been populated, so it should only + be
called from the :meth:`syncrunner` function. + + Based on __sync()""" +
+ hook = self.getconf('presynchook', '') # Is it important? +
self.callhook(hook) + + if self.utf_8_support and
self.remoterepos.getdecodefoldernames(): + raise
OfflineImapError("Configuration mismatch in account " + + "'%s'. "%
self.getname() + + "\nAccount setting 'utf8foldernames' and repository "
+ + "setting 'decodefoldernames'\nmay not be used at the " + + "same
time. This account has not been updated.\n" + + "Please check the
configuration and documentation.", + OfflineImapError.ERROR.REPO) + +
try: + self.remoterepos.getfolders() +
self.remoterepos.dropconnections() + self.localrepos.getfolders() + +
#XXX: This section should be checked and + # rewritten (copy-pasted from
callhook to handle try) + except (KeyboardInterrupt, SystemExit): +
raise + except Exception as e: + self.ui.error(e, exc_info()[2],
msg="Calling hook") + + hook = self.getconf('postupdateconfhook', '') #
Is this right? + self.callhook(hook) +
#XXX: This function should likely be refactored. This should not be
#passed the
# account instance.
diff --git a/offlineimap/folder/IMAP.py b/offlineimap/folder/IMAP.py
index ead4396..3a716e4 100644
--- a/offlineimap/folder/IMAP.py
+++ b/offlineimap/folder/IMAP.py @@ -96,6 +96,9 @@ class
IMAPFolder(BaseFolder):
name = imaputil.utf8_IMAP(name) return name
+ def getimapname(self): + return self.imap_name +
# Interface from BaseFolder
def suggeststhreads(self):
singlethreadperfolder_default = False diff --git
a/offlineimap/folder/Maildir.py b/offlineimap/folder/Maildir.py index
f061bcd..795f4d3 100644
--- a/offlineimap/folder/Maildir.py
+++ b/offlineimap/folder/Maildir.py @@ -295,7 +295,6 @@ class
MaildirFolder(BaseFolder):
uid, self._foldermd5, self.infosep, ''.join(sorted(flags)))
return uniq_name.replace(os.path.sep, self.sep_subst)
-
def save_to_tmp_file(self, filename, content):
"""Saves given content to the named temporary file in the 'tmp'
subdirectory of $CWD.
@@ -416,6 +415,9 @@ class MaildirFolder(BaseFolder):
self.ui.debug('maildir', 'savemessage: returning uid %d' % uid)
return uid
+ def getmessagefilename(self, uid): + return
self.messagelist[uid]['filename'] +
# Interface from BaseFolder
def getmessageflags(self, uid):
return self.messagelist[uid]['flags'] diff --git
a/offlineimap/repository/Base.py b/offlineimap/repository/Base.py index
2ad7708..60f926c 100644
--- a/offlineimap/repository/Base.py
+++ b/offlineimap/repository/Base.py @@ -63,6 +63,24 @@ class
BaseRepository(CustomConfig.ConfigHelperMixin):
self.foldersort = self.localeval.eval(
self.getconf('foldersort'), {'re': re})
+ def getmetadata(self): + if self.metadatadir and self.mapdir and
self.uiddir: + return self.metadatadir, self.mapdir, self.uiddir + else:
+ self.metadatadir = os.path.join(self.config.getmetadatadir(), +
'Repository-' + self.name) + if not os.path.exists(self.metadatadir): +
os.mkdir(self.metadatadir, 0o700) + self.mapdir =
os.path.join(self.metadatadir, 'UIDMapping') + if not
os.path.exists(self.mapdir): + os.mkdir(self.mapdir, 0o700) + # FIXME:
self.uiddir variable name is lying about itself. + # (Still true with
this new method???) + self.uiddir = os.path.join(self.metadatadir,
'FolderValidity') + if not os.path.exists(self.uiddir): +
os.mkdir(self.uiddir, 0o700) + return self.metadatadir, self.mapdir,
self.uiddir +
def restore_atime(self):
"""Sets folders' atime back to their values after a sync
diff --git a/offlineimap/repository/LocalStatus.py
b/offlineimap/repository/LocalStatus.py index a651fd2..4ad2315 100644
--- a/offlineimap/repository/LocalStatus.py
+++ b/offlineimap/repository/LocalStatus.py @@ -53,6 +53,30 @@ class
LocalStatusRepository(BaseRepository):
# self._folders is a dict of name:LocalStatusFolders().
self._folders = {}
+ def getmetadata(self): + # class and root for all backends. +
self.backends = {} + self.backends['sqlite'] = { + 'class':
LocalStatusSQLiteFolder, + 'root':
os.path.join(self.account.getaccountmeta(), 'LocalStatus-sqlite') + } +
self.backends['plain'] = { + 'class': LocalStatusFolder, + 'root':
os.path.join(self.account.getaccountmeta(), 'LocalStatus') + } + + if
self.account.getconf('status_backend', None) is not None: + raise
OfflineImapError( + "the 'status_backend' configuration option is not
supported" + " anymore; please, remove this configuration option.", +
OfflineImapError.ERROR.REPO + ) + # Set class and root for sqlite. +
self.setup_backend('sqlite') + + if not os.path.exists(self.root): +
os.mkdir(self.root, 0o700) +
def _instanciatefolder(self, foldername):
return self.LocalStatusFolderClass(foldername, self) #
Instanciate.
diff --git a/offlineimap/repository/Maildir.py
b/offlineimap/repository/Maildir.py index 0db728c..5c2685d 100644
--- a/offlineimap/repository/Maildir.py
+++ b/offlineimap/repository/Maildir.py @@ -130,6 +130,30 @@ class
MaildirRepository(BaseRepository):
else:
raise
+ def moveroot(self, newroot, update_mbnames = False): + '''Move local
repository root folder to a new path. + + This function is called by
the config updating process. + + TODO: update mbnames according to new
root''' + + self.ui.moveroot(self.name, newroot) + if
self.account.dryrun: + return + try: + os.renames(self.root, newroot) +
self.root = newroot + except OSError or OfflineImapError as e: +
self.ui.error(e, exc_info()[2], + "Moving root from '%s' to '%s'"% +
(self.root, newroot)) + raise + + if update_mbnames: +
self.ui.warn("Updating mbname for moving root folder NOT YET
IMPLEMENTED") + # raise + #XXX TODO +
def deletefolder(self, foldername):
self.ui.warn("NOT YET IMPLEMENTED: DELETE FOLDER %s"%
foldername)
diff --git a/offlineimap/ui/UIBase.py b/offlineimap/ui/UIBase.py index
731d6f1..f46eb35 100644
--- a/offlineimap/ui/UIBase.py
+++ b/offlineimap/ui/UIBase.py @@ -344,6 +344,91 @@ class
UIBase(object):
self.debug('', "Copying folder structure from %s to %s" %\
(src_repo, dst_repo))
+ ############################## Config updating + + def
updateconf(self, oldconf, newconf): + """Output that we start the
updating process.""" + + self.acct_startimes['updateconf'] =
time.time() + self.logger.info("*** Updating config file %s from source
%s"% + (oldconf, newconf)) + + def updateconfdone(self): + """Output
that we finished the updating process.""" + + sec = time.time() -
self.acct_startimes['updateconf'] + del
self.acct_startimes['updateconf'] + self.logger.info("*** Finished
updating config. Total time elapsed: %d:%02d"% + (sec // 60, sec % 60))
+ + def updateconfacct(self, account): + """Output that we start
updating folder structure for account.""" + +
self.acct_startimes['updateconfacct'] = time.time() +
self.logger.info("*** Updating folder structure for account '%s'"% +
account) + + def updateconfacctdone(self, account): + """Output that we
finished updating folder structure for account.""" + + sec =
time.time() - self.acct_startimes['updateconfacct'] + del
self.acct_startimes['updateconfacct'] + self.logger.info("*** Finished
account '%s' in %d:%02d"% + (account, sec // 60, sec % 60)) + + def
getfoldercontent(self, oldfolder, newfolder, move=False): + """Log
'Moving folder content...'.""" + +
self.acct_startimes['getfoldercontent'] = time.time() + prefix = "Moving
" if move else "Copying " + prefix = "[DRYRUN] " + prefix if self.dryrun
else prefix + self.info(('{0}folder content from "{1}" to "{2}"'.format(
+ prefix, oldfolder, newfolder))) + + def nbmessagestosend(self,
folder_basename, nbofmessages, movecontent): + """ Output the number of
messages (files) this operation will move.""" + + indent =
'{:>4}'.format('') + prefix = "[DRYRUN] " if self.dryrun else "" +
action = 'Moving ' if movecontent else 'Copying ' + if nbofmessages ==
0: + self.info("{0}Folder empty.".format(indent)) + else: +
self.info(("{0}{1}{2}{3} messages to folder '{4}'".format( + indent,
prefix, action, nbofmessages, folder_basename))) + + def
sendingmessage(self, uid, num, num_to_send, src, destfolder,
movecontent): + """Output a log line stating which message we copy."""
+ + indent = '{:>4}'.format('') + prefix = "[DRYRUN] " if self.dryrun
else "" + action = 'Moving ' if movecontent else 'Copying ' +
self.logger.debug("%s%s%s message UID %s (%d/%d) %s:%s -> %s:%s"% ( +
indent, prefix, action, uid, num, num_to_send, src.repository, src, +
destfolder.repository, destfolder)) + + def getfoldercontentdone(self):
+ """Output that we finished operation on folder content for old
folder.""" + + indent = '{:>4}'.format('') + sec = time.time() -
self.acct_startimes['getfoldercontent'] + del
self.acct_startimes['getfoldercontent'] + self.logger.info("%s Operation
completed in %d:%02d"% + (indent, sec // 60, sec % 60)) + + def
moveroot(self, account, newroot): + """Output that we are moving the
root folder for the account.""" + + prefix = "[DRYRUN] " if self.dryrun
else "" + self.logger.info("*** {0}Moving root folder for account '{1}'
to {2}".format( + prefix, account, newroot)) + + def
movemetadatadir(self, account, newmetadatadir): + """Output that we are
moving the metadatadir for the account.""" + + self.logger.info("***
Moving metadata directory for account '%s' to %s"% + (account,
newmetadatadir)) +
############################## Folder syncing
def makefolder(self, repo, foldername):
"""Called when a folder is created."""
--
2.11.0
>From e31fc24b7f827156f673ffb675f1f360ff5398d0 Mon Sep 17 00:00:00 2001
From: Hubert Pineault <hpineault at riseup.net> Date: Mon, 4 Feb 2019
00:22:00 -0500 Subject: [PATCH 5/7] Alterations to existing methods
needed by the update
process
account.SyncableAccount.syncrunner():
The update process uses two SyncableAccount instance (new and old) for
each account found in both config files. To initialize repos, we use
syncrunner in order to loop for imap connection and deal with
exceptions. When in an update process, syncrunner will call
self.get_folderlist(). Exceptions, error messages and loop handling
are altered to deal with updating.
I think this method should be left to its original state and the whole
repos initialization for the update process should be in a distinct
method.
folder.IMAP.IMAPFolder.__init__():
Add self.imap_name, which store original imap name returned in utf7
encoding.
With the implentation of utf8 decoding option (utf8foldernames), the
imap name gets replaced if the option is enabled. The update process
needs to know the original imap return (in utf7 encode) in order to
match a config file where the utf8foldernames is enabled with a config
file where it's disabled. (Note that the new methods self.getimapname
returns self.imap_name)
repository.Base.Baserepository.__init__():
Remove initialization of self.uiddir and self.mapdir. Call
self.getmedata() where initialization is handled and the new property
self.metadatadir is added.
repository.LocalStatus.LocalStatusRepository.__init__():
Same logic as for Baserepository.__init__(), but specific for
localrepos. Remove initialization of self.backend.
Signed-off-by: Hubert Pineault <hpineault at riseup.net>
---
offlineimap/folder/IMAP.py | 6 ++++-- offlineimap/repository/Base.py |
13 +++---------- offlineimap/repository/LocalStatus.py | 25
+------------------------ 3 files changed, 8 insertions(+), 36
deletions(-)
diff --git a/offlineimap/folder/IMAP.py b/offlineimap/folder/IMAP.py
index 3a716e4..ee7ab39 100644
--- a/offlineimap/folder/IMAP.py
+++ b/offlineimap/folder/IMAP.py @@ -49,9 +49,11 @@ class
IMAPFolder(BaseFolder):
# querying the IMAP server, while False is used when
# creating a folder object from a locally available utf_8
# name)
# In any case the given name is first dequoted.
- name = imaputil.dequote(name)
+ self.imap_name = imaputil.dequote(name) # For update-conf mode
if decode and repository.account.utf_8_support:
- name = imaputil.IMAP_utf8(name)
+ name = imaputil.IMAP_utf8(self.imap_name) + else: + name =
self.imap_name
self.sep = imapserver.delim super(IMAPFolder,
self).__init__(name, repository) if
repository.getdecodefoldernames():
diff --git a/offlineimap/repository/Base.py
b/offlineimap/repository/Base.py index 60f926c..0a58612 100644
--- a/offlineimap/repository/Base.py
+++ b/offlineimap/repository/Base.py @@ -34,16 +34,9 @@ class
BaseRepository(CustomConfig.ConfigHelperMixin):
self.localeval = account.getlocaleval() self._accountname =
self.account.getname() self._readonly =
self.getconfboolean('readonly', False)
- self.uiddir = os.path.join(self.config.getmetadatadir(),
- 'Repository-' + self.name) if not os.path.exists(self.uiddir):
- os.mkdir(self.uiddir, 0o700) self.mapdir =
- os.path.join(self.uiddir, 'UIDMapping') if not
- os.path.exists(self.mapdir):
- os.mkdir(self.mapdir, 0o700)
- # FIXME: self.uiddir variable name is lying about itself.
- self.uiddir = os.path.join(self.uiddir, 'FolderValidity') if
- not os.path.exists(self.uiddir):
- os.mkdir(self.uiddir, 0o700)
+ + self.mapdir = self.uiddir = self.metadatadir = None +
self.getmetadata()
self.nametrans = lambda foldername: foldername
self.folderfilter = lambda foldername: 1
diff --git a/offlineimap/repository/LocalStatus.py
b/offlineimap/repository/LocalStatus.py index 4ad2315..605093a 100644
--- a/offlineimap/repository/LocalStatus.py
+++ b/offlineimap/repository/LocalStatus.py @@ -26,30 +26,7 @@ from
offlineimap.error import OfflineImapError
class LocalStatusRepository(BaseRepository):
def __init__(self, reposname, account):
BaseRepository.__init__(self, reposname, account)
-
- # class and root for all backends.
- self.backends = {} self.backends['sqlite'] = {
- 'class': LocalStatusSQLiteFolder, 'root':
- os.path.join(account.getaccountmeta(),
- 'LocalStatus-sqlite')
- } self.backends['plain'] = {
- 'class': LocalStatusFolder, 'root':
- os.path.join(account.getaccountmeta(), 'LocalStatus')
- }
-
- if self.account.getconf('status_backend', None) is not None:
- raise OfflineImapError(
- "the 'status_backend' configuration option is not
- supported" " anymore; please, remove this configuration
- option.", OfflineImapError.ERROR.REPO
- )
- # Set class and root for sqlite.
- self.setup_backend('sqlite')
-
- if not os.path.exists(self.root):
- os.mkdir(self.root, 0o700)
-
+ self.getmetadata()
# self._folders is a dict of name:LocalStatusFolders().
self._folders = {}
--
2.11.0
>From 16af7c323f00233315e384f36c797d88da13d676 Mon Sep 17 00:00:00 2001
From: Hubert Pineault <hpineault at riseup.net> Date: Mon, 4 Feb 2019
19:16:41 -0500 Subject: [PATCH 6/7] Main update method. Loop through
each account, and
prepare for update.
For the update process, we use two SyncableAccount instance, one for
original config file (oldaccount) and one for the new config file
(newaccount). Before initializing newaccount, we need to move
oldaccount maildir root and metadata folder to a backup location. So
we first initialize old account to get folder names and we move
them. We then initialize newaccount, maildir and metadata folders will
be created automatically. Once both accounts are initialized, we sync
folder structure for newaccount and then call
newaccount.get_content_from_account() to procede to copying or moving
content from old folder structure to new folder structure.
The whole update process should be rewritten in a new class
UpdatableAccount inheriting from Account. The UpdatableAccount class
should have these repos defined: statusrepos, oldlocalrepos,
newlocalrepos, remoterepos. To avoid having to pass two imap list
command and two independent remoterepos, we will likely have to write
a new method to replace SyncableAccount.get_local_folder(remotefolder)
which seems incompatible with two sets of nametrans.
Signed-off-by: Hubert Pineault <hpineault at riseup.net>
---
offlineimap/folder/Maildir.py | 67 +++++++++++++++ offlineimap/init.py
| 188 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 255
insertions(+)
diff --git a/offlineimap/folder/Maildir.py
b/offlineimap/folder/Maildir.py index 795f4d3..bab8284 100644
--- a/offlineimap/folder/Maildir.py
+++ b/offlineimap/folder/Maildir.py @@ -551,3 +551,70 @@ class
MaildirFolder(BaseFolder):
self.ui.warn(("Inconsistent FMD5 for file `%s':"
" Neither `%s' nor `%s' found")
% (filename, oldfmd5, self._foldermd5))
+ + def sendcontentto(self, dstfolder, movecontent=False, dryrun=False):
+ from shutil import move, copy2 + + srcfolder = self + folderpath =
os.path.join(srcfolder.root, srcfolder.name) + + for d in
os.listdir(folderpath): + if os.path.isdir(os.path.join(folderpath, d)):
+ messages = os.listdir(os.path.join(folderpath, d)) + num = 0 +
totalnum = len(messages) +
self.ui.nbmessagestosend(os.path.join(dstfolder.name, d), + totalnum,
movecontent) + + for f in messages: + #TODO: Show progression + + if
os.path.isdir(os.path.join(folderpath, d, f)): + '''There should not be
any folder in here. If there + is, should we also deal with it? In a
sane folder + structure, this should not happen. Or am I wrong?''' + +
self.ui.info("A folder was found in source folder '{0}'" + "while trying
to move its content. Ignoring" + "'{1}'. This indicate that the
original" + "folder structure is probably corrupt in" + "some
way. Please investigate.", + format(os.path.join(srcfolder.name, d), f))
+ continue + + dstfile = os.path.join(dstfolder.root, dstfolder.name, d,
f) + if movecontent: + #TODO: Prevent or ask file overwrite +
move(os.path.join(folderpath, d, f), + dstfile) + else: + num += 1 + if
os.path.exists(dstfile): + self.ui.ignorecopyingmessage(f, srcfolder,
dstfolder) + continue + if not dryrun: + copy2(os.path.join(folderpath,
d, f), + dstfile) + + if (movecontent + and
len(os.listdir(os.path.join(folderpath, d))) == 0 + and not dryrun): +
os.rmdir(os.path.join(folderpath, d)) + elif movecontent and not dryrun:
+ self.ui.info("Folder {0} is not empty. Some files were not moved". +
format(os.path.join(folderpath, d))) + + else: + '''If there is a file
in srcfolder, should we also deal with it? In a + sane folder structure,
this should not happen. Or am I wrong? + move(os.path.join(folderpath,
d), + os.join(dstfolder.root, dstfolder.name, f))''' +
self.ui.ignorecopyingmessage(self._parse_filename(d)['UID'], +
srcfolder, dstfolder) + + if movecontent and len(os.listdir(folderpath))
== 0 \ + and not dryrun: + os.rmdir(folderpath) + elif movecontent and
not dryrun: + self.ui.info("Folder {0} is not empty. Some files were not
moved.". + format(folderpath)) diff --git a/offlineimap/init.py
b/offlineimap/init.py index e82d808..b31f7d1 100644
--- a/offlineimap/init.py
+++ b/offlineimap/init.py @@ -628,6 +628,194 @@ class
OfflineImap(object):
prof.dump_stats(os.path.join(
profiledir, "%s_%s.prof"% (dt, account.getname())))
+ def __updateconf(self, list_oldaccounts, list_newaccounts,
profiledir): + """Executed only in singlethreaded mode. + + For each
account found in new config source file, + get folder structure from
both files and change + local repository accordingly. + + Assumes that
list_newaccounts are only existing account + in old config file, and
that config_filename and newconfig_filename + are defined. + + :param
accs: A list of accounts that should be synced + """ + if profiledir: +
self.ui.error("Profile mode in config update is not implemented yet!")
+ # Profile mode. + raise NotImplementedError + +
self.ui.updateconf(self.config_filename, self.newconfig_filename) + + #
For each account in new config file, initiate both + # account (old and
new). Then set a temporary dir for + # the new folder structure + for
accountname in list_newaccounts: + updatedone = False +
self.ui.updateconfacct(accountname) + + try: + # Disable remote folder
creation + conf_account_remoterepos = 'Repository ' + \ +
self.newconfig.get("Account " + accountname, + 'remoterepository') +
self.config.set(conf_account_remoterepos, + 'createfolders', + 'False')
+ self.newconfig.set(conf_account_remoterepos, + 'createfolders', +
'False') + localrepos = 'Repository ' + \ + self.config.get("Account " +
accountname, + 'localrepository') + newconf_localfolders =
os.path.expanduser( + self.newconfig.get(localrepos, 'localfolders')) +
failedupdaterestore = os.path.expanduser( +
self.newconfig.get('general', 'failedupdaterestore')) + + # Set
temporary dir for dryrun. Otherwise, old and new accounts + # will point
to the same dir. + if self.newconfig.getboolean('general', 'dry-run'):
+ if failedupdaterestore: + newtmpmetadatadir = failedupdaterestore +
else: + tmpfolder_name = '.tmp-update-dryrun' + metadatadir =
os.path.expanduser(self.newconfig. + getdefault("general", +
"metadata", + "~/.offlineimap")) + tmpfolder = os.path.join(metadatadir,
+ tmpfolder_name) + if not os.path.exists(tmpfolder): +
os.makedirs(tmpfolder, 0o700) + newtmplocalfolders =
os.path.join(metadatadir, + tmpfolder, +
os.path.basename(newconf_localfolders)) + + newconf_localrepos =
'Repository ' + \ + self.config.get("Account " + accountname, +
'localrepository') + self.newconfig.set(newconf_localrepos, +
'localfolders', + newtmplocalfolders) + newtmpmetadatadir =
os.path.join(metadatadir, + tmpfolder, + 'metadata') +
self.newconfig.set('general', 'metadata', newtmpmetadatadir) + +
threading.currentThread().name = \ + "Account getfolder %s"% accountname
+ + ### Old account + oldaccount = accounts.SyncableAccount(self.config,
+ accountname) + oldaccount.syncrunner() + + ### Proceed to update + #
Check for CTRL-C or SIGTERM (not sure if it's ok). + if
oldaccount.abort_NOW_signal.is_set(): + break + + # Backup metadata and
point old account to it + metadatadir =
os.path.expanduser(oldaccount.metadatadir) + metadatabak =
os.path.join(metadatadir, + "UpdateBackup_" + accountname) +
oldaccount.movemetadatadir(metadatabak) + + # Move oldlocalrepo to a
backup folder + from datetime import datetime + updatetime =
datetime.today().strftime('%y%m%d.%H%M') + maildir =
oldaccount.localrepos.root + mailbak = self.getbackupname( +
os.path.join(metadatabak, '{0}.{1}'.format( + os.path.basename(maildir),
updatetime))) + oldaccount.localrepos.moveroot(mailbak) +
oldaccount.localrepos.forgetfolders() +
oldaccount.localrepos.getfolders() + + ### New account + if
failedupdaterestore \ + and not self.newconfig.getboolean('general',
'dry-run'): + from shutil import move +
move(os.path.join(failedupdaterestore, +
os.path.basename(newconf_localfolders)), + newconf_localfolders) + for
item in os.listdir(failedupdaterestore): + s =
os.path.join(failedupdaterestore, item) + d = os.path.join(metadatadir,
item) + move(s, d) + os.rmdir(failedupdaterestore) + + newaccount =
accounts.SyncableAccount(self.newconfig, + accountname) +
newaccount.syncrunner() + + # Check for CTRL-C or SIGTERM (not sure if
it's ok). + if newaccount.abort_NOW_signal.is_set(): + break + + #
Create the new folder structure +
newaccount.remoterepos.sync_folder_structure(newaccount.localrepos, +
newaccount.statusrepos) + # Moving old content into new structure +
newaccount.get_content_from_account(oldaccount) + + except: + try: +
newaccount + except: + pass + else: + # Backup failed update metadata +
failedmetadatabak = self.getbackupname( +
os.path.join(os.path.expanduser(newaccount.metadatadir), +
"FailedUpdate_{0}.{1}". + format(accountname, updatetime))) +
newaccount.movemetadatadir(failedmetadatabak) + # Backup failed update
maildir + mailbasename = os.path.basename(newaccount.localrepos.root) +
failedmaildirbak = os.path.join(failedmetadatabak, mailbasename) +
newaccount.localrepos.moveroot(failedmaildirbak) + try: + oldaccount +
except: + pass + else: + # Move back old account maildir and metadata to
their original dir + if oldaccount.metadatadir != metadatadir: +
oldaccount.movemetadatadir(metadatadir) + try: + oldaccount.localrepos +
except: + pass + else: + if oldaccount.localrepos.root != maildir: +
oldaccount.localrepos.moveroot(maildir) + raise + + finally: +
oldaccount._unlock() + newaccount._unlock() + if
self.newconfig.getboolean('general', 'dry-run') \ + and
os.path.exists(tmpfolder): + from shutil import rmtree + try: +
rmtree(tmpfolder) + except IOError: + raise #TODO Message error +
self.ui.updateconfacctdone(accountname) + + self.ui.updateconfdone() + +
def getbackupname(self, folder): + i = 0 + while os.path.exists(folder):
+ i += 1 + folder = '{0}.{1}'.format(folder, i) + return folder + +
def __serverdiagnostics(self, options):
self.ui.info(" imaplib2: %s (%s)"% (imaplib.__version__,
imaplib.DESC)) for accountname in
self._get_activeaccounts(options):
--
2.11.0
>From 4398c0b3df24c5d2e9ee3c4a467f1d9fc6b9a7b5 Mon Sep 17 00:00:00 2001
From: Hubert Pineault <hpineault at riseup.net> Date: Wed, 27 Feb 2019
22:59:35 -0500 Subject: [PATCH 7/7] Get content from old folder
structure and copy it to new
structure
Two new methods are added. this is where the actual update is done.
SyncableAccount.get_content_from_account(oldaccount, move):
Called by the newaccount to get content from oldaccount. Loop through
remote folders, skiping ignored folders and folders not yet synced in
oldaccount maildir structure. For each folder, get old and new local
folder names, then call self.__get_folder_content().
SyncableAccount.__get_folder_content(oldaccount, old_localfolder,
new_localfolder, new_remotefolder,
movecontent):
Initialize message list from both local folders (old and new) and run
folder.Base.syncmessagesto(). Unfortunatly, this is not much faster
than downloading the whole account from imap. I've tried writing a new
method for moving files instead of copying messages by adapting
__syncmessagesto_copy(), but it wasn't much faster either. I need to
know more about mbnames to improve performance. I little help on this
subject would be greatly appreciated.
Signed-off-by: Hubert Pineault <hpineault at riseup.net>
---
offlineimap/accounts.py | 134 ++++++++++++++++++++++++++++++++++
offlineimap/folder/Maildir.py | 165
+++++++++++++++++++++++++++--------------- 2 files changed, 239
insertions(+), 60 deletions(-)
diff --git a/offlineimap/accounts.py b/offlineimap/accounts.py index
0ae8b0d..6544b98 100644
--- a/offlineimap/accounts.py
+++ b/offlineimap/accounts.py @@ -559,6 +559,140 @@ class
SyncableAccount(Account):
hook = self.getconf('postupdateconfhook', '') # Is this right?
self.callhook(hook)
+ def get_content_from_account(self, oldaccount): + newaccount = self +
+ old_remote_hash, new_remote_hash = {}, {} + old_local_hash,
new_local_hash = {}, {} + + for folder in
oldaccount.remoterepos.getfolders(): +
old_remote_hash[folder.getimapname()] = folder + + for folder in
oldaccount.localrepos.getfolders(): + old_local_hash[folder.getname()] =
folder + + for folder in newaccount.remoterepos.getfolders(): +
new_remote_hash[folder.getimapname()] = folder + + for folder in
newaccount.localrepos.getfolders(): + new_local_hash[folder.getname()] =
folder + + # Loop through remote folder and get local folder
correspondance + for new_remote_imapname, new_remote_folder in
new_remote_hash.items(): + if not new_remote_folder.sync_this: +
self.ui.debug('', "Ignoring filtered folder in new config '%s'" +
"[%s]"% (new_remote_folder.getname(), + newaccount.remoterepos)) +
continue # Ignore filtered folder. + if not new_remote_imapname in
old_remote_hash.keys(): + self.ui.debug('', "Ignoring filtered folder in
old config '%s'" + "[%s]"% (new_remote_folder.getname(), +
newaccount.remoterepos)) + continue # Ignore filtered folder. + + #
Apply old remote nametrans and fix serparator. + old_remote_folder =
old_remote_hash[new_remote_imapname] + old_local_name =
old_remote_folder.getvisiblename().replace( +
oldaccount.remoterepos.getsep(), + oldaccount.localrepos.getsep()) + if
old_local_name not in old_local_hash.keys(): + self.ui.debug('',
"Ignoring unsynced folder '%s'" + "[%s]"% (new_remote_folder.getname(),
+ newaccount.remoterepos)) + continue # Ignore unsynced folder. +
old_local_folder = oldaccount.get_local_folder(old_remote_folder) + + #
Check for CTRL-C or SIGTERM (not sure if it's ok). + if
(oldaccount.abort_NOW_signal.is_set() + or
newaccount.abort_NOW_signal.is_set()): + break + + if not
newaccount.localrepos.getconfboolean('readonly', False): + if not
newaccount.dryrun: + new_local_folder =
newaccount.get_local_folder(new_remote_folder) + else: +
new_local_folder = new_remote_folder.getvisiblename().replace( +
newaccount.remoterepos.getsep(), + newaccount.localrepos.getsep()) +
self.__get_folder_content(oldaccount, old_local_folder, +
new_local_folder, new_remote_folder) + +
newaccount.localrepos.restore_atime() +
mbnames.writeIntermediateFile(self.name) # Write out mailbox names. + +
def __get_folder_content(self, oldaccount, old_localfolder, +
new_localfolder, new_remotefolder): + """Get the content from
old_local_folder""" + + newaccount = self + dststatusrepos =
newaccount.statusrepos + srcstatusrepos = oldaccount.statusrepos +
remoterepos = newaccount.remoterepos + movecontent =
self.config.getboolean('general', 'movecontent') + old_localfolder_name
= old_localfolder.name + new_localfolder_name = new_localfolder if
self.dryrun \ + else new_localfolder.name + +
newaccount.ui.getfoldercontent(old_localfolder_name, +
new_localfolder_name, + movecontent) + + if self.dryrun: +
newaccount.ui.getfoldercontentdone(old_localfolder_name) + return + + #
Load status folders. + srcstatusfolder =
srcstatusrepos.getfolder(old_localfolder_name. +
replace(old_localfolder.getsep(), + srcstatusrepos.getsep())) +
srcstatusfolder.openfiles() + dststatusfolder =
dststatusrepos.getfolder(new_remotefolder.getvisiblename(). +
replace(remoterepos.getsep(), dststatusrepos.getsep())) +
dststatusfolder.openfiles() + + #TODO: check that local folder does not
contain local sep + ''' # The remote folder names must not have the
local sep char in + # their names since this would cause troubles while
converting + # the name back (from local to remote). + sep =
localrepos.getsep() + if (sep != os.path.sep and + sep !=
remoterepos.getsep() and + sep in remotefolder.getname()): +
self.ui.warn('', "Ignoring folder '%s' due to unsupported " + "'%s'
character serving as local separator."% + (remotefolder.getname(),
localrepos.getsep())) + continue # Ignore unsupported folder name.''' +
+ try: + # Add the folder to the mbnames mailboxes. +
mbnames.add(newaccount.name, newaccount.localrepos.root, +
new_localfolder.getname()) + + # At this point, is this test necessary?
+ if not newaccount.localrepos.getconfboolean('readonly', False): +
old_localfolder.sendcontentto(new_localfolder, srcstatusfolder,
dststatusfolder, movecontent) + else: + self.ui.debug('', "Not sending
content to read-only repository '%s'"% +
newaccount.localrepos.getname()) + +
newaccount.localrepos.restore_atime() + + except (KeyboardInterrupt,
SystemExit): + raise + except Exception as e: + self.ui.error(e,
msg="ERROR while getting folder content for %s: %s"% +
(new_localfolder.getvisiblename(), traceback.format_exc())) + raise #
Raise unknown Exceptions so we can fix them. + else: +
newaccount.ui.getfoldercontentdone() + finally: + for folder in
["old_localfolder", "new_localfolder"]: + if folder in locals(): +
locals()[folder].dropmessagelistcache() + for folder in
["srcstatusfolder", "dststatusfolder"]: + if folder in locals(): +
locals()[folder].closefiles() +
#XXX: This function should likely be refactored. This should not be
#passed the
# account instance.
diff --git a/offlineimap/folder/Maildir.py
b/offlineimap/folder/Maildir.py index bab8284..b54879d 100644
--- a/offlineimap/folder/Maildir.py
+++ b/offlineimap/folder/Maildir.py @@ -552,69 +552,114 @@ class
MaildirFolder(BaseFolder):
" Neither `%s' nor `%s' found")
% (filename, oldfmd5, self._foldermd5))
- def sendcontentto(self, dstfolder, movecontent=False,
- dryrun=False):
- from shutil import move, copy2
+ def sendcontentto(self, dstfolder, srcstatusfolder, dststatusfolder, +
movecontent=False, dryrun=False): + '''We avoid using
Maildir.cachemessagelist() in order to drastically + improve
performance. Instead, we use statusrepos and + listdir().''' + from
shutil import move, copy2, rmtree
srcfolder = self
- folderpath = os.path.join(srcfolder.root, srcfolder.name)
-
- for d in os.listdir(folderpath):
- if os.path.isdir(os.path.join(folderpath, d)):
- messages = os.listdir(os.path.join(folderpath, d)) num
- = 0 totalnum = len(messages)
- self.ui.nbmessagestosend(os.path.join(dstfolder.name,
- d),
- totalnum, movecontent)
-
- for f in messages:
- #TODO: Show progression
-
- if os.path.isdir(os.path.join(folderpath, d, f)):
- '''There should not be any folder in here. If
- there is, should we also deal with it? In a
- sane folder structure, this should not
- happen. Or am I wrong?'''
-
- self.ui.info("A folder was found in source
- folder '{0}'"
- "while trying to move its
- content. Ignoring" "'{1}'. This
- indicate that the original"
- "folder structure is probably
- corrupt in" "some way. Please
- investigate.",
- format(os.path.join(srcfolder.name,
- d), f))
- continue
+ srcstatusfolder.cachemessagelist() +
dststatusfolder.cachemessagelist() + srcfolderpath =
os.path.join(srcfolder.root, srcfolder.name) + dstfolderpath =
os.path.join(dstfolder.root, dstfolder.name) + savemsgtostatusfolder =
True + + # To speed up process, when dststatusfolder is empty, we copy
status + # file and test on maildir folder for existing messages. If + #
dststatusfolder is not empty, we procede to savemessage. + if
len(dststatusfolder.getmessageuidlist()) == 0 \ + and
len(srcstatusfolder.getmessageuidlist()) > 0: + savemsgtostatusfolder =
False + + if savemsgtostatusfolder: + sendmsglist = [uid for uid in
srcstatusfolder.getmessageuidlist() + if not
dststatusfolder.uidexists(uid)] + else: + sendmsglist =
srcstatusfolder.getmessageuidlist() + num_of_msg = len(sendmsglist)
- dstfile = os.path.join(dstfolder.root,
- dstfolder.name, d, f) if movecontent:
- #TODO: Prevent or ask file overwrite
- move(os.path.join(folderpath, d, f),
- dstfile) else:
- num += 1 if os.path.exists(dstfile):
- self.ui.ignorecopyingmessage(f, srcfolder,
- dstfolder) continue
- if not dryrun:
- copy2(os.path.join(folderpath, d, f),
- dstfile)
-
- if (movecontent
- and len(os.listdir(os.path.join(folderpath, d))) ==
- 0
- and not dryrun):
- os.rmdir(os.path.join(folderpath, d))
- elif movecontent and not dryrun:
- self.ui.info("Folder {0} is not empty. Some files
- were not moved".
- format(os.path.join(folderpath, d)))
+ if not savemsgtostatusfolder and num_of_msg > 0: + try: +
dststatusfolder.closefiles() + copy2(srcstatusfolder.filename,
dststatusfolder.filename) + except OSError: + savemsgtostatusfolder =
True + pass
- else:
- '''If there is a file in srcfolder, should we also deal
- with it? In a sane folder structure, this should not
- happen. Or am I wrong? move(os.path.join(folderpath,
- d),
- os.join(dstfolder.root, dstfolder.name, f))'''
- self.ui.ignorecopyingmessage(self._parse_filename(d)['UID'],
- srcfolder, dstfolder)
-
- if movecontent and len(os.listdir(folderpath)) == 0 \
+ self.ui.nbmessagestosend(dstfolder.name, num_of_msg, movecontent) + +
filelist = {} + try: + for f in os.listdir(os.path.join(srcfolderpath,
'cur')) \ + + os.listdir(os.path.join(srcfolderpath, 'new')): + uidmatch
= re_uidmatch.search(f) + if uidmatch: +
filelist[int(uidmatch.group(1))] = f + except OSError: + pass + + #TODO
: + # -Ignore UIDs??? + + for num, uid in enumerate(sendmsglist): +
#TODO: Bail out on CTRL-C or SIGTERM. + '''if
offlineimap.accounts.Account.abort_NOW_signal.is_set(): + break''' +
try: + # Should we check that UID > 0? + # With Maildir, there
shouldn't be any UID = 0, or am I wrong? + num += 1 + filename =
filelist[uid] + flags = srcstatusfolder.getmessageflags(uid) +
dir_prefix = 'cur' if 'S' in flags else 'new' + srcfilepath =
os.path.join(srcfolderpath, dir_prefix, filename) + dstfilepath =
os.path.join(dstfolderpath, dir_prefix, filename) + +
self.ui.sendingmessage(uid, num, num_of_msg, + srcfolder, dstfolder,
movecontent) + + if dryrun: + continue + if movecontent: + #TODO:
Prevent or ask file overwrite + move(srcfilepath, dstfilepath) + else: +
if os.path.exists(dstfilepath): + self.ui.ignorecopyingmessage(filename,
srcfolder, dstfolder) + elif not dryrun: + copy2(srcfilepath,
dstfilepath) + except Exception as e: + self.ui.info( + "Error while
sending content of folder {0}, to {1} : '{2}'". +
format(srcfolder.name, dstfolder.name, e)) + raise + + if
savemsgtostatusfolder: + labels = srcstatusfolder.getmessagelabels(uid)
+ # Should we save rtime and mtime? + dststatusfolder.savemessage(uid,
None, flags, 0, 0, labels) + dststatusfolder.save() + + for folder in
["srcfolder", "srcstatusfolder", "dststatusfolder"]: + if folder in
locals(): + locals()[folder].dropmessagelistcache() + + if num_of_msg ==
0: + num = 0 + + if movecontent \ + and num == num_of_msg \
and not dryrun:
- os.rmdir(folderpath)
+ try: + rmtree(srcfolderpath) + except OSError: + self.ui.warn( +
"Error while removing source folder {0}, but no \ + messages were left
in it.".format(srcfolder.root)) + pass
elif movecontent and not dryrun:
- self.ui.info("Folder {0} is not empty. Some files were not
- moved.".
- format(folderpath))
+ self.ui.warn( + "Folder {0} is not empty. Some files were not moved. \
+ Please investigate source statusfolder.".format(srcfolder.root))
--
2.11.0
More information about the OfflineIMAP-project
mailing list