Fresh copy of large maildir
Sebastian at SSpaeth.de
Wed Jul 6 09:16:08 BST 2011
On Tue, 05 Jul 2011 22:11:31 +0200, Christoph Höger <christoph.hoeger at tu-berlin.de> wrote:
> I have some fairly large maildirs now (> 2GB) and running offlineimap
> plainly on them is quite ... slow.
> 1. Is there some optimization for such a task (e.g. compressing bulks of
> messages) already present or would that be a completely new work?
Not sure if you are talking about the initial sync where you actually
have to transfer loads of content. For this, I guess we would benefit
most by supporting IMAP extensions that enable compression or sending
multiple message in one go, rather than requesting each
individually. Supporting this would indeed be nice. However, the common
case is when you already synced and only a (relatively) few messages
have changed, is the case that I am most concerned with. And indeed
there are a few cases which speed up sync considerably.
The things that help:
1) Use maxconnections > 1 (the default used to be 1 and we recently
changed it to 2). You probably have that.
2) Use folderfilters. The quickest sync is a sync that can ignore
folders. I sort my inbox into monthly folders, and ignore everyfolder
that is more than 2-3 months old, this lets me only inspect a
fraction of my Mails on every sync. If you haven't done this yet, do
3) Use quick sync. A regular sync will request all flags and all UIDs of
all mails in a folder which takes quite some time. A quick sync only
compares the number of messages in a folder on the IMAP side (it will
detect flag changes on the Maildir side of things though). A quick
sync on my smallish account will take 7 seconds rather than 40
seconds for a regular one. So I run a cron script that does a regular
sync once a day, and does quick syncs inbetween.
There is a number of areas in which offlineimap can improve. I am
thinking off allowing to do n quicksyncs between each full sync even
if you are not in auto-refresh mode. This will make using quick syncs
much more convenient. Also I am thinking of allowing to use different
folderfilter settings depending of whether you run a quick or a full
I was actually thinking of posting these "performance" tweaks on the
wiki or including them in the manual as they can help a lot. What do
> 2. Rsync usually works fine, although it naturally keeps filenames.
> Therefore I cannot simply rsync two maildirs (offlineimap will ignore
> the present messages and effectively double the size). Could offlineimap
> re-index a present folder offline?
rsync between IMAP-server and Maildir works but, as we store the FMD5
and the UID of the IMAP server in the filename, you need them to be the
same. I know that some servers use the same format, so it could
work. How do the filenames on your IMAP server look like?
OfflineImap currently does not look at the content of files at all, so
it could not re-index stuff if the filenames don't match. I really want
us to implement a "recovery mode" that is able to reconstruct our
LocalStatus cache in case it gets corrupted or missing (and also that
you can simply move in an existing Maildir and have it
reconstructed). This will likely imply parsing mails for Message-ID and
content and searching the IMAP server, matching up files. But this is
still some time in the future, I guess.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 197 bytes
Desc: not available
More information about the OfflineIMAP-project