Fresh copy of large maildir

[Sebastian Spaeth]

On Tue, 05 Jul 2011 22:11:31 +0200, Christoph Höger <christoph.hoeger at tu-berlin.de> wrote:
> I have some fairly large maildirs now (> 2GB) and running offlineimap 
> plainly on them is quite ... slow.

Really? :-).

> 1. Is there some optimization for such a task (e.g. compressing bulks of 
> messages) already present or would that be a completely new work?

Not sure if you are talking about the initial sync where you actually
have to transfer loads of content. For this, I guess we would benefit
most by supporting IMAP extensions that enable compression or sending
multiple message in one go, rather than requesting each
individually. Supporting this would indeed be nice. However, the common
case is when you already synced and only a (relatively) few messages
have changed, is the case that I am most concerned with. And indeed
there are a few cases which speed up sync considerably.
The things that help:

1) Use maxconnections > 1 (the default used to be 1 and we recently
   changed it to 2). You probably have that.
2) Use folderfilters. The quickest sync is a sync that can ignore
   folders. I sort my inbox into monthly folders, and ignore everyfolder
   that is more than 2-3 months old, this lets me only inspect a
   fraction of my Mails on every sync. If you haven't done this yet, do
   it :).
3) Use quick sync. A regular sync will request all flags and all UIDs of
   all mails in a folder which takes quite some time. A quick sync only
   compares the number of messages in a folder on the IMAP side (it will
   detect flag changes on the Maildir side of things though). A quick
   sync on my smallish account will take 7 seconds rather than 40
   seconds for a regular one. So I run a cron script that does a regular
   sync once a day, and does quick syncs inbetween.

   There is a number of areas in which offlineimap can improve. I am
   thinking off allowing to do n quicksyncs between each full sync even
   if you are not in auto-refresh mode. This will make using quick syncs
   much more convenient. Also I am thinking of allowing to use different
   folderfilter settings depending of whether you run a quick or a full
   sync.

I was actually thinking of posting these "performance" tweaks on the
wiki or including them in the manual as they can help a lot. What do
people think?

> 2. Rsync usually works fine, although it naturally keeps filenames. 
> Therefore I cannot simply rsync two maildirs (offlineimap will ignore 
> the present messages and effectively double the size). Could offlineimap 
> re-index a present folder offline?

rsync between IMAP-server and Maildir works but, as we store the FMD5
and the UID of the IMAP server in the filename, you need them to be the
same. I know that some servers use the same format, so it could
work. How do the filenames on your IMAP server look like?

OfflineImap currently does not look at the content of files at all, so
it could not re-index stuff if the filenames don't match. I really want
us to implement a "recovery mode" that is able to reconstruct our
LocalStatus cache in case it gets corrupted or missing (and also that
you can simply move in an existing Maildir and have it
reconstructed). This will likely imply parsing mails for Message-ID and
content and searching the IMAP server, matching up files. But this is
still some time in the future, I guess.

Sebastian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110706/ca266d73/attachment-0001.sig>