Speed of syncing several folders at the same time vs sequentially

[Sebastian Spaeth]

On Wed, 20 Jul 2011 13:45:17 -0700, Marc MERLIN <marc at merlins.org> wrote:
> maxconnections = 1
> is faster than syncing 6 folders at a time.
> 
> How does the maxconnections code work?
> Should it fork off 6 python processes, or is that 6 threads to download the
> 6 folders at the same time

It will not fork off processes. Instead it keeps a pool of up to 6
connections and spawns a thread whenever it needs one, waiting for one
thread to become available. Some ideas:

1) Use the sqlite backend for the Status database. As Stewart pointed
out, you'll write plenty of MB of data, and as only one thread can write
at a time, you'll be waiting for the status db to become available a
lot.

2) Google might decide to punish in terms of bandwidth limit if you open
too many connections, no idea about the sensible limits to gmail.

3) offlineimap will sync up to "maxconnections" folders simultanously,
using "maxconnections" threads. I am not sure if accessing so many
different folders at the same time could impact things. I believe it
would make sense to limit the number of concurrently synced folders to 2
and use "maxconnections" threads to speed up syncing within these folders...
In the worst case, we are doing lots of folder "SELECT"ing in our
connection pool to jump between those 6 folders.

Sebastian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110808/bf234135/attachment.sig>