Bug report: Too many messages fetched at once with maxage

[Sebastian Spaeth]

> Hi there,
> 
> syncing large folders fails for me with offlineimap, at least with maxage=365
> in my offlineimaprc.  It seems the query returns too many values for the
> active messages on busy mailing list folders and the subsequent fetch isn't
> split into multiple parts.

> error: FETCH command error: BAD ['Error in IMAP command FETCH: Too long argument.']. Data: EMOK193 FETCH 6112,6113,6114,6115,6116,6117,6118,6119,6120,6121,6122,6123,6124,6125,6126,6127,6128,6
> 
> I guess cachemessagelist in folder/IMAP.py should split the long string, which
> is 65823 characters long, into parts and do multiple fetches.  It seems that
> the results could just be appended to the response list then?

Hi there,
interesting bug. :-)

I see that the Dovecot default for IMAP commands is 65536 bytes, so your
command of 65823 just broke that limit... It should definitely be
fixed. I see that we already split the command for "processmessageflags"
changes if the UID list is too long, but we are not doing anything about
fetches (or deletes for that matter). I see 2 solutions:

1) Just split the request into multiple fetches as your patch suggests,
and 2) collapse continuous message ranges. I see that the fetch includes
large parts like this 6112,6113,6114,6115,6116,6117,6118,6119,... so we
could be doing:

FETCH 6112:6119,6300,6301:6350

(I have not tested if that actually works, but that should be a valid
IMAP sequence set). This would reduce the length of our IMAP command
significantly. We could then still see if the command is longer than
65536 bytes.

Fortunately, we already have a function that collapses lists of UIDs in
this way. It is the undocumented(!) function listjoin() in imaputil.py,
so fixing things this way should be relatively easy.

What do people think of this approach?
I set up http://trac.sspaeth.de/issue27 to track this one.

Sebastian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110808/d6d719a7/attachment.sig>