Bug report: Too many messages fetched at once with maxage

[Philipp Kern]

On Sun, Jul 17, 2011 at 12:11:32AM +0200, Philipp Kern wrote:
> syncing large folders fails for me with offlineimap, at least with maxage=365
> in my offlineimaprc.  It seems the query returns too many values for the
> active messages on busy mailing list folders and the subsequent fetch isn't
> split into multiple parts.
> 
> Here's the backtrace:
> 
>  WARNING: ERROR in syncfolder for thrall folder debian.lists.release: Traceback (most recent call last):
>   File "/usr/lib/pymodules/python2.6/offlineimap/accounts.py", line 347, in syncfolder
>     remotefolder.cachemessagelist()
>   File "/usr/lib/pymodules/python2.6/offlineimap/folder/IMAP.py", line 191, in cachemessagelist
>     response = imapobj.fetch(messagesToFetch, '(FLAGS UID)')[1]
>   File "/usr/lib/pymodules/python2.6/offlineimap/imaplib2.py", line 692, in fetch
>     return self._simple_command(name, message_set, message_parts, **kw)
>   File "/usr/lib/pymodules/python2.6/offlineimap/imaplib2.py", line 1492, in _simple_command
>     return self._command_complete(self._command(name, *args), kw)
>   File "/usr/lib/pymodules/python2.6/offlineimap/imaplib2.py", line 1272, in _command_complete
>     raise self.error('%s command error: %s %s. Data: %.100s' % (rqb.name, typ, dat, rqb.data))
> error: FETCH command error: BAD ['Error in IMAP command FETCH: Too long argument.']. Data: EMOK193 FETCH 6112,6113,6114,6115,6116,6117,6118,6119,6120,6121,6122,6123,6124,6125,6126,6127,6128,6
> 
> I guess cachemessagelist in folder/IMAP.py should split the long string, which
> is 65823 characters long, into parts and do multiple fetches.  It seems that
> the results could just be appended to the response list then?

The band-aid fix would be this:

    msgs = messagesToFetch.split(',')
    response = []
    while len(msgs) > 0:
        currentMessagesToFetch = ','.join(msgs[:1024])
        msgs = msgs[1024:]
        response.extend(imapobj.fetch(currentMessagesToFetch, '(FLAGS UID)')[1])

But it's obviously incorrect for >60 chars IDs.  And maybe also a bit
inefficient.  Apart from that it works fine for me.

Ciao
Philipp Kern
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110719/262b0e39/attachment-0001.sig>