crash on sync of large maildir

[Mark A. Hershberger]

My INBOX maildir hold mail for the last 10 years — no Inbox Zero for me!

;)

I've used offlineimap very successfully for a long time, but recently
something started happening to screw it up.  To try and regain some
sanity (and at the advise of John Goerzen), I got a fresh version from
git but I was still running into a problem — though this one was
different.

Now I have the following problem:

    $ offlineimap -f INBOX
    OfflineIMAP 6.3.3
    Copyright (C) 2002 - 2010 John Goerzen <john at complete.org>

    This software comes with ABSOLUTELY NO WARRANTY; see the file
    COPYING for details.  This is free software, and you are welcome
    to distribute it under the conditions laid out in COPYING.
    ***** Processing account nichework.com
    Copying folder structure from IMAP to MappedIMAP
    Establishing connection to tunnel:nice env MAILDIR=/home/mah/maildir/everybody.org /usr/lib/dovecot/imap.
    Establishing connection to mail.nichework.com:993.
    Syncing INBOX: IMAP -> MappedIMAP
    Copy message -2 MappedIMAP[INBOX] -> IMAP[INBOX]
    Copy message -6 MappedIMAP[INBOX] -> IMAP[INBOX]
    Copy message -5 MappedIMAP[INBOX] -> IMAP[INBOX]
    Copy message -4 MappedIMAP[INBOX] -> IMAP[INBOX]
    Thread 'New msg sync from INBOX' terminated with exception:
    Traceback (most recent call last):
    File "/usr/lib/pymodules/python2.7/offlineimap/threadutil.py", line 159, in run
    Thread.run(self)
    File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
    File "/usr/lib/pymodules/python2.7/offlineimap/folder/Base.py", line 264, in syncmessagesto_neguid_msg
    self.deletemessage(uid)
    File "/usr/lib/pymodules/python2.7/offlineimap/folder/UIDMaps.py", line 238, in deletemessage
    self._mb.deletemessage(self, self.r2l[uid])
    File "/usr/lib/pymodules/python2.7/offlineimap/folder/IMAP.py", line 567, in deletemessage
    self.deletemessages_noconvert([uid])
    File "/usr/lib/pymodules/python2.7/offlineimap/folder/IMAP.py", line 578, in deletemessages_noconvert
    self.addmessagesflags_noconvert(uidlist, ['T'])
    File "/usr/lib/pymodules/python2.7/offlineimap/folder/IMAP.py", line 499, in addmessagesflags_noconvert
    self.processmessagesflags('+', uidlist, flags)
    File "/usr/lib/pymodules/python2.7/offlineimap/folder/IMAP.py", line 531, in processmessagesflags
    assert r[0] == 'OK', 'Error with store: ' + '. '.join(r[1])
    AssertionError: Error with store: UID STORE failed


    Last 9 debug messages logged for New msg sync from INBOX prior to exception:
    imap: Returned object from fetching 780665: ('OK', ['72967 (UID 780665 BODY[] "")'])
    imap: savemessage: called
    imap: savemessage: using date None
    imap: savemessage: content is: '2'
    imap: savemessage: returning new UID 406
    imap: savemessage: called
    imap: savemessage: using date None
    imap: savemessage: content is: '2'
    imap: savemessage: returning new UID 786370
    imap(mah): Info: Connection closed bytes=14751/16239

Three things:

1) It copies about the same number of emails each time and then bails.
2) It creates a bogus email in my inbox consisting of a single
   character: “2”.  No From headers or To headers… just the number 2.

I delete the bogus email from my maildir locally and (probably because
I've confused my local IMAP by deleting the non-mail file), it crashes again:

    $ offlineimap -f INBOX
    OfflineIMAP 6.3.3
    Copyright (C) 2002 - 2010 John Goerzen <john at complete.org>

    This software comes with ABSOLUTELY NO WARRANTY; see the file
    COPYING for details.  This is free software, and you are welcome
    to distribute it under the conditions laid out in COPYING.
    ***** Processing account nichework.com
    Copying folder structure from IMAP to MappedIMAP
    Establishing connection to tunnel:nice env MAILDIR=/home/mah/maildir/everybody.org /usr/lib/dovecot/imap.
    Establishing connection to mail.nichework.com:993.
    Syncing INBOX: IMAP -> MappedIMAP
    Deleting 6 messages (407, 408, 409, 410, 411, 412) in MappedIMAP[INBOX]
    WARNING: ERROR attempting to sync flags for account nichework.com:Traceback (most recent call last):
      File "/usr/lib/pymodules/python2.7/offlineimap/folder/Base.py", line 486, in syncmessagesto
        action(dstfolder, statusfolder)
      File "/usr/lib/pymodules/python2.7/offlineimap/folder/Base.py", line 397, in syncmessagesto_delete
        folder.deletemessages(deletelist)
      File "/usr/lib/pymodules/python2.7/offlineimap/folder/UIDMaps.py", line 242, in deletemessages
        self._mb.deletemessages(self, self._uidlist(self.r2l, uidlist))
      File "/usr/lib/pymodules/python2.7/offlineimap/folder/UIDMaps.py", line 77, in _uidlist
    return [mapping[x] for x in items]
    KeyError: 408L

    WARNING: ERROR in syncfolder for nichework.com folder INBOX: Traceback (most recent call last):
      File "/usr/lib/pymodules/python2.7/offlineimap/accounts.py", line 353, in syncfolder
        remotefolder.syncmessagesto(localfolder, statusfolder)
      File "/usr/lib/pymodules/python2.7/offlineimap/folder/Base.py", line 486, in syncmessagesto
        action(dstfolder, statusfolder)
      File "/usr/lib/pymodules/python2.7/offlineimap/folder/Base.py", line 397, in syncmessagesto_delete
        folder.deletemessages(deletelist)
      File "/usr/lib/pymodules/python2.7/offlineimap/folder/UIDMaps.py", line 242, in deletemessages
        self._mb.deletemessages(self, self._uidlist(self.r2l, uidlist))
      File "/usr/lib/pymodules/python2.7/offlineimap/folder/UIDMaps.py", line 77, in _uidlist
    return [mapping[x] for x in items]
    KeyError: 408L

    imap(mah): Info: Disconnected: Logged out bytes=139/18719
    ***** Finished processing account nichework.com

Running again after this works the same as the initial invocation.

Adding -1, to disable threading does let it run a little longer
usually.  But then it just ends up creating more of these bogus non-mail
files and in a seemingly infitite loop that times out.

Interestingly, it looks like the first thing it does when the -1 is
added is add 5-6 of those bogus “2” emails to both the local and remote
IMAP.


-- 
http://hexmode.com/

War begins by calling for the annihilation of the Other,
    but ends ultimately in self-annihilation.