[PATCH] More detailed error output on corrupt UID mapping files

[chris coleman]

> Thanks Sebastian and Nicolas.
> 
> The corrupt data has been found.
> 
> You're right it was indeed in the huge file ~/.offlineimap/Repository-IMAPLocal<username>/UIDMapping/INBOX.Sent  
> 
> Here are a few lines before and after :
> 16731:4777
> 16732:4778
> 16733:4779
> 16734:  P    35:4781
>
 16736:4782
> 16737:4783
> 16738:4784
> They're consecutive UIDs so I re-entered them and filled in the missing numbers in the sequence.
> 
> With the INBOX.Sent file repaired in this user, I re-ran it, and we're down to this same basic error on both mailboxes:
> 
> 
> "WARNING: ERROR in syncfolder for username1: null byte in argument for long()"
> "WARNING: ERROR in syncfolder for username2: null byte in argument for long()"
> 
> 
> Do you have a patch for this "ERROR in syncfolder", to pinpoint the line number and filename ?

The error most likely occurs in offlineimap/offlineimap/folder/IMAP.py
around line 162 in the cachemessagelist() function

What happens is that we select a folder in line 111 in:

imaptype, imapdata = imapobj.select(self.getfullname(), readonly = 1, force = 1)

and later in line 162 we go through the returned data, trying to get
 the
highest UID number that the select call returned:

                for msgid in imapdata:
                    maxmsgid = max(long(msgid), maxmsgid)

If you could add some debug code in the form

                for msgid in imapdata:
                    try:
                        maxmsgid = max(long(msgid), maxmsgid)
                    except:
                        raise Exception(str(imapdata))

and let us see what exactly had been returned in imapdata we could try
to work this out.

Let me know if you need more help in making the code changes. I
 hope we
get this one nailed down.

Sebastian

============

I'm on 6.3.2-0 because it's the highest that will run on debian 5.0 (lenny).

The catch is, 6.3.2-0 code for offlineimap/offlineimap/folder/IMAP.py is using imapobj not imapdata - so it seems like your patch would need modification to fit.

Here is the code excerpted from lines 175-188:

                try:
                    # 1. Some mail servers do not return an EXISTS response
                    # if the folder is empty.  2. ZIMBRA servers can return
                    # multiple
 EXISTS replies in the form 500, 1000, 1500,
                    # 1623 so check for potentially multiple replies.
                    maxmsgid = 0
                    for msgid in imapobj.untagged_responses['EXISTS']:
                        maxmsgid = max(long(msgid), maxmsgid)
                    messagesToFetch = '1:%d' % maxmsgid;
                except
 KeyError:
                    return
                if maxmsgid < 1:
                    #no messages; return
                    return

Your suggestion on how to patch this ?

Chris

---------------------------------------------------------------

Sebastian, Nicolas,

I went ahead with your suggestion and patched those lines (in version 6.3.2-0  folder/IMAP.py ) by wrapping it with the try catch bock.

Running it triggered a recompile.  

But it didn't output any further details on the filename/line number where any bad data is.  

Here's the patch just FYI, from line 174:

            else:
                try:
                    # 1. Some mail servers do not return an EXISTS response
                    # if the folder is empty.  2. ZIMBRA servers can return
                    # multiple EXISTS replies in the form 500, 1000, 1500,
                    # 1623 so check for potentially multiple replies.
                    maxmsgid = 0
                    for msgid in imapobj.untagged_responses['EXISTS']:
                        try:        #new
                            maxmsgid = max(long(msgid), maxmsgid)
                        except:    #new
                            raise Exception(str(imapobj))        #new
                    messagesToFetch = '1:%d' % maxmsgid;
                except KeyError:
                    return
                if maxmsgid < 1:
                    #no messages; return
                    return
            # Now, get the flags and UIDs for these.
            # We could conceivably get rid of maxmsgid and just say
            # '1:*' here.
            response = imapobj.fetch(messagesToFetch, '(FLAGS UID)')[1]

Must get better info on where the bad data is located, and repair these 2 username's data, ASAP.

I see the actual error message gets output in the file offlineimap/accounts.py, function syncfolder, line 380:
    except:
        ui.warn("ERROR in syncfolder for %s folder %s: %s" % \
                (accountname,remotefolder.getvisiblename(),sys.exc_info()[1]))

These are available: localrepos, localfolder, remoterepos, remotefolder.  

How about we output details of the line number and filename right here in the call to ui.warn ????

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110503/382ef27c/attachment-0001.html>