<DKIM> Re:<DKIM> AssertionError in cachemessagelist

[Isaac Bennetch]

On 12/22/14, 11:15 PM, Nicolas Sebrecht wrote:
> On Mon, Dec 22, 2014 at 09:49:03PM -0500, Isaac Bennetch wrote:
> 
>>> The patch I wrote introduce support for non full ASCII cache file.
>>
>> I have done so and it appears to work correctly -- both accounts are
>> downloading many old messages and I do not see the error message.
> 
> Thanks for the feedback. So, the enconding is clearly the root cause.
> 
>> This is particularly odd to me because, as far as I can tell, I do not
>> have any unicode characters in my Gmail labels. Of course, I could be
>> wrong, but nothing is obvious when looking through the list.
> 
> Thinking more deeply about that I think you're right.
> 
> The correct reason is that labels are retrieved from raw mails. If the
> mail is not in ASCII, we open the mail and take strings from it without
> checking nor fixing the encoding. Then, follows a cascade of unexpected
> types which will hurt at some random point.
> 
> Also, I'm pretty sure that this could be the root cause of other
> unsolved issues reported to the mailing list for years. Such unsolved
> issues are hard to fix because they are raised at any random place in
> the code, depending of contexts, and could lead to surprising behaviors
> looking as if totally unrelated to this.
> 
> The bad news is that I know there is potentially a lot of wrong
> assumptions about the types. Also, thanks to python for changing the
> type of the strings behind our back to better match the encoding of the
> files.
> 
> It sucks. I'll try to find a good way to fix that for the long term
> during the holidays. SQLite might be touched with encoding issues, too.
> 
>> Thank you for the detailed explanation on cache files as well as the
>> unbelievably fast patch. Is there some further testing I can do?
> 
> Keep testing. One sync is not enough to test the patch, for sure. It has
> to be tested over multiple syncs with actual changes to the cache file.
> 
> I'd say, it's worth testing for some days or weeks depending on the flow
> rate of mails in your mailbox and the rate you sync with offlineimap.
> 
> I'll start with an arbitrary pending period of one week before merging
> the patch. I'll be much more confident if you can confirm that sync
> happened and you really checked that the cache file changed several
> times.

I've been using this patched version since you sent it, syncing every
couple of days, so perhaps 8-10 runs total. About number of
messages/volume, my most recent sync, run this afternoon pulled in ~600
messages from one account and ~150 from another. I haven't had an error
message nor have I noticed any oddities with my mail. It certainly
appears as if everything is working properly.

I wasn't clear from the git logs whether you have merged this patch to
the main branch, but either way thought some feedback would be useful to
you.

> KEEP MAKING BACKUPS of your mails and cache files, often. Before each
> sync is the best. The situation is worse for you, now. With the fix
> applied to the cache file, you might experience bugs that nobody had a
> chance to hit before. Think that the assert() could have prevented other
> bugs to ever happen.
> 
> Ok, I tend to be alarming. I know what I do. Don't be surprised. ,-)

:-)

> I'd rather you to be aware of what could happen with a BIG WARNING
> without anything bad happening than not saying anything and you to
> discover a seriously damaging issue on your mails.
> 
> Please, keep going on so we can fix it together! :-)
> 

Thanks for your help
~isaac