<DKIM> Re:<DKIM> AssertionError in cachemessagelist

[Nicolas Sebrecht]

On Mon, Dec 22, 2014 at 09:49:03PM -0500, Isaac Bennetch wrote:

> > The patch I wrote introduce support for non full ASCII cache file.
> 
> I have done so and it appears to work correctly -- both accounts are
> downloading many old messages and I do not see the error message.

Thanks for the feedback. So, the enconding is clearly the root cause.

> This is particularly odd to me because, as far as I can tell, I do not
> have any unicode characters in my Gmail labels. Of course, I could be
> wrong, but nothing is obvious when looking through the list.

Thinking more deeply about that I think you're right.

The correct reason is that labels are retrieved from raw mails. If the
mail is not in ASCII, we open the mail and take strings from it without
checking nor fixing the encoding. Then, follows a cascade of unexpected
types which will hurt at some random point.

Also, I'm pretty sure that this could be the root cause of other
unsolved issues reported to the mailing list for years. Such unsolved
issues are hard to fix because they are raised at any random place in
the code, depending of contexts, and could lead to surprising behaviors
looking as if totally unrelated to this.

The bad news is that I know there is potentially a lot of wrong
assumptions about the types. Also, thanks to python for changing the
type of the strings behind our back to better match the encoding of the
files.

It sucks. I'll try to find a good way to fix that for the long term
during the holidays. SQLite might be touched with encoding issues, too.

> Thank you for the detailed explanation on cache files as well as the
> unbelievably fast patch. Is there some further testing I can do?

Keep testing. One sync is not enough to test the patch, for sure. It has
to be tested over multiple syncs with actual changes to the cache file.

I'd say, it's worth testing for some days or weeks depending on the flow
rate of mails in your mailbox and the rate you sync with offlineimap.

I'll start with an arbitrary pending period of one week before merging
the patch. I'll be much more confident if you can confirm that sync
happened and you really checked that the cache file changed several
times.

KEEP MAKING BACKUPS of your mails and cache files, often. Before each
sync is the best. The situation is worse for you, now. With the fix
applied to the cache file, you might experience bugs that nobody had a
chance to hit before. Think that the assert() could have prevented other
bugs to ever happen.

Ok, I tend to be alarming. I know what I do. Don't be surprised. ,-)

I'd rather you to be aware of what could happen with a BIG WARNING
without anything bad happening than not saying anything and you to
discover a seriously damaging issue on your mails.

Please, keep going on so we can fix it together! :-)

-- 
Nicolas Sebrecht