Bugreport: maxage deleting old email from destination

Fri Jul 29 11:42:26 BST 2011

This problem has became urgent, because we have reached the gmail
limit, so I had to delete some messages from gmail (still trying to
preserve them at the local mirror).
My live email mirror runs at offlineimap 6.3.1 so I thought it may not
have the maxage bug. I've set maxage=180 and ran the sync. I also
remembered the FIRST (oldest) message in inbox. After the sync the
message was still there! So I thought 6.3.1 was not affected and
changed maxage to 120 to be able to actually delete old mail from
gmail. Next sync:

Syncing March-April: Gmail -> MappedIMAP
Deleting 245 messages (204, 205, 206, 207, 208, 210, 211, 212, 213,
214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 228,
229, 230, 231, 232, 233, 235, 236, 237, 238, 239, 240, 241, 242, 243,
244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 257, 259,
262, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276,
277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290,
291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 302, 304, 305, 306,
307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320,
321, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335,
336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 347, 348, 351, 352,
353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366,
367, 368, 369, 370, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381,
382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395,
396, 397, 398, 399, 400, 401, 403, 405, 406, 493, 494, 784, 981, 984,
985, 986, 987, 988, 989, 991, 992, 993, 994, 995, 996, 999, 1000,
1001, 1002, 1003, 1004, 1005, 1006, 1008, 1009, 1010, 1011, 1012,
1013, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023, 1024, 1026,
1029, 1032, 1033, 1034, 1035, 1036, 1038, 1039, 1040, 1041, 1043,
1044, 1045, 1046, 1047, 1048, 1049, 1051, 1052) in Gmail[March-April],
LocalStatus[March-April]
Syncing Misc: Gmail -> MappedIMAP
Syncing Priority: Gmail -> MappedIMAP
Syncing Sent Messages: Gmail -> MappedIMAP
Syncing Trash: Gmail -> MappedIMAP
Syncing [Gmail]/Drafts: Gmail -> MappedIMAP
Syncing [Gmail]/Sent Mail: Gmail -> MappedIMAP
Deleting 41 messages (671, 672, 673, 674, 675, 676, 677, 678, 679,
680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693,
694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707,
708, 709, 710, 711) in Gmail[[Gmail]/Sent Mail],
LocalStatus[[Gmail].Sent Mail]
Syncing [Gmail]/Spam: Gmail -> MappedIMAP

The inbox was still untouched! I have no idea what that means, but it
definitely is not correct.

Also, I am really tired of the bug with downloading large messages
from gmail (I mentioned that a couple of months earlier and I am still
getting it in both 6.3.1 and 6.3.4rc3 (next))

Thread 'New msg sync from March-April' terminated with exception:
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/offlineimap/threadutil.py",
line 158, in run
    Thread.run(self)
  File "/usr/lib/python2.4/threading.py", line 422, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.4/site-packages/offlineimap/folder/Base.py",
line 215, in syncmessagesto_neguid_msg
    successuid = tryappend.savemessage(uid, message, flags, rtime)
  File "/usr/lib/python2.4/site-packages/offlineimap/folder/IMAP.py",
line 347, in savemessage
    date, content)[0] == 'OK')
AssertionError

Last 12 debug messages logged for New msg sync from March-April prior
to exception:
imap: Returned object from fetching 394: ('OK', [('377 (UID 394 BODY[]
{487582}', '<body here>'
imap: savemessage: new content length is 487685

This happens on some messages and the only way to fix is to delete the
message from the source server.

Also, when developing our gmail-imap related software we found another
bug with gmail, which is not offlineimap related. Recently they
changed something so that mail from March was returned AFTER mail from
July. The order was definitely incorrect, I'm not sure about the UIDs.
I find these messages strange too:

Copy message -1 MappedIMAP[March-April] -> Gmail[March-April],
LocalStatus[March-April]
Copy message -383 MappedIMAP[March-April] -> Gmail[March-April],
LocalStatus[March-April]
Copy message -382 MappedIMAP[March-April] -> Gmail[March-April],
LocalStatus[March-April]
Copy message -381 MappedIMAP[March-April] -> Gmail[March-April],
LocalStatus[March-April]

What are the numbers and why are they negative?

As a conclusion, the idea behind offlineimap is great, but
unfortunately the software is extremely unrealiable and we will be
working to build a system that avoids imap-to-imap syncronisations to
avoid using offlineimap at all. Mostly that's gmails's fault I
suspect, but it's too expensive to support this system currently, so
we are dumping both gmail and offlineimap.