Bug report: Too many messages fetched at once with maxage

[Mike Dawson]

Hi,

Actually it should be applied to both ends - I tested that with IMAP,
Gmail and Maildir - only thing that occurs to me is perhaps it's a
timezone issue between the server and the client - as the way the IMAP
search query works is to ask for SINCE day x ...  But AFAIK IMAP servers
should work with GMT.

Regards,

-Mike

-- 
Mike Dawson
CEO
PAIWASTOON Networking Services Ltd.
www.paiwastoon.af

"Making IT work for Afghanistan"
Tel: +93 798 258 092
Email: mike at paiwastoon.com.af




On Fri, 2011-07-22 at 13:13 -0700, chris coleman wrote:
> 
> 
> Sounds like the "maxage" algorithm is defective: it's ignoring the
> source messages older than maxage, but trying to process destination
> messages regardless of "age"..  
> 
> 
> The algorithm bug should be fixed to exclude from consideration all
> messages older than maxage on BOTH the source AND destination.  
> 
> 
> 
> 
> =========================
> 
> That's just a test run, it's ok if I spoil the local copy.
> The problem is that offlineimap seems to ignore the old messages at
> the source repo and tries to delete them from the destination (because
> they are 'missing' at source). It also fails with an exception which
> is a problem too.
> 
> 2011/7/22 Mike Dawson <mike at paiwastoon.com.af>:
> > Hi,
> >
> > Not entirely sure about what might be going on with the message
> delete
> > or the value returned.
> >
> > The way maxage works is that it only looks at what is within that
> age on
> > both sides.  Any messages that are not within maxage will be ignored
> > completely.
> >
> > Thus if I were you I would backup the local copy entirely, then
> delete
> > the messages from gmail, and then run offlineimap with maxage ...
> >
> > -Mike
> >
> > --
> > Mike Dawson
> > CEO
> > PAIWASTOON Networking Services Ltd.
> > www.paiwastoon.af
> >
> > "Making IT work for Afghanistan"
> > Tel: +93 798 258 092
> > Email: mike at paiwastoon.com.af
> >
> >
> >
> >
> > On Fri, 2011-07-22 at 09:26 +0400, Ivan Semin wrote:
> >> I am having problems with maxage too (not sure it's the same bug
> though).
> >> I have a gmail account which is 90% full. I am doing a one-way sync
> >> (readonly=1) from gmail to my local backup server. Now I want to
> >> delete all messages that are >3 months old from gmail, but I don't
> >> want them deleted from the local box. So I set maxage=90 and run
> the
> >> sync again. I did NOT delete any messages from gmail yet! That's
> what
> >> offlineimap 6.3.4-rc2 is doing:
> >>
> >> Deleting 533 messages (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
> 14,
> >> 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
> >> 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48,
> >> 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65,
> >> 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82,
> >> 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99,
> >> 100, 101, 102, 103, 104, 105, 106, 108, 109, 110, 111, 112, 113,
> 114,
> >> 115, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138,
> 140,
> >> 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166,
> 168,
> >> 169, 171, 172, 173, 174, 175, 176, 178, 179, 181, 182, 184, 186,
> 188,
> >> 190, 192, 194, 196, 198, 200, 202, 203, 205, 207, 209, 211, 213,
> 215,
> >> 217, 219, 220, 222, 224, 226, 228, 230, 231, 233, 235, 237, 238,
> 240,
> >> 242, 244, 246, 248, 250, 252, 254, 256, 258, 260, 262, 264, 266,
> 268,
> >> 269, 271, 273, 275, 277, 279, 281, 283, 285, 287, 289, 290, 291,
> 292,
> >> 294, 295, 297, 299, 301, 304, 305, 307, 308, 310, 312, 314, 316,
> 319,
> >> 320, 322, 324, 326, 328, 330, 332, 334, 336, 338, 339, 341, 342,
> 344,
> >> 346, 347, 349, 350, 351, 352, 354, 355, 357, 359, 361, 363, 365,
> 368,
> >> 369, 371, 373, 375, 377, 379, 381, 383, 385, 387, 389, 391, 393,
> 395,
> >> 397, 399, 401, 403, 405, 407, 408, 410, 412, 414, 416, 418, 419,
> 421,
> >> 422, 423, 424, 425, 427, 429, 430, 431, 432, 434, 436, 438, 440,
> 442,
> >> 443, 445, 447, 449, 451, 453, 455, 457, 459, 461, 463, 465, 467,
> 468,
> >> 469, 471, 473, 475, 477, 479, 481, 483, 485, 487, 489, 491, 493,
> 495,
> >> 497, 499, 501, 503, 504, 505, 506, 509, 510, 511, 512, 513, 514,
> 516,
> >> 518, 521, 522, 524, 526, 528, 530, 532, 533, 536, 537, 540, 541,
> 543,
> >> 545, 547, 549, 552, 553, 555, 558, 559, 561, 563, 565, 567, 583,
> 584,
> >> 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597,
> 599,
> >> 601, 602, 603, 604, 605, 607, 610, 611, 616, 617, 618, 619, 620,
> 621,
> >> 624, 625, 628, 629, 631, 632, 634, 636, 638, 640, 641, 642, 643,
> 646,
> >> 647, 648, 649, 651, 654, 655, 661, 662, 664, 666, 667, 668, 669,
> 670,
> >> 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683,
> 684,
> >> 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697,
> 698,
> >> 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711,
> 712,
> >> 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725,
> 726,
> >> 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739,
> 740,
> >> 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753,
> 754,
> >> 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767,
> 768,
> >> 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781,
> 782)
> >> in MappedIMAP[[Gmail].Sent Mail]
> >> WARNING: ERROR attempting to sync flags for account Data:Traceback
> >> (most recent call last):
> >>   File
> "/usr/lib/python2.6/site-packages/offlineimap/folder/Base.py",
> >> line 413, in syncmessagesto
> >>     action(dstfolder, statusfolder)
> >>   File
> "/usr/lib/python2.6/site-packages/offlineimap/folder/Base.py",
> >> line 329, in syncmessagesto_delete
> >>     folder.deletemessages(deletelist)
> >>   File
> "/usr/lib/python2.6/site-packages/offlineimap/folder/UIDMaps.py",
> >> line 253, in deletemessages
> >>     self._mb.deletemessages(self._uidlist(self.r2l, uidlist))
> >>   File
> "/usr/lib/python2.6/site-packages/offlineimap/folder/UIDMaps.py",
> >> line 87, in _uidlist
> >>     return [mapping[x] for x in items]
> >> KeyError: 1L
> >>
> >> WARNING: ERROR in syncfolder for Data folder [Gmail]/Sent Mail:
> >> Traceback (most recent call last):
> >>   File "/usr/lib/python2.6/site-packages/offlineimap/accounts.py",
> >> line 347, in syncfolder
> >>     remotefolder.syncmessagesto(localfolder, statusfolder)
> >>   File
> "/usr/lib/python2.6/site-packages/offlineimap/folder/Base.py",
> >> line 413, in syncmessagesto
> >>     action(dstfolder, statusfolder)
> >>   File
> "/usr/lib/python2.6/site-packages/offlineimap/folder/Base.py",
> >> line 329, in syncmessagesto_delete
> >>     folder.deletemessages(deletelist)
> >>   File
> "/usr/lib/python2.6/site-packages/offlineimap/folder/UIDMaps.py",
> >> line 253, in deletemessages
> >>     self._mb.deletemessages(self._uidlist(self.r2l, uidlist))
> >>   File
> "/usr/lib/python2.6/site-packages/offlineimap/folder/UIDMaps.py",
> >> line 87, in _uidlist
> >>     return [mapping[x] for x in items]
> >> KeyError: 1L
> >>
> >> Syncing [Gmail]/Spam: Gmail -> MappedIMAP
> >> ***** Finished processing account Data
> 
>