<DKIM> maxage causes loss of local email

[Nicolas Sebrecht]

On Thu, Mar 05, 2015 at 01:54:11AM -0500, Janna Martl wrote:

> There's more to this than I thought at first.

Yep. :-)

The whole maxage logic is not very appearing at first read. We must blam
the lack of good OO code, BTW. We don't have proper "Mail" class with
good methods attached about maxage. I think moving from Python
primitives to more objects would be a good thing, even for the
collections.

> On Tue, Mar 03, 2015 at 12:16:47PM +0100, Nicolas Sebrecht wrote:
> >So, in folder/Maildir.py we set the timestamp with
> >
> >  def savemessage(self, uid, content, flags, rtime):
> >      <...>
> >      if rtime != None:
> >          os.utime(os.path.join(self.getfullname(), tmpname), (rtime, rtime))
> >      <...>
> 
> At this point, for me, rtime = None, so this isn't doing anything.
> This function is called by copymessageto() in Base.py, and the rtime
> it passes to savemessage() is set in the following lines:
> 
>             rtime = self.getmessagetime(uid)
>             if dstfolder.utime_from_message:
>                 content = self.getmessage(uid)
>                 rtime = emailutil.get_message_date(content, 'Date')
> 
> I have two things to say about these lines:
> 
> (1) In line 1, self.getmessagetime() (here self = the remote folder) returns
> None for me. This is because in:
> 
>     def cachemessagelist(self):
> 		...
> 		rtime = imaplibutil.Internaldate2epoch(messagestr)
> 		self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime}
> 
> times seem to be gotten by parsing messagestr, but my messagestr's
> look like (UID 45437 FLAGS ()) and don't contain anything resembling a
> date.

Ok. So, could you provide a sample of what you get at this point? IMAP
servers are so different fomr one another. I don't get why each
different behaviour has not been documented. Today, we are lacking a lot
of documentation. Patches about specific cases got piled up without care
over time.

> (2) In lines 2-4 of the above, utime_from_message is False. This
> seems to be an undocumented configuration option (introduced in commit
> 3bc66c0858ec) whose default option is False.

Plain true. We must document this option. Also, the option is wrongly
assigned to the "general" section. "general" should be a fallback if the
account doesn't have the option.

>                                              If this is set to True
> instead, then lines 3 and 4 actually happen: now rtime is the
> internal date (which is not changed by the later call to os.utime),
> but the filename still has the download date.

Ah? So, we also have to understand why this rtime get ignored.

> This is bad, because Maildir.py's _iswithinmaxage() still checks the
> filename to determine the age of a local message.
> 
> >  def _scanfolder(self):
> >      <...>
> >      for dirannex, filename in files:
> >          <...>
> >          if maxage and not self._iswithinmaxage(filename, maxage):
> >              continue
> >          <...>
> 
> >Janna, could you confirm that maxage is honored for you and that
> >_scanfolder() excludes mails from your Maildir?
> 
> So: if I set utime_from_message = True, then the message ends up on the
> local messagelist with the correct time, during the run in which it is
> downloaded.

Ok.

>             But if offlineimap is run again later, it uses
> _scanfolder() to reconstruct the local messagelist, and the only
> maxage-related exclusion is done via _iswithinmaxage(), which only
> looks at filenames.

So, setting the rtime into the access/modification time was done for
another reason. Just checked and found 03488ba81b0a

    Sync INTERNALDATE <-> mtime

    The attached patch adds syncing the INTERNALDATE of IMAP folders with
    the mtime of messages in maildir folders.
    I want this to happen, because I'm running a dovecot over the maildirs
    synced by offlineimap, and that uses the mtime as the INTERNALDATE.
    When using mutt to view messages I generally sort based on the received
    date, which for IMAP folders is the INTERNALDATE.

    Since this is the first real coding I've done in Python the patch may
    need to be cleaned up some, but it's working pretty well for me. I've
    added new messages to each side, and the received date has been
    preserved going both ways.

This gets back to 2006.

I could add a TODO file. A lot of ways of improvements have been clearly
identified recently.

In fact, there are so much to do that I wonder if a full rewrite is not
the better option. This was discussed some years ago but led to nothing
because too many people had would have chosen a different language.

I tend to think that OfflineIMAP did not started fully right from the
beginning and later improvements did not help, adding poorly written
code/features on top of the whole mess.

It's a pitty to get so confused about such a feature and having the
implementation spread over so many "random" places in the code. :-/

-- 
Nicolas Sebrecht