offlineimap and dovecot-antispam: APPEND issues

[chris coleman]

I can foresee a time, maybe soon, when the OLI algorithm might naturally add the capability of working globally - especially on IMAP servers that support modern features that would make global sync relatively simple to implement.

One feature that would help achieve this is an IMAP server that implements the 2010 IETF draft proposal of the single MOVE command, which allows message to be moved from one folder to another, on the same IMAP server.   One command instead of about 4, and zero upload/download of the moved messages.  This would perform about 1000x faster than today's upload/download method, and greatly reduce the chances of a race condition/deadlock with other connected IMAP clients.


Basic algorithm: between runs, retain the unique keys of all the messages in all of the local and remote mailboxes.  The next time you run, use this key information to detect changes in the mailbox contents on both sides.  For all messages that have moved, simply do a mass MOVE or a COPY - from one mailbox to another - hundreds of messages can be moved/copied at once.  A few seconds to run this, instead of  hundreds of seconds.

Also - if IMAP servers were to support something like snapshots (or equivalent), this would help tremendously to reduce the occurrence of runtime errors, due to other clients changing the mailbox state, message state, and flags, during the run of OLI.




Daniel wrote:
chris coleman wrote on Wed, Jan 11, 2012 at 08:22:56 -0800:
> I don't recall the reason we're moving messages from one mailbox to
> another with APPEND+DELETE instead of COPY+DELETE.  
> 

Because the sync algorithm works on a folder-by-folder basis and not
globally.

At least, that's what Sebastian tells me.

> Might've originally been an attempt to preserve the UID of the message
> when going from one mailbox to another - but UID is lost across
> mailboxes so it can't actually be that.  
> 
> Sebastian and Nicholas should have a better idea why APPEND+DELETE
> instead of COPY+DELETE.
> 
> Chris
> 
> 
> 
> Micah wrote:
> 
> Hello,
> 
> I've recently implemented the dovecot-antispam plugin and was dismayed
> to find that it has a particularly bad failure situation that only
> occurs for Offlineimap users... of which I am one. This seems
> unfortunate, and should be resolved. So I'm writing here to see what can
> be done about this, and CC'ing the dovecot-antispam maintainer in the
> process. Hopefully a good dialoge about the problem and potential ways
> forward will result.
> 
> The summary of the problem is that offlineimap operates in a particular
> way with regard to APPEND operations that make it impossible for the
> dovecot-antispam plugin to detect certain spam training situations. 
> 
> The problem is when training filters. When a user is looking through
> their Spam folder, and finds a message that was incorrectly tagged as
> spam and decides to move it out of their Spam folder. The way
> offlineimap handles this situation is that when it runs it does an
> APPEND+DELETE, and this results in a situation that makes the
> dovecot-antispam plugin not be able to detect that this is happening and
> thus will not train the message as not-spam (ham). If the user does this
> move in a webmail client, or other mail client, things work fine.
> 
> The dovecot-antispam man page has this warning to say about APPEND
> operations, which is what offlineimap does:
> 
> ALLOWING APPENDS?
> 
>        You should be careful with allowing APPENDs to SPAM folders. The
>        reason for possibly allowing it is to allow not-SPAM --> SPAM
>        transitions to work with offlineimap. However, because with
>        APPEND the plugin cannot know the source of the message, multiple
>        bad scenarios can happen:
> 
>         1. SPAM --> SPAM transitions cannot be recognised and are trained
> 
>         2. the same holds for Trash --> SPAM transitions
> 
>        Additionally, because we cannot recognise SPAM --> not-SPAM
>        transitions, training good messages will never work with APPEND.
> 
> I'm hoping that this is enough information for offlineimap developers to
> understand what the issue is, but if not I've CC'd the dovecot-antispam
> developer if further clarification is needed.
> 
> micah
> 
> -
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20120112/bfea61a7/attachment-0003.html>