[PATCH] Draft of maxfetch option

[Nicolas Sebrecht]

On Thu, May 05, 2016 at 07:41:41PM +0200, Łukasz Żarnowiecki wrote:

> Signed-off-by: Łukasz Żarnowiecki <dolohow at outlook.com>
> ---
> 
> This is just a draft for you to take a look if this option actually make
> any sense.  I thought about it and I think it might be useful for
> someone.  I am wondering if it is possible to pass search condition like
> you did in "_msg_to_fetch" for "imapobj.search".  If not, I probably
> should use "cachemessagelist" where I can slice the list, but I guess
> the messages are not sorted by date.  Any hint would be valuable.
> 
> Please take a look I tell me what you think.

For each sync, offlineimap retrieves the list of messages of both sides
(local and remote).  Those collections are compared with the cache which
is the base of the 3-way merge algo to know what to do: what are the new
emails, which one are deleted, flag changes, etc.

When we strip a collection we must be sure to consider the same starting
point (UID) for all the collections. Otherwise, the comparisons are
broken and the result is this would delete wrong emails in the worst
case. Getting things right in this area might be hard. This is what
maxage is doing.

Compared to maxage, maxfetch is different in two ways:

- offer more grained control to the max number of messages to consider
  for a sync;

- move forward the oldest the starting point (UID) for each sync when
  necessary.

IOW, I think the maxfetch feature is very similar to maxage.  Checking
what is done for maxage would probably help:

* you might be interested by this blog post

http://www.offlineimap.org/devel/2015/04/14/why-we-changed-maxage.html

* the code running maxage.


I have no idea if it's worth. I'd say if it helps you, it's enough to
worth it.  However, this might only help if there are a lot of new
messages since the previous sync /and/ the user only cares about the
latest ones.

> diff --git a/offlineimap.conf b/offlineimap.conf
> index 6f05dcc..706bee7 100644
> --- a/offlineimap.conf
> +++ b/offlineimap.conf
> @@ -364,6 +364,22 @@ remoterepository = RemoteExample
>  
>  # This option stands in the [Account Test] section.
>  #
> +# maxfetch enables you to sync only the last specified number of emails.

There are two times we can decide (compute) a common UID to become the
starting point:

1. At "discover" time: when we fetch the collection in Maildir. In this
case, we don't know how many emails are synced.

2. Once both collections are compared.

I'm not sure what's the best way to get this implemented easily.

> +#                                                                        This is
> +# useful when you have limited bandwidth or when you know that the messages
> +# could be rotated on the IMAP server.
> +#
> +# This works per folder. For OfflineIMAP it will be like these messages do not
> +# exist.
> +#
> +# maxfetch is allowed only when the local folder of type Maildir.
> +#
> +# The maxfetch option expects an integer.
> +#
> +#maxfetch = 3

<...>

> diff --git a/offlineimap/folder/Maildir.py b/offlineimap/folder/Maildir.py
> index 77a774b..718b233 100644
> --- a/offlineimap/folder/Maildir.py
> +++ b/offlineimap/folder/Maildir.py
> @@ -152,6 +152,7 @@ class MaildirFolder(BaseFolder):
>          """
>  
>          maxsize = self.getmaxsize()
> +        maxfetch = self.getmaxfetch()
>  
>          retval = {}
>          files = []
> @@ -162,6 +163,8 @@ class MaildirFolder(BaseFolder):
>                           filename in os.listdir(fulldirname))
>  
>          date_excludees = {}
> +        if maxfetch:
> +            files = files[-maxfetch:]

I don't think the 'files' collection is sorted by UID order!

>          for dirannex, filename in files:
>              if filename.startswith('.'):
>                  continue # Ignore dot files.

-- 
Nicolas Sebrecht