Automated tests using dovecot

[Nicolas Sebrecht]

Sorry for the delay. I was in holidays and I think there was more
important patches to be merged before reviewing this. Also, I wanted to
take a bit more time before giving my anwser. :-)

On Mon, Jun 06, 2011 at 07:53:55AM -0400, Ethan Glasser-Camp wrote:

> I know that patches generally get sent to the mailing list, but this
> is a few hundred K of very WIP stuff so I'm just going to point to
> my github branch:
> 
> https://github.com/glasserc/offlineimap/tree/volatile%2Fautomated-tests

Not a problem. As I understand, this work is WIP and experimental.

> This commits a bash-based test harness imported from notmuch, which
> was originally imported from git.  It demonstrates a test that
> brings up a local, non-root dovecot instance, delivers a mail to it,
> and then syncs that mail using offlineimap.
> 
> This doesn't go as far as Nicolas suggested in [1], because there's
> no chroot and at least once before I got everything setup correctly,
> offlineimap tests tried to sync against my actual, real mailbox.
> Nevertheless it's pretty simple and better than nothing :)
> 
> Feedback would be gratefully received.

First of all, I didn't test the tests. I don't want to install any
system-wide imap server on my laptop and the HDD on my very old server
crashed, so I have to wait for a new full server. I've read the scripts.

I'd like to say I still have a WIP for the tests suites but I know I'm
very long to provide it.

Please, remember that a test suites is a succes if it can widely be
used. So, it has to be as easy as possible and well-documented.


Anyway, your work made me think a bit deeper on how to set up the tests
suites.

I think we should actually split it into two independant problems:

 - providing a IMAP server for the tests.
 - the tests suites with the scenarios.


#1. Providing a IMAP server for testing IMAP clients.

I'm sure we are not alone to face against this problem. There are a lot
of IMAP clients outside. I've already looked at how other projects solve
it but couldn't find any answer. My guess is that nobody did the work as
it's not so easy and most developers do the tests against an IMAP server
configured or set up on their own.

There are 3 ways to test an IMAP client:

  - use any IMAP server such as gmail, gmx, etc. (this is somewhat
    limited as we tester can't check what was done on the server);
  - install and configure a system-wide IMAP server;
  - use a chrooted environment.

In the long-term, we should be fine with any of the two latter
possibilities, at least. The main consequence is that each solution
should also provide the same API to easily check the results. I'm
thinking about shell scripts, here. Something like:

  cooked.sh --number --new-mail # returns the number of new mail only
                                # (since last checkpoint)
  cooked.sh --create-ckeckpoint
  cooked.sh --plain --all       # return all emails as plain text
  ...

  cook.sh --new < mail.sample   # add a new mail on server without any
                                # IMAP client

Side note:
 for the chrooted environment I think about writing a independant
 project we could rely on (using a dedicated repository, etc). This way,
 we could let testers choose what they want and call others projects to
 join our efforts.


#2. Writing scenarios.

Writing scenarios should be comfortable with both system-wide and
chrooted IMAP servers. It stands for OfflineIMAP only, of course. It
would use the API provided by (1).

We still have the time to choose the language for it. I'm not totally
convinced that bash scripts is the better. They suck at beeing easily
portable and our code-base is in python (so, we could expect more python
contributors).


That's why I think this new tests suites is very good but should be
improved to:

  a) treat both problems each in a independant manner;
  b) write a python-based test suites


Be convinced I'm _NOT_ talking about great-advanced software.

For (a), we just need public conventions to store and configure the
server on one side and write server-oriented little shell scripts.
To do so, I propose to use these paths:

  /test/   # root test directory
    |
    |
     `tests-suites.conf  # where testers configure the environment(s) they like
    |
     `update-tests-suites.sh  # fetch choosen environment, configuration
    |                         # sample files from an external repository
    |
     `start-tests.sh
    |
     `scenarios   # our scenarios for OfflineIMAP
    |
     `imap-server/  # where to store external repositories and
           |        # high-level API scripts for cooking
           |
           `system-wide/
           |    |
           |     `dovecot-vXYZ/  # configuration file samples, low-level
           |    |                # scripts for cooking, etc
           |    |
           |     `imapserver2-vXYZ/
           |
            `environments
                |
                 `dovecot-vXYZ/  # dovecot chrooted environment, low-level
                |                # scripts for cooking, etc
                |
                 `imapserver2-vXYZ/


For (b) (write a python based tests suites), we don't need any
public/official tests suites since it's just about writing _stories_.


It should not be hard work to achieve this. Again, I'm not thinking
about something advanced. You could start a new public repository (which
I could maintain in the longer run) for the server side configuration
files and low-level scripts.

What do you think?

-- 
Nicolas Sebrecht