Automated tests using dovecot

[Ethan Glasser-Camp]

On 06/09/2011 02:37 PM, Nicolas Sebrecht wrote:
> First of all, I didn't test the tests. I don't want to install any
> system-wide imap server on my laptop and the HDD on my very old server
> crashed, so I have to wait for a new full server. I've read the scripts.

Could you clarify what you would be willing to install on your laptop?  
It's possible to install dovecot without it running as a system service 
-- that's what I do.  Are you unwilling to have dovecot installed at all?

> There are 3 ways to test an IMAP client:
>
>    - use any IMAP server such as gmail, gmx, etc. (this is somewhat
>      limited as we tester can't check what was done on the server);
>    - install and configure a system-wide IMAP server;
>    - use a chrooted environment.

I think the approach I took is closest to #3, but again, I didn't use an 
actually chroot with a full OS install available on it.  I just followed 
the Dovecot wiki's instructions on running "rootless", i.e. as a 
non-privileged user.  While a chroot is a nice idea, I wonder if it's 
really necessary or even useful.  What's the payoff?  Even in a chroot, 
we still can't run on port 143 (for example) without having root privileges.

> In the long-term, we should be fine with any of the two latter
> possibilities, at least. The main consequence is that each solution
> should also provide the same API to easily check the results. I'm
> thinking about shell scripts, here. Something like:
>
>    cooked.sh --number --new-mail # returns the number of new mail only
>                                  # (since last checkpoint)
>    cooked.sh --create-ckeckpoint
>    cooked.sh --plain --all       # return all emails as plain text
>    ...
>
>    cook.sh --new<  mail.sample   # add a new mail on server without any
>                                  # IMAP client

I like this approach, and it feels cleaner and more flexible, but it 
also does sound like a little more work.  I am afraid we are going to 
fall into the trap of "let's build a test harness", instead of "let's 
use a test suite to test our real project" :)

>       `tests-suites.conf  # where testers configure the environment(s) they like

Meaning, "Please test against Gmail with these credentials", "Please 
test against rootless Dovecot", and so on?

>      |
>       `update-tests-suites.sh  # fetch choosen environment, configuration
>      |                         # sample files from an external repository

Does this mean, "download and install Dovecot"?  I think that's outside 
the scope of this test harness.

>       `imap-server/  # where to store external repositories and
>             |        # high-level API scripts for cooking
>             |
>             `system-wide/
>             |    |
>             |     `dovecot-vXYZ/  # configuration file samples, low-level
>             |    |                # scripts for cooking, etc
>             |    |
>             |     `imapserver2-vXYZ/

So cook.sh would be a wrapper around simpler scripts that actually 
implement the functionality (deliver, count mail, etc.)?

> It should not be hard work to achieve this. Again, I'm not thinking
> about something advanced. You could start a new public repository (which
> I could maintain in the longer run) for the server side configuration
> files and low-level scripts.
>
> What do you think?

I think it's a great idea but somewhat ambitious considering our limited 
pool of manpower.  Is it really likely that other projects will join in 
and contribute?  The perfect is the enemy of the good.

Ethan