Automated tests using dovecot

[Nicolas Sebrecht]

On Tue, Jun 14, 2011 at 07:01:53AM -0400, Ethan Glasser-Camp wrote:
> 
> On 06/09/2011 02:37 PM, Nicolas Sebrecht wrote:
> >First of all, I didn't test the tests. I don't want to install any
> >system-wide imap server on my laptop and the HDD on my very old server
> >crashed, so I have to wait for a new full server. I've read the scripts.
> 
> Could you clarify what you would be willing to install on your
> laptop?  It's possible to install dovecot without it running as a
> system service -- that's what I do.  Are you unwilling to have
> dovecot installed at all?

If some IMAP server can be installed and run as non-privileged user,
it's just fine.

> >There are 3 ways to test an IMAP client:
> >
> >   - use any IMAP server such as gmail, gmx, etc. (this is somewhat
> >     limited as we tester can't check what was done on the server);
> >   - install and configure a system-wide IMAP server;
> >   - use a chrooted environment.
> 
> I think the approach I took is closest to #3, but again, I didn't
> use an actually chroot with a full OS install available on it.  I
> just followed the Dovecot wiki's instructions on running "rootless",
> i.e. as a non-privileged user.  While a chroot is a nice idea, I
> wonder if it's really necessary or even useful.  What's the payoff?
> Even in a chroot, we still can't run on port 143 (for example)
> without having root privileges.
> 
> >In the long-term, we should be fine with any of the two latter
> >possibilities, at least. The main consequence is that each solution
> >should also provide the same API to easily check the results. I'm
> >thinking about shell scripts, here. Something like:
> >
> >   cooked.sh --number --new-mail # returns the number of new mail only
> >                                 # (since last checkpoint)
> >   cooked.sh --create-ckeckpoint
> >   cooked.sh --plain --all       # return all emails as plain text
> >   ...
> >
> >   cook.sh --new<  mail.sample   # add a new mail on server without any
> >                                 # IMAP client
> 
> I like this approach, and it feels cleaner and more flexible, but it
> also does sound like a little more work.  I am afraid we are going
> to fall into the trap of "let's build a test harness", instead of
> "let's use a test suite to test our real project" :)
> 
> >      `tests-suites.conf  # where testers configure the environment(s) they like
> 
> Meaning, "Please test against Gmail with these credentials", "Please
> test against rootless Dovecot", and so on?

Yes, things like that. Also, it could be "use this URL for checking out
a test environment", etc.

> >     |
> >      `update-tests-suites.sh  # fetch choosen environment, configuration
> >     |                         # sample files from an external repository
> 
> Does this mean, "download and install Dovecot"?

Mainly. I mean somehting like:

  git_update_repo () {
    cd "${TEST_PATH}/imap-server"
    if [ -d $1 ]; then
      # update
    else
      # git clone

  git_update_repo "dovecot_as_user"
    

>                                                  I think that's
> outside the scope of this test harness.

FMHO, it allows:
- to split the IMAP server environment and scenarios
- to launch tests against older releases

> >      `imap-server/  # where to store external repositories and
> >            |        # high-level API scripts for cooking
> >            |
> >            `system-wide/
> >            |    |
> >            |     `dovecot-vXYZ/  # configuration file samples, low-level
> >            |    |                # scripts for cooking, etc
> >            |    |
> >            |     `imapserver2-vXYZ/
> 
> So cook.sh would be a wrapper around simpler scripts that actually
> implement the functionality (deliver, count mail, etc.)?

Yes. :-)

> >It should not be hard work to achieve this. Again, I'm not thinking
> >about something advanced. You could start a new public repository (which
> >I could maintain in the longer run) for the server side configuration
> >files and low-level scripts.
> >
> >What do you think?
> 
> I think it's a great idea but somewhat ambitious considering our
> limited pool of manpower.

I don't think, so. Again, I'm not asking for something advanced. Just
keep a long-term goal and model in mind to start the good way. In the
short-term, it means splitting things out and using a dedicated
repository.

>                            Is it really likely that other projects
> will join in and contribute?

I don't know. Nobody knows until it's done. :-)

-- 
Nicolas Sebrecht