Test suite ?

[Nicolas Sebrecht]

[ Could you please NOT set the Mail-Followup-To header. This breaks
  threads. ]

On Mon, Aug 08, 2011 at 04:20:34PM +0200, Vladimir Marek wrote:
> Hi,

Hi,

> I'm thinking about making test suite. I even wrote proof of concept IMAP
> server which has no logic but can be scripted.
> 
> But first, what and how should be tested?

I think this is what you're looking for:

  Subject: RFC: starting a unit tests suites
  From: Nicolas Sebrecht <nicolas.s-dev at laposte.net>
  Date: Fri, 15 Apr 2011 21:20:58 +0200
  Message-ID: <20110415192058.GC14542 at vidovic>

> I can see three parts - IMAP, Maildir and state directory
> 
> IMAP
>  - simple socket server
>  - scripted scenarios

I think you're a bit confused, here. Scenarios is on the client side
whereas everything else you put in this section is about the server.

>  - single threaded only

Why? It looks like a strong limitation, to me.

>  - should we check order of commands sent? Or should it be rather state
>    machine (find corresponding output to a given input)? Strictly
>    ordered commands means tighter checks, but does OfflineIMAP always
>    send commands in the same order?

Should we care about low-level issues _in a test suites_? My answer is
NO! Please, don't!

First, because we aren't working at the IMAP level. Imaplib2 does this.
If we decide to switch of library, everything done in this area is lost.

Second, because we care about emails. I mean we have to know what was
uploaded, downloaded, modified, etc.

> Maildir
>  - file names contain random strings, so before comparing to known good
>    state first sanity them?
>  - the mails themselves can contain random X-OfflineIMAP header. That
>    can be sanitized too.

I have no idea why we would want to check that. It works just fine for
everybody and never heard about issues until the last (fixed) "windows
support" due to a very unusual configuration.

>  - compare list of files

One email or a list of emails is the same thing, to me.

>  - compare their contents

Yes. An email is a content. What we need to know for a test suites is
mainly the body.

> State directory
>  - no research done yet

If you're talking about the cache, I guess we don't care.

> Anyone had thoughts about test suite?

This has been discussed, recently. Ethan even provided a patch. I didn't
merge it due to the issues discussed in this thread:

  Subject: Automated tests using dovecot
  From: Ethan Glasser-Camp <glasse at cs.rpi.edu>
  Date: Mon, 06 Jun 2011 07:53:55 -0400
  Message-ID: <4DECBFD3.1090505 at cs.rpi.edu>

This thread is what you're looking for, I guess.

Notice I was wrong and did not know that a user-level installation of
Dovecot is possible. So, a dovecot (whatever) installation into a
dedicated git repository is easy.

-- 
Nicolas Sebrecht