Test suite ?

[Vladimir Marek]

> [ Could you please NOT set the Mail-Followup-To header. This breaks
>   threads. ]

I'm sorry about that. I turned off Mail-Followup-To for this alias
(hopefully).

> > I'm thinking about making test suite. I even wrote proof of concept IMAP
> > server which has no logic but can be scripted.
> > 
> > But first, what and how should be tested?
> 
> I think this is what you're looking for:
> 
>   Subject: RFC: starting a unit tests suites
>   From: Nicolas Sebrecht <nicolas.s-dev at laposte.net>
>   Date: Fri, 15 Apr 2011 21:20:58 +0200
>   Message-ID: <20110415192058.GC14542 at vidovic>


Thank you, I'll take close look at this later. I'm glad I'm not the only
one wanting to have some tests.



> > I can see three parts - IMAP, Maildir and state directory
> > 
> > IMAP
> >  - simple socket server
> >  - scripted scenarios
> 
> I think you're a bit confused, here. Scenarios is on the client side
> whereas everything else you put in this section is about the server.

Maybe we are thinking about different tests here? What I had in mind was
something like:

Let's build fake IMAP server which does not support APPENDUID and
SEARCH. Now run OfflineIMAP with exactly this config and see if it
synchronizes my mails correctly. Let's check that it still works against
(scripted) Gmail like server.

So I'm thinking about script driven IMAP-like server.


> >  - single threaded only
> 
> Why? It looks like a strong limitation, to me.

Because that would be more difficult to implement:) Also it will be
harder to make OfflineIMAP behave consistently between two test runs.



> >  - should we check order of commands sent? Or should it be rather state
> >    machine (find corresponding output to a given input)? Strictly
> >    ordered commands means tighter checks, but does OfflineIMAP always
> >    send commands in the same order?
> 
> Should we care about low-level issues _in a test suites_? My answer is
> NO! Please, don't!
> 
> First, because we aren't working at the IMAP level. Imaplib2 does this.
> If we decide to switch of library, everything done in this area is lost.

That's probably poor explanation on my side. My question is if
OfflineIMAP is always guaranteed to send the IMAP commands in the same
order. And yes, if we switch Imaplib2 for something else (not likely to
happen I guess), I would expect some tests to break. But is that bad?
Maybe we had certain sequence of IMAP commands because of some broken
IMAP server?


> Second, because we care about emails. I mean we have to know what was
> uploaded, downloaded, modified, etc.

Agreed. My first intention was to try to make sure that happens with all
possible strange IMAP servers. But maybe that's too much work ...



> > Maildir
> >  - file names contain random strings, so before comparing to known good
> >    state first sanity them?
> >  - the mails themselves can contain random X-OfflineIMAP header. That
> >    can be sanitized too.
> 
> I have no idea why we would want to check that. It works just fine for
> everybody and never heard about issues until the last (fixed) "windows
> support" due to a very unusual configuration.

I'm glad to hear that, IMAP part is probably the most sensitive area.
Still would be nice to have test if OfflineIMAP didn't lost content of
the email. But I agree with you that it has very low priority.


> >  - compare list of files
> 
> One email or a list of emails is the same thing, to me.

I meant something like

cd maildir; find . | sort; comare that to known good result. To see if
the mail is still delivered to 'new' directory as it used to be before.



> >  - compare their contents
> 
> Yes. An email is a content. What we need to know for a test suites is
> mainly the body.

ok



> > State directory
> >  - no research done yet
> 
> If you're talking about the cache, I guess we don't care.

I had two cases in mind which might be worth testing. But it's really
just brainstorming

 - UIDVALIDITY changed, make sure that everything is fine
 - if we implement folder deletion (both local and remote), make sure we
   delete cache too



> > Anyone had thoughts about test suite?
> 
> This has been discussed, recently. Ethan even provided a patch. I didn't
> merge it due to the issues discussed in this thread:
> 
>   Subject: Automated tests using dovecot
>   From: Ethan Glasser-Camp <glasse at cs.rpi.edu>
>   Date: Mon, 06 Jun 2011 07:53:55 -0400
>   Message-ID: <4DECBFD3.1090505 at cs.rpi.edu>
> 
> This thread is what you're looking for, I guess.

Thank you for the pointer.


> Notice I was wrong and did not know that a user-level installation of
> Dovecot is possible. So, a dovecot (whatever) installation into a
> dedicated git repository is easy.

I don't know Dovecot, but I hoped to be able to simulate strange IMAP
servers and cases in the tests. Like Gmail, UIDVALIDITY changed, not
supported SEARCH command. If there would be a way to make Dovecot behave
exactly as we like, that would be excellent!

Thank you for your inputs, I'll read through the threads you told me
about.
-- 
	Vlad