[PATCH] Re: This is what I call test suite for now. What a mouthful :-).

[Sebastian Spaeth]

On Sun, 12 Dec 2010 21:15:36 +0100, Nicolas Sebrecht <nicolas.s-dev at laposte.net> wrote:
> Think of OfflineIMAP being a box. We don't care about what happens
> inside this box, do we? What we know is that the tool is deterministic.
> Then, we can give predefined inputs to the box, let it work and only
> check for the output. The box could use whatever strategy or magic. We
> don't know what they are and we don't _want_ to know.

Thanks for the explanation. I do see your point now. I just happen to
disagree. :-) I do want high-level as well as low-level tests. For all
that I care, we could just run the highlevel tests if they succeed. In
that case we really are not interested how the box works. But in case
high-level does not work, I believe lower-level tests are warranted. In
that case I want to know if a change in the syncing strategy cause the
failure, or a bug in a specific backend implementation.

I agree that high-level tests are what we *usually* want.

> This approach is all about the behaviour. Testing the behaviour is what
> is the most important to test.  We don't want to know what is the
> internal syncing strategy and how it stands because it doesn't matter.

It matters when trying to find out what exactly broke, when something
breaks :). It doesn't matter if everything works fine.

> I think you're missing my point, here. Doing it in shell, python or
> anything else is not sensible.

I might have missed it and I am still not getting it, sorry. We need to
invoke offlineimap *somehow* in order to test it. If not in python, then
via shell. How else should we invoke it?
 
> Folder.syncfolder() or anything else found in the code is not that
> interesting to test at that point. What /could/ be done later (if most
> behaviour cases are tested) is to provide more fine-grained tests
> against internals strategies (drivers, synchronisation engine, etc).

Folder.syncfolder is *the* high-level API that should make sure that a
remote folder is syncronized to a local folder after the invocation. It
*is* the box, the rest is just box wrapping. So if that works,
offlineimap works. If that fails, offlineimap fails. I find that a
convenient level to test.

What you seem to propose is a bash testsuite that invokes
command line offlineimap in a series of commands, using real config
files and accounts. This sounds very much like git has, so if that is
the test suite to be adopted it would make sense to import their test
suite thingie. I will just not be the one doing that, for that I know
and like shell too little and love python too much :).

Sebastian