[PATCH] Re: This is what I call test suite for now. What a mouthful :-).

[Nicolas Sebrecht]

On Sun, Dec 12, 2010 at 02:56:29PM -0600, Sebastian Spaeth wrote:
> 
> On Sun, 12 Dec 2010 21:15:36 +0100, Nicolas Sebrecht <nicolas.s-dev at laposte.net> wrote:
> > Think of OfflineIMAP being a box. We don't care about what happens
> > inside this box, do we? What we know is that the tool is deterministic.
> > Then, we can give predefined inputs to the box, let it work and only
> > check for the output. The box could use whatever strategy or magic. We
> > don't know what they are and we don't _want_ to know.
> 
> Thanks for the explanation. I do see your point now. I just happen to
> disagree. :-) I do want high-level as well as low-level tests. For all
> that I care, we could just run the highlevel tests if they succeed. In
> that case we really are not interested how the box works. But in case
> high-level does not work, I believe lower-level tests are warranted. In
> that case I want to know if a change in the syncing strategy cause the
> failure, or a bug in a specific backend implementation.
> 
> I agree that high-level tests are what we *usually* want.

Not usually, it *is* what we need! :-)

You're taking the problem the wrong way. The normal case is when test
suites succeed. If not, we obviously would like to know a bit more. The
next usefull information usually comes using a bissection tool through
the commits to find where the regression is introduced. Regression is
the key word. A good test suites help to find regressions.

When a regression is found, we then look more deeply inside the code
around the lines the patch touches. This is really how to work with test
suites. We start from high level because it prevents from wrong tests.

If had to start from a low level, it would be very easier to have wrong
tests: they highly depends of the code. So a patch may be good and the
test just not updated or raising wrong (initial) assumptions of how the
working code should be.

> I might have missed it and I am still not getting it, sorry. We need to
> invoke offlineimap *somehow* in order to test it. If not in python, then
> via shell. How else should we invoke it?

We can call a program from any other program written in any language.

> Folder.syncfolder is *the* high-level API that should make sure that a
> remote folder is syncronized to a local folder after the invocation. It
> *is* the box, the rest is just box wrapping.

So don't squeeze the box. Use the entire box. ,-)
Wrappers are code by themselves there is no reason to ignore.

> What you seem to propose is a bash testsuite that invokes
> command line offlineimap in a series of commands, using real config
> files and accounts.

Yes! Except it may be the shell or anything else. The good news when
another language is used is that nobody can write tests sticked to the
software implementation.

>                     This sounds very much like git has, so if that is
> the test suite to be adopted it would make sense to import their test
> suite thingie.

You're right when you say that this is what git has. They work like that
because it is definitely the best thing to do (for all the reasons I
explained before).

Now, I'm not sure we want to take their test suites tool. It has some
limitations due to the use of a shell and the libraries. Shell scripts
are not very portable. Python is much better at that game, for example.

-- 
Nicolas Sebrecht