[PATCH] Re: This is what I call test suite for now. What a mouthful :-).

[Nicolas Sebrecht]

On Sun, Dec 12, 2010 at 08:26:54AM -0600, Sebastian Spaeth wrote:
> On Sun, 12 Dec 2010 14:13:08 +0100, Nicolas Sebrecht <nicolas.s-dev at laposte.net> wrote:
> > As said in another mail, I think this is not what we want. OfflineIMAP
> > doesn't even provide this feature. Why would we need to test it?
> 
> Because to *test the syncing strategy* the backend doesn't matter. If we
> want to find out, what happens if we at the same time delete a remote
> message and fiddle with local flags, it doesn't matter wether we slowly
> sync over a real IMAP connection of use a quick local maildir.
> 
> Of course we will also need to test IMAP connections, to see whether our
> IMAP implementation has bugs. But most of the tests will really only
> need to be run locally.
> 
> > It may be worth trying to NOT use python for the test suites at a first
> > step. Something in pure shell or something could be enough for most
> > test cases. Also, I tend to say so because of what I've read in your
> > tests. We don't want to not fall into the "design test suites" pattern
> > trap (as opposed to the "behaviour test suites" pattern which is what we
> > care most).(*)
> 
> I do not get that, sorry. Could you elaborate on what you mean by those
> patterns?

Think of OfflineIMAP being a box. We don't care about what happens
inside this box, do we? What we know is that the tool is deterministic.
Then, we can give predefined inputs to the box, let it work and only
check for the output. The box could use whatever strategy or magic. We
don't know what they are and we don't _want_ to know.

In the case of IMAP to maildir story, the inputs are the configuration
file and what is provided by the IMAP server. The ouput is the given
maildir.

This approach is all about the behaviour. Testing the behaviour is what
is the most important to test.  We don't want to know what is the
internal syncing strategy and how it stands because it doesn't matter.

> > For example, we should not set up the client configuration using
> > internal class or anything like that. We don't care to test the X class
> > or Y function as it stand in the current code. This code is subject to
> > refactoring as much as desirable. Refactoring code shouldn't have to
> > impact the test suites. Instead, we should set up a temporary
> > configuration file (customizable from tests to tests) that OfflineIMAP
> > would use (with the -c option).
> 
> Oh, I very much want to test low-level as well as high-level
> functionality. Also I want to focus on high-level functionality, of
> course. But as long as those high-level APIs are not up to par, I go as
> deep as needed to set up things. I do test highlevel functionality, such
> as Folder.syncfolder()...
> 
> Of course, I also want to have tests that simply invoke offlineimap. I
> don't see what implementing that in bash vs python has to do with it
> though.

I think you're missing my point, here. Doing it in shell, python or
anything else is not sensible.

Folder.syncfolder() or anything else found in the code is not that
interesting to test at that point. What /could/ be done later (if most
behaviour cases are tested) is to provide more fine-grained tests
against internals strategies (drivers, synchronisation engine, etc).


-- 
Nicolas Sebrecht