[PATCH] Re: This is what I call test suite for now. What a mouthful :-).

[Sebastian Spaeth]

On Sun, 12 Dec 2010 14:13:08 +0100, Nicolas Sebrecht <nicolas.s-dev at laposte.net> wrote:
> As said in another mail, I think this is not what we want. OfflineIMAP
> doesn't even provide this feature. Why would we need to test it?

Because to *test the syncing strategy* the backend doesn't matter. If we
want to find out, what happens if we at the same time delete a remote
message and fiddle with local flags, it doesn't matter wether we slowly
sync over a real IMAP connection of use a quick local maildir.

Of course we will also need to test IMAP connections, to see whether our
IMAP implementation has bugs. But most of the tests will really only
need to be run locally.

> It may be worth trying to NOT use python for the test suites at a first
> step. Something in pure shell or something could be enough for most
> test cases. Also, I tend to say so because of what I've read in your
> tests. We don't want to not fall into the "design test suites" pattern
> trap (as opposed to the "behaviour test suites" pattern which is what we
> care most).(*)

I do not get that, sorry. Could you elaborate on what you mean by those
patterns?
 
> For example, we should not set up the client configuration using
> internal class or anything like that. We don't care to test the X class
> or Y function as it stand in the current code. This code is subject to
> refactoring as much as desirable. Refactoring code shouldn't have to
> impact the test suites. Instead, we should set up a temporary
> configuration file (customizable from tests to tests) that OfflineIMAP
> would use (with the -c option).

Oh, I very much want to test low-level as well as high-level
functionality. Also I want to focus on high-level functionality, of
course. But as long as those high-level APIs are not up to par, I go as
deep as needed to set up things. I do test highlevel functionality, such
as Folder.syncfolder()...

Of course, I also want to have tests that simply invoke offlineimap. I
don't see what implementing that in bash vs python has to do with it
though.
 
> > +    def __init__(self):
> > +        #list of files (or directories that we have to cleanup on exit
> > +        self._tmpfiles = []
> 
> Static temporary files maping is hard to maintain and tend to introduce
> artefacts. We should blindly wipe the target paths when starting the
> test suites.

? We do, I use that to store the locations of the temporary  metadir and
maildir root folders, which are wiped on exiting the tests (we should
keep it when a test fails though). I don't use it to store the location
of each and every file.

Sebastian