[PATCH] Re: This is what I call test suite for now. What a mouthful :-).

[Nicolas Sebrecht]

On Fri, Dec 10, 2010 at 10:47:52PM +0100, Sebastian Spaeth wrote:
> 
> It contains 2 working and one non-functional test so far and of course some test infrastructure. Works against current master. Run with "python test.py" :)
> 
> Signed-off-by: Sebastian Spaeth <Sebastian at SSpaeth.de>
> ---
>  tests/test.py |  267 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 267 insertions(+), 0 deletions(-)
>  create mode 100644 tests/test.py
> 
> diff --git a/tests/test.py b/tests/test.py
> new file mode 100644
> index 0000000..7aabc78
> --- /dev/null
> +++ b/tests/test.py
> @@ -0,0 +1,267 @@
> +import offlineimap
> +from offlineimap.repository import Repository
> +import random
> +import tempfile
> +import unittest
> +import os
> +import shutil
> +import threading
> +
> +class Folder2FolderScaffold:
> +    """This provides 'scaffolding' for testing Maildir<->Maildir syncing"""

As said in another mail, I think this is not what we want. OfflineIMAP
doesn't even provide this feature. Why would we need to test it?

The more I think about a good strategy for the test suites the more I
think we need to provide a locahost IMAP server. This could be done
through a predefined chrooted environment. This way, we could easily set
up known test cases in the server side.

What I would do is:

0) have a OfflineIMAP client configuration template
1) have a "chrootable" IMAP server environment template
2) build the target environment from here
3) customize the server to provide the test cases
4) set up the client configuration and customize as needed
4) do the test(s)
5) check the result(s)
6) come back to (2) everywhere it's needed for other stories

It may be worth trying to NOT use python for the test suites at a first
step. Something in pure shell or something could be enough for most
test cases. Also, I tend to say so because of what I've read in your
tests. We don't want to not fall into the "design test suites" pattern
trap (as opposed to the "behaviour test suites" pattern which is what we
care most).(*)

For example, we should not set up the client configuration using
internal class or anything like that. We don't care to test the X class
or Y function as it stand in the current code. This code is subject to
refactoring as much as desirable. Refactoring code shouldn't have to
impact the test suites. Instead, we should set up a temporary
configuration file (customizable from tests to tests) that OfflineIMAP
would use (with the -c option).


* In case of pure shell, we could consider is to not "import
  OfflineIMAP" at all. Of course, this is a bit drastic and could not be
  applicable to test some highly core implementations. Core implementation
  tests is the only test case where the "design test suites" pattern has
  to be considered.

> +    def __init__(self):
> +        #list of files (or directories that we have to cleanup on exit
> +        self._tmpfiles = []

Static temporary files maping is hard to maintain and tend to introduce
artefacts. We should blindly wipe the target paths when starting the
test suites.


-- 
Nicolas Sebrecht