RFC: starting a unit tests suites

[Nicolas Sebrecht]

Hi,

We all know we need a tests suites. Here are some thoughts about the
best style of tests suites we want. Two approaches are in competition.

Thanks for your comments.


1) unit testing

  We test each function, module, API we want to.

  Advantages: 

    * can be implemented at any level (low level for drivers or high
      level for processes)
    * add documentation to the code
    * can be launched locally without the need to have a IMAP server

  Drawbacks:

    * can easily go wrong on code refactoring (function signature
      changing, etc)
    * can easily be poorly implemented and not really test what it
      should
    * test portions of code, sometimes relying on dummy interfaces never
      used in real cases
  

2) black box testing

  Here, we consider the program as a black box giving it inputs and
  checking for its output.

  Advantages: 

    * not impacted by code refactoring
    * test code through the features, not portions code
    * nearest from real use cases

  Drawbacks:

    * need a local dedicated IMAP server: the test suites must work
      out-of-the-box with a precompiled chrooted environment in order to
      change the inputs
    * because it needs a precompiled chrooted environment it would only
      be available on systems supporting the chroot command with a x86
      compatible processor
    * would need to share and download a whole compiled environment each
      tester would have to download (probably using git)
    * can't test low-level implementations
      ( side note: is it a true drawback? Knowing a regression is
        introduced, we then bisect to gain more clues. )

-- 
Nicolas Sebrecht