[sane-devel] Proposal for integration tests

[Povilas Kanapickas]

Hey,

I would like to propose adding integration tests to guard against 
accidental regressions especially in rarer scanners that few SANE 
developers have access to. We could run these tests on each pull request 
on gitlab without any scanner hardware.

Given limited development resources and large variety of hardware that 
SANE supports, a major requirement of the tests would be that they are 
easy to create and use, otherwise no one will bother to deal with them.

Therefore, I would like to propose a simple way of testing SANE by 
recording low-level communication with the scanner, replaying it at the 
test time and checking whether SANE behaves in an equivalent way.

Additionally, I would like to propose that we distribute the recording 
functionality with release SANE builds. This way we could collect 
recordings from our users for scanners that no SANE developers have 
access to and then add to the test database.

Finally, I expect that there will be a need for tools for simplify and 
automate the process of working with the recorded data. I would like to 
propose that we use python as a implementation language for any of such 
tools.

Distribution of recording functionality together with release versions 
of SANE would require that we are able to change the communication 
backend on-demand. The best place to do that it seems to be at sanei 
layer. For example, in case of sanei_usb.h, I propose that instead of 
calling functions from that header we pass a pointer to a struct 
containing function pointers that match the current functionality 
one-to-one. Then, depending on startup options, we could create a USB 
backend that supports either of the following options:

  - calls directly to sanei_usb.c.
  - calls directly to sanei_usb.c, but records all parameters and 
responses to a file.
  - reads expected parameters and responses from a file and compares 
them with the current parameters. An error is recorded if anything does 
not match.

If we agree with the above proposal, the use of the new, dynamic APIs 
would be completely opt-in. For example, the current API of sanei_usb.h 
would be left unmodified. Once the viability is proved with a single 
backend we could gradually introduce it to more backends.

What do you think about this?

Regards,
Povilas