[sane-devel] Proposal for integration tests

[Olaf Meeuwissen]

Hi Povilas,

You asked me off-list to reply, here you go ;-)

# Actually, I've been kind of waiting for others to chime in first.
# Glad to see that René Rebe, an ex-SANE developer, eventually did but
# would have liked to see some more opinions before my reply :-(.

Povilas Kanapickas writes:

> Hey,
>
> I would like to propose adding integration tests to guard against
> accidental regressions especially in rarer scanners that few SANE
> developers have access to. We could run these tests on each pull request
> on gitlab without any scanner hardware.

I think this is a *really* *cool* idea.  But, as René mentioned, the
devil will be in the details.

> Given limited development resources and large variety of hardware that
> SANE supports, a major requirement of the tests would be that they are
> easy to create and use, otherwise no one will bother to deal with them.

ACK.  Using them should be as easy as `make check` ;-)

# ... and only run tests for the backends that have been compiled.

> Therefore, I would like to propose a simple way of testing SANE by
> recording low-level communication with the scanner, replaying it at the
> test time and checking whether SANE behaves in an equivalent way.

Recording low-level communication can be done with Wireshark for network
and USB scanners.  This requires *zero* work on SANE's side.

# OK, so maybe we would need some documentation or pointers to such on
# how to make captures with Wireshark.

Apart from use in integration and regression tests, these captures can
also be useful in reproducing odd behaviour observed by our users.  If
they can just provide a capture, anyone would be able to reproduce the
behaviour.  Especially nice when looking at those pesky timing issues.

> Additionally, I would like to propose that we distribute the recording
> functionality with release SANE builds. This way we could collect
> recordings from our users for scanners that no SANE developers have
> access to and then add to the test database.

As per suggestion above, Wireshark already does a very good job at
recording network and USB traffic, runs on Unix and Windows and is
well-known and well-supported.  Its default PCap file format is very
easy to understand and work with too.

Personally, I wouldn't bother with SCSI, parallel and serial devices
anymore.  That is, unless someone is willing to put in the effort and
implements something that can output device I/O to PCap files for such
scanners.

> Finally, I expect that there will be a need for tools for simplify and
> automate the process of working with the recorded data. I would like to
> propose that we use python as a implementation language for any of such
> tools.

Not too charmed with your choice of Python.  Nothing wrong with the
language itself but I have next to no experience with it.  Don't know
about other project members but you probably want some people familiar
with it around if this is going to be something long(er)-term.

As for PCap files, there is a C library to deal with them and, as said,
the file format is pretty simple.  Most all project members are (should
be?) familiar with C.

> Distribution of recording functionality together with release versions
> of SANE would require that we are able to change the communication
> backend on-demand. The best place to do that it seems to be at sanei
> layer. For example, in case of sanei_usb.h, I propose that instead of
> calling functions from that header we pass a pointer to a struct
> containing function pointers that match the current functionality
> one-to-one. Then, depending on startup options, we could create a USB
> backend that supports either of the following options:
>
>   - calls directly to sanei_usb.c.
>   - calls directly to sanei_usb.c, but records all parameters and
> responses to a file.
>   - reads expected parameters and responses from a file and compares
> them with the current parameters. An error is recorded if anything does
> not match.

# You basically suggest an OOP approach to device I/O.  I have some
# experience with that (see the third party epkowa backend code if
# interested).

That would be a noble end goal but I'd start with something a bit more
hackish (and in line with René's suggestions) to get a feel for what's
needed/involved and expand from there.  A rough roadmap would look a
bit like this, I guess

 - implement the sanei_usb API to use the recorded device I/O;
   control what gets used with a compile time switch
 - get that to work with a scanner you're familiar with first
   (for a simple definition of "work")
 - get some feedback on your code here (or over on GitLab.com, assuming
   you're using a fork or dedicated branch for your work on this)
   # Please do *not* do any of the work on our master branch!
 - collect a few more captures to work with (for other scanners)
 - adjust your implementation to get those to work too
 - push your sanei_usb implementation for replaying of recorded
   device I/O behind a dynamic switch

# BTW, not all supported USB scanners use sanei_usb ... :-/
# The ones that don't are pretty old ... let's not care about those for
# now (and probably forever).

> If we agree with the above proposal, the use of the new, dynamic APIs
> would be completely opt-in. For example, the current API of sanei_usb.h
> would be left unmodified. Once the viability is proved with a single
> backend we could gradually introduce it to more backends.

Once you have implemented the dynamic opt-in functionality in sanei_usb
it is immediately usable by *all* backends that use sanei_usb.  That is
I think the switching belongs in the sanei_usb code.  There should not
be any code changes in any backend to accommodate this functionality.
Backends should be completely unaware of what they are talking to (as
long as whatever that is talks the expected protocol).

> What do you think about this?

See above ;-)

Hope this helps,
--
Olaf Meeuwissen, LPIC-2            FSF Associate Member since 2004-01-27
 GnuPG key: F84A2DD9/B3C0 2F47 EA19 64F4 9F13  F43E B8A4 A88A F84A 2DD9
 Support Free Software                        https://my.fsf.org/donate
 Join the Free Software Foundation              https://my.fsf.org/join