[sane-devel] Backend regression testing

[Olaf Meeuwissen]

Hi Povilas, Ralph,

Povilas Kanapickas writes:

> Hi Ralph,
>
> On 11/5/19 2:47 AM, Ralph Little wrote:
>> Hi,
>> Has anyone looked into scripted backend regression testing?
>>
>> I was day dreaming over the weekend, and it occurred to me that it might
>> be possible to script the testing of backend conversations over USB and
>> Ethernet based perhaps on Wireshark traces.
>>
>> Often we don't have hardware to do regression testing, but if there was
>> some way that we could to automated scripting by playing a selection of
>> scripted conversations by hijacking the comms end, then we could
>> accumulate a library of repeatable conversations.
>
> We already capability for this on USB side, check out
> https://gitlab.com/sane-project/backends/merge_requests/93. This will
> work for any backend.

I remembered a lot more discussion on this than what's in !93 so I added
a pointer to !55 which has all that and pointers to more.  FWIW, anyone
wanting to play with this, see #108 (and #138?) as well.

> I have a library of regression tests for genesys scanners. It's useful
> for a backend maintainer to prevent accidental changes from creeping in.
> However, even benign changes need updates of tests and for that you need
> hardware to be available nearby. Additionally, the size of the test data
> is huge, which means that it probably would not be possible to use
> gitlab to store the data. My test repository is already larger than 50
> gigabytes and it's less than 20 scanners.

Test data storage is something that we'll have to sort out eventually.
Backend maintainer(s) come and go, other folks want to hack on their own
well-meant fixes and improvements, etc.

I most definitely do *not* want the test data be a part of the backends
repository.  As for storing it on GitLab.com, I'd have to check if there
are any limitations (and whether using LFS would make sense).

Hope this helps,
--
Olaf Meeuwissen, LPIC-2            FSF Associate Member since 2004-01-27
 GnuPG key: F84A2DD9/B3C0 2F47 EA19 64F4 9F13  F43E B8A4 A88A F84A 2DD9
 Support Free Software                        https://my.fsf.org/donate
 Join the Free Software Foundation              https://my.fsf.org/join