[Nut-upsdev] Testing NUT

[Arnaud Quette]

2008/11/3 Arjen de Korte <nut+devel at de-korte.org>:
> Citeren Arnaud Quette <aquette.dev at gmail.com>:
>
>> The biggest show stopper there is the ability to emulate devices
>> (serial, USB, network (though these last should be easier), ...).
>
> I doubt that emulating devices is worth the effort. For many we don't have
> access to the protocol and timing anyway, so this would basically be a
> reversed driver. Since invariably this would have to be made by the same
> person as the driver, you risk missing vital parts/timing requirements in
> the protocol.

I was more thinking about a "simple replayer" that would record and simply .
that way, we don't need the protocol knowledges, and even more, we
mustn't use it.
what we want is a virtual device. The next part is to have a kind of
pattern matching algo to give the right answer when asked for.
 I've not gone deep enough into this, but you get the idea...

The big drawback of this approach is that it will somehow fails when
we add features.
ie if you request a new HID path, that wasn't implemented at the time
of the recording session, the test will not 100 % pass!
but it there depends on the comparison base: driver output or upsc output.

> I'm in favor of more testing before releasing things, but when it comes to
> the drivers, this would have to be done with actual hardware to be worth the
> effort. The majority of regressions we see is in the drivers, not the server
> or clients.

bingo. BUT since we rely on a community of benevolent, generally using
the UPS in production, it's not that easy!

I'm trying to solve this by:
- working with Kjell on a script which assists the test reports. This
would make users report more easy and suitable / reliable for our
needs.
ideally, this feature in a GUI would appeal more testers. and Python
would be a good candidate.
- thinking about a way to couple buildbot + some Eaton resources
(which now covers a good bunch of drivers / units ;-) and some
scripting.
The interesting point here is that the soon to be commited PDU support
will help us in automating power failure / restoration and  the needed
automation ^_^
The other interesting bit is that the work I've within Ubuntu QA
(before mentioned in this thread) should be a good base to achieve
this more quickly...

-- Arnaud