[Debian-med-packaging] Regression tests.

[Charles Plessy]

Le Sat, May 04, 2013 at 11:04:12PM +0200, Andreas Tille a écrit :
> On Fri, May 03, 2013 at 08:46:53AM +0900, Charles Plessy wrote:
> > > do you think that the result might be wrong even on other 64bit
> > > architectures?
> > 
> > Hi Thorsten,
> > 
> > I can not tell precisely for this package, but for all the packages I upload, I
> > want to underline that I have no evidence that they work as intended on other
> > architectures, as they may also differ on other aspects than the number of
> > bits, for instance endianness.
> 
> Shouldn't we ask upstream to provide reasonable test cases to verify
> that the binary is working properly?  This should not only be useful
> for different architectures but also for new upstream versions.

Hi Andreas

We definitely can.  But given how diverse our upstreams are, from skilled
programmers to bench biologists, not to mention upstreams who eventually became
promoted to managerial positions and do not have time to enhance their software
anymore, I do not think that asking will give us a high coverage.  Many
upstreams will accept patches, and this is another possible way to push forward
test suites as a standard component of software releases.

If there are people interested to work in these directions, I suggest to write
tests that fit well with the autopkgtest platform.  I try to do it for the packages
I maintain, especially when the upstream README already provide examples that
are easy to turn into small scripts.

    http://dep.debian.net/deps/dep8/

Even the most trivial tests are useful.  For instance, we had t-coffee completely broken
on ARM for months before we realise, by enabling a test that was not doing much more than
running the executable with test data, without inspecting the result.  But the test showed
us that t-coffee was freezing and consuming all CPU ressource indefinitely. 

Have a nice day,

-- 
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan