[Pkg-samba-maint] On the selftest in debian work

[Jelmer Vernooij]

On Fri, May 06, 2016 at 06:33:45AM +1200, Andrew Bartlett wrote:
> I've been thinking about how to get the best out of the selftest on
> debian work.  
> 
> The key, as I see it, is to get a quick, reliable quicktest into Samba
> git master, then try again here.  
> 
> Currently quicktest is neither of these, and I'm afraid of the
> infection of Samba flapping tests (our full 4 hour make test regularly
> fails, perhaps 50% of the time at bad points) infecting the debian
> release process.  One simple task is to get the st/subunit file from a
> successful build, and using tools like subunit-ls and a spreadsheet,
> work out a set of tests that runs in 10mins (aim for 5 to start, then
> add some other useful tests to increase coverage up to 10mins).
> 
> Then compare that list against the known set of sometimes flapping
> tests, and remove those.
> 
> This can then be used in script/autobuild.py once the patch lands to
> run some tests in a new samba-03 build (and so that will check it keeps
> working). 
> 
> Once we have a list of tests that are both quick and reliable, we
> should maintain a local debian patch that also excludes (use the skip
> file) tests that are simply unsuitable on debian build machines, such
> as those currently running the hosts out of disk. 
> 
> That, long and painful process, but I think it is the best way to
> proceed.  In terms of debian, it should be in experimental in the short
> term, given we are blocking the new packages from migrating to testing.
> 
> I'm very happy to continue to give advice (and info like the current
> long-term flapping list) on this.
Agreed that experimenting with running the testsuite should be happening in
experimental rather than sid (where it risks blocking other changes).

However, the work to get to get the test runs stable should really happen
upstream and not in the Debian packaging. This is not specific to Debian.
The upstream testsuite shouldn't fail regularly. Until it is stable there,
I don't see a point in trying to get it to run on Debian.

If we end up with a (nontrivial) Debian-specific whitelist or blacklist of
tests we're increasing the maintainance burden for a packaging team that
is already stretched, keeping those lists up to date.