[Pkg-samba-maint] On the selftest in debian work

[Andrew Bartlett]

On Thu, 2016-05-05 at 18:44 +0000, Jelmer Vernooij wrote:
> On Fri, May 06, 2016 at 06:33:45AM +1200, Andrew Bartlett wrote:
> > I've been thinking about how to get the best out of the selftest on
> > debian work.  
> > 
> > The key, as I see it, is to get a quick, reliable quicktest into
> > Samba
> > git master, then try again here.  
> > 
> > Currently quicktest is neither of these, and I'm afraid of the
> > infection of Samba flapping tests (our full 4 hour make test
> > regularly
> > fails, perhaps 50% of the time at bad points) infecting the debian
> > release process.  One simple task is to get the st/subunit file
> > from a
> > successful build, and using tools like subunit-ls and a
> > spreadsheet,
> > work out a set of tests that runs in 10mins (aim for 5 to start,
> > then
> > add some other useful tests to increase coverage up to 10mins).
> > 
> > Then compare that list against the known set of sometimes flapping
> > tests, and remove those.
> > 
> > This can then be used in script/autobuild.py once the patch lands
> > to
> > run some tests in a new samba-03 build (and so that will check it
> > keeps
> > working). 
> > 
> > Once we have a list of tests that are both quick and reliable, we
> > should maintain a local debian patch that also excludes (use the
> > skip
> > file) tests that are simply unsuitable on debian build machines,
> > such
> > as those currently running the hosts out of disk. 
> > 
> > That, long and painful process, but I think it is the best way to
> > proceed.  In terms of debian, it should be in experimental in the
> > short
> > term, given we are blocking the new packages from migrating to
> > testing.
> > 
> > I'm very happy to continue to give advice (and info like the
> > current
> > long-term flapping list) on this.
> Agreed that experimenting with running the testsuite should be
> happening in
> experimental rather than sid (where it risks blocking other changes).
> 
> However, the work to get to get the test runs stable should really
> happen
> upstream and not in the Debian packaging. This is not specific to
> Debian.
> The upstream testsuite shouldn't fail regularly. Until it is stable
> there,
> I don't see a point in trying to get it to run on Debian.
> 
> If we end up with a (nontrivial) Debian-specific whitelist or
> blacklist of
> tests we're increasing the maintainance burden for a packaging team
> that
> is already stretched, keeping those lists up to date.

I think we are in violent agreement.  

The only thing I want in Debian is a list of tests that don't work on
debian build boxes, but are still reasonable elsewhere (and even then,
that may as well go upstream).

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba