[Pkg-samba-maint] On the selftest in debian work

[Andrew Bartlett]

G'Day,

I've been thinking about how to get the best out of the selftest on
debian work.  

The key, as I see it, is to get a quick, reliable quicktest into Samba
git master, then try again here.  

Currently quicktest is neither of these, and I'm afraid of the
infection of Samba flapping tests (our full 4 hour make test regularly
fails, perhaps 50% of the time at bad points) infecting the debian
release process.  One simple task is to get the st/subunit file from a
successful build, and using tools like subunit-ls and a spreadsheet,
work out a set of tests that runs in 10mins (aim for 5 to start, then
add some other useful tests to increase coverage up to 10mins).

Then compare that list against the known set of sometimes flapping
tests, and remove those.

This can then be used in script/autobuild.py once the patch lands to
run some tests in a new samba-03 build (and so that will check it keeps
working). 

Once we have a list of tests that are both quick and reliable, we
should maintain a local debian patch that also excludes (use the skip
file) tests that are simply unsuitable on debian build machines, such
as those currently running the hosts out of disk. 

That, long and painful process, but I think it is the best way to
proceed.  In terms of debian, it should be in experimental in the short
term, given we are blocking the new packages from migrating to testing.

I'm very happy to continue to give advice (and info like the current
long-term flapping list) on this.

Thanks,

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba