[Pkg-samba-maint] On the selftest in debian work

[Andrew Bartlett]

On Sat, 2016-05-07 at 13:22 +0200, Mathieu Parent wrote:
> 2016-05-06 23:04 GMT+02:00 Andrew Bartlett <abartlet at samba.org>:
> > On Fri, 2016-05-06 at 15:57 +0200, Mathieu Parent wrote:
> > > 2016-05-05 20:44 GMT+02:00 Jelmer Vernooij <jelmer at jelmer.uk>:
> > > > On Fri, May 06, 2016 at 06:33:45AM +1200, Andrew Bartlett
> > > > wrote:
> > > [...]
> > > > However, the work to get to get the test runs stable should
> > > > really
> > > > happen
> > > > upstream and not in the Debian packaging. This is not specific
> > > > to
> > > > Debian.
> > > > The upstream testsuite shouldn't fail regularly. Until it is
> > > > stable
> > > > there,
> > > > I don't see a point in trying to get it to run on Debian.
> > > 
> > > I agree. but currenly the quicktest only fails on the raw.write
> > > tests,
> > > under 32-bit only.
> > > 
> > > I've removed this test (actually 3 tests) on my local repo,
> > > currently
> > > rebuilding on i386.
> > > 
> > > If everything goes well, I'll upload this. Let's see if the
> > > flappy
> > > test is still here (it was on amd64 reproducible buildd).
> > > 
> > > I'm sorry that my uploads have stopped samba 4.4 to go to
> > > testing.
> > > Will fix this soon.
> > 
> > I'm really sorry, but we need to first work on this upstream, then
> > in
> > experimental.  There are still too many flapping tests to do this
> > reliably in something like Debian, certainly for unstable.
> > 
> > Have a read of
> > https://lists.samba.org/archive/samba-technical/2016-May/113894.htm
> > l fo
> > r the sorry state of affairs.
> > 
> > We first need to get this reliable upstream, then in experimental,
> > as
> > Jelmer and I have discussed.  It is a very worthwhile task, but not
> > a
> > small one.
> > 
> > Because of the nature of the Samba flapping tests, you need to have
> > a
> > large sample, not just if a build works once.
> > 
> > To be clear, I'm currently running 10 builds in large VMs on the
> > Catalyst cloud to try and get a reasonable statistical overview of
> > the
> > tests for a bug I'm fixing at work.  If the builds run on slow or
> > busy
> > hosts, we can expect spurious failures.
> 
> THis is sad.

I know.  The best way forward is probably to re-work quicktest into a
mode that is more 'reliabletest', that uses an even smaller subset of
tests, but with startup and teardown done per test.  We could back up
the AD DC provision at startup, and re-play the test with that each
time.  That is much more likely to be reliable, and then we can cut the
tests down to something that runs in a realistic amount of time. 

In any case, that work belongs on samba-technical.

BTW, the ctdb tests have similar issues.

> I'll upload again with make quicktest return code ignored (once built
> locally).

Thanks.

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba