Bug#848859: FTBFS randomly (failing tests)

[Santiago Vila]

On Wed, Jan 04, 2017 at 08:44:17AM +0100, Ole Streicher wrote:

> > It's in Release Policy: Packages *must* autobuild *without* failure.
> > 
> > If a package fails to build from time to time, that's a failure.
> 
> Packages actually *do* fail from time to time, when I look into my
> autobuilder. Not due to the package, but due to glitches within the
> buildd infrastructure. Would you consider this a failure?

If the package is not to blame, of course not.

I'm speaking about packages which intrinsically fail with a
probability p such that 0 < p < 1. Funny example of what
I call "instrinsically":

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838828

No matter how much glitch-free is the autobuilder you use to build the
above package, it will fail to build 1 every 147 times on average,
mathematically, because the test is wrongly designed.

> >> I totally agree that catching random failures
> >> is a good quality measure, but this is IMO severity "important" at maximum.
> > 
> > Well, would you say it's RC if it fails 99% of the time?
> > I guess you would.
> 
> I would consider a bug RC if it actually doesn't build on our buildds.

Aha! But *that* is what is not written in policy anywhere.

Not only it's not written anywhere, it's invalidated by current practice
every day. Examples here:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?include=subject%3AFTBFS;submitter=lamby%40debian.org

or here:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?include=subject%3AFTBFS;submitter=lucas%40debian.org

or even here:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?include=subject%3AFTBFS;submitter=sanvila%40debian.org


Are you proposing that Lucas Nussbaum, Chris Lamb or myself stop
reporting FTBFS bugs as serious unless we can point to a failed build
log at buildd.debian.org?

That restricted way of reporting bugs surely may not be right.

> [...]
> Doing release QA just before the release leads to quick hacks to keep
> things there, while a continious QA really solves them.

Well, I started doing QA more than a year ago, to check for
"dpkg-buildpackage -A". As a side effect, I started to report each
and every package which FTBFS for whatever reason.

Really, we need more people doing QA, and not stop doing it "because
we are near the freeze".

Thanks.