Bug#848859: FTBFS randomly (failing tests)

[Santiago Vila]

On Thu, Jan 05, 2017 at 11:06:50AM +0100, Ole Streicher wrote:
> Hi all,
> 
> On 04.01.2017 20:57, Santiago Vila wrote:
> > I still want to build all packages and have 0 or 1 failures,
> > so in this case the probability should be 1/50/2, i.e. 1%.
> > 
> > I think this is still feasible.
> 
> My experience is that the buildds that are currently in use provide more
> build problems than the packages themself. BTW, why don't you count this
> as RC?

Can you clarify the question? I don't understand what you refer exactly.

> [statistical]
> >> The "fix" for such cases is the increasing of the threshold or disabling
> >> the test completely. Because you can do nothing with it due to the
> >> nature of numerical simulations.
> 
> Just disabling would be bad, since the test still can point to some
> problem in the implementation, like optimization problems or such.
> 
> IMO the correct solution here would be to remove the randomness and to
> seed with a fixed value (which is known to give a result within the
> expectations).

Indeed. Sometimes I've asked that in my bug reports: "I would love to
provide a way to reproduce it, but you are using a different seed each
time..."

Removing randomness is certainly the right path to achieve both builds
that always work and also reproducible builds.

> > But as far as there are people in this project who consider that a
> > package which FTBFS on single-CPU machines more than 50% of the time
> > is ok "because it does not usually fail in buildd.debian.org",
> > we are doomed.
> 
> We have 10 archs, and with a 50% chance of failure you will not get any
> version built. Even when the buildds try it two or three times.

It's actually more subtle than that.

The package may be ready to be built on multi-core machines, but not on
single-core machines. So, the probability that I measure (greater than 50%),
might not be the same as the probability on official autobuilders.

This is one of the reasons I insist so much that the rule "If it does
not fail on official autobuilders, it's not RC" is wrong.

We pride ourselves to be the Universal Operating System. We can't make
multi-core a requirement to build. That's also mathematically
and logically wrong, because in an abstract sense, building a package
is like following an algorithm. It may be slower or faster, but
it should always succeed regardless of the number of CPU cores.


Then there is the other subtle thing: The package may be Arch:all, in
which case a 50% of probability (this time independent on the number
of CPUs) may go unnoticed very easily, either because the maintainer
uploaded the .deb him/herself, or, if the maintainer uploaded
in source-only form, we can just be lucky for that time.

> > The problem I see with this threshold thing is that every maintainer
> > seems to have his own threshold, different from the others.
> > 
> > In case we decide about RC-ness depending on probability of failure:
> > What threshold do you think we should use for a single package and why?
> > 
> > [ I say that 1% of failure is the maximum we should allow, and I've
> >   explained why, but I would love to hear your opinion on this ].
> 
> I would not put any direct number here, but be pragmatic: We need to
> build the package from source, and this has to be done on all supported
> architectures, and on the buildds. As long as this is done, I see no
> reason for an RC bug. [...]

My objection to consider the official buildds the "standard" by which
RC-ness is measured is that it makes packages to have an implicit
"build-depends: buildd.debian.org".

We provide software which is free to be modified. If people is going
to modify it and then it fails because the package only builds ok in
buildd.debian.org, then we are removing one of the essential freedoms,
the freedom to modify it, and the package becomes de-facto non-free,
even if we still distribute it in main.

That's why I think that packages must build ok on any autobuilder which
is not misconfigured, not just on buildd.debian.org.

Thanks.