Bug#848859: FTBFS randomly (failing tests)

[Santiago Vila]

On Tue, 27 Dec 2016, Ole Streicher wrote:

> Hi Santiago,

Hello Ole. Thanks for your reply. Please don't forget to Cc: me if you
expect your message to be read.

> > In particular, if something happens 1 every 20 times on average, the
> > fact that it did not happen when you try 10 times does not mean in any
> > way that it may not happen.
> 
> 
> I must however say that I don't see why a package that fails to build
> once in 20 builds would have a release critical problem:

It's in Release Policy: Packages *must* autobuild *without* failure.

If a package fails to build from time to time, that's a failure.

> There is nothing in our policy that requires a reproducible or even a
> always successful build.

Please don't mix "reproducible" with "always successful build".

They are different things and nobody is making "reproducible builds"
policy yet.

But "always successful" has always been release policy:

https://release.debian.org/stretch/rc_policy.txt

Please read the paragraph saying "packages must autobuild without
failure". That paragraph is the very reason FTBFS bugs are serious
since a loooong time.

> I totally agree that catching random failures
> is a good quality measure, but this is IMO severity "important" at maximum.

Well, would you say it's RC if it fails 99% of the time?
I guess you would.

Would you say it's RC if it fails 0.0000001% of the time?
I guess you would not, since you have just said that 0.05% does not
deserve to be serious.

So, you are implicitly saying that packages which FTBFS with a
low probability does not deserve a serious bug.

However, there is no threshold at all in policy, and if you think
about it, any such threshold would be quite arbitrary indeed:
Why would 50% of failure be RC while 49% of failure is not?

[ To me, the mere fact that we are able to *measure* the probability
  of failure already means that it is too high ].


Anyway, there will be plenty of time to discuss about this because I'm
going to downgrade all the FTBFS-randomly bugs I've detected until
Release Managers say something about the subject.

Then I expect most (if not all) of them will become serious and RC
again.

> And it would be nice to have this QA test not just before the release,
> but already early in the release cycle. This would help a lot to avoid
> stressful bugfixes just before the freeze.

Well, if you see my list of FTBFS-randomly bug here:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=ftbfs-randomly;users=sanvila@debian.org

you will see that I started to report FTBFS-randomly bugs several
months ago, not last week, and not last month.

What would really help is to have somebody else reporting these bugs,
because apparently very few people want to report them.

Also, you can't seriously "complain" that a bug reporter didn't not
report something earlier! Of course that it would be nice to do this
kind of QA stuff at every point in the release cycle, but as the same
software that we maintain, my bug reports come with no warranty, not
even the warranty that the bug is reported "as soon as it exists"! :-)

Thanks.