Bug#848859: FTBFS randomly (failing tests)

[Anton Gladky]

2017-01-04 13:26 GMT+01:00 Santiago Vila <sanvila at unex.es>:
> No matter how much glitch-free is the autobuilder you use to build the
> above package, it will fail to build 1 every 147 times on average,
> mathematically, because the test is wrongly designed.

That is not always true. If you look in many tests from numerical
simulation packages, there is usually a "threshold" for test result
which should not be exceeded. And the test result varies in
the limits, which are set by upstream authors. This result
can be different even on the same machine, running the simulation
several time. And it is normal.

The "fix" for such cases is the increasing of the threshold or disabling
the test completely. Because you can do nothing with it due to the
nature of numerical simulations.

> Really, we need more people doing QA, and not stop doing it "because
> we are near the freeze".

If you are maintaining the package several years, fixing most of its
bugs, hoping to see it in release and trying to escape major changes
several months before the freeze.. Sure, it will actively be defended
from maintainers if some pseudo-reasons for its removal appear just
before the freeze. This fact has to be considered as well.

Best regards

Anton