Bug#848859: FTBFS randomly (failing tests)

[Ole Streicher]

Hi Santiago,

On 04.01.2017 01:41, Santiago Vila wrote:
> On Tue, 27 Dec 2016, Ole Streicher wrote:
> Hello Ole. Thanks for your reply. Please don't forget to Cc: me if you
> expect your message to be read.

OK, however I usually assume that a bug submitter actually reads the
messages for his submitted bugs.

>>> In particular, if something happens 1 every 20 times on average, the
>>> fact that it did not happen when you try 10 times does not mean in any
>>> way that it may not happen.
>>
>>
>> I must however say that I don't see why a package that fails to build
>> once in 20 builds would have a release critical problem:
> 
> It's in Release Policy: Packages *must* autobuild *without* failure.
> 
> If a package fails to build from time to time, that's a failure.

Packages actually *do* fail from time to time, when I look into my
autobuilder. Not due to the package, but due to glitches within the
buildd infrastructure. Would you consider this a failure?

>> I totally agree that catching random failures
>> is a good quality measure, but this is IMO severity "important" at maximum.
> 
> Well, would you say it's RC if it fails 99% of the time?
> I guess you would.

I would consider a bug RC if it actually doesn't build on our buildds.

>> And it would be nice to have this QA test not just before the release,
>> but already early in the release cycle. This would help a lot to avoid
>> stressful bugfixes just before the freeze.
> 
> Well, if you see my list of FTBFS-randomly bug here:
> 
> https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=ftbfs-randomly;users=sanvila@debian.org
> 
> you will see that I started to report FTBFS-randomly bugs several
> months ago, not last week, and not last month.
> 
> What would really help is to have somebody else reporting these bugs,
> because apparently very few people want to report them.
> 
> Also, you can't seriously "complain" that a bug reporter didn't not
> report something earlier! Of course that it would be nice to do this
> kind of QA stuff at every point in the release cycle, but as the same
> software that we maintain, my bug reports come with no warranty, not
> even the warranty that the bug is reported "as soon as it exists"! :-)

What I do complain about is that doing this just before the release adds
additional pressure to the package maintainers: The problem for us
ordinary people is that most of the time my packages look fine, without
serious problems, and just before the release all those RC bugs pop up
in bunches, with quite some time pressure to solve them. If the RC bugs
were randomly distributed over time, it would be more time to solve
them, and the quality of the fixes would be better: for example, in the
moment, I would just disable build time tests that randomly fail, while
usually I would try to see why they fail and fix that. In the moment,
there is no time for this.

Doing release QA just before the release leads to quick hacks to keep
things there, while a continious QA really solves them.

Best regards

Ole