[Debian-science-sagemath] Testing logic and output

[Tobias Hansen]

On 09/14/2017 03:24 PM, Ximin Luo wrote:
> Tobias Hansen:
>> Hi,
>>
>> I noticed that sometimes it's not really clear to me why builds fail or
>> don't fail after doctesting. For someone else (who never looked at our
>> debian/rules) it must be even more confusing.
>>
>> It would be cool if our debian/rules prints what it's doing and why, e.g.
>>
>> x test failures out of y allowed - pass / fail
>> x test failures out of 0 allowed for documentation tests - pass / fail
>> There were critical failures (e.g. timeout, segfault, etc.) - fail
>> There were up to three timeouts - rerun tests without parallelization
>> x test failures out of z allowed during reruns - pass / fail
>>
>> (Maybe we should also choose y and z to be the same number...)
>>
>> Ximin could you give this a shot? Your makefile fu seems to be stronger.
>> Otherwise I'll try.
>>
> Hey, I have a bunch of other stuff I need to do first (ocaml and rustc stuff) so it would be good if you tried this.
>
> You should edit the run_tests_with_retry function/macro, edit the "then :" parts to instead output some nice diagnostic messages like "ignoring failures because [etc]", be careful with the wording and the negations.
>
> In the check_test_log function, the -s flag is for make(1) and suppresses some diagnostics messages, perhaps you can leave it out. I think I originally added it in there because I was capturing the output and had to omit the extra messages, but it looks like this isn't the case currently.
>
> You can also add extra messages to the had-few-failures/had-not-too-many-failures recipes. The main thing is that each line is a shell script and if it returns non-zero the whole recipe fails. So perhaps something like:
>
> had-few-failures:
> 	echo "Testing $@ ..."
> 	if ! test1; then echo "failed because XXX"; false; fi
> 	if ! test2; then echo "failed because YYY"; false; fi
> 	[..]
>
> If you start a shell-script line with "@" then make(1) will not print the shell-script before executing it (I don't recommend this though). If you start the line with "-" then make(1) will ignore a non-zero exit-code and continue executing the subsequent lines. Each line runs in a separate shell so you can't easily share variables across them. Also errors within each line are not ignored, so something like this succeeds:
>
> target:
> 	false; [etc]
>
> To get a fail-fast semantic within each line do this:
>
> target:
> 	set -e; command-that-might-fail; [etc]
>
> Let me know if you need other hints.
>
> X
>

Thanks for the hints, after some experimenting I managed to put
something together.

Best, Tobias