[Debian-science-sagemath] Testing logic and output

[Ximin Luo]

Tobias Hansen:
> Hi,
> 
> I noticed that sometimes it's not really clear to me why builds fail or
> don't fail after doctesting. For someone else (who never looked at our
> debian/rules) it must be even more confusing.
> 
> It would be cool if our debian/rules prints what it's doing and why, e.g.
> 
> x test failures out of y allowed - pass / fail
> x test failures out of 0 allowed for documentation tests - pass / fail
> There were critical failures (e.g. timeout, segfault, etc.) - fail
> There were up to three timeouts - rerun tests without parallelization
> x test failures out of z allowed during reruns - pass / fail
> 
> (Maybe we should also choose y and z to be the same number...)
> 
> Ximin could you give this a shot? Your makefile fu seems to be stronger.
> Otherwise I'll try.
> 

Hey, I have a bunch of other stuff I need to do first (ocaml and rustc stuff) so it would be good if you tried this.

You should edit the run_tests_with_retry function/macro, edit the "then :" parts to instead output some nice diagnostic messages like "ignoring failures because [etc]", be careful with the wording and the negations.

In the check_test_log function, the -s flag is for make(1) and suppresses some diagnostics messages, perhaps you can leave it out. I think I originally added it in there because I was capturing the output and had to omit the extra messages, but it looks like this isn't the case currently.

You can also add extra messages to the had-few-failures/had-not-too-many-failures recipes. The main thing is that each line is a shell script and if it returns non-zero the whole recipe fails. So perhaps something like:

had-few-failures:
	echo "Testing $@ ..."
	if ! test1; then echo "failed because XXX"; false; fi
	if ! test2; then echo "failed because YYY"; false; fi
	[..]

If you start a shell-script line with "@" then make(1) will not print the shell-script before executing it (I don't recommend this though). If you start the line with "-" then make(1) will ignore a non-zero exit-code and continue executing the subsequent lines. Each line runs in a separate shell so you can't easily share variables across them. Also errors within each line are not ignored, so something like this succeeds:

target:
	false; [etc]

To get a fail-fast semantic within each line do this:

target:
	set -e; command-that-might-fail; [etc]

Let me know if you need other hints.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git