Unreproducible test logs in output

[Ximin Luo]

I've been reviewing Debian's "required" and "build-essential" package sets. One new issue that's come up, is that the gcc-6 and binutils packages both put test logs in their binary output.

Of course this has no chance of ever being reproducible, due to parallelism, build paths, and many many things. I don't think it would be reasonable to try to work towards making this reproducible.

The justification for *including* them is that many very very large programs have test suites that aren't expected to completely pass all the time. Now, sometimes it's possible to ignore specific failing tests, and that is what I have been personally doing for the rust compiler in Debian. 

However, in other cases there are just too many, and the maintainer(s) physically don't have enough time to ignore these tests, since nothing else useful will get done. I guess this is the case for gcc-6 and binutils, and I know this is the case for another very large program I help maintain in Debian, sagemath with ~300k test cases.

In these cases, it is much easier for the package maintainer to run the tests, pass the build even if their are failures, and distribute the test logs along with the output so that they can be examined later. Indeed I've been very tempted to do this myself for sagemath.

How shall we handle this in the context of reproducible builds?

As a first proposal, perhaps we could allow package maintainers to define a set of "non-installable binary output" that does not have to be reproducible. This would only be used in very exceptional circumstances such as the ones described above, they can't be installed via the package manager, but they can be downloaded and manually extracted to analyse or distribute test results with.

OTOH if we tell package maintainers "don't put test logs in binary output", we need to offer an alternative that caters to the needs of very very large packages, as described earlier. A simple interface to download the build logs of a given binary package, might be sufficient.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git