Bug#1019742: reprotest: add a variation that sets DEB_BUILD_OPTIONS=nocheck

[Vagrant Cascadian]

On 2022-09-14, Philip Hands wrote:
> I suggest adding a 'nocheck' variation, that sets DEB_BUILD_OPTIONS=nocheck
> during the build,

Sounds reasonable!

> and enabling it by default.

Less sure...

> The reason for doing so is that one could imagine that a package produces
> differing results depending upon whether the tests were run or not

Indeed!

> but also
> (given that the tests will have passed during the normal build) the tests
> failing during the varied build seems unlikely to be identifying faults that are
> worth fixing, and so is just a waste of cycles.

How do you know weather the bugs it is identifying are worth fixing? It
could also identify non-deterministic failures, or failures triggered by
specific build environment configurations...

> This idea is prompted by `busybox` where the tests fail in the varied scenario,
> despite the fact that the package is reproducible.

> Here they are failing:
>
>   https://salsa.debian.org/installer-team/busybox/-/jobs/3227197
>
>   (among other things, du produces weird results when the `fileordering`
>    variation is active, claiming the 1MB directoy is 2MB so the tests fail, so
>    the varied package is not produced, so we don't get to see that it was
>    reproducible:
>    https://salsa.debian.org/installer-team/busybox/-/blob/master/testsuite/du/du-m-works
>    )

Consistantly? Then, maybe the test needs to be improved?


> I found a couple of ways of making the issue go away:
>
>   1) disabling the 'fileordering' variation, thus:
>
>     https://salsa.debian.org/installer-team/busybox/-/commit/17387890c73388e1f56a6ae9fbc79783095b4e86
>
>     https://salsa.debian.org/installer-team/busybox/-/jobs/3233259
>
>   2) telling the package to skip the tests when doing the variations:
>
>     https://salsa.debian.org/installer-team/busybox/-/commit/5260442e8ceea220fa36bdda169978d15108f781
>
>     which is setting this in the salsa-ci.yml:
>       SALSA_CI_REPROTEST_ARGS: --variations=environment.variables+=DEB_BUILD_OPTIONS=nocheck
>
>     https://salsa.debian.org/installer-team/busybox/-/jobs/3235476
>
>
> Option 2) is what I'm suggesting making into a default variation.
>
> If nothing else it will speed up testing of packages with extensive test suits.

I think it's a valuable feature, but I'm not entirely sure weather it
should be default or not...


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20220914/147bb8b1/attachment.sig>