[Qa-jenkins-dev] Bug#786644: reproducible builds should vary whether nocheck is added to DEB_BUILD_OPTIONS

[Vagrant Cascadian]

Some elements of this issue have changed in the last couple years, so
following up with a few updated points below...

On 2015-05-24, Stuart Prescott wrote:
>> > > I'd expect that setting DEB_BUILD_OPTIONS=nocheck on a package build
>> > > should not change the resulting binary packages. It might make the build
>> > > succeed despite being broken, but if it succeeds without nocheck, it
>> > > should be no different when enabled.
>> > 
>> > Policy is, however, silent on whether that is the correct behaviour or
>> > not.
>> 
>> Policy is silent on many aspects of the distribution, in many cases
>> because they are obviously correct or buggy. Here I'd say the former.
>
> I would disagree that it is obvious that nocheck should not change the 
> contents of the package. I think it's entirely reasonable that test results or 
> test outputs could be shipped as examples or in the documentation. I think 
> it's entirely reasonable that nocheck would thus cause a difference in the 
> package.
>
> Other DEB_BUILD_OPTIONS are supposed to cause the resultant .deb to be 
> different (nostrip and noopt) so it's not true that the package contents have 
> to be invariant on DEB_BUILD_OPTIONS. I can even conceive of situations (such 
> as parallel compressors) where parallel=n could change the output.

The reproducible builds infrastructure does now vary parallelism, fwiw.


>> > Clarifying policy as to what the correct behaviour should be seems to be a
>> > necessary first step.
>> 
>> I don't see why, in this specific case less so when the request is
>> being done on a feature (reproducibility) that is neither in policy.
>
> Precisely because reproducibility is not mandated in policy or even encouraged 
> in devref, I think it is important to only test variations that are commonly 
> agreed-upon as being worth being robust to. Agreement can be evidenced by 
> discussion on mailing lists or by codifying in Policy.

As of debian-policy 4.1, reproducibility is in policy, so this has
changed in the intervening years... though without any specific mention
of DEB_BUILD_OPTIONS.


> So far, there is broad agreement that builds should be robust to when the 
> build was started, the path to the build, the user that runs the build etc. 
> Rapid progress has been made in improving the robustness against variations on 
> these things with the cooperation of many maintainers precisely because they 
> are widely agreed.
...
>> The check does not impose anything on package maintainers, like
>> migration blocking or similar. So even if there was a substantial
>> amount of packages that would fail that test, it would still be useful
>> information for the reproducible effort IMO.
>
> Except that reproducibility is boolean valued and this boolean is exposed to 
> maintainers. There is either a nice little tick on the maintainer's QA page or 
> there is a nasty little cross. If nasty little crosses come from what is 
> considered to be poor tests generating incorrect results, history tells us 
> that the entire column will be ignored. That makes including tests for which 
> there is not broad agreement a net loss for each maintainer, for 
> reproducible.d.n, and for the project.

Now, the qa pages only expose the results of the testing suite
(currently "buster") to the end-users, which is more conservative and
varies fewer things.

If nocheck variation was added to the reproducible builds test
infrastructure, it could be added to tests only on unstable and
experimental.


Some packages take considerably longer running the testsuite than
building the actual package, although I don't have any specific
references at this point off the top of my head.


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/qa-jenkins-dev/attachments/20171030/3600cfa3/attachment.sig>