Bug#1019742: Bug#786644: should reproducible builds vary nocheck?

[Helmut Grohne]

On Sat, Dec 14, 2024 at 08:47:47PM +0000, Rebecca N. Palmer wrote:
> Should we merge #786644 and #1019742?  Or should we consider #1019742 to be
> "have the option" and #786644 to be "enable it by default"?

As the submitter of #786644, it was certainly meant as performing the
testing by default not merely having it as an option.

> I'm willing to try implementing this, if we agree that having it is a good
> idea.  Maybe use _PROFILES rather than _OPTIONS and allow it to be more
> general than just nocheck?

Allowing other variations also is useful. For instance,
pkg.unbound.libonly also is supposed to be reproducible in the sense
that any package that is being produced matches the ones that are
produced when building without build profiles, but the set of packages
produces ends up being smaller. So generally, being able to verify
reproducibility of a particular profile is a useful feature separately.

> There is a way to vary nocheck but continue to check for test failures in
> the less-standard environment: make the *first* build the nocheck build.
> (timezone is precedent that the first build is allowed to do non-default
> things.  No strong opinion on whether we actually want that.)

Sounds ok to me.

> On whether nocheck even should always mean no change to output:
> 
> DEB_BUILD_PROFILES=nocheck (which requires also setting
> DEB_BUILD_OPTIONS=nocheck) *is* defined (at
> https://wiki.debian.org/BuildProfileSpec#Registered_profile_names ; Policy
> doesn't describe build profiles at all, #757760) as not changing the build
> result.  (Packages that ship test results/tools and want to make this
> optional are probably supposed to use DEB_BUILD_PROFILES=noinsttest
> instead.)

Confirmed.

> However, I'm not aware of any systematic attempts to check this, and hence,
> I don't know how often it is violated in practice.  There appears to have
> been at least one mass rebuild with nocheck, but it only seems to have filed
> bugs for packages that failed to build *at all* with nocheck (e.g.
> #1086765).

I note that a nocheck ftbfs is considered release-critical by the
release team, because the autoremover disregards dependencies annotated
<!nocheck> and expects that you can at any time remove such dependencies
by reducing testing of the package. If the package ftbfs with nocheck,
that property no longer holds and testing no longer is self-contained.

That said, nobody is testing this systematically, so I guesstimate
something between 50 and 500 packages that ftbfs in such a way in
unstable right now. Reproducibility likely degrades in even more cases
as the compiler flags may change and they influence the buildid. Since
cross building uses nocheck, I have been filing a couple of nocheck
ftbfs bugs maybe one to two per month on average.

> Hence, I suggest implementing the option but not initially making it the
> default, doing a large (maybe archive-wide) run with it enabled, and only
> then deciding whether to enable it by default.

Reasonable.

> On the time/resource savings:
> 
> I suggest that this run also record the duration of each build, to provide
> an estimate of the time/resource saving.

Good idea.

So thanks for working on these old bugs!

Helmut