[Debian-salsa-ci] reprotest: inadvertent misconfiguration in salsa-ci config

[James Addison]

Hello,

A few hundred packages that use reprotest in Salsa-CI appear to be
misconfigured; the remainder of this message explains the problem, and
asks for help figuring out what to do.

Context
-------
The reprotest[1] utility tests reproducibility of .deb package builds
by performing two comparative builds with selective differences in the
environment.

As documented[2], the extent of build-env difference can be customized
using the 'variations' command-line argument, that has a default value
of 'all', or similarly the 'vary' argument.  These arguments can be
used together, and they support plus-or-minus symbols as value
prefixes (+/-) to indicate whether a variance factor is being added or
removed.

The reprotest commandline is parsed in sequence from left-to-right,
with each 'vary' argument applied like a patch -- amending existing
settings -- while in contrast each 'variations' argument performs a
complete reset of the variance context.

To examine/confirm reprotest's behaviour locally I can recommend its
'--dry-run' argument, instructing it to print what it would do without
performing any build actions.

Problem: misconfiguration case
------------------------------
Although the single argument '--variations=-timezone' could reasonably
be expected to disable a single form of variance (timezone) during a
test, in fact it resets the variance context to empty (it does not
contain 'all', begins with an empty context, and then attempts
performs a no-op removal of timezone from that).

This could allow packages to succeed when they would otherwise fail if
the intended level of build variation was enabled.

This misconfiguration has occurred in practice, and based on some code
searches (example[3]) I believe that around 400 Debian packages are
affected by this.

Resolution
----------
My working assumption is that packages that have a single
negative-variations entry (like the -timezone example above) intended
to disable solely the named factor during reprotest testing.

To resolve this it seems that we could:

  * Update the salsa-ci.yml files in each affected case to replace
'--variations=-' with '--vary=-'.
  * Update reprotest to handle a single-disabled-varations-value as a
special case - treating it as vary and/or emitting a warning.
  * Treat removal of a variance factor from an already-empty-context
as an error.
  * Radically, remove the ability for packages to customize their
reprotest arguments at all.

To readers of these lists: does this analysis and set of assumptions
make sense, and if so: do you prefer/recommend any of the suggested
approaches, or have alternative suggestions of your own?

Thank you,
James

[1] - https://salsa.debian.org/reproducible-builds/reprotest

[2] - https://salsa.debian.org/reproducible-builds/reprotest/-/blob/6cb0328ea422e12d115737714627850745f93a71/README.rst?plain=1#L299-311

[3] - https://codesearch.debian.net/search?q=path%3Asalsa-ci.yml+SALSA_CI_REPROTEST_ARGS%3A+%27--variations%3D-build-path%27&literal=1