Proposal for making Multi-Arch:same binNMU-safe

[Helmut Grohne]

Hello,

I am reaching out to the wanna build team (where I see changes needed),
reproducible builds (as a driving force for deterministic build
artifacts), sbuild team (where I see changes needed) and cross builders
(as the main consumers of packages marked Multi-Arch: same).

Problem
=======

When requesting a binNMU, the changelog entry is generated on the build
daemon. The timestamp used is the time the build was started. When
building a Multi-Arch: same package for multiple architectures, this
timestamp typically differs between architectures. Due to reproducible
builds, this timestamp is then transferred to SOURCE_DATE_EPOCH where it
is used by various build systems for creating reproducible artifacts.
Often times, it ends up in e.g. C header files. Those end up being
reproducible, but architectures reproduce varying contents. This tends
to break coinstallation.

Earlier, SOURCE_DATE_EPOCH was seeded from the most recent non-binNMU
timestamp, but that happened to break Ian's backup system. It used to
make Multi-Arch: same work though.

Proposed solution
=================

Use the timestamp of when the binNMU was requested in the binNMU
changelog.

When running wanna-build --binNMU (e.g. via wb wrapper), a row is
inserted into the transactions table for each architecture setting the
action to --binNMU. The first change here is using the same timestamp
for all those rows. The time column already is not unique and this would
coalesce a few more timestamps to the same values.

When a build daemon "takes" a build using API version 1, it receives a
YAML object containing metadata about the build being performed. This
happens to include the binNMU increment and the changelog text. This
should be augmented with a new field named binNMU-timestamp. Existing
autobuilders deal with unknown extra fields and ignore it.

To get the timestamp during the take action, there are two possible
approaches. It mainly draws information from the packages table, but
does not record the relevant timestamp. Therefore, another SQL query
would be needed to query the transactions table for the most recent time
a --binNMU was requested for the same
package/version/architecture/distribution combination. This can work
given that a newer binNMU cancels out an earlier one.

Alternatively, the packages table is augmented with a new column
recording the timestamp when the binNMU was requested. This would be
readily available in the subroutine add_one_building and could be
emitted. In my opinion, this would be slightly cleaner at the cost of a
schema change.

>From here we leave the wanna-build source code and continue in sbuild.
It already has --binNMU-timestamp and all that the buildd code would
have to do here is check for the newly added field and pass it along
using the new option. When the field is not provided, the option is
skipped. Thus, this change would also be backwards compatible.

Updated wanna-build would work with old buidds and updated buildds would
work with old wanna-build. Only when both are updated, the timestamp
would be changed and an entire class of Multi-Arch: same file conflicts
would go away.

Discussion
==========

What do you think about this solution?

buildd team:
 * Would you be ok with the projected extensions?
 * Do you prefer the schema change or the extra query?

reproducible team:
 * Do you see any issues arising from the projected changes?

sbuild team:
 * Would you be ok with the projected changes?

all:
 * Do you see any reason for why this would not work?

Let me answer for cross builders that this would fix somewhere between
10 and 100 M-A:same file conflicts. This always is a latent problem that
only surfaces once a package is binNMUed and therefore the full scale is
difficult to gauge.

Ian's backup system would be happy with this as far as I can tell.

Helmut