Getting condor builds reproducible

[Vagrant Cascadian]

On 2026-05-22, Tim Theisen wrote:
> Now that reproducible builds are mandatory, I am working on getting 
> condor to meet those requirements.

Great!

> I recently got condor to build reproducibly, as you can see on Salsa:
>
>      https://salsa.debian.org/hpc-team/condor/-/pipelines
>
> And on the package info page, in the  testing migrations section, it 
> shows that the amd64 build is reproducible.
>
>      https://tracker.debian.org/pkg/condor
>
> However, on the reproducible results page, it shows the same version is 
> not reproducible.
>
> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/condor.html
>
> So, why is there this disagreement? Is there one that is the 
> authoritative answer?

Yeah, as Holger mentioned, the reproduce.debian.net results are the ones
that might actually block migration... and the specific packages
blocking migration are linked to on the tracker page:

  https://reproduce.debian.net/i386/api/v1/builds/208001/artifacts/508612/diffoscope
  https://reproduce.debian.net/i386/api/v1/builds/208001/artifacts/508613/diffoscope
  https://reproduce.debian.net/i386/api/v1/builds/208001/artifacts/508614/diffoscope


> The disagreement seems to be with the NT_GNU_BUILD_ID. I thought that 
> the default rules would take care of that with -ffile-prefix-map. One 
> suggestion that I was was to link with build_id is sha1. I don't know if 
> that is warranted.

NT_GNU_BUILD_ID is just a sign that something was different in the
binary... most likely not anything -ffile-prefix-map would fix anymore;
in the past, the buildd.debian.org infrastructure used randomized paths
and our testing infraqstructure did also, but that is no longer the
case. It is almost certainly something other than the build path...


> Also, once a package is reproducible on one platform, I would expect it 
> to be reproducible on other platforms/CPU architectures.

There are often architecture-specific issues.


> It claims that there was a regression in reproducibility on i386 it that 
> is blocking promotion. None of the recent changes should caused this.

It is entirely possible that there is some non-deterministic
issue... such as an ordering issue with a reasonable chance to come out
reproducible, but some N out of 100 times comes out different.

A hunch would be to check the version of the kernel that was used in
both builds, as there have been a lot of kernel security updates
lately... and those versions can sometimes get embedded in builds. That
is not something that is intentionally varied, but due to security
updates, sometimes that can cause issues.


> What kinds of issues cause a package to not build reproducibly on arm64, 
> when it build reproducibly on the amd64 platform.

Not sure I have identified general classes of issues, but differences in
platforms do come up related to little/big/middle endian issues or
32-bit vs. 64-bit, etc. We have a fairly small number of tests and
history across multiple architectures, so I would not assume it is
architecture specific...

I did perform a few local builds of condor yesterday, but was not able
to identify the nature of the differences in the diffoscope output, but
there are still some lingering reproducible builds issues with condor.

Also tried using: reprotest --auto-build to "bisect" the issues, and the
control and experiment-0 builds (e.g. done with "everything" the same)
were successful, but the experiment-1 ("everything" different) crashed
the build ... so I was not able to get to the point where it tries to
only change one thing at a time (e.g. kernel version, time, timezone,
locale, etc.) ...

I could try again with fewer variations to see if I could narrow down
the issue. reprotest, like tests.reproducible-builds.org, is a little
more aggressive out of the box with in intentional variations...
compared to reproduce.debian.net, which does not intentionally vary
anything (though time will always vary, and kernel version, specific cpu
processor (e.g. amd vs. intel), etc. might still be unintentionally
different).


live well,
 vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20260523/7c3ae2ed/attachment.sig>