Getting condor builds reproducible

[Vagrant Cascadian]

On 2026-05-23, Vagrant Cascadian wrote:
> On 2026-05-22, Tim Theisen wrote:
>> Now that reproducible builds are mandatory, I am working on getting 
>> condor to meet those requirements.
...
>> It claims that there was a regression in reproducibility on i386 it that 
>> is blocking promotion. None of the recent changes should caused this.
>
> It is entirely possible that there is some non-deterministic
> issue... such as an ordering issue with a reasonable chance to come out
> reproducible, but some N out of 100 times comes out different.
>
> A hunch would be to check the version of the kernel that was used in
> both builds, as there have been a lot of kernel security updates
> lately... and those versions can sometimes get embedded in builds. That
> is not something that is intentionally varied, but due to security
> updates, sometimes that can cause issues.

Ok, my hunch has a little more background to go on that this is at least
part of the issue...

During a local build I see various build arguments passing the kernel
version:

  /usr/bin/c++ -DCONDOR_VERSION=\"25.10.1\" -DHAVE_CONFIG_H
  -DLINUX=\"LINUX_6.12.90+DEB13-AMD64\" ...

This is probably a permutation of this issue:

  https://tests.reproducible-builds.org/debian/issues/unstable/captures_kernel_version_via_CMAKE_SYSTEM_issue.html

Probably caused by the use of CMAKE_SYSTEM_VERSION:

  CMakeLists.txt-string( TOUPPER "${CMAKE_SYSTEM_NAME}" OS_NAME )
  CMakeLists.txt:string( TOUPPER "${CMAKE_SYSTEM_VERSION}" OS_VER )
  CMakeLists.txt-string( TOUPPER "${CMAKE_SYSTEM_PROCESSOR}" SYS_ARCH )

Not sure if just removing that line entirely, or providing a "REDACTED"
string would be an option here... we have the buildd.debian.org logs to
get information like that if needed, rather than embedding in the
binaries.

It is possible some machines (e.g. the debrebuild salsa-ci pipeline) are
using the same kernel version across builds, so would not trigger this.

buildd.debian.org and reproduce.debian.net might end up with a different
version depending on the timing and frequency of kernel updates being
available and applied, which might lead to inconsistant reproducibility
results.

I think tests.reproducible-builds.org may not actually be varying the
kernel version, though at least historically the intention is to vary
the kernel version (e.g. use a backports kernel and a stable kernel
across two builds).


> I did perform a few local builds of condor yesterday, but was not able
> to identify the nature of the differences in the diffoscope output, but
> there are still some lingering reproducible builds issues with condor.
...
> I could try again with fewer variations to see if I could narrow down
> the issue. reprotest, like tests.reproducible-builds.org, is a little
> more aggressive out of the box with in intentional variations...
> compared to reproduce.debian.net, which does not intentionally vary
> anything (though time will always vary, and kernel version, specific cpu
> processor (e.g. amd vs. intel), etc. might still be unintentionally
> different).

I tried again, but ran into diskspace issues. *sigh*

But at least I noticed the likely kernel version issues when reading the
build log! :)


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20260524/fe4d1d95/attachment.sig>