Getting condor builds reproducible

[Vagrant Cascadian]

On 2026-05-24, Vagrant Cascadian wrote:
> On 2026-05-23, Vagrant Cascadian wrote:
>> On 2026-05-22, Tim Theisen wrote:
>>> Now that reproducible builds are mandatory, I am working on getting 
>>> condor to meet those requirements.
> ...
>>> It claims that there was a regression in reproducibility on i386 it that 
>>> is blocking promotion. None of the recent changes should caused this.
>>
>> It is entirely possible that there is some non-deterministic
>> issue... such as an ordering issue with a reasonable chance to come out
>> reproducible, but some N out of 100 times comes out different.
>>
>> A hunch would be to check the version of the kernel that was used in
>> both builds, as there have been a lot of kernel security updates
>> lately... and those versions can sometimes get embedded in builds. That
>> is not something that is intentionally varied, but due to security
>> updates, sometimes that can cause issues.
>
> Ok, my hunch has a little more background to go on that this is at least
> part of the issue...
>
> During a local build I see various build arguments passing the kernel
> version:
>
>   /usr/bin/c++ -DCONDOR_VERSION=\"25.10.1\" -DHAVE_CONFIG_H
>   -DLINUX=\"LINUX_6.12.90+DEB13-AMD64\" ...
>
> This is probably a permutation of this issue:
>
>   https://tests.reproducible-builds.org/debian/issues/unstable/captures_kernel_version_via_CMAKE_SYSTEM_issue.html
>
> Probably caused by the use of CMAKE_SYSTEM_VERSION:
>
>   CMakeLists.txt-string( TOUPPER "${CMAKE_SYSTEM_NAME}" OS_NAME )
>   CMakeLists.txt:string( TOUPPER "${CMAKE_SYSTEM_VERSION}" OS_VER )
>   CMakeLists.txt-string( TOUPPER "${CMAKE_SYSTEM_PROCESSOR}" SYS_ARCH )
>
> Not sure if just removing that line entirely, or providing a "REDACTED"
> string would be an option here... we have the buildd.debian.org logs to
> get information like that if needed, rather than embedding in the
> binaries.

It seems to build by replacing OS_VER with "REDACTED" ... although
passes:

  -DLINUX=\"LINUX_\"

And it dramatically reduced the diffoscope output (from hundreds of
megabytes to ~10MB)! Checking another build to see if I can figure out
what is left...


> buildd.debian.org and reproduce.debian.net might end up with a different
> version depending on the timing and frequency of kernel updates being
> available and applied, which might lead to inconsistant reproducibility
> results.

Yes, this is exactly what happened on i386:

  https://buildd.debian.org/status/fetch.php?pkg=condor&arch=i386&ver=25.10.1%2Bdfsg-1&stamp=1778880751&raw=0

  -DLINUX=\"LINUX_6.12.86+DEB13-AMD64\"

And on reproduce.debian.net:

  https://reproduce.debian.net/i386/api/v1/builds/208996/log

  -DLINUX=\"LINUX_6.12.88+DEB13-AMD64\"

For the corresponding amd64 logs, they were running the same
kernel. Probably the same for other architectures too...


> I think tests.reproducible-builds.org may not actually be varying the
> kernel version, though at least historically the intention is to vary
> the kernel version (e.g. use a backports kernel and a stable kernel
> across two builds).

We should really ensure tests.r-b.o is systematically testing different
kernel versions... since buildd.debian.org/reproduce.debian.net may run
arbitrary kernel versions that will be out of sync on occasion, we want
some way to catch that.

I do not think it is reasonable for reproduce.debian.net to run the
original running kernel version (E.g. running a security vulnerable
kernel) in the build environment just to get reproducible builds...


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20260524/7d0694ff/attachment.sig>