Getting condor builds reproducible

[Tim Theisen]

Hello Vagrant,

This you for this insight. I will take care of the -DLINUX issue with 
condor. I can work directly with the developers to see what all the 
ramifications are.

...Tim

On 5/24/26 19:03, Vagrant Cascadian wrote:
> On 2026-05-24, Vagrant Cascadian wrote:
>> On 2026-05-23, Vagrant Cascadian wrote:
>>> On 2026-05-22, Tim Theisen wrote:
>>>> Now that reproducible builds are mandatory, I am working on getting
>>>> condor to meet those requirements.
>> ...
>>>> It claims that there was a regression in reproducibility on i386 it that
>>>> is blocking promotion. None of the recent changes should caused this.
>>> It is entirely possible that there is some non-deterministic
>>> issue... such as an ordering issue with a reasonable chance to come out
>>> reproducible, but some N out of 100 times comes out different.
>>>
>>> A hunch would be to check the version of the kernel that was used in
>>> both builds, as there have been a lot of kernel security updates
>>> lately... and those versions can sometimes get embedded in builds. That
>>> is not something that is intentionally varied, but due to security
>>> updates, sometimes that can cause issues.
>> Ok, my hunch has a little more background to go on that this is at least
>> part of the issue...
>>
>> During a local build I see various build arguments passing the kernel
>> version:
>>
>>    /usr/bin/c++ -DCONDOR_VERSION=\"25.10.1\" -DHAVE_CONFIG_H
>>    -DLINUX=\"LINUX_6.12.90+DEB13-AMD64\" ...
>>
>> This is probably a permutation of this issue:
>>
>>    https://tests.reproducible-builds.org/debian/issues/unstable/captures_kernel_version_via_CMAKE_SYSTEM_issue.html
>>
>> Probably caused by the use of CMAKE_SYSTEM_VERSION:
>>
>>    CMakeLists.txt-string( TOUPPER "${CMAKE_SYSTEM_NAME}" OS_NAME )
>>    CMakeLists.txt:string( TOUPPER "${CMAKE_SYSTEM_VERSION}" OS_VER )
>>    CMakeLists.txt-string( TOUPPER "${CMAKE_SYSTEM_PROCESSOR}" SYS_ARCH )
>>
>> Not sure if just removing that line entirely, or providing a "REDACTED"
>> string would be an option here... we have the buildd.debian.org logs to
>> get information like that if needed, rather than embedding in the
>> binaries.
> It seems to build by replacing OS_VER with "REDACTED" ... although
> passes:
>
>    -DLINUX=\"LINUX_\"
>
> And it dramatically reduced the diffoscope output (from hundreds of
> megabytes to ~10MB)! Checking another build to see if I can figure out
> what is left...
>
>
>> buildd.debian.org and reproduce.debian.net might end up with a different
>> version depending on the timing and frequency of kernel updates being
>> available and applied, which might lead to inconsistant reproducibility
>> results.
> Yes, this is exactly what happened on i386:
>
>    https://buildd.debian.org/status/fetch.php?pkg=condor&arch=i386&ver=25.10.1%2Bdfsg-1&stamp=1778880751&raw=0
>
>    -DLINUX=\"LINUX_6.12.86+DEB13-AMD64\"
>
> And on reproduce.debian.net:
>
>    https://reproduce.debian.net/i386/api/v1/builds/208996/log
>
>    -DLINUX=\"LINUX_6.12.88+DEB13-AMD64\"
>
> For the corresponding amd64 logs, they were running the same
> kernel. Probably the same for other architectures too...
>
>
>> I think tests.reproducible-builds.org may not actually be varying the
>> kernel version, though at least historically the intention is to vary
>> the kernel version (e.g. use a backports kernel and a stable kernel
>> across two builds).
> We should really ensure tests.r-b.o is systematically testing different
> kernel versions... since buildd.debian.org/reproduce.debian.net may run
> arbitrary kernel versions that will be out of sync on occasion, we want
> some way to catch that.
>
> I do not think it is reasonable for reproduce.debian.net to run the
> original running kernel version (E.g. running a security vulnerable
> kernel) in the build environment just to get reproducible builds...
>
>
> live well,
>    vagrant

-- 
Tim Theisen (he, him, his)
Release Manager
Center for High Throughput Computing
University of Wisconsin - Madison
3695 Morgridge Hall
1205 University Ave
Madison, WI 53706