usrmerge testing in our CI

[Vagrant Cascadian]

On 2023-06-17, Holger Levsen wrote:
> On Sat, Jun 17, 2023 at 02:08:20PM -0700, Vagrant Cascadian wrote:
>> From what I recall looking at the log posted in irc it might be
>> sufficient to "apt autoremove usrmerge" after the fact, as usrmerge is
>> not really reversible... e.g. the /bin -> /usr/bin and other symlinks
>> should continue to persist.
>
> yes, we have done something like this in the past. for now, I just
> wanted to get the builds back first.
>
> and this seems to have worked!
>
> quoting https://tests.reproducible-builds.org/debian/index_performance.html
>
> packages tested on 2023-06-16 	2 	0 	1 	0
> packages tested in the last 24h 	3665 	6043 	2322 	1652

After another fix, yes, it is working! :)


>> > as said originally: we are the reproducible builds project, not the 
>> > Debian QA project trying to find as many issues as we can.
>> It is a judgement call just how much of the variations are "just" QA and
>> how much are important reproducible builds issues...
>> 
>> I mean, sure, the argument can be made that a usrmerge and non-usrmerge
>> environment are not the same build environment, even if they otherwise
>> have the same packages installed!
>
> well, for sure that's a different environment. hence some want to test,
> whether builds in those still result in the same result. :)

Exactly...


>> I see the purpose of testing exceptional things as actual reproducible
>> builds issues, and even if unlikely to occur in Debian, they might help
>> out some other project that makes different choices than Debian.
>
> and I'd like to focus on setting up rebuilders, finally and properly.
>  
> or, in other words: I want to stop doing^wfocussing on QA&CI and improve
> the real world.
>
> and I'm also sure we can do both. but until we *have* rebuilders and
> a working snapshot mirror, I'd personally like to spent time on CI.
> And don't get me wrong: I want to keep maintaining the CI, but with
> less priority and with less priority on introducing or maintaining
> variations which have little real world impact.
>
> If you want to rebuild^wreproduce a build, you'll install whatever
> packages where installed in the 1st build. voila.

Sure, we have agreement on that, at least on the surface! :)


>> > I'd rather have us focus on real reproducibility issues, than issues
>> > with a variation we, Debian, don't care about.
>> I would also like that we focus on real reproducibility issues, with
>> variations that we, Reproducible Builds and Debian, care about. :P
>
> I honestly don't care as much about our CI build results as what
> Debian publishes on ftp.d.o - CI builds are just a mean to an end.

Well, I think the most important feature of the CI builds are for
catching reproducibility regressions! Which happen far more often than I
would like...


>> >> > I'm less clear about whether we should cease testing bookworm. It
>> >> > doesn't seem right for the CI to claim that various [bookworm]
>> >> > packages are "reproducible in bookworm", when the presence of usrmerge
>> >> > (or lack thereof) in bookworm means that they can still vary.
>> > bookworm wasn't build with usrmerge variation, but rather usrmerge
>> > was explicitly disabled on the builds.
>> For clarity, you mean on buildd.debian.org?
>
> no. I mean in our CI, for the 2nd build.
>
> the code I disabled yesterday was:
>
>    case "${SUITE}" in
>        buster|bullseye)        ;;
>        *)          pbuilder_options+=(--extrapackages usrmerge)
>                    ;;
>    esac
>
> (see 675175178cc)

That looks to me like bookworm was at least intended to be built with
usrmerge enabled, as it was in the "*" rather than "buster|bullseye"
match.


>> > so one can also say that it doesnt seem right to introduce a variation
>> > which never occured.
>> I am near-absolutely-positive that it did occur on many developer
>> machines, many of whom hopefully signed and uploaded their .buildinfo
>> files...
>
> those builds didnt make it to testing, because the release team requires
> that packages are build on buildds.

But the developer .buildinfo files are available. That is what I am
talking about.

Longer-term, I would love to see the workflow for uploading to Debian
be something more like:

  Developer builds and pushes source + .buildinfo to debian infrastructure,
  buildd builds the .buildinfo on two different buildd machines,
  if all three match, upload to the actual archive...

Obviously, we are quite a long ways from there...

The ship has basically sailed for bookworm, for the most part, and so if
trixie switches to usrmerge on the buildd machines, then I guess we are
basically done with usrmerge (and everything else is a bug to sort out).

The question is weather we want to help with finding those stupid bugs,
but it is maybe proving to be technically more annoying than it might be
worth to keep testing usrmerge...


>> Even some builds on buildd.debian.org were done with usrmerge enabled,
>> as debootstrap at various points defaulted to creating a usrmerge chroot
>> by default. And then did not. And then did. And I am not even sure what
>> the default is anymore.
>
> AIUI this was tracked and fixed, so that no builds with usrmerge enabled
> were let into testing.

Even thorough enough to catch build-depends in those odd windows of
usrmerge-ness?


>>  So even if people were using sbuild, pbuilder,
>> etc. for a clean build environment, they very likely uploaded packages
>> and corresponding .buildinfo files throughout the bookworm release cycle
>> that were built in both usrmerge and non-usrmerge environments...
>
> no, see above. only source uploads could migrate.

No for the .deb packages, yes for the developer .buildinfo files (which
are technically maintained kind of outside the archive ... meh.)
although the buildd .buildinfo files are the ones we expect to match the
.deb in the archive...


>> Yes most of those packages did not actually make it into bookworm,
>> thanks to an awesome release team policy, but I would not be surprised
>> if some that used build-depends on some of these maintainer-built
>> packages ended up in bookworm... and of course a few builds that
>> happened on buildds with usrmerge enabled might have also slipped
>> through?
>
> no.

My guess is still maybe... but I am entirely open to being wrong about
that, and it is not so important to spend much time on it either!


> also, for rebuilding the release: the usrmerge package would be listed
> in those .buildinfo files. You can check them:
>
> https://buildinfos.debian.net/buildinfo-pool_bookworm_amd64.list
> https://buildinfos.debian.net/buildinfo-pool_bookworm_all-amd64.list
>
> wget them all and grep for usrmerge. I'd be curious for the results!

Whee!

Those are just the buildd built .buildinfo files? I remember there was
some syntax for the maintainer ones (detected by source+ARCH maybe?)


>> Well, yes... we are not testing against packages in the archive at all,
>> really, we are stress-testing in an intentionally challenging
>> environment with the intention to find reproducibility bugs... and
>> reproducible builds testing also tends to unveil real bugs that might be
>> hard to find any other way that through applying reproducible builds.
>> 
>> I know we would all love to (also?) be able to have tests that attempt
>> to rebuild what is in the archive, but as I understand that is a bit
>> stalled out right now...
>
> yes. so lets fix that.

Yes, I think we are in agreement there! :)

It seems like the main issue is a reliable snapshot.debian.or or
alternative, and then actually performing rebuilds with matching
packages.


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20230619/c5c75ed6/attachment.sig>