usrmerge testing in our CI

[Holger Levsen]

On Sat, Jun 17, 2023 at 02:08:20PM -0700, Vagrant Cascadian wrote:
> From what I recall looking at the log posted in irc it might be
> sufficient to "apt autoremove usrmerge" after the fact, as usrmerge is
> not really reversible... e.g. the /bin -> /usr/bin and other symlinks
> should continue to persist.

yes, we have done something like this in the past. for now, I just
wanted to get the builds back first.

and this seems to have worked!

quoting https://tests.reproducible-builds.org/debian/index_performance.html

packages tested on 2023-06-16 	2 	0 	1 	0
packages tested in the last 24h 	3665 	6043 	2322 	1652
 
> I think at several times during the bookworm cycle the techniques had to
> be adjusted in order to actually test usrmerge variations... and this
> just seems like a new surprise in a long list of surprises...

sure.
 
> > as said originally: we are the reproducible builds project, not the 
> > Debian QA project trying to find as many issues as we can.
> It is a judgement call just how much of the variations are "just" QA and
> how much are important reproducible builds issues...
> 
> I mean, sure, the argument can be made that a usrmerge and non-usrmerge
> environment are not the same build environment, even if they otherwise
> have the same packages installed!

well, for sure that's a different environment. hence some want to test,
whether builds in those still result in the same result. :)
 
> I see the purpose of testing exceptional things as actual reproducible
> builds issues, and even if unlikely to occur in Debian, they might help
> out some other project that makes different choices than Debian.

and I'd like to focus on setting up rebuilders, finally and properly.
 
or, in other words: I want to stop doing^wfocussing on QA&CI and improve
the real world.

and I'm also sure we can do both. but until we *have* rebuilders and
a working snapshot mirror, I'd personally like to spent time on CI.
And don't get me wrong: I want to keep maintaining the CI, but with
less priority and with less priority on introducing or maintaining
variations which have little real world impact.

If you want to rebuild^wreproduce a build, you'll install whatever
packages where installed in the 1st build. voila.

> > I'd rather have us focus on real reproducibility issues, than issues
> > with a variation we, Debian, don't care about.
> I would also like that we focus on real reproducibility issues, with
> variations that we, Reproducible Builds and Debian, care about. :P

I honestly don't care as much about our CI build results as what
Debian publishes on ftp.d.o - CI builds are just a mean to an end.
 
> >> > I'm less clear about whether we should cease testing bookworm. It
> >> > doesn't seem right for the CI to claim that various [bookworm]
> >> > packages are "reproducible in bookworm", when the presence of usrmerge
> >> > (or lack thereof) in bookworm means that they can still vary.
> > bookworm wasn't build with usrmerge variation, but rather usrmerge
> > was explicitly disabled on the builds.
> For clarity, you mean on buildd.debian.org?

no. I mean in our CI, for the 2nd build.

the code I disabled yesterday was:

   case "${SUITE}" in
       buster|bullseye)        ;;
       *)          pbuilder_options+=(--extrapackages usrmerge)
                   ;;
   esac

(see 675175178cc)
 
> > so one can also say that it doesnt seem right to introduce a variation
> > which never occured.
> I am near-absolutely-positive that it did occur on many developer
> machines, many of whom hopefully signed and uploaded their .buildinfo
> files...

those builds didnt make it to testing, because the release team requires
that packages are build on buildds.
 
> Even some builds on buildd.debian.org were done with usrmerge enabled,
> as debootstrap at various points defaulted to creating a usrmerge chroot
> by default. And then did not. And then did. And I am not even sure what
> the default is anymore.

AIUI this was tracked and fixed, so that no builds with usrmerge enabled
were let into testing.

>  So even if people were using sbuild, pbuilder,
> etc. for a clean build environment, they very likely uploaded packages
> and corresponding .buildinfo files throughout the bookworm release cycle
> that were built in both usrmerge and non-usrmerge environments...

no, see above. only source uploads could migrate.

> Yes most of those packages did not actually make it into bookworm,
> thanks to an awesome release team policy, but I would not be surprised
> if some that used build-depends on some of these maintainer-built
> packages ended up in bookworm... and of course a few builds that
> happened on buildds with usrmerge enabled might have also slipped
> through?

no.

also, for rebuilding the release: the usrmerge package would be listed
in those .buildinfo files. You can check them:

https://buildinfos.debian.net/buildinfo-pool_bookworm_amd64.list
https://buildinfos.debian.net/buildinfo-pool_bookworm_all-amd64.list

wget them all and grep for usrmerge. I'd be curious for the results!

> Well, yes... we are not testing against packages in the archive at all,
> really, we are stress-testing in an intentionally challenging
> environment with the intention to find reproducibility bugs... and
> reproducible builds testing also tends to unveil real bugs that might be
> hard to find any other way that through applying reproducible builds.
> 
> I know we would all love to (also?) be able to have tests that attempt
> to rebuild what is in the archive, but as I understand that is a bit
> stalled out right now...

yes. so lets fix that.


-- 
cheers,
	Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Covid. Changing hearts and minds since 2019. (@1goodtern)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20230617/1559075b/attachment.sig>