[Pkg-rust-maintainers] rustc_1.84.0+dfsg1-1~bpo12+1~bup_source.changes REJECTED

[Fabian Grünbichler]

On Wed, Jan 22, 2025, at 5:34 AM, Michael Tokarev wrote:
> 22.01.2025 01:01, Micha Lenk wrote:
>> 
>> Hi Michael,
>
> Hi!  Thank you for handling this, and for your comments and questions.

Hi as well,

Fabian (the main rustc maintainer) chiming in ;)

>> The backported version 1.84.0+dfsg1-1 is not in testing (yet).
>
> Yes.  I expected it to be a bit more difficult to get into bpo, this
> is why I uploaded it a bit earlier.  But now thinking about it, -
> it was a mistake, I should've waited for it to migrate to testing.
>
>> Additionally, I'd really appreciate some hints on how to validate the included
>> stage0 tarballs (once 1.84.0+dfsg1-1 entered testing, I would like to know!).
>> 
>> Would you mind to explain a little bit on debian-backports at l.d.o why it is
>> worth to allow such a big delta in a backport?
>> What alternatives were considered?
>
> Rust is problematic to bootstrap.  I can only be bootstrapped with a previous
> version of itself (1.83 in this case) being available as a stage0 compiler.
> This is the reason why it hasn't been backported before - we didn't back-
> port it in time at the first upload after bookworm has been released - at
> that time it the previous version has been available to compile the next
> version, the next version would be used to compile next-to-next version and
> so on - with waiting in backports-NEW each time because each next version
> introduces different name for the supporting package (like librust-1.84 etc).
>
> There are currently two possible ways to handle this, plus a few varietes:
>
> 1. using the pre-compiled version from the upstream as the stage0.  These
>     versions are especially provided by the upstream for this very purpose.
>     This is basically the standard way to bootstrap rust.
>
>     This is exactly what this upload does.  I grabbed the upstream stage0
>     binaries for the 3 architectures (please note: this backport will only
>     be available for the architectures where stage0 is available, not for
>     all architectures in debian).  There's a d/rules rule especially for
>     this purpose, to download upstream-published stage0 for the given list
>     of architectures, and pack it as rust_$version-stage0.orig.tar.gz --
>     see debian/make_orig-stage0_tarball.sh and debian/get-stage0.py
>
>     Now, rustc will build itself using these stage0 pre-compiled binaries
>     to bootstrap itself.  This is the way how it is done to bootstrap
>     rust on debian (ports), and/or when a previous version of the compiler
>     is somehow not available.
>
>     It is interesting - current build procedure does NOT have any way
>     to validate the stage0 binaries it is downloading.  It looks like
>     a serious omission which should be addressed in the rust source
>     package, by maybe verifying gpg signatures with the already
>     available debian/upstream/signing-key.asc.

most points above are correct, but the prebuilt stage0 from upstream is verified. the trust anchor/chain looks like this:

1.) when importing a new upstream version, the upstream source tarball is verified via the upstream signing key (GPG atm, with plans to switch to another mechanism in the future)
2.) that upstream source tarball is heavily pruned (it would be 4GB otherwise, including a full copy of gcc and llvm)
3.) the upstream source tarball (both in pristine form, as well as after repacking/pruning) contains a file containing the information about the prebuilt stage0 artifacts provided by upstream (`src/stage0`)
4.) the debian/rules target and corresponding script to download the stage0 for (re)bootstrapping purposes uses rustc own build entry point (somewhat confusingly called bootstrap and existing as a python script and a rust binary, the former builds and calls the latter)
5.) this bootstrap tool from upstream uses the `src/stage0` file to validate the checksums of downloaded stage0 components before moving them into place

as a result the `debian/rules` target can only download the stage0 artifacts matching the ones upstream used (and as a result, is also limited to those architectures where upstream provides such a prebuilt stage0 - atm this excludes armel and mips64el among the release architectures).

> 1.a there's a build profile in rust package in debian which enables
>     downloading pre-built stage0 from upstream, instead of using the
>     bundled ones as rust_$version-stage0.orig.tar.gz.  This, in my
>     view, is worse than the previous variant, due to several reasons.

the machinery described above exists exactly to avoid this problem and have a (public) record of the prebuilt artifacts used to (re)bootstrap, as opposed to doing whatever on a DD machine to get an initial binary package.

> 1.b there's rustup package.  This one installs pre-built binaries from
>     the upstream site directly for the use.

rustup as currently packaged is not co-installable with regular rustc, and has most of the problems described above (no track record, ..)

> 2. Having version of the compiler available for a different architecture,
>     by using cross-compiling for this architecture (any rust installation
>     can generate binaries for all supported architectures).
>
>     Basically, it's possible to build (one way or another) rustc for amd64
>     for example, and cross-build it for other interesting architectures.
>     This is actually what I *tried* to do, - and to provide binaries of
>     rust built by me in the first upload to bpo.  I would've used just one
>     of the upstream pre-built binaries for bootstrapping on amd64, and
>     would upload just the .deb files (for several architectures at once)
>     to bookworm-backports.

cross builds should work again now (with 1.84.0+dfsg1-2) :)

>     This has significant - in my view - drawback: it basically hides the
>     fact that the initial bootstrapping is done using an upstream pre-built
>     binary.  I think it is better to be explicit here, with the actual
>     pre-built binaries being in the debian archive readily available for
>     any verification.

I agree as well. cross-building is unavoidable when (re)bootstrapping architectures where no prebuilt stage0 exists.

>     Besides hiding the actual pre-built binary in use, this way actually
>     doesn't work currently, - there's a prob in upstream cross-environment
>     bootstrapping which is being worked with.  Maybe we should try to fix
>     this one before trying to upload rust to bookworm-backports.
>
> That's basically it.
>
> Theoretically it is possible (maybe) to build rust for bookworm-backports
> using rust in trixie.  The problem here is that the resulting binaries
> require libc6 from trixie to run, - maybe this can be worked around by
> providing static binaries, so glibc from bookworm can be used when
> actually compiling rust.  This would be seriously complicated though,
> since we'll need some serious assistance from you to use .debs from
> trixie to build stuff for bookworm-backports.

I don't think that this would be any better than the current approach to be honest ;)

starting with trixie I'd like to provide each version via backports as soon as they hit testing. that way we shouldn't have to do prebuilt-stage0 builds at all.

> This thing is kinda difficult.  Even without the foreign binaries, rust
> brings enough maintenance tasks by having packages named after version,
> so each new version (about every 6 months) needs a new round in the NEW
> queue.  During bookworm freeze we haven't updated rust for a long time,
> so had to perform multiple updates after the release, to catch up the
> upstream, - it was kinda fun to prepare the next version, upload, wait
> for the acceptance in NEW, wait till it is built on all architectures
> and migrate (and fix all possible issues in between), and repeat for
> the next version upload to repeat the whole cycle..

bin-NEW turnaround (for rustc at least) is pretty fast nowadays (and I hope the same will be true for backports once I start uploading rustc regularly there? ;)). a new upstream release happens every 6 *weeks*, not months. and like you said, each version needs to be built and uploaded in turn if we want to avoid the rebootstrap dance with the prebuilt (or downloaded during build time) stage0.

hope this clears things up!
Fabian