[Pkg-rust-maintainers] Rust Packages, BoF meeting notes

[Angus Lees]

I'm trying to think through some various "obvious" alternatives to the
current debcargo approach, and figured I'd dump them here in case it
triggers useful discussion:

(I've used the term cargo "features" below.  I realised after writing most
of it that I probably should say "optional dependencies", since features
include both optional dependencies and conditional compilation.  I'm
ignoring that last bit, because that doesn't affect packaging
dependencies...)

*** Alternative: Ignore cargo features, install superset of build-deps

Build-dep on all possible source packages (superset of all cargo
features).  This was initially explored and rejected in favour of the
current debcargo features approach - but maybe we can go back to this with
enough work on the edge cases.

- Removes all the trivial binary "feature" packages (yay).
- Building a Rust binary package will download _lots_ of extra build-deps
(eg: multiple TLS implementations), many of which will be ignored.
Hopefully none conflict (pure-Rust shouldn't ?? but they might pull in
non-Rust deps that do conflict)
- Circular Rust dependencies can't be built without bootstrap hacks because
Debian tools don't deal with cycles in *build-dep* dependencies when
building from scratch (boo).

Possibly(??) some extra automated tooling around that last point
(bootstrapping circular build-deps) works out to be better for the project
overall than the some extra automated tooling to help with the explosion of
trivial "feature" binary packages.  I think this is probably the most
obvious point where the different points-of-view are arriving at different
conclusions currently.

*** Alternative: Lift "features" into application build-deps, remove
library deps

Rethinking everything from first principles:
- When building a Rust application, we "just" need to get the Rust source
for the current package and all the library dependency packages onto disk
somewhere, and then invoke cargo.  Conceptually, it's not that hard.
- Installing other packages at build-time means build-deps, without a
significant change to the way Debian works.
- That means Rust library source has to be in binary packages.
- The obvious way to do that is to have a 1:1 Rust library source->binary
package for each upstream Rust crate.

(So far so good - now we get into the controversial parts.)

Split the Rust world into "library" (source bundle) and "application"
(compiled binary[*]) Debian source packages.

[*] "Anything that actually gets compiled" also includes dylib plugins and
other exotic things if we want to be strict.  I'm using "applications" to
include all these.

"Libraries" don't actually get compiled - they're just repackaged Rust
source.  So these packages don't _actually_ need any dependencies in the
technical Debian sense.  If we remove all the dpkg dependencies, then we
have no problems with circular dependencies - they really are just source
tarballs trivially repackaged into .debs.

"Application" packages now have to do all the hard work: They need to pull
in all the source libraries they need (including transitive dependencies),
_and resolving features_.  Resolving features at the last compilation step
is good, because it's actually how the Rust/cargo system works anyway.  We
need to write a debcargo-replacement that expands out the list of
transitive dependencies _taking into account features_ into an explicit
list of Debian "library" package build-deps.

- Removes all the trivial binary "feature" packages (yay. But see [+])
- No circular build-dep dependencies (yay).
- Applications will now have enormous build-dep lists (imagine mozilla).
.. It's the same _actual_ build-dependencies they would have any other way,
just flattened a bit. Is this a problem?
- Every change in a transitive dependency needs a change in the
end-application build-deps .. but maybe this doesn't matter, since it
needed to be rebuilt anyway (because static linking).
- Uploads to library packages are basically untested, since no compilation
happens (that's the point).  This is probably ok if tests are done some
other way (autopkgtest?)
- [+] I think we need some new trivial _source_ packages for crates that
include both "application" and "library".  The "application" source package
will depend on the library package (in the usual way described above) and
invoke cargo as required to get the desired application binary.  I don't
know whether there will be more or fewer than the feature packages overall,
but they should not churn NEW as much (since once they exist, they will
tend to continue to exist, and there is probably at most one per upstream
crate.)

*** Alternative: Improve Debian source tools

Some changes I haven't thought enough about.  Maybe we make it so you can
build-depend on *source* packages or something equally radical (introduce a
new category of "archive" package?) - since source packages are "trivially"
packaged in a way that allows tools to unpack a circular dependency without
issues.

That seems hard, but maybe worth pursuing given this "new" rust/golang
pattern of deferring just about all compilation until the "application"
package.  I think the right solution here would be bigger than Rust and
outside the scope of this thread.


On Mon, 19 Oct 2020 at 06:05, Joerg Jaspert <ftpmaster at debian.org> wrote:

> On 15925 March 1977, Geert Stappers wrote:
>
> >
> http://meetbot.debian.net/debian-rust/2020/debian-rust.2020-08-26-18.58.log.html#l-102
> > is what I'm aware of.
>
> Thats more/better than just the video, thanks.
>
> Well, reading through that, let me comment on some points:
>
>  - Yes, we *can* allow some packages to bypass bin-NEW. That list is
>  naturally strictly limited.
>  - No, thats currently not up for rust.
>  - Sure, tech-ctte is a possible way to override us, but I hope we can
>  get something sorted between us without that hammer.
>  - Yes, empty packages that are nothing but a container for
>  Depends/Provides lines are a problem, and yes, due to getting the
>  indexes and database sizes needlessly up.
>  - Insanely huge provides are also not good, but maybe better than the
>  packages. Tools that break ought to be fixed.
>  - Or maybe we can add something new, no idea. Lets see.
>
> >> and more important, did any action afterwards happen to come up with
> >> a nicer way?
> >> And then, which way would that way be?
> > Good question.
> > I do hope it is the impulse that this project, Rust in Debian, was
> > waiting for.
>
> > With the help of favorite $DEITY feel those who care about this
> > project
> > comfortable enough[1] to speak freely in the up coming discussion.
>
> Noone will be hit for suggesting something, so whoever has something to
> say, speak up. Or nothing will go forward, which isn't good.
>
> >> Or the other way round, for someone that has only a very basic
> >> understanding
> >> of rust - whats the goal here that should be supported? Maybe we can
> >> come up
> >> with a better way that doesn't let our tools or Package files explode
> >> needlessly?
> > In case it stays quiet in here:
> >   Next scheduledd IRC-meeting in #debian-rust is
> >   Wednesday October 28th  19:00  UTC
> > Inviting FTP-Masters for that meeting.
>
> I put it in my calendar, but can't make a hard promise. I try, though.
> Also, if we stay on the current point, I doubt it will gain anyone much.
> So hopefully more will happen by mail before then.
>
> --
> bye, Joerg
>
> _______________________________________________
> Pkg-rust-maintainers mailing list
> Pkg-rust-maintainers at alioth-lists.debian.net
>
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-rust-maintainers
>


-- 
 - Gus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-rust-maintainers/attachments/20201019/4df869f7/attachment.html>