[Pkg-rust-maintainers] debcargo and dh-cargo ready for production use; sponsor request

[Josh Triplett]

On Mon, Dec 05, 2016 at 01:25:46PM +0000, Luca BRUNO wrote:
> On Sunday, 4 December 2016 23:47:08 UTC Josh Triplett wrote:
> > I just tagged and pushed version 1 of dh-cargo
> > <https://anonscm.debian.org/git/pkg-rust/dh-cargo.git> and version 1.0.0
> > of debcargo <https://anonscm.debian.org/git/pkg-rust/debcargo.git>, and
> > published the latter to <https://crates.io/crates/debcargo>.  I've
> > addressed everything needed to use them in production, and I've locally
> > tested using them to package numerous crates.  Could a Debian Developer
> > in pkg-rust-maintainers please review and upload dh-cargo 1?
> 
> Thanks for your great work! I would however suggest to:
>  * start drafting the policy this implements, at
>     https://wiki.debian.org/Teams/RustPackaging/Policy

Good point.  I'd thought about starting to write this down; doing so
before uploading seems like a good idea.

>  * hold dh-cargo and debcargo till the policy is finalized
>  * Target those at experimental, or anyway hold them out of stretch

Agreed.  (In hindsight, I should have thought of this before
tagging; I may force-push the dh-cargo tag with this change.)

Also, I just added an option to debcargo to set the target distribution,
for when you want to upload a crate package to experimental or similar.

> Also, care to join #debian-rust on OFTC?

Sure.

> > debcargo includes all the dependencies for the "default"
> > feature in the main librust-cratename-dev package; for non-default
> > features, debcargo generates a librust-cratename+featurename-dev
> > package, which pulls in any additional dependencies needed for that
> > feature.
> 
> This looks convoluted and will pollute the package namespace (and also
> relies on the fact that "featurename" charset is a subset of the package name 
> charset), but I don't have any better ideas at the moment.

I did specifically review the characters allowed in crate/feature names
versus the characters allowed in Debian package names.  Debian package
names additionally allow '+' and '.' (and allow starting with a number).
I used '+' for features (and any other future package variants), and '.'
for version numbers included in the package name.

Most crates don't have non-default features; after building the 40
crates needed to package ripgrep, only 7 have non-default features,
producing a total of 9 empty dependency packages.  Half of those come
from packages providing SIMD/AVX support, and based on a conversation
with the author of ripgrep and much of the SIMD support, those features
will go away once Rust has better support for runtime detection of SIMD
CPU features.

> > Because many application crates have dependency trees that involve more
> > than one version of the same library (for instance, during the current
> > upstream transition from thread_local 0.2.7 and thread-id 2.0.0 to
> > thread_local 0.3.0 and thread-id 3.0.0), I added a --multi option to
> > include the ABI version in the source and binary package names,
> > producing co-installable packages.  debcargo still defaults to
> > generating unversioned package names, but you can pass --multi when
> > packaging a library crate that you need to support more than one version
> > of. 
> 
> This looks like an inverted default, and possible something that could be 
> better handled by versioned provides.

I'm using versioned Provides for this, but having more than one
version simultaneously installed requires distinct package names (and
source package names).

> Care to draft out in the policy how
> semver compatibility will map to deb dependencies resolution?

Will do.

debcargo maps every kind of semver dependency a Cargo.toml file can
contain into a corresponding Debian package dependency; for instance,
"1.2.3" (AKA "^1.2.3") becomes (>= 1.2.3), (<< 2).  For 0.1.2, it
becomes (>= 0.1.2), (<< 0.2).

When you pass --multi, debcargo includes -major.0 in the package name
if major != 0, or -0.minor if major == 0.  This corresponds to the
definition of compatibility Cargo uses, defined at
http://doc.crates.io/specifying-dependencies.html#caret-requirements .

> > debcargo will still generate a binary package by
> > default for crates that don't also build a library, and for some library
> > packages we'll want to pass --bin, such as for future packaging of cargo
> > (which includes both a library and the "cargo" binary).
> 
> How do you handle path dependency on same-repository crates?
> For example, cargo depends on fake versions of crates-io, see
> https://github.com/rust-lang/cargo/issues/2989#issuecomment-240040153

Every crate that people can install necessarily gets published as a
.crate file on crates.io, and debcargo uses cargo to download those.
You can ignore the path dependencies; any path dependency on crates.io
must also have a version requirement, and when cargo resolves those as a
dependency or non-local "cargo install cratename" (rather than as the
root crate with "cargo build" or similar), it ignores the path and uses
the version.

> > As discussed in my previous mail, I've reworked dh-cargo to never invoke
> > cargo directly on unpacked .crate sources; instead, it always sets up a
> > directory repository and invokes "cargo install cratename".  This allows
> > it to reliably build all crates, even those whose Cargo.toml files use
> > workspaces, path dependencies, and other interesting features that apply
> > when Cargo runs directly on a source tree (rather than pulling in
> > sources from a registry).
> 
> This is a much saner way of doing things compared to what all the hoops I 
> jumped through to bootstrap cargo :)

I hope that next time you need to update cargo, you can use debcargo to
do so, and avoid bundling its dependencies.

> > Note that none of this automation solves the issue of upstream crates
> > that embed library sources directly in the crate sources.  For those, we
> > may want to strip out the embedded sources to ensure they don't get
> > used, but in any case we'll definitely want to build against the
> > corresponding system libraries instead.
> 
> I guess some discussion with upstream around best-practices for vendored 
> native libs should happen. It will make our life easier to detect (and filter) 
> those.

Agreed.  I've had a bit of that discussion upstream, but we need to have
more.

> > That will require adding
> > appropriate Depends on a C library -dev package to the Rust -sys package
> > for that library.  (We may want to add a debcargo option for that in the
> > future.)  This also requires some care to make sure we depend on an
> > appropriate version corresponding to the embedded sources (which in some
> > cases represent a git snapshot of the library).
> > 
> > Many packages will autodetect a system version of the library
> > if available.  A few packages, such as libgit2-sys, require an
> > environment variable set to detect a system version;
> 
> https://github.com/alexcrichton/git2-rs/pull/80

Yeah, I've seen that. :(

I hope we can come up with something more acceptable upstream, or at the
least, some way to switch to a standardized environment variable instead
of a crate-specific one.

> > we can either set that environment variable in the debian/rules of
> > applications depending on those packages (which seems wrong), set that
> > environment variable in dh-cargo (better), or create a
> > /usr/share/cargo/env.d or similar to provide such snippets (most
> > general, but most complex).
> 
> The env flag needs to be transitively passed down to dependencies, so
> a debian/rules export won't probably work.

I meant in the debian/rules of an application package that transitively
depended on libgit2-sys.  But I do agree that that seems suboptimal.  If
we have to go this route, I'd rather implement /usr/share/cargo/env.d. .

- Josh Triplett