[Pkg-rust-maintainers] debcargo and dh-cargo ready for production use; sponsor request

[Josh Triplett]

I just tagged and pushed version 1 of dh-cargo
<https://anonscm.debian.org/git/pkg-rust/dh-cargo.git> and version 1.0.0
of debcargo <https://anonscm.debian.org/git/pkg-rust/debcargo.git>, and
published the latter to <https://crates.io/crates/debcargo>.  I've
addressed everything needed to use them in production, and I've locally
tested using them to package numerous crates.  Could a Debian Developer
in pkg-rust-maintainers please review and upload dh-cargo 1?

(debcargo packaging can come later; we can and should package it using
itself, but we don't need it packaged to start using it to generate
packages.  For now, you can `cargo install debcargo --root ~/.local`,
and make sure your PATH includes ~/.local/bin .)

You can run `debcargo package cratename` to generate a Debian source
package for a crate from crates.io, ready for review and (in most cases)
ready for upload.

Summarizing the steps to package a Rust application:

- cargo install debcargo --root ~/.local (and add to your PATH if needed)
- debcargo package application-crate
- debcargo package library-crate-dependency (recursively for all
  not-yet-packaged crates, until all dependencies satisfied)
- debcargo package --multi library-crate-dependency 'version expr'
  (for any library where you need multiple versions simultaneously)
- debuild all the library crates, and then install them all with dpkg -i
  (you don't need to install them incrementally, as library crates have
  no Build-Depends other than dh-cargo).
- debuild the application crate.
- Install and test.
- Send a mail to debian-devel summarizing all the new crates packaged
  (to avoid spamming debian-devel with individual ITPs for every
  individual crate).
- dput all the library crates and the application crate.

(A similar procedure will work for anyone looking to build a PPA for
their Rust application.)

I think we now have everything we need to start packaging applications
written in Rust.

Some changes since my last mail about debcargo and dh-cargo:

debcargo and dh-cargo now handle Cargo "features" for crates.  A Cargo
crate can declare various features, which enable/disable features of the
package, and pull in various additional dependencies or features of
dependencies.  debcargo includes all the dependencies for the "default"
feature in the main librust-cratename-dev package; for non-default
features, debcargo generates a librust-cratename+featurename-dev
package, which pulls in any additional dependencies needed for that
feature.  When generating dependencies, debcargo will translate a
dependency on a specific crate feature into a dependency on the
corresponding feature package.  (This relies heavily on dpkg's support
for versioned Provides, but that has worked since pre-jessie dpkg.)
I've structured the dependencies such that if we ever need to split the
"default" feature out of the primary librust-cratename-dev package, we
can do that without changing any other packages that depend on it.

Because many application crates have dependency trees that involve more
than one version of the same library (for instance, during the current
upstream transition from thread_local 0.2.7 and thread-id 2.0.0 to
thread_local 0.3.0 and thread-id 3.0.0), I added a --multi option to
include the ABI version in the source and binary package names,
producing co-installable packages.  debcargo still defaults to
generating unversioned package names, but you can pass --multi when
packaging a library crate that you need to support more than one version
of.  debcargo accepts a version constraint as a second argument to
specify the version you want to package, so if you want to package both
versions of thread-id, try `debcargo package --multi thread-id` and then
`debcargo package --multi thread-id '<3'`

To handle crates containing both a library and a binary, where we might
only want to ship the library and not the binary, debcargo now defaults
to omitting the binary from a crate that has both, unless you pass
--bin.  (For instance, the Rust "gcc" crate would otherwise ship a
binary package named "gcc", containing a binary "gcc-shim".)  You can
also rename the binary package using --bin-name, to avoid conflicts with
existing packages.  debcargo will still generate a binary package by
default for crates that don't also build a library, and for some library
packages we'll want to pass --bin, such as for future packaging of cargo
(which includes both a library and the "cargo" binary).

As discussed in my previous mail, I've reworked dh-cargo to never invoke
cargo directly on unpacked .crate sources; instead, it always sets up a
directory repository and invokes "cargo install cratename".  This allows
it to reliably build all crates, even those whose Cargo.toml files use
workspaces, path dependencies, and other interesting features that apply
when Cargo runs directly on a source tree (rather than pulling in
sources from a registry).

debcargo now scans through sources for copyright notices and
automatically includes them in debian/copyright, which eliminates the
copyright-without-copyright-notice lintian warning (at least for sources
that have copyright notices at all, which many don't bother with).  Note
that this does produce some false positives for crates that have license
files describing third-party licenses that apply to upstream-built
binary packages that embed libraries but not to binary packages that use
system libraries; for instance, cargo's LICENSE-THIRD-PARTY lists many
copyright notices and licenses that wouldn't apply to Cargo packages we
built via debcargo.  We may need to tweak this over time as we package
more crates.  (I would rather add piles of heuristics in debcargo than
tweak any of the generated packaging. :) )

debcargo-generated packages generate a description from the upstream
crate description.  This often produces good results, but occasionally
needs tweaking, most commonly to address excessively long short
descriptions or short descriptions that continue into the long
description.  (For the short description, debcargo uses the portion of
the description up to the first ". " or newline, whichever comes first.)
I've proposed
<https://internals.rust-lang.org/t/pre-rfc-adding-a-summary-short-description-to-cargo-crates/4436>
upstream in the hopes of a long-term solution; meanwhile, I would
suggest submitting patches to individual crates to improve their
"description" fields to produce better Debian package descriptions.
I would encourage submitting description improvements upstream, rather
than tweaking them downstream.

I've submitted a patch to Lintian to handle upstream sources in
/usr/share/cargo/registry, including tests: <https://bugs.debian.org/845201>
I would suggest that we not upload crate packages to Debian until that
lintian patch goes in.

I've filed <https://bugs.debian.org/845576> on ftp.debian.org requesting a
"rust" section, and sent
<https://lists.debian.org/debian-devel/2016/12/msg00097.html> to discuss
any potential release impact of introducing new archive sections.  In
the meantime, we can use "libdevel".

Finally, while working on this, I ran into one issue in upstream Cargo,
for which I've submitted an upstream pull request at
<https://github.com/rust-lang/cargo/pull/3369>.  That issue arises
whenever a crate's dependencies would pull in a specific non-default
feature in another crate that requires additional crates as
dependencies.  We'll need to get that fixed in order to package some
Rust applications (such as ripgrep), though many others will work
without it.

Note that none of this automation solves the issue of upstream crates
that embed library sources directly in the crate sources.  For those, we
may want to strip out the embedded sources to ensure they don't get
used, but in any case we'll definitely want to build against the
corresponding system libraries instead.  That will require adding
appropriate Depends on a C library -dev package to the Rust -sys package
for that library.  (We may want to add a debcargo option for that in the
future.)  This also requires some care to make sure we depend on an
appropriate version corresponding to the embedded sources (which in some
cases represent a git snapshot of the library).

Many packages will autodetect a system version of the library
if available.  A few packages, such as libgit2-sys, require an
environment variable set to detect a system version; for those, I'd
suggest we seek an upstreamable solution, but in the absence of that, we
can either set that environment variable in the debian/rules of
applications depending on those packages (which seems wrong), set that
environment variable in dh-cargo (better), or create a
/usr/share/cargo/env.d or similar to provide such snippets (most
general, but most complex).

Josh Triplett