[Pkg-rust-maintainers] Initial prototype of dh-cargo

[Josh Triplett]

I just pushed an initial prototype of dh-cargo
(https://anonscm.debian.org/cgit/pkg-rust/dh-cargo.git), along with some
improvements to debcargo
(https://anonscm.debian.org/cgit/pkg-rust/debcargo.git).  I've
successfully used them to build a few Rust crates, including crates with
dependencies.  I tested the package builds in an environment with no
network access, to confirm that they don't touch the network.

Note that the directory registry feature (and some subsequent bugfixes
to it) requires cargo 0.14, and Debian currently has cargo 0.11.  For
testing, until a Debian package of cargo 0.14 or newer exists, I'd
suggest force-installing dh-cargo and bind-mounting a binary of cargo
0.14 or newer (such as one built from source) over /usr/bin/cargo.
(Also note that without that dependency satisfied, builds will still
appear to work, but they'll download dependencies from the network
rather than from the directory registry.)

A few issues remaining before using these in production, starting with
the most critical:

First, after some extensive discussion with Alex Crichton (upstream
developer on Rust and Cargo), it turns out that the contents of .crate
files can't, in general, support running any cargo commands on them
directly (such as "cargo test", "cargo build", or even "cargo package"),
for a variety of reasons.  Cargo behaves differently when handling a
dependency than when handling the main package.  When handling the main
package, Cargo wants to handle workspaces, path dependencies, and
various other features that it handles differently when processing a
build-dependency, and Cargo makes no promises about further such
features that may crop up in the future.  In general, running any cargo
command directly on a source (rather than on a dependency) may require
bits that don't get packaged in a .crate, and that only exist in the
upstream source repository.  Changing that would require a major
overhaul of Cargo and crates.io, and upstream doesn't have any interest
in doing so in the short-term, if at all.  It happens to work in many
cases, but not all.

The practical effect of this for library crates: the package can't run
the testsuite via "cargo test".  For any crate where we care about
running the testsuite, and where it doesn't happen to work from the
.crate, we'd have to obtain and package the source from the upstream
version control repository instead.  (In particular, we may want to do
that for crates that link to a C library via FFI, if we make them use a
system version of the library.)  Apart from that, though, I can work
around it by changing dh-cargo to avoid invoking cargo for library
crates.

For binary crates, the situation gets a bit more complex, since we do
need to use cargo to build and install their binaries.  Again, "cargo
install" (with its implicit "--path .") usually works but cargo doesn't
guarantee that.  So, if we want this to work reliably, and we don't want
to package the source from version control, then we'd need to put the
source in a directory registry and use "cargo install $cratename"
instead.

I plan to rework debcargo and dh-cargo to accommodate that, but I wanted
to go ahead and provide the prototype code first for people to play
with.  It does work for many crates; for instance, try "debcargo
quick-error", "debcargo ansi_term", build both, install both, then
"debcargo colorparse", and build that.

Other minor issues:

- debcargo currently assumes you want all dev-dependencies in
  Build-Depends, and all dependencies for all package features in
  Depends.  This will sometimes produce the right result (since you need
  the dev-dependencies to run "cargo test"), but may cause issues in
  bootstrapping (some crates have cycles between dev-dependencies and
  dependencies), or in some cases require a crate feature that only
  works with nightly (such as clippy).  Given the above issue that
  prevents running the testsuite anyway, I'll likely ignore
  dev-dependencies for now.

- Lintian reports warnings like extra-license-file on the upstream
  sources packaged in /usr/share/cargo/registry, which it should ignore:
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845201
  (We want to package the verbatim upstream sources, matching the hash
  provided in the .cargo-checksum.json file; in theory, if we repacked
  them and controlled for the usual sources of variance such as
  timestamps, we could reproduce the .crate file exactly.)

- The debcargo-generated packaging mostly complies with Policy and
  lintian, but has a couple of small lintian warnings:
  - new-package-should-close-itp-bug (debcargo doesn't know the ITP bug
    number)
  - copyright-without-copyright-notice (debcargo generates an accurate
    debian/copyright with all the license texts, but most crates don't
    have copyright notices, and for the few that do debcargo doesn't
    have any automated way to capture them.)

- debcargo uses the crate description to generate a package Description,
  which works OK, but sometimes needs some changes before uploading.
  (debcargo uses the first sentence/line of the description as the
  summary, and subsequent sentences/lines as the long description, along
  with a bit of boilerplate about Rust/Cargo.)  In some cases, we may
  want to contribute an improved description back upstream to the crate
  metadata; I hope we can avoid maintaining new descriptions downstream
  most of the time.

- debcargo should support generating an ITP for the package, using the
  package metadata to fill out the standard template. (We may or may not
  want to file ITPs for most library crates, but we probably want to
  file them for binary crates.)

- debcargo only generates crate packaging from scratch; it doesn't
  update existing packaging for new versions.  So, if you want to
  preserve old changelog entries, or any tweaks made to the package, for
  now you'll need to generate new packaging and then copy over the
  changelog entries and tweaks.  I plan to add an update mechanism at
  some point.