[Pkg-rust-maintainers] Packaging software written in Rust

[Josh Triplett]

On Tue, Jul 19, 2016 at 11:32:17PM +0000, Angus Lees wrote:
> On Mon, 18 Jul 2016 at 15:31 Josh Triplett <josh at joshtriplett.org> wrote:
> > On Sun, Jul 17, 2016 at 08:57:40PM -0700, Josh Triplett wrote:
> > > I can understand wanting to support local distro patches. We should be
> > very careful doing so, as it seems questionable to patch a crate while
> > keeping the same version number; that breaks some of Cargo's assumptions
> > about reproducibility. I'd rather have a separate crate and a replace
> > stanza. However, if we absolutely had to, directory registries seem to
> > support that.
> >
> 
> Remember the *primary* reason we are bothering to do any of this is to be
> able to support a hypothetical security fix to an existing Rust library
> package 5 years from now.  We *must* be able to rebuild with patches.

I absolutely agree.  I'm just arguing that if we patch version 1.2.3 of
some crate, the patched version shouldn't use version number 1.2.3; it
should use, for instance, version 1.2.3+debian1 or similar, to make it
possible to identify what crate the package built with.  (Cargo and
the Semantic Versioning standard allow that.)  Anything that declares a
dependency on ^1.2, ^1.2.3, ~1.2.3, 1.2.*, or >1.2.3 will still allow
that.  (And we have to rebuild binary crates to pull in the new library
anyway, whether we change the version number or not.)

Having a distinct version number when we patch a crate will make it
feasible to tell when a build actually picks up that patched crate, such
as in the cargo output in the build log for a bin crate.

> > - Sources that want to build multiple crates, with path dependencies.
> > I'll need to look at the resulting .crate files to see if we can just treat
> > them as separate packages.
> >
> > We can; I tested this, and I could "cargo package" each crate
> > independently, which produced separate .crate files with a disjoint set
> > of included sources.  I think we can either ship the resulting crates
> > independently, or ship a package with multiple .crate.orig.tar.gz files
> > that builds all the crates.  The former seems simpler, though the latter
> > might help if a package expresses an exact version constraint on the
> > crates it bundles.  Let's go with the former for now, until we run into
> > a package that makes that problematic.
>
> From my earlier exploration, it seemed ~common for a single upstream github
> repo to produce multiple crates.io crates, each with separate version
> numbers, and for the github repo to not have any sort of tag/branch that
> corresponded with whatever was uploaded to crates.io.  My conclusion at the
> time was that we had to package what was published on crates.io, and
> basically ignore whatever original common source tree they may or may not
> have been produced from.

Agreed.  I think we'll generally want to take the .crate file, rename it
to .orig.tar.gz, and add a generated Debian diff.  I don't think we can
reasonably package crates directly from their repositories,
unfortunately.

> > - -sys crates that link C code. Some of these support building against
> > the system library, some don't. Fortunately, libgit2-sys does. But we'll
> > have to figure out a way to systematically specify system libraries for all
> > crates.
> >
> > It looks like libgit2-sys supports an environment variable to use
> > pkg-config to find libgit2; if set, it completely ignores its built-in
> > copy of libgit2, which means we can avoid shipping that copy entirely.
> >
> > However, that means we'll need that environment variable set when
> > building packages of bin crates that incorporate libgit2-sys (by way of
> > git2-rs).  That seems like a pain.
> >
> > One possible proposal: librust-libgit2-sys-dev can have a Depends on
> > libgit2-dev, and can ship a file
> > /usr/share/rust/env.d/libgit2-sys-$version.conf (near the installed
> > Cargo directory registry in /usr/share/rust/cargo/), which contains
> > "LIBGIT2_SYS_USE_PKG_CONFIG=1".
> >
> 
> .. or we just patch the Debian version of librust-libgit2-sys-dev's build.rs
> to always use the system version.

While that's always an option, I'd like to avoid having to repeatedly
patch each new upstream version of a library if at all possible.  (If we
can get the patch upstream, great, but I'm not wildly hopeful there.)
That said, see below; I think this is a much less widespread problem
than I'd first thought.

> Realistically, we probably want to strip
> the vendored C sources out of most of the *-sys crates when packaging,
> since Build-Deps can guarantee the system library is present, and actually
> removing the files means we can avoid auditing them for licenses, etc.

> Many of the other *-sys crates I looked at earlier used pkg-config if
> available, so using the system library versions Just Worked.

I took a quick look at a large selection of -sys crates, and as far as I
can tell, only two (libgit2-sys and one of the readline crates) shipped
their own copy of the upstream source as part of the .crate, and only
libgit2-sys required an environment variable to find and use a system
version by default.

Given that, I think we could reasonably ask libgit2 to support a
standard environment variable (like CARGO_USE_SYSTEM_DEPS), set that one
variable unconditionally, and ask any new crates to support that.  I
think upstream would probably take such a patch.

I also don't think, for trivial cases like those where the source is
unlikely to have license problems, that it's worth the trouble of
patching out the bundled source, compared to the benefit of using an
unmodified .orig.tar.gz from the .crate file.  But on the other hand, it
also wouldn't be that bad for just those two crates.

On a related note, we're probably also going to want a way for lib
crates that pull in system libraries to run "cargo test" at build time,
to make sure they function with the system version of the library.  Many
of them do *not* include any version specifications in their pkg-config
invocations.

(I also found one that downloaded source with curl at build time,
without verification of any kind, and then built it.  *That* one we'll
want to patch, and ideally get the patch upstream.)

- Josh Triplett