[Pkg-rust-maintainers] Packaging software written in Rust

[Josh Triplett]

On Wed, Jul 20, 2016 at 06:12:24AM +0000, Angus Lees wrote:
> (Agreed on all counts)
> 
> On Wed, 20 Jul 2016 at 13:08 Josh Triplett <josh at joshtriplett.org> wrote:
> > On Tue, Jul 19, 2016 at 11:32:17PM +0000, Angus Lees wrote:
> > > On Mon, 18 Jul 2016 at 15:31 Josh Triplett <josh at joshtriplett.org>
> > wrote:
> > > > On Sun, Jul 17, 2016 at 08:57:40PM -0700, Josh Triplett wrote:
> > > > > I can understand wanting to support local distro patches. We should
> > be
> > > > very careful doing so, as it seems questionable to patch a crate while
> > > > keeping the same version number; that breaks some of Cargo's
> > assumptions
> > > > about reproducibility. I'd rather have a separate crate and a replace
> > > > stanza. However, if we absolutely had to, directory registries seem to
> > > > support that.
> > > >
> > >
> > > Remember the *primary* reason we are bothering to do any of this is to be
> > > able to support a hypothetical security fix to an existing Rust library
> > > package 5 years from now.  We *must* be able to rebuild with patches.
> >
> > I absolutely agree.  I'm just arguing that if we patch version 1.2.3 of
> > some crate, the patched version shouldn't use version number 1.2.3; it
> > should use, for instance, version 1.2.3+debian1 or similar, to make it
> > possible to identify what crate the package built with.  (Cargo and
> > the Semantic Versioning standard allow that.)  Anything that declares a
> > dependency on ^1.2, ^1.2.3, ~1.2.3, 1.2.*, or >1.2.3 will still allow
> > that.  (And we have to rebuild binary crates to pull in the new library
> > anyway, whether we change the version number or not.)
> >
> > Having a distinct version number when we patch a crate will make it
> > feasible to tell when a build actually picks up that patched crate, such
> > as in the cargo output in the build log for a bin crate.
> >
> 
> Oh, I hadn't thought about propagating the Debian source version into the
> library - interesting...  That means we'll appear "different" from cargo's
> pov even for unmodified crates (1.2.3-4 instead of 1.2.3), but I imagine if
> that triggers an issue with an over-strict dependency, then it was an issue
> we wanted to fix anyway...

I don't actually think we should propagate the Debian source version
into the library, *unless* we change it.  If we ship unmodified 1.2.3,
let's call it 1.2.3; if we have to patch the source, we can call it
1.2.3+debian1, 1.2.3+debian2, ...

> Debian has elsewhere solved the "what build statically linked what" issue
> with Built-Using headers in the binary package, fwiw.  We should make sure
> we properly add these, so the standard tools do the right thing.

Good call; we can make the build tools for bin crates generate substvars.

> > .. or we just patch the Debian version of librust-libgit2-sys-dev's
> > build.rs
> > > to always use the system version.
> >
> > While that's always an option, I'd like to avoid having to repeatedly
> > patch each new upstream version of a library if at all possible.  (If we
> > can get the patch upstream, great, but I'm not wildly hopeful there.)
> > That said, see below; I think this is a much less widespread problem
> > than I'd first thought.
> >
> 
> Applying patches and later re-patching onto a new upstream is what the
> Debian toolchain is reasonably good at, for what it's worth.  It's what
> just about every package does, after all.

I'd actually be interested to know what fraction of Debian packages have
to patch something outside of the debian/ directory; I'd be surprised if
it was the majority of packages.

> Obviously we can explore this further as we gain experience.

Agreed.

> > Realistically, we probably want to strip
> > > the vendored C sources out of most of the *-sys crates when packaging,
> > > since Build-Deps can guarantee the system library is present, and
> > actually
> > > removing the files means we can avoid auditing them for licenses, etc.
> >
> > > Many of the other *-sys crates I looked at earlier used pkg-config if
> > > available, so using the system library versions Just Worked.
> >
> > I took a quick look at a large selection of -sys crates, and as far as I
> > can tell, only two (libgit2-sys and one of the readline crates) shipped
> > their own copy of the upstream source as part of the .crate, and only
> > libgit2-sys required an environment variable to find and use a system
> > version by default.
> >
> > Given that, I think we could reasonably ask libgit2 to support a
> > standard environment variable (like CARGO_USE_SYSTEM_DEPS), set that one
> > variable unconditionally, and ask any new crates to support that.  I
> > think upstream would probably take such a patch.
> >
> > I also don't think, for trivial cases like those where the source is
> > unlikely to have license problems, that it's worth the trouble of
> > patching out the bundled source, compared to the benefit of using an
> > unmodified .orig.tar.gz from the .crate file.  But on the other hand, it
> > also wouldn't be that bad for just those two crates.
> >
> 
> I'm glad to hear that vendoring the upstream source is less common than I
> thought.
> Again, we probably want to choose to repack on a case-by-case basis - I
> imagine the upstream size and licensing complexity might vary wildly.

Absolutely.  For instance, if a library gets included and we know the
Debian version of that library had to repack upstream, we should
definitely remove that vendored source.

> Last time I checked there also weren't any signed checksums that we could
> use to verify crates.io downloads (other than the implicit HTTPS
> signature).  It would be nice if we could preserve a bit-for-bit identical
> upstream however.

While they sadly don't include signatures,
https://github.com/rust-lang/crates.io-index/ does include checksums,
which allows us to cross-check between crates.io and github.com.  That
said, I'd love to see the "bors" bot pushing signed commits to that
repository.

- Josh Triplett