[Pkg-rust-maintainers] debcargo and dh-cargo ready for production use; sponsor request

[Josh Triplett]

On Thu, Dec 08, 2016 at 11:55:00AM +0000, Ximin Luo wrote:
> Josh Triplett:
> > [..]
> > 
> > You need libssl1.0-dev installed instead.  The latest stable version of
> > Cargo on crates.io remains 0.14, and the dependency tree there requires
> > libssl 1.0.  This problem will go away when cargo 0.15 hits crates.io.
> > 
> 
> OK, thanks for that. I also had to install libgit2-dev libssh2-1-dev cmake (libgit2-sys needs cmake), and possibly other its dependencies were already on my system. I noticed
> 
>    Compiling cargo v0.14.0
> 
> and that when I run `debcargo package debcargo` I get:
> 
>  librust-cargo-0.14+default-dev,
> 
> Luca tells me this is the cargo library and not the cargo binary that we already have. But I guess we should drop the "+default" from the name?

No, this reflects a crate depending on the "default" feature of another
crate; librust-cargo-0.14-dev has a versioned Provides for
librust-cargo-0.14+default-dev.  All feature dependencies, even
"default", use the appropriate feature package, to avoid making
assumptions about whether the feature appears in the main library crate
package (which has a Provides) or in one of the separate feature
packages.

> > [..]
> > 
> >>> [..] debcargo now defaults
> >>> to omitting the binary from a crate that has both, unless you pass
> >>> --bin. [..] You can
> >>> also rename the binary package using --bin-name, to avoid conflicts with
> >>> existing packages.  debcargo will still generate a binary package by
> >>> default for crates that don't also build a library, and for some library
> >>> packages we'll want to pass --bin, such as for future packaging of cargo
> >>> (which includes both a library and the "cargo" binary).
> >>>
> >>
> >> This is probably something that you should prompt the developer for, otherwise they might not notice. That is, if the crate has a binary package then force the developer to supply a --bin-name. Or generate a default one but have them manually confirm it, like how dh_make works.
> > 
> > debcargo prints a warning message if it skips a binary for a library
> > crate, and the message explicitly says to provide --bin if they want the
> > binary packaged too.
> > 
> 
> $ debcargo package cargo
> Ignoring binaries from lib crate; pass --bin to package: cargo
> $ debcargo package cargo --bin
> File already exists: rust-cargo-0.14_0.14.0.orig.tar.gz
> 1
> 
> OK, this doesn't seem so bad. I was worried the warning would be buried within a sea of other messages. However, I then have to find and manually rm those files; you could instead add an --overwrite option.

For the .orig.tar.gz, I could just compare it to the .crate file and
only error out if it differs.

Overwriting the actual source directory seems sufficiently dangerous
(since you might have edited it) that I think it makes sense to require
the user to delete or move aside the source directory first.  I could
add a message saying to remove it if you want to regenerate it, though.

Or, alternatively, I could have an option to write the new source
directory with a .debcargo suffix or similar, which would make it easy
to diff and merge in.  And I do like the idea of generating it into a
dedicated git branch, too.

> >> +1 to extra heuristics. Also, as discussed on IRC, you should make it as close to DEP-5 as possible, ideally DEP-5. I would have more specific comments and concrete suggestions on how to do this, but I couldn't run it yet due to the openssl error above.
> > 
> > I followed a very similar format, and included all the information
> > available from crate metadata and scans of the crate contents.  I
> > also referenced the ftpmaster-posted guides for writing complete
> > copyright files.
> > 
> 
> After running it and taking a look:
> 
> By similar I meant looks-wise. The point of DEP-5 is to be machine-readable, and it's not clear how to do that for yours. If you don't want to spend time on this, I will see if I can do it. But I really do think we should be aiming for a DEP-5-like format.

As mentioned on IRC, we often don't have enough information to fill out
DEP-5, because many crates (intentionally) don't include copyright
notices, just license notices, and DEP-5 makes the "Copyright:" field
mandatory.  I don't think it makes sense to use an
almost-but-not-quite-DEP-5 format, because any tools expecting a
machine-readable DEP-5 would fail to parse that format; better to look
obviously not machine-readable.  If you'd like to see these copyright
files use DEP-5, I'd suggest proposing a revision to DEP-5 to make the
"Copyright:" field optional, for the case when no copyright notices
exist upstream, which would allow us to generate a DEP-5-formatted
copyright file for every crate.  In that case, I'd happily implement
DEP-5 generation.

> > If we can't successfully manage that, then as another alternative that I
> > think I'd prefer, I'd like to make debcargo parse one or more input
> > configuration files that specify tweaks for various crate packages.
> > Temporary description overrides, additional dependencies, --bin and
> > --bin-name values, dependency version overrides, etc.  That way, in the
> > spirit of "don't check generated files into version control", the entire
> > auto-generated Debian packaging can stay *out* of version control, and
> > we can just check in the configuration files that control the
> > automation.  We could then maintain a single repository with the
> > half-dozen configuration files needed across a hundred crates, rev them
> > as needed, and run some kind of "debcargo --recursive" to generate all
> > the new or updated source packages.  (Those configuration files would
> > then shrink or disappear as we find ways to upstream those tweaks.)
> > 
> 
> I can see where you're going with this. It's a good idea, but I don't think this will work well for all packages. We should be *prepared* for something like the git workflow I described, and work in-tree directly on the generated package.
> 
> There's lots of things one might want to do for potential rust software, that I don't think would fit well into your single-repo approach. For example:
> 
> - multi-language software that contains cargo crates and other things

Many of those won't be able to use the majority of debcargo, and may not
have uploaded their Rust bits to crates.io.  They also wouldn't work
with the current dh-cargo.  So, I think we'll have to look at the first
couple of these and figure out how to handle them.

In the case of Firefox, we'll almost certainly need to patch it
significantly anyway, because upstream will likely vendor all the crates
they depend on, while we'll want to build against the Debian-packaged
versions.

> - random other things that cargo doesn't directly handle
>   - system services
>   - system config files
>   - ad-hoc documentation like manpages
> 
> To account for all of these things in your "single-repo" idea, you would have to effectively re-implement a lot of Debian packaging infrastructure inside debcargo.

Not at all; for those cases, I'd just add a directory that provides a
debian/ overlay for a package.  Then you could provide debhelper
configuration files, maintainer scripts (such as for alternatives), or
anything else you want.

> We can certainly try to put *most* packages into a form that would fit your single-config-file approach. It is rather annoying how the current Debian way involves many small files, much of the time. A naive solution could just be to cat all the files in debian/* into a single file, with an appropriate separator between the contents - and this would be completely independent of debcargo or other languages.
> 
> But I don't think it's a feasible goal to try this for *every* package. A single config file works better for small packages, but the Debian way works better for larger, more complex packages.

I certainly don't want to force every package into that model.  I do
want to try that model first, and only go the other route when we find
something that doesn't fit.

- Josh Triplett