[Pkg-rust-maintainers] debcargo and dh-cargo ready for production use; sponsor request

[Ximin Luo]

Josh Triplett:
> [..]
> 
> You need libssl1.0-dev installed instead.  The latest stable version of
> Cargo on crates.io remains 0.14, and the dependency tree there requires
> libssl 1.0.  This problem will go away when cargo 0.15 hits crates.io.
> 

OK, thanks for that. I also had to install libgit2-dev libssh2-1-dev cmake (libgit2-sys needs cmake), and possibly other its dependencies were already on my system. I noticed

   Compiling cargo v0.14.0

and that when I run `debcargo package debcargo` I get:

 librust-cargo-0.14+default-dev,

Luca tells me this is the cargo library and not the cargo binary that we already have. But I guess we should drop the "+default" from the name?

> [..]
> 
>>> [..] debcargo now defaults
>>> to omitting the binary from a crate that has both, unless you pass
>>> --bin. [..] You can
>>> also rename the binary package using --bin-name, to avoid conflicts with
>>> existing packages.  debcargo will still generate a binary package by
>>> default for crates that don't also build a library, and for some library
>>> packages we'll want to pass --bin, such as for future packaging of cargo
>>> (which includes both a library and the "cargo" binary).
>>>
>>
>> This is probably something that you should prompt the developer for, otherwise they might not notice. That is, if the crate has a binary package then force the developer to supply a --bin-name. Or generate a default one but have them manually confirm it, like how dh_make works.
> 
> debcargo prints a warning message if it skips a binary for a library
> crate, and the message explicitly says to provide --bin if they want the
> binary packaged too.
> 

$ debcargo package cargo
Ignoring binaries from lib crate; pass --bin to package: cargo
$ debcargo package cargo --bin
File already exists: rust-cargo-0.14_0.14.0.orig.tar.gz
1

OK, this doesn't seem so bad. I was worried the warning would be buried within a sea of other messages. However, I then have to find and manually rm those files; you could instead add an --overwrite option.

>> +1 to extra heuristics. Also, as discussed on IRC, you should make it as close to DEP-5 as possible, ideally DEP-5. I would have more specific comments and concrete suggestions on how to do this, but I couldn't run it yet due to the openssl error above.
> 
> I followed a very similar format, and included all the information
> available from crate metadata and scans of the crate contents.  I
> also referenced the ftpmaster-posted guides for writing complete
> copyright files.
> 

After running it and taking a look:

By similar I meant looks-wise. The point of DEP-5 is to be machine-readable, and it's not clear how to do that for yours. If you don't want to spend time on this, I will see if I can do it. But I really do think we should be aiming for a DEP-5-like format.

I don't see an issue in editing the copyright notices to add that extra colon, that is already common for DEP-5 and has been passed through debian-legal@ (I assume).

>> [..]
>>
>> How does debcargo (or even cargo) declare dependencies on non-rust software? You mentioned on IRC that this is a big hole at the moment, but I wonder if we can begin to patch this from the Debian / debcargo side. Is there a "convention" currently in the rust ecosystem for doing this?
> 
> Not as machine-parseable metadata, though some discussions have taken
> place on how to fix that.  Right now, packages that link to non-Rust
> libraries typically 1) bundle the source code (often as a git
> submodule), and 2) have a build.rs script that checks pkg-config,
> falling back to the bundled code.
> 
> The most prominent exception to that appears in libgit2-sys, which only
> checks pkg-config if you set a libgit2-sys-specific environment
> variable.  (Their justification: because libgit2 has such an unstable
> API that the system version almost certainly won't work.)
> 
> Most of the calls to pkg-config tend to not include a version, either.
> 
> One possible heuristic: I could scan build.rs, look for a call to the
> pkg-config crate, and attempt to match that call to an installed
> package.  I could also emit a warning if the pkg-config call doesn't
> include a version.
> 

We started a discussion here:

https://github.com/onur/docs.rs/pull/73

>> [..]
>>
>> librust-foo.git$ git status
>> On branch debcargo-auto
>>
>> librust-foo.git$ ls -la debian/
>> [.. original debcargo-generated packaging ..]
>>
>> librust-foo.git$ git checkout master
>> librust-foo.git$ ls -la debian/
>> [.. debcargo generated package, but with extra manual edits ..]
>>
>> librust-foo.git$ debcargo update
>> [.. downloads latest version, generates packaging for it, commits it on top of the debcargo branch, tries to merge it back into master ..]
>> [.. sort of like how `gbp import-orig` works ..]
>>
>> MERGE CONFLICT, ruh-roh!!
>>
>> [.. hack hack hack ...]
>> librust-foo.git$ git add -- [etc etc]
>> librust-foo.git$ git commit -a
>> # great success
> 
> I'd thought about the "debcargo update" workflow.  I hadn't thought of
> using a separate branch and merging; I like that idea.
> 
> That said, I don't want to give up on fully automated packaging quite
> yet.  If at all possible, I'd prefer to find a way to put the necessary
> metadata into the upstream crate, ideally in a reliably machine-readable
> fashion, but at a minimum in a heuristically matchable fashion.
> 
> If we can't successfully manage that, then as another alternative that I
> think I'd prefer, I'd like to make debcargo parse one or more input
> configuration files that specify tweaks for various crate packages.
> Temporary description overrides, additional dependencies, --bin and
> --bin-name values, dependency version overrides, etc.  That way, in the
> spirit of "don't check generated files into version control", the entire
> auto-generated Debian packaging can stay *out* of version control, and
> we can just check in the configuration files that control the
> automation.  We could then maintain a single repository with the
> half-dozen configuration files needed across a hundred crates, rev them
> as needed, and run some kind of "debcargo --recursive" to generate all
> the new or updated source packages.  (Those configuration files would
> then shrink or disappear as we find ways to upstream those tweaks.)
> 

I can see where you're going with this. It's a good idea, but I don't think this will work well for all packages. We should be *prepared* for something like the git workflow I described, and work in-tree directly on the generated package.

There's lots of things one might want to do for potential rust software, that I don't think would fit well into your single-repo approach. For example:

- multi-language software that contains cargo crates and other things
- random other things that cargo doesn't directly handle
  - system services
  - system config files
  - ad-hoc documentation like manpages

To account for all of these things in your "single-repo" idea, you would have to effectively re-implement a lot of Debian packaging infrastructure inside debcargo. I don't think that's what you intended nor what we'd want.

It also increases the build-test-debug-fix loop time - you'd need to go through debcargo every single time, without being able to manually run specific parts of the build process. This will get very very annoying for larger, more complex packages that need heavier patching.

We can certainly try to put *most* packages into a form that would fit your single-config-file approach. It is rather annoying how the current Debian way involves many small files, much of the time. A naive solution could just be to cat all the files in debian/* into a single file, with an appropriate separator between the contents - and this would be completely independent of debcargo or other languages.

But I don't think it's a feasible goal to try this for *every* package. A single config file works better for small packages, but the Debian way works better for larger, more complex packages.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git