[Pkg-rust-maintainers] Bug#942898: debcargo: reduce the impact of debcargo-built crates on the package index, and facilitate debian packaging of crates

[Daniel Kahn Gillmor]

Package: debcargo
Version: 2.4.0-1
Control: affects -1 ftp.debian.org

Hi!  I'm trying to summarize in this report the state of conversation i
had today between members of the FTP team (and others on #debian-ftp)
and members the debian Rust packaging team.

We seem to be in a bit of an impasse, and i'm hoping that we can use
this report to find a concrete way out of the impasse.  This report
explains my understanding of the problem and proposes one way forward.
I'd be very happy for constructive engagement with this report.  If you
think i'm wrong, help explain what would be better, or why my proposed
changes aren't acceptable.


Background
----------

debcargo creates .deb packages corresponding to Rust crates.  I'm going
to focus here on the librust-* .deb binary packages, and ignore the
packages that ship tools that have, for example, a command-line
interface.

Each source package corresponds to a Rust crate, but it might produce
multiple librust-* binary packages -- one for each crate "flavor".

the librust-*.deb files contain mainly a distribution of .rs source
code in /usr/share/cargo/registry/…

Most of the "+flavor" .debs basically have an additional source
dependency on some other rust crate, and also ship a symlink back to the
folder of .rs source code from the standard (non "+flavor") .deb.

For each binary package shipped, it also defines a list of Provides:
that include version numbers in the name.



Problem Statement(s)
--------------------

From talking with the FTP team, the concern appears to be that the large
number of packages, and the large number of Provides produced for some
of the packages is seen as a potential problem for the apt Packages
index.

In particular, a larger Packages index:

 * increases the cost of data transfer for every debian system that does
   updates

 * means that apt has to do more complicated dependency resolution

I have not yet gotten measurements of what kinds of costs we're talking
about with respect to this shared resource.  If someone could provide
some numbers and a methodology for getting them, that would be useful in
figuring out whether any proposed change contributes a substantial
solution to the problem.  Kinds of measurements that might make the
risks a bit clearer for leaning on this shared resource too heavily:

 - size of Packages file
 - RAM needed by aptitude to ingest the Packages file
 - CPU time taken by aptitude to do dependency resolution



Additionally, from the Rust team's perspective, when a crate upstream
proposes a feature, this creates a new binary .deb, which triggers
inclusion in the NEW queue.  The delay in the NEW queue causes friction
which makes packaging Rust-related projects harder to do.

I've experienced this myself, where i spend about an hour or two on rust
packaging, and then find myself waiting for weeks before i can do the
next hour of work -- this isn't conducive to effective maintenance.



Proposed Solutions
------------------

AIUI, the FTP team thinks that debcargo could reduce the impact on the
shared resource of the Packages index by adopting one or both of the
tactics described below:

 0) reduce the number of "+feature" .debs produced by each crate,
    perhaps by creating two base .debs for each package: one with no
    "features" and one that bundles together all of the features that
    are not mutually-exclusive.  Any features that are mutually
    exclusive would still get their own separate "+feature" .deb.

 1) drop version numbering from the Provides: entries for standard
    packages -- this should reduce the number of Provides: by a
    substantial fraction.  Given that crates are expected to hew to
    semantic versioning, a generated version number range should be
    sufficient to declare an API-compatible version dependency.



Concerns
--------

A concern I've heard about tactic (0) is that this approach might result
in dependency loops, as in:

    https://github.com/sfackler/cargo-tree/issues/34#issuecomment-394266843

It's conceivable that apt could figure out how to deal with those loops,
and given that these librust-*-dev .debs are basically just sourcecode,
these loops will only affect Rust developers (who might be expected to
have beefier machines than regular end users).

Such a change would also reduce the number of packages


A concern i've heard about tactic (1) is that it makes it harder to ship
a two versions of a crate in a given debian distro.  For example, we
might want to ship librust-foo-dev version 1.x alongside librust-foo-dev
version 2.x.  If they have the same package name (librust-foo-dev) then
they can't both be in the archive at once.

We currently do ship librust-foo-dev (meaning "the latest version"), and
we create a new package librust-foo-1-dev for when we intend to maintain
a stable API for an older release.  I don't think anyone wants that to
change (many C library packages use the same pattern based on their
SONAMEs).

The use of Provides: makes it so that a package can directly Depend: on
the package with the API in the name, rather than depending on a
versioned range.  But perhaps the fix is to make the dependencies
marginally more complex, rather than stuffing lots of Provides in each
binary package.  For example, a package that depends on the foo crate at
1.2 or higher could use:

   Depend: librust-foo-dev (>= 1.2) (<< 2) || librust-foo-1-dev (>= 1.2)


Do the above approaches seem feasible?  Would they satisfy both the FTP
team and the Rust packaging team?  Are they implementable?

Sorry that i don't have the capacity to actually implement them myself,
i just want to identify a way forward that will make both Debian and the
Rust ecosystem more compatible.

Regards,

        --dkg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/pkg-rust-maintainers/attachments/20191022/8ae2ca04/attachment.sig>