[Pkg-rust-maintainers] Bug#945542: debcargo -- Randomly adds and removes binary packages

[Daniel Kahn Gillmor]

Hi all--

I've been trying to make sense of the current struggles around rust
packaging in Debian, and this bug report (#945542) seems to be where
they're crystallzing.  (it's not clear to me how this bug report is
supposed to be distinct from #942898, which already captures some of
these details.  if the reporter or maintainer wants to merge them, i
think that'd be reasonable)

Full disclosure: i *want* rust software to work in a well-established
way in Debian. i am involved in helping with some of the rust packaging.
and at the same time, i am also frustrated at the immaturity and flux
that i see in some parts of the Rust ecosystem.  And i also find myself
frustrated with some of the frameworks involved with NEW queue
processing and unstable-to-testing migration, which i think are
occasionally counter-productive to debian's goals as a whole, and are
wasteful of the limited time that all of us (on the Rust team and the
FTP team) have to spend on Debian.

So i acknowledge that there is legitimate friction here, and i hope we
can work through it in a way that makes both Debian and the Rust ecosystem
healthier in the long run.

Identifying problems
--------------------

The claims that have been listed here (but actually seem distinct to me)
are:

 a) A small number of Rust packages provide very long Provides lines,
    which break some other infrastructure (e.g. reprepro)

 b) Many Rust source packages produce a handful of additional binary
    packages that are empty but have different dependencies, as a way of
    capturing crate "features" without introducing a large mass of
    interlocked circular dependencies.

 c) The subject of this bug report is "Randomly adds and removes binary
    packages".  I but haven't been able to find evidence of randomness
    here -- the binary packages that do trigger additional passes
    through NEW are not random -- they represent evolution of the
    features offered by upstream software.  Surely this is similar to
    evolution of other upstream software that is packaged for Debian
    (though perhaps the scale is different).

 d) There is an assertion that binary packages are specifically
    problematic for the FTP team.  For example:
    
On Sat 2019-11-30 12:15:55 +0100, Bastian Blank wrote:
> This problem is already listed since a long time in
> https://ftp-master.debian.org/REJECT-FAQ.html.

    But reading this page, the closest thing i can find is its
    description of "Package Split", which reads:

>>> You split a package too much or in a broken way. Well, broken or too
>>> much is a wide definition, so this is a case-by-case thing, but you
>>> should really think about a split before you do it. For example it
>>> doesn't make any sense to split a 50k arch:all package from a 250k
>>> arch:any one. Or splitting a package for only one file, depending on
>>> the main package. Yes, big dependency chains can be a reason. Or big
>>> documentation split into one -doc package. The point there is big.

    Bastian, if you meant something else, please correct me here!


Evaluating solutions
--------------------

(a) appears to be quite real, as lengthy Provides: lines have caused
failures in other pieces of software.  However, I'm a bit surprised that
i didn't easily find a bug report that either references or affects
reprepro, one of the packages known to have line-length limitations.
Someone who has encountered this bug in the wild and has documented it
should make sure that it is adequately represented on its own as a
specific bug report (or more than one), so we can peel this particular
situation off from the other issues.

(b) appears to be important to make it possible to avoid significant
bootstrapping difficulties with rust packages in the debian
ecosystem. Ximin has already explained some of the issues related to
re-bootstrapping a bunch of indpendent packages that have circular
dependencies so i won't go into it in more detail, but the goal here is
to facilitate package maintenance and require less manual intervention.
These are good goals.

There are potential performance concerns about what these extra binary
packages do to consumers of the archive -- both for mirrors and for
typical debian installations.  I asked for some clear profiling of these
performance issues over on #942898, but no one concerned about the
performance loss has stepped up to provide measurements.  I believe
these risks are real, but i also believe that there are real advantages
(to both Rust and Debian) to making sure that packaging rust software
for debian is easy and straightforward.

I'm not convincd that (c) represents a real problem, or at least it's
not one unique to the Rust ecosystem.  While shifts in upstream designs
may be annoying/frustrating to the debian maintainers of this software
and to the FTP team, the changes in stated feature sets by each version
of a rust crate upstream is no different than a C library package that
pushes changes to the SONAME more rapidly.  As packagers, we should be
applying pressure on our upstreams to adhere to some reasonable measures
of stability.  we can help to explain to them why the costs of
instability and complexity are probably not in their project's (or their
users) best interests in many cases.  For example, I do that already in
rust projects like https://gitlab.com/sequoia-pgp/sequoia/issues/356.
But, there will inevitably be some legitimate instability or complexity.
this is software.  and as packagers and distributors, some of our work
is to absorb and systematize that instability and flexibility.  At least
Rust is trying to explicitly grapple with these variations via feature
flags and semantic versioning.  Some other language ecosystems don't
even have an understanding of these issues or a way to talk about them,
and they just get swept under the rug.  We shouldn't be penalizing a
language that tries to grapple with them explicitly.

As for (d), reasonable people might disagree about whether the quoted
text describes the Rust crate packaging produced by debcargo.  I don't
think it describes the situation (except perhaps for the distinct
concern raised in (a)).  In particular, these package splits are not
made for any unfounded arbitrary reason.  they have clearly articulated
goals, and they are situated in the broader context of debian package
management.  The main concern is even explicitly called out as a
legitimate justification for package splits -- "big dependency chains
can be a reason".

> The problem is not about severities, but about the Rust team to
> acknowledge that there is a problem.

There's clearly a problem, because Rust-based software isn't getting
into Debian at the rate that it probably should.  But it doesn't seem to
me that the problem is on the Rust team alone to solve, or solely the
fault of the current debcargo packaging infrastructure.  And it's
certainly not a problem that would warrant removal of decargo from a
future debian release.

We have varied, competing interests in Debian, and it works best when we
all understand where each other is coming from, and we work together to
find a balancing point, along with exerting pressure on our upstreams to
provide sensible, maintainable software.

I'd love to see some concrete proposals put forward to move this out of
the current stalemate.  Perhaps evidence of the claimed performance
problems would help that work progress?

Regards,

        --dkg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/pkg-rust-maintainers/attachments/20191130/69804188/attachment-0002.sig>