[Piuparts-devel] Pending mass bug filing for broken symlinks detected by piuparts

[Dave Steele]

On Sun, Jun 2, 2013 at 9:54 AM, Andreas Beckmann <anbe at debian.org> wrote:
> So let's dig into the lists (only the .csv you sent is usable),

So you retried the spreadsheet after I changed permissions, with no
success? It sure would be great if you could resolve some of these
issues directly with less discussion needed.

> ...let me
> start with the serious ones and pick a random one ...
>
> The d-d mail should clearly detail the different categories we sort the
> broken links into and a list of common causes that could have lead to
> this link ...
>

Ok, looking into separate defined categories, based on a more detailed
automated analysis of this class of failures, makes sense.

> (from analyzing libuim-dev, libsquizz-dev)
>
>   Tag: broken-symlink-libdevel-so
>   Severity: serious
>   Broken link: /usr/lib/libFOOBAR.so -> libFOOBAR.so.42
>   Problem: libFOOBAR-dev package with libFOOBAR.so link misses
>            Depends:libFOOBAR42
>   Reason: forgotten dependency
>   Reason: multiple libraries built from src:FOO, maybe split into
>           separate packages after some time, but the (single) -dev
>           package was not adjusted to depend on all packages

So what you've really described here is the case of
broken-symlink-libdevel-so-forgotten-dependency, to be joined by
broken-symlink-libdevel-so-expects-recommends and many other brethren,
right?

If I can change this 'Reason' in the category definition to say
something like "forgotten dependency, missing Recommends, or link to
reverse dependency", pointing to a category broad enough to cover all,
we may be fine.

Otherwise, we may have a flat out disagreement here. The bug filing
has a responsibility to verify that the problem exists, that it is
assigned to the proper package, and that the severity is correct. For
these shared library issues, that can be satisfied by verifying that
the package created the symlink (I believe this is done), that it
sources from global shared library directory (need to verify link vs.
target here), and that the symlink is broken (piuparts does a fine job
at that). The bug filing does not need to have a requirement to go
into further analysis of the mechanism of the breakage, or to point to
the particular related package involved with the problem (if there is
one), especially if that requires manual research on EACH OF 86
PACKAGES. If it can be automated without too much trouble, that is
another question, that I'll look into a bit.

> And for filing a serious bug, a manual inspection of the package should
> be made, and as a result we should be able to mention the missing
> dependency in this case.

Disagree. I think we can trust that piuparts identified the right
package, for cases in sid-fail-broken-symlinks where the broken
symlink appears only once, and also that it is not a requirement to
identify the missing dependency/reverse dependency/recommends in the
bug filing.

I have gone beyond this, by inspection at the spreadsheet level that
the symlink ownership by the package 'makes sense'. You've pointed out
below that a toolchain could be responsible. Spreadsheet-level
inspection is a reasonable filter for catching these.

Yes, it is possible for there to be cases in THE 86 PACKAGES that the
expertise of the maintainer will be involved in targeting the
ultimately right package. I do not believe that is inappropriate.

Perhaps we need a mechanism for arbitrating this? Or a plan that
recognizes that I am not up for doing this for 86 PACKAGES. Otherwise
we could be looking at another brick wall.

As a separate issue, this raises additional questions on how you wish
to list the maintainers section. That remains open.

> That seems to cover most of the broken links in /usr/lib, and I fully
> agree to file them as serious.
> I would be a bit more careful with broken links in "package specific"
> directories like /usr/lib/PACKAGE/, but if a missing dependency is
> obvious, that should be serious as well.
>

Ok. I'll look harder at that.

>
> Looking at xmanpages-ja from your list, that is also serious:
>
>  /usr/share/man/ja/man3/XLockDisplay.3.gz ->  XInitThreads.3x
>
> the link target is missing the .gz extension (and policy says somewhere
> that links to compressed files must have the same extension)
>

Ok, will research. That would be the second category, in my book.

> rebuilding in sid fixes this issue

Guaranteed? Only if dh_installman is doing the work, right? Is this
because dh_installman changed behavior recently? Otherwise, more
explanation please.

> Are there more packages of this type?

Meaning this compressed file issue? I'll look into it.

> There I would suggest to do "sourceful-rebuild-NMUs" (similarly what was
> done to switch large gnome/kde packages to xz to get more onto CD1)

The maintainer should get 1st shot at fixing the bug, IMO. The report
can contain recommendations associated with the category, if
appropriate.

> lots of kde packages have a broken symlink to ../../common
> I would expect that is an error in the toolchain used, please
> investigate and file a bug against the toolchain (check experimental
> first, might be fixed there)

Ok, that would go in Anomalies for the time being. (and can be
deferred, should non-serious discussion be deferred)

>
> there are lots of broken links to jquery.js and friends - please
> investigate, this may be a toolchain bug, too (or a "missed" transition)
> (this should be analyzed in stable, too)

I did look at this, for at least a sampling of cases. jquery was a
recommended package.

> libpythonX.Y-dbg has many broken links in /usr/include, please
> investigate, probably a missing Depends on a corresponding -dev package?
> maybe a reason this is missing?
>

Ok

>
> anything involving /etc/alternatives is a false positive and a separate
> bug should be filed already
>

Meaning it should be removed from consideration here? Ok

>
> src:mpich causes several broken links in /usr/bin
>
> =======
>
> From looking at the list of broken links and digging into some of the
> packages I conclude:
>
> * serious bugs should be files for broken links in /usr/bin, no d-d
> discussion needed (4 source packages)

Ok, will prepare.

> * a bug should be filed for xmanpages-ja, no d-d discussion needed,
> intent-to-NMU

Will prepare bug(s). I don't intend to NMU (can't actually - you want
this?). xmanpages-ja will still come up again at the 'normal'
conversation.

>
> * we should have a specific d-d discussion about filing broken .so
> symlinks in -dev packages as serious - how many would that be?

Glad you asked :-). 82 BINARY PACKAGES, 66 source at last count.

Sorry, you said -dev packages. I can't right now. Try grepping the csv
on serious and -dev.

> Rationale: a -dev pkg that cannot be used to link against its library is
> useless

or more simply a violation of "Packages containing shared libraries
must be constructed with a little care to make sure that the shared
library is always available". A violation is a violation. Serious is
serious. More nuance is not required.

> For the bug template:
> Subject: $PKG: Broken .so symlink due to missing Depends: $DEP1,$DEP2,..
>

Per previous objection, I see this as

Subject: $PKG: Broken .so symlink

> * we should not have a general broken-symlink discussion on d-d right now

Sounds right to me, but Holger has said otherwise. Holger, do you
agree with this?

> * the other broken symlinks need further investigation
>   (if they are toolchain bugs, filing against the affected packages
>    would be very wrong)

If discussion is deferred, then they can be out of scope for the time
being. The list will probably need to be regenerated at the time of
consideration.

> If we want to escalate broken symlinks in piuparts, we should limit this
> to broken /usr/lib{,/<triplet>}/*.so for now.

The current ruling as I understand it is to defer escalation until
after the discussion. I'll resurrect the issue.

> But maybe the adequate check for broken symlinks is more adequate for
> this is this is easily limited to the broken package, while the piuparts
> check is "global" (which is a good thing to find broken non-shipped but
> buggy-maintscript-created stuff).
> And the adequate output is getting parsed already :-)

I haven't been playing close attention to the adequate work. If there
are adequate considerations, I'll need more detail.


The Project page has been updated with tasks and issues.