[Piuparts-devel] Piuparts state-dependency-failed-testing analysis
Scott Schaefer
saschaefer at neurodiverse.org
Sun Oct 30 15:23:25 UTC 2011
On 10/29/2011 08:05 PM, Dave Steele wrote:
> In the past week, the number of state-dependency-failed-testing has
> increased by 510, to 3409. Of those, about 80% are blocked by the
> first package in the list, and 75% are blocked only by that package.
>
> On Sat, Oct 22, 2011 at 11:03 PM, Dave Steele <dsteele at gmail.com
> <mailto:dsteele at gmail.com>> wrote:
>
>
> I find myself frequently called back to the Piuparts
> waiting-to-be-tested package list at
> http://piuparts.debian.org/sid/state-dependency-failed-testing.html,
> and got curious about what is keeping that list as long as it is.
> Following is an analysis.
>
> There are currently close to 3000 packages
> in state-dependency-failed-testing that can be traced to one or
> more state-failed-testing packages. That number actually went down
> quite a bit in the last week, when libgtk2.0-bin passed, clearing
> hundreds of packages for testing.
>
> There are less than 300 packages in state-failed-testing holding
> up those tests. The blocking distribution is typically very
> skewed. At any given time, half of the packages
> in state-dependency-failed-testing may be held there, directly or
> indirectly, by 3 to 5 packages. Eliminating libgtk2.0-bin last
> week exposed libreadline6 as the new top blocker.
>
> Here's the output of a script that scrapes
> http://piuparts.debian.org/sid/state-dependency-failed-testing.html and
> analyzes blocking packages:
>
> # ./piublocker
> dependency failed - 2899
> failed testing - 274
>
> blocking free cum package
> 1588 1005 1894 libreadline6
> 451 35 1854 sgml-data
> 429 1 1466 docbook-xsl
> 191 153 1286 texlive-base
> 155 79 1182 ca-certificates-java
> 125 106 1071 php5-cli
> 89 29 1042 gnustep-base-common
> 60 0 982 gnustep-back0.20
> 49 6 973 menu
> 49 4 957 libcommons-httpclient-java
> 42 0 915 python-ibus
> 40 15 897 libcommons-beanutils-java
> 35 34 862 gosa
> 35 1 827 blends-common
> 33 33 794 liquidsoap
> 30 3 791 libspring-core-java
> 28 20 771 libwebkit1.1-cil
> 28 1 750 libatinject-jsr330-api-java
> 26 26 724 drupal6
> 26 7 703 antlr
> 24 24 679 cyrus-common
> 21 13 664 libmx-1.0-2
> 20 15 645 openssh-client
> 17 0 630 libhttpclient-java
> 17 17 613 drizzle
> 16 16 597 dovecot-core
> 14 0 597 sat4j
> 14 6 591 proftpd-basic
> 14 0 577 libjasper-java
> 14 11 566 libhttp-date-perl
> 14 7 559 libgcj-bc
> 13 4 546 libmono-addins-gui0.2-cil
> 12 10 535 postfix
> 12 3 523 libusrp0
> 12 9 512 libgtkhtml-4.0-common
> 11 6 501 libncurses5-dev
> 11 6 490 libgtkglext1
> ...
>
> This output says that there are currently 2899 packages
> in state-dependency-failed-testing traceable to a
> state-failed-testing package (that doesn't exactly match Piupart's
> count of 2920). 274 packages are responsible for that blocking.
> More than half of them (1588) are blocked by a single package,
> libreadline6. 1005 of those packages would be cleared for testing
> by removing only libreadline6 from the list of blockers. Possibly,
> at least some of those exposed packages may have blocking numbers
> in the 1000 range (e.g. 'python' is in that list).
>
> The "cum" shows how many packages are left as each subsequent top
> blocker is cleared. From that number, you can see that the
> dependency list could be cut in half by removing the top 4
> packages on the list. 80% would be cleared by passing the top 29
> packages.
>
> It seems worth the effort to identify and focus fixing the top
> packages on this list, over time. There should be opportunities
> for low-hanging fruit whenever the state-dependency-failed-testing
> list gets over about 1000-1500 entries.
>
> The source for the script piublocker is at
> https://github.com/davesteele/piublocker
>
>
>
> _______________________________________________
> Piuparts-devel mailing list
> Piuparts-devel at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/piuparts-devel
Wow. ** IF ** I am analyzing this correctly, there is really nothing
wrong with dependencies of libreadline6; this is in fact a "bug" in
piuparts (or more appropriately, a "bug" in the operational policies of
how piuparts is run) ...
Looking at log of piuparts' testing of libreadline6
(http://piuparts.debian.org/sid/fail/libreadline6_6.2-7.log) the
proximate cause of the error is:
1) dpkg --purge --pending fails to purge package libtinfo5, due to
(installed) libreadline6 having Depends --> libtinfo5. Just prior, each
of the following also failed:
dpkg --purge libtinfo5
dpkg --remove --pending
dpkg --remove libtinfo5
2) Normally, the package being tested would have also been target of
dpkg --remove and dpkg --purge. That is, piuparts would normally have
executed:
dpkg --remove libreadline6 (prior to dpkg --remove --pending)
dpkg --purge libreadline6 (prior to dpkg --purge --pending)
Which would have allowed dpkg --remove/--purge --pending to succeed and
libreadline6 to have "passed" (and 1000+ additional packaegs to be
tested !!)
3) piuparts did not execute dpkg --remove/--purge libreadline6 because
it was already installed in the original chroot.tar.gz image file [per
original dpkg --get-selections]. However, the version in the
chroot.tar.gz was 6.2.2, which had no Depends -> libtinfo5. Thus,
libtinfo5 was not in the chroot.tar.gz image -- it was thus installed,
and an attempt was made to remove/purge it as part of testing of the
libreadline6 package.
Again, assuming I have analyzed this correctly, the simplest 'solution'
is to remove the chroot image and force piuparts to generate a new one,
which will have 6.2.7 version of libreadline6, and will thus include
libtinfo5 in the .tar.gz image.
In summary, this is really a consequence of the operational tradeoff of
the frequency of regenerating the chroot.tar.gz image. I think
piuparts.d.o has this set to 30 days; perhaps this should be changed to
7 or even 3-4 ??
Long-term, I can see 2-3 alternatives for how to deal with this "bug".
However, I would like someone (Holger ??) to review above analysis (and
remove .tar.gz image on piuparts.d.o to test 'solution') before proceeding.
More information about the Piuparts-devel
mailing list