[Piuparts-devel] Bug#698526: Bug#698526: Sort known issues by reverse dependency count

[Dave Steele]

The rest of my proposed changes for known problem handling are pushed,
for review.
A rebase is needed before merging. I will do this at your request.


The following serial branch heads are involved:

well-known - I've added tolerance for missing files and packages, and
added PTS links

fast-problems - replaced grep shell calls with python re. Per the commit:

    Run with full .kpr replacement is 2 1/2 minutes vs. 28 minutes for
    grep, per section, with stale file buffers, and idle slaves.
    Subsequent runs are 15 seconds vs. 60 seconds. Replacing the
    packagesdb rdep sort with an alpha sort reduces that to 5 seconds.

fast-report - detect_well_known_errors is morphed into the piupartslib module
    'known_problems', and is and called from piuparts-report. Report
runs always include
    issues and error summaries now.

report_problem_integration - replace linktarget_by_template with known_problem
    module support. All problem definition information is encoded in
the conf file.

piatti-problems - known_problems uses the packaged dir for the problem files. A
    new known-problem-directory config parameter lets piatti set it
back to under /org.

Commits:

piatti-problems
    1b61655 piuparts-report - Add known-problem-directory config for piatti.
report_problem_integration
    a8360ec piuparts-report - Add a special Problem case for unknown failures.
    dc39b89 piuparts-report - replace linktarget_by_template with Problem class.
    4db2254 piuparts-report - add known Problems class list to Section.
    108bbfd Add piuparts-report linktarget_by_template information to
known_problems.
fast-report
    bdc0939 Mv detect_well_known_errors to piupartslib - call from
piuparts-report.
fast-problems
    4394b8f detect_well_known_errors - Changelog entry for re speedup.
    c398289 Remove COMMAND parameter from known_problems.
    4e4e011 detect_well_known_errors - Generate 'grep' help command
from INCLUDE.
    87696e9 detect_well_known_errors - Use python re for fast kpr generation.
    2991e15 known_problems - Add INCLUDE parameters for re-based searching.
well-known
    955a6a2 Close the 698526 python detect_well_known_errors wishlist bug.
    5b61a03 detect_well_known_errors - Add PTS link to issue/error entries.
    fe4e400 detect_well_known_errors - handle having the pkgsdb entry disappear.
    895e035 detect_well_known_errors - Tolerate missing .kpr files.
    967e27d detect_well_known_errors - Tolerate deleted log files.
    427aa41 detect_well_known_errors - restore recheck and
recheck-failed options.
    a4553bc Bump the required python version to 2.7.
    a12f676 detect_well_known_errors - display the reverse dependency count.
    500e97f detect_well_known_errors - sort known errors/issues by rdep count.
    7de4eb9 detect_well_known_errors - integrate the package templates.
    d066bb3 detect_well_known_errors - Template HTML format fix.
    76b8ce2 detect_well_known_errors - Copyright notice.
    b8af3e4 detect_well_known_errors.py - move to detect_well_known_errors.
    25b9351 Remove bash detect_well_known_errors.
    ece5e4e detect_well_known_errors.py - change ext's to create kpr
and tpl files.
    39837e9 detect_well_known_errors.py - print failures to match bash script
    198c65e detect_well_known_errors - Create html tpl files
    4049338 detect_well_known_errors.py - Create Failure Mgr class to
hold kpr fails.
    d04c1bb detect_well_known_errors.py - Create missing kpr files
    1880598 detect_well_known_errors.py - add class for handling known problems
    9b25943 detect_well_known_errors.py - establish the problem file location.
    53df049 detect_well_known_errors.py - clean obsolete kpr and bug files
    cdd8803 Add skeleton for python replacement of detect-well-known-errors
    601e6a7 start with 0.50
    df94975 release as 0.49

In addition, there is a fixup branch that contains changes that need
to be rebased into
well-known.

fixup-well
    8fc8df2 fixup - fix method arguments for recheck* parameters
    b897262 fixup - fix filtered()
    a1b381d fixup - delete the comment that .kprn is temporary


Andreas, per your wishlist:

On Sun, Jan 20, 2013 at 7:56 AM, Dave Steele <dsteele at gmail.com> wrote:
> On Sun, Jan 20, 2013 at 6:56 AM, Andreas Beckmann <debian at abeckmann.de> wrote:
> ...
>>
>> What I'd like to see is (in probable order of implementation)
>> * piuparts-report "discovering" all existing known problem descriptions
>> instead of hardcoding them

Done, by pulling in the detect_well_known_errors code as a module, and
using it's Problem
class.

>>   - need to add ordering information somehow, perhaps by adding a
>>     number prefix:  42_foo_not_found_issue.conf
>>     or by adding a variable with a sort key inside
>>     (there should be a bug or some todo entries about this)

Done, using a PRIORITY key in the problem files, seeded with the order
of linktarget_by_template in piuparts-report.

>>   - needs to move title information from piuparts-report to .conf

Done, using a new "EXPLAIN" field in the problem definitions.

>> * piuparts-report generating the known problem reports, allowing access
>>   to packagedb etc. for "better" reports, making .tpl files obsolete

Done. The .tpl files are still being created (by piuparts-report now), just
before they are consumed and deleted. piuparts-reports also is taking over
.kpr management, since that is now a fast operation. This made for a much
simpler patch, and html issue/error output that is always present, and
up-to-date.

>> * getting rid of error/issue redundancies

No changes.

>> * computing the .kpr with python re instead of grep

Done. 10x faster in the worst case, and 5-10 seconds per section for typical
runs, excluding packagesdb time.

>> * adjusting the .conf and .kpr formats to what is actually needed
>>

Done.


If you want to try it out, check out a later commit, and copy over
piuparts-report, piupartslib/known_problems.py, and the known_problem
configuration files to the installed locations. Then run
piuparts-report.