[Piuparts-devel] piuparts-report/known problem Integration Status

[Dave Steele]

In the interest of completeness, and closure, I've reworked and
rebased the controversial report_problem_integration and skip_kpr
branches. To review, they integrate the processing of the
newly-reworked Python detect_well_known_errors script into
piuparts-report, consolidating known problem logic into one source
file. It was submitted as part of the known problem work in January.

https://github.com/davesteele/piuparts/commits/report_problem_integration
https://github.com/davesteele/piuparts/commits/skip_kpr

Concerning the report_problem_integration branch:

The large piuparts-report linktarget_by_template structure is
eliminated, replaced with information spread through the known problem
conf files. The following TODO known_problem items are resolved:
  - use a number prefix for sorting
  - add title information
  - piuparts-report: "discover" the available known_problems, dont hardcode the
    list

With this branch, piuparts-report output will always include issue
summaries, at a typical cost of 5-10 seconds per section. Offsetting
that cost, the detect_well_known_errors script step is eliminated.
Overall report processing time is again reduced. (the PackagesDB
object is shared between piuparts-report and known-problem
calculation, eliminating duplicate processing for rrdep metrics)

report_problem_integration touches about 30 lines of code, not
counting linktarget_by_template.

-------------------------

On the negative side, .tpl files are still in the work flow, and the
new known_problems module maintains a redundant, independent list of
packages and states, based on master's log file contents and location.
It currently needs knowledge of the log directory tree to determine
and cache known problem results, and to retrieve bug info. These
branches don't consider, nor do they preclude, any work on these
issues.

--------------------------

This is the path forward that I would propose for dealing with the ...
development opportunities ... longer term:

- Create an abstract cache file class, tied to a PackagesDB/LogDB
instance. The class would include methods to delete obsolete cache
files, to access existing cache files, and to identify/create missing
caches.
- Create a bug access class as a subclass of the cache file class. Add
a method replacing get_bug_text() in known_problems, taking the
package name as an argument. Remove references to bug file names or
paths, outside of this class, from known_problems.
- Create a known problem handling class as a subclass of the cache
file class. It would encapsulate the known_problem Problem and
FailureManager classes, and provide an external interface to get a
list of problem types and problems, and a sorting function. The .kpr
cache files are created on-the-fly, as needed. Remove references to
kpr file names and paths from known_problems.
- Move the HTML template code to piupart-report, replacing the .tpl
interface. Use a PackagesDB instance in lieu of a log file list. The
known_problems module should be clear of references to Master's
directory tree at this point. That also means moving the
responsibility to create Master's pass, fail, etc. directories
elsewhere.
- Rename what's left of known_problems to something more apropos, like
package_info. What remains at that point are the cache, bug, problem,
and failure classes.

Going farther down the rabbit hole, piuparts-analyze, and probably
other scripts, would be looking for the same treatment - in this case
by replacing file tree information with a streamlined PackagesDB and
an expanded version of the bug class. Once the bug class is handling
data creation, piuparts-analyze and a resurrected
detect_well_known_errors could morph into a simple, optional,
update-caches script (to offload piuparts-report).

It would not be my preference to use the skip_kpr branch as a part of
this migration, though it would be needed to totally remove problem
processing from piuparts-report.

I don't expect to pursue this path to completion at this point.
Hopefully, it is useful as a reference for whatever final solution is
implemented, perhaps hastening the day that known problem processing
is integrated in piuparts-report.