[Piuparts-devel] piupartsreport timings (was: Re: Bug#698526: Sort known issues by reverse dependency count)

[Dave Steele]

Sorry, that was for the list.

On Thu, Feb 21, 2013 at 7:38 PM, Dave Steele <dsteele at gmail.com> wrote:
> On Thu, Feb 21, 2013 at 1:05 PM, Andreas Beckmann <anbe at debian.org> wrote:
>> On 2013-02-21 15:55, Dave Steele wrote:
>>>  There's no significant
>>> performance issue.
>>
>> Timings ...
>
> Clarification - I was speaking specifically about the .tpl interface here.
>
>>
>> OK, lets take a look at piatti, too:
>> http://lists.alioth.debian.org/pipermail/piuparts-reports/Week-of-Mon-20130218/002970.html
>>
>> 00:00 start
>> report_newly_bugged_packages
>> Thu Feb 21 00:22:13 UTC 2013
>> detect_well_known_errors
>> Thu Feb 21 00:34:15 UTC 2013
>> piuparts-report
>> 00:46 Running section lenny2squeeze
>> +Traceback
>>
>> 00:35 total: 38476
>> 00:35 source: 18358
>> 00:35 Writing package templates in sid
>> 00:36 Writing maintainer summaries in /org/piuparts.debian.org/htdocs/sid
>> 00:36 Writing section index page
>>
>> Hmm. Interesting. why is that so much faster there?
>>
>
> Indeed. My numbers are closer to piatti's results.
>
> Using stock piuparts-report, and sort-issues-by-rdep-fast
> detect_well_known_errors.
>
> With no tpls to run:
> 19:01 Running section sid
>
> 19:01 total: 38476
> 19:01 source: 18358
> 19:01 Writing package templates in sid
> 19:01 Writing maintainer summaries in /var/lib/piuparts/htdocs/sid
> 19:01 Writing section index page
> 19:01 analysis template initdscript_lsb_header_issue.tpl does not exist.
>
>
> After a detect_well_known_errors run (~90 seconds over 4 sections,
> stale files, about a dozen new kpr's/section)
> 19:08 Running section sid
>
> 19:09 total: 38476
> 19:09 source: 18358
> 19:09 Writing package templates in sid
> 19:09 Writing maintainer summaries in /var/lib/piuparts/htdocs/sid
> 19:09 Writing section index page
> 19:09 analysis template initdscript_lsb_header_issue.tpl does not exist.
>
> That's 2:28 total over 4 sections. That suggests 86 sections in less
> than an hour.
>
> Watching the messages scroll by, "writing package templates" doesn't
> even look like the heavy hitter - successfully-tested feels like it
> takes longer (the log entries need to have seconds resolution)
>
> Your detect_well_known_errors run is maybe faster than I expected.
> Your piuparts-report run seems way slow.
>
> Could it be available buffer space?
>
> top - 19:12:31 up 16 days,  8:25,  6 users,  load average: 0.65, 0.78, 0.51
> Tasks: 229 total,   3 running, 225 sleeping,   0 stopped,   1 zombie
> %Cpu(s): 16.4 us,  3.8 sy,  0.0 ni, 77.9 id,  1.7 wa,  0.0 hi,  0.2 si,  0.0 st
> KiB Mem:  10236532 total,  9265276 used,   971256 free,   640960 buffers
> KiB Swap: 20903932 total,   153916 used, 20750016 free,  5019796 cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM     TIME+ COMMAND
> 26835 piupart+  20   0  675m 552m 7660 R  98.1  5.5   0:22.16 piuparts-report
>  3410 root      20   0  325m 115m 8264 S   3.3  1.2 153:30.72 Xorg