[Piuparts-devel] new ideas for testing ...

[Andreas Beckmann]

On 2013-05-14 20:53, Holger Levsen wrote:
> one of the bottlenecks then becomes archive_old_logs, which really should be 
> rewritten to archive old logs when new logs are submitted. we're going 
> through, what, 175k logfiles now, to archie 200-1000, each day?

archive_old_logs already takes care for this, it only consider logs
newer than a stamp from the previous run as candidates that can have old
logs ... while there is still potential for (constant factor)
improvement, this is not urgent

IIRC this was one of the first scripts that I optimized from

O(num_logs)

down to

O(num_new_logs_since_last_run) + O(num_archived_logs)

there is at least one call to ls for each candidate ...


archiving and compressing in total 1000-3000 logs from roughly 60-80
sections on my heavily loaded machine takes 10-15 minutes (time does
*not* correlate to count), this runs twice a day

piuparts-report takes about 6 hours


Andreas

PS: the initial compressing run may have been really slow