[Piuparts-devel] speeding up archive_old_logs

[Holger Levsen]

Hi,

On Dienstag, 6. Dezember 2011, Andreas Beckmann wrote:
> piatti.git feature/speedup-archive_old_logs
> 
>     incrementally archive old logs
> 
>     only look at logs that changed since the last archiving run to
>     determine the packages to check - no need to look at 30000+
>     unchanged packages from stable every day
>     (actually also look at logs up to one day older to avoid races)

this made me think: if we would only create the source packages html pages for 
the packages which have been tested in the last 24h, piuparts-report could be 
speed up further: /htdocs/*/source contains 20k files for sid (and 16k for 
squeeze) - of which only a very few change each day.

/htdocs/*/maintainer contains 3.5k files for sid, so their is less potential, 
but still.

btw, piuparts-reports is already down to 3h runtime (from 6h previously), 
which is pretty cool, so tomorrow I plan to add the squeeze2wheezy suite! 

(Today is still for retesting successful logs in wheezy with the old /etc/mtab 
issue... then, once squeeze2wheezy has been tested, the same then will be done 
the /etc/mtab logs in sid.)


cheers,
	Holger