[Piuparts-devel] piuparts-report excessive memory usage - solved

[Andreas Beckmann]

On 2013-02-21 15:53, Holger Levsen wrote:
> On Donnerstag, 21. Februar 2013, Andreas Beckmann wrote:
>> And having the rpy jobs only finish at the end is probably also a cause
>> for the vast memory usage of piuparts-report, at the end it currently
>> takes nearly 6 GB ...
> 
> ouch.

Thanks python for this feature:

>>> def foo(a=[]):
...     a.append(1)
...     return len(a)
...
>>> foo()
1
>>> foo()
2
>>> foo()
3
>>> foo([])
1
>>> foo([])
1
>>> foo([])
1

So I still had a global packagedb cache (that would not be garbage
collected), even after disabling it (i.e. no longer passing a
cache=globalcachedict parameter and relying on the cache={} default)
half a year ago ... seeing that I better spend some time to refetch and
recompute (5-10 seconds per section, maybe more for multistep upgrades)
than the memory to cache them ... and every new (partial) distro added a
bit more cruft there. Maybe caching only the k most recent full distros
would be sufficient (for k = 1 or perhaps k <= 3)).

857.93user 605.14system 2:22:00elapsed 17%CPU (0avgtext+0avgdata
1115736maxresident)k
541256inputs+18550000outputs (275major+17759736minor)pagefaults 0swaps

Nice - 4.5 GB *less*

Andreas