Bug#504413: KeyError: problem

[John Wright]

tags 504413 + pending
thanks

On Tue, Nov 04, 2008 at 03:20:16PM +0100, Pietro Abate wrote:
> On Tue, Nov 04, 2008 at 02:43:22AM -0700, John Wright wrote:
> > Anyway, I plan to pull this into the master branch and upload it after
> > I've tested it a bit more (and probably written a few more unit tests).
> 
> It seems that using share_storage = False (fast parser and implicit copy) gives
> the best result. I think this should be the default as it's backward compatible 
> and it's iteration-safe.

I wrote some unit tests, found and fixed a nasty bug with
use_apt_pkg=True and shared_storage=False (key order wasn't being
preserved).  I also figured out how to make the gpg verification stuff
play nice with this, so I'm a bit more comfortable pushing it back up
now.

I made a couple of small optimizations that resulted in pretty
significant performance advantages over what I had before.  As you can
see, except for "raw" Deb822 paragraphs (with none of the
multivalued-fields magic), there is little difference between using
shared_storage=False and shared_storage=True.  Since the performance is
so similar, I've made use_apt_pkg=True, shared_storage=False the
default.  I've attached a little benchmark script and the results on my
system.

-- 
John Wright <jsw at debian.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: benchmark.py
Type: text/x-python
Size: 1117 bytes
Desc: not available
Url : http://lists.alioth.debian.org/pipermail/pkg-python-debian-maint/attachments/20081104/1b5bc885/attachment.py 
-------------- next part --------------
jswright at neptune:~/debian/python-debian/benchmark$ python benchmark.py 
Class: <class 'debian_bundle.deb822.Deb822'>
use_apt_pkg=True, shared_storage=True
Elapsed time: 0:00:02.137641
use_apt_pkg=True, shared_storage=False
Elapsed time: 0:00:06.978174
use_apt_pkg=False, shared_storage=False
Elapsed time: 0:00:23.644512

Class: <class 'debian_bundle.deb822.Packages'>
use_apt_pkg=True, shared_storage=True
Elapsed time: 0:00:12.842322
use_apt_pkg=True, shared_storage=False
Elapsed time: 0:00:18.014803
use_apt_pkg=False, shared_storage=False
Elapsed time: 0:00:28.468205

jswright at neptune:~/debian/python-debian/benchmark$ python benchmark.py 
Class: <class 'debian_bundle.deb822.Deb822'>
use_apt_pkg=True, shared_storage=True
Elapsed time: 0:00:02.183086
use_apt_pkg=True, shared_storage=False
Elapsed time: 0:00:07.134066
use_apt_pkg=False, shared_storage=False
Elapsed time: 0:00:23.840483

Class: <class 'debian_bundle.deb822.Packages'>
use_apt_pkg=True, shared_storage=True
Elapsed time: 0:00:12.793268
use_apt_pkg=True, shared_storage=False
Elapsed time: 0:00:17.962679
use_apt_pkg=False, shared_storage=False
Elapsed time: 0:00:28.890604

jswright at neptune:~/debian/python-debian/benchmark$ python benchmark.py 
Class: <class 'debian_bundle.deb822.Deb822'>
use_apt_pkg=True, shared_storage=True
Elapsed time: 0:00:02.184487
use_apt_pkg=True, shared_storage=False
Elapsed time: 0:00:06.981271
use_apt_pkg=False, shared_storage=False
Elapsed time: 0:00:23.387675

Class: <class 'debian_bundle.deb822.Packages'>
use_apt_pkg=True, shared_storage=True
Elapsed time: 0:00:13.834592
use_apt_pkg=True, shared_storage=False
Elapsed time: 0:00:18.019999
use_apt_pkg=False, shared_storage=False
Elapsed time: 0:00:28.221635