Bug#779937: GDP shouldn't process all yaml files all the time

[Alexandre Detiste]

2015-03-13 1:09 GMT+01:00 Simon McVittie <smcv at debian.org>:
> Next idea: YAML is pretty complicated, JSON is much simpler while still
> human-editable, maybe we can use that? That turns out to be simple to
> do, and is another factor of 20 speedup. I'll push the results shortly.

This also seems much less intrusive than the pickled arg parser.

Some people have analysed that in deep:
http://www.csc.kth.se/utbildning/kth/kurser/DD143X/dkand11/Group2Mads/victor.hallberg.malin.eriksson.report.pdf

> I'd like to keep using YAML for the source files, because it's a lot
> more pleasant to write

I agree !

>> That's only the package name, aliases, longname & demo_for tags.
>
> I think a factor of 400 should be enough to keep your RPi happy until we
> add quite a lot more games :-)

This gets around 2,5 seconds and certainly around the double for the RPi;
much less than things like aptitude's GUI.

I've added a minimal process indicator to give user some feedback.
I didn't bother with isatty() for now...
would people ever run GDP unattended / through a pipe ?

     for jsonfile in glob.glob(os.path.join(DATADIR, '*.json')):
 +      print('.', end='', flush=True)
     ...
 +  print('\r', end='', flush=True)
     return games


> If we need more, I suspect kicking out the md5sums etc. into separate
> files (probably in plain md5sums format, or whatever), and loading them
> "lazily", would get us another significant speedup. File lists could
> maybe be kicked out too; I expect the rest of the metadata for games,
> packages and CD tracks is small enough to not matter either way.
> Combining all the remaining metadata into one big JSON file (perhaps a
> map from game name to data) might also be good.

Ok, there's no hurry for this at all; it felt wrong that GDP took more time
to load up than RTCW on the same computer; but now it's perfectly acceptable.

Thanks,