Bug#931239: please improve performance of hopenpgp-tools on large certificates

[Daniel Kahn Gillmor]

Package: hopenpgp-tools
Version: 0.21.3-1
Severity: wishlist

I'm looking at performance tests on large (spammed/flooded)
certificates.  hopenpgp-tools consumes more CPU than GnuPG by a factor
of 2×, 5×, or 10× depending on the operation.

I provide these figures as a target for hopenpgp to meet or beat, if
possible.

During these tests, xxx is the dearmored form of my flooded OpenPGP
certificate, as it is found on the SKS keyservers:

  wget -O - \
   'http://keys.mayfirst.org/pks/lookup?op=get&search=0xF20691179038E5C6' | \
   gpg --dearmor > xxx

it's about 17MiB in size, bloated with junk (see
https://dkg.fifthhorseman.net/blog/openpgp-certificate-flooding.html for
more details)

All tests were run on a quad-core Intel(R) Core(TM) i5-2540M CPU @
2.60GHz, and all data is in a tmpfs, so there should be no disk latency
to worry about.

----------

Here is a performance comparison of just listing the contents of the
packet.  hopenpgp-tools takes 10× more time:

0 dkg at alice:/tmp/cdtemp.7QJ3xD$ time hot dump < xxx > /dev/null
hot (hopenpgp-tools) 0.21.3
Copyright (C) 2012-2019  Clint Adams
hot comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions.

real	0m4.750s
user	0m4.725s
sys	0m0.025s
0 dkg at alice:/tmp/cdtemp.7QJ3xD$ time pgpdump < xxx > /dev/null

real	0m0.516s
user	0m0.403s
sys	0m0.104s
0 dkg at alice:/tmp/cdtemp.7QJ3xD$ time gpg --list-packets < xxx > /dev/null

real	0m0.511s
user	0m0.506s
sys	0m0.005s
0 dkg at alice:/tmp/cdtemp.7QJ3xD$ 
----------


----------
Here is a comparison of adding or removing armor.  hot takes about 5× as
much time as gpg:

1 dkg at alice:/tmp/cdtemp.bOnXz6$ time gpg --enarmor < xxx > xxx.gpg.asc

real	0m0.239s
user	0m0.222s
sys	0m0.017s
0 dkg at alice:/tmp/cdtemp.bOnXz6$ time hot armor --armor-type pubkeyblock < xxx > xxx.hot.asc
hot (hopenpgp-tools) 0.21.3
Copyright (C) 2012-2019  Clint Adams
hot comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions.

real	0m0.978s
user	0m0.917s
sys	0m0.061s
0 dkg at alice:/tmp/cdtemp.bOnXz6$ time gpg --dearmor < xxx.gpg.asc  > /dev/null

real	0m0.332s
user	0m0.317s
sys	0m0.016s
0 dkg at alice:/tmp/cdtemp.bOnXz6$ time hot dearmor < xxx.gpg.asc  > /dev/null
hot (hopenpgp-tools) 0.21.3
Copyright (C) 2012-2019  Clint Adams
hot comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions.

real	0m1.657s
user	0m1.616s
sys	0m0.036s
0 dkg at alice:/tmp/cdtemp.bOnXz6$ 
---------------


---------------
and here is an attempt to look at parsing the data in more detail:

they're both pretty bad, but gpg is faster by a factor of ~2×:

0 dkg at alice:/tmp/cdtemp.bOnXz6$ time gpg --show-keys < xxx  | wc
      8      35     383

real	0m34.374s
user	0m34.297s
sys	0m0.080s
0 dkg at alice:/tmp/cdtemp.bOnXz6$ time hokey lint < xxx  | wc
hokey (hopenpgp-tools) 0.21.3
Copyright (C) 2012-2019  Clint Adams
hokey comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions.
     42     219    2054

real	0m57.681s
user	0m57.387s
sys	0m0.284s
0 dkg at alice:/tmp/cdtemp.bOnXz6$ 


It should not take a modern CPU more than a few seconds to produce ~2KiB
of output out of 17MiB of input!

---------------

Regards,

        --dkg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/pkg-haskell-maintainers/attachments/20190628/a2e7153f/attachment.sig>