[pkg-gnupg-maint] Bug#852106: Bug#852106: gpgme1.0: Build allocates 200 GB as a normal thing

[Daniel Kahn Gillmor]

Hi Santiago--

On Sat 2017-01-21 12:40:06 -0500, Santiago Vila wrote:
> I have a cron job monitoring "Committed_AS" in /proc/meminfo every time
> my autobuilders are running, that way I know how much memory each
> package requires.
>
> Well, building gpgme1.0 allocates 100 GB, 200 GB and sometimes 500 GB.

yikes!  I've never noticed that, but i can replicate it now that i'm
looking for it.  I'm a bit surprised a cronjob would have caught it,
because i only see the effect for a few seconds at most.  but it's
definitely a significant spike in Committed_AS.

> Then try to build this package in such machine, and it will fail.
>
> I have detected only one other package with a problem like this.
> If you are curious:
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848589
>
> Every other package which I tried builds fine with only 20 GB of RAM,
> so this is clearly an anomaly.

I would have guessed that this happens in gpgme due to the C++ library
build which is now part of libgpgme -- i know that g++ and the linker
can both be pretty beastly.  and snap-align is also C++.

however, i did a bit more inspection, and i see these commitments only
during the test suite.  in particular, on a system with 8GiB of RAM, my
RAM commitments moved from a normal level of:

Committed_AS:     7360592 kB

    to:

Committed_AS:   108890120 kB

during the execution of tests/gpg/t-thread-keylist-verify.


this test makes 200 threads, 100 of which do a gpg verification
operation, and the other 100 do a keylist operation; that's when the RAM
allocation spikes.

Interestingly, just before that happens is tests/gpg/t-thread-keylist,
which doesn't seem to allocate nearly as much RAM despite being of the
same general form (it just does keylist without the simultaneous verify,
though).

Each thread does spawn an additional gpg process, so there's certainly
memory committments there.  But even with 200 concurrent processes, that
amounts to 0.5GiB of RAM commitments per process, which seems large to
me.

I'm open for suggestions for how to address this.  We could just drop
THREAD_COUNT from 100 to 10 to reduce the RAM consumption, but that
would mean minimizing some of the concurrency that the test suite is
(rightly) trying to judge itself against.  I'd rather not remove those
kinds of checks if we don't need to.

Any suggestions about what the right thing to do is?  

    --dkg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-gnupg-maint/attachments/20170123/b3267b84/attachment.sig>