[Pkg-puppet-devel] Splitting puppetq?

[Nigel Kersten]

On Fri, Jan 29, 2010 at 12:41 PM, Russ Allbery <rra at debian.org> wrote:
> micah anderson <micah at riseup.net> writes:
>> Digant C Kasundra <digant at stanford.edu> wrote:
>
>>> Absolutely.  We commissioned the puppetq work specifically for the
>>> purpose of having storeconfigs running on a different server than the
>>> puppetmasters.  When we turned on storeconfigs on the puppetmasters, we
>>> found that the puppetmasters could no longer handle the load and would
>>> take a long time compiling manifests, sometimes so long that they would
>>> timeout.
>
>> I'm curious if you recall at what the tipping point for this was. I
>> haven't seen this problem with a couple hundred nodes and one
>> puppetmaster with storeconfigs turned on. I assume I will hit it at some
>> point, but dont know when that will be.
>
> We have about 500 nodes and we knocked it over dead, but we also have to
> restart our (0.24) Puppet masters every four hours because of the massive
> memory leaks somewhere in Ruby, and it's not clear that other people are
> running into that either.  I'm not sure exactly why we seem to have more
> problems there than others.
>

Russ, I reduced the impact of memory leaks significantly by:

a) moving to RubyEE as the ruby stack
b) moving to passenger, with the backends set to restart after x
thousand requests.

I concur about number of resources. I can crash my puppetmasters with
our base config and 10 clients puppeting against them if I have the
full storeconfigs (not thin) and no queuing.