[Pkg-puppet-devel] Bug#780664: puppetmaster fills up the database connection pool.

[bertagaz]

Package: puppetmaster
Version: 3.7.2-3
Severity: important
User: tails-dev at boum.org
Usertags: infra

Hi,

When enabling storeconfig in puppetmaster, the database connection pool gets
filled up by inactive connections opened by previous puppet agents runs. After
the maximum size of the pool have been reached (default to 5 in ActiveRecord),
the puppet agents can't retrieve the catalog anymore and exit with this error:

  err: Could not retrieve catalog from remote server: Error 400 on SERVER:
  could not obtain a database connection within 5.000 seconds (waited
  5.000 seconds)

With the sqlite3 database backend, the puppetmaster opens 5 FDs to the
sqlite database:

  root at puppet-git:~/# lsof /var/lib/puppet/state/clientconfigs.sqlite3
  COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
  puppet  17031 puppet   12u   REG  253,1 14777344 131800 /var/lib/puppet/state/clientconfigs.sqlite3
  puppet  17031 puppet   13u   REG  253,1 14777344 131800 /var/lib/puppet/state/clientconfigs.sqlite3
  puppet  17031 puppet   14u   REG  253,1 14777344 131800 /var/lib/puppet/state/clientconfigs.sqlite3
  puppet  17031 puppet   15u   REG  253,1 14777344 131800 /var/lib/puppet/state/clientconfigs.sqlite3
  puppet  17031 puppet   16u   REG  253,1 14777344 131800 /var/lib/puppet/state/clientconfigs.sqlite3

The same seems to happen with the MySQL database backend:

  mysql> show processlist\g
  +----+--------+-----------+--------+---------+------+-------+------------------+
  | Id | User   | Host      | db     | Command | Time | State | Info             |
  +----+--------+-----------+--------+---------+------+-------+------------------+
  | 44 | puppet | localhost | puppet | Sleep   |  117 |       | NULL             |
  | 45 | root   | localhost | NULL   | Query   |    0 | NULL  | show processlist |
  | 46 | puppet | localhost | puppet | Sleep   |   98 |       | NULL             |
  | 47 | puppet | localhost | puppet | Sleep   |   79 |       | NULL             |
  | 48 | puppet | localhost | puppet | Sleep   |   59 |       | NULL             |
  | 49 | puppet | localhost | puppet | Sleep   |   40 |       | NULL             |
  +----+--------+-----------+--------+---------+------+-------+------------------+
  6 rows in set (0.00 sec)


This sound a lot like a bug reported upstream a while ago [1], but it's not
easy to track if something came out with upstream's tracker move.

I've tried several changes in lib/puppet/rails.rb:

* replace ActiveRecord::Base.clear_active_connections! by
ActiveRecord::Base.clear_all_connections!
* remove the ActiveRecord::Base.allow_concurrency = true setting, as proposed
in the upstream ticket

But this didn't fix anything. Each time an agent runs, the puppetmaster
initiates a new connection to the database rather than using the active one,
even if it is the same agent than the previous run.

bert.

[1] http://projects.puppetlabs.com/issues/3238