[Pkg-nagios-devel] Bug#311695: Rather difficult to reproduce

[Andres Salomon]

On Thu, 2005-06-02 at 20:33 -0700, Don Armstrong wrote:
> The only way you can get this bug to occur is if you're configuring
> the database while nagios is running, and manage to create the user so
> nagios can connect, and not give it the appropriate grant statements
> to actually do what it is supposed to do to the database.
> 
> Otherwise, you just get a string of messages like the following:
> 
> root@archimedes:/# less /var/log/nagios/nagios.log 
> [1117766679] Nagios 1.3 starting... (PID=20707)
> [1117766910] Nagios 1.3 starting... (PID=21683)
> [1117766910] Error: Could not connect to MySQL database '' on host '' using username '' and password 'XXXXXX'.  Status data will not be saved!
> [1117766910] Error: Could not connect to MySQL database '' on host '' using username '' and password 'XXXXXX'.  Retention data will not be processed or saved!
> [1117766910] Error: Could not re-connect to database server on host '' for status data.  I'll keep trying every 60 seconds...
> [1117766910] Error: Could not connect to MySQL database '' on host '' using username '' and password 'XXXXXX' for comment data!
> [1117766910] Error: Could not connect to MySQL database '' on host '' using username '' and password 'XXXXXX' for downtime data!
> 


Ah-ha, figured it out.

Steps to reproduce:

1) Create a woody chroot.  Install mysql-server.
2) Change your sources.list to use sarge instead of woody.
Dist-upgrade.
3) Put "deb http://snapshot.debian.net/archive/2005/04/16/debian sarge
main" in your sources.list.  Apt-get update && apt-get install
mysql-server=4.0.24-2 libmysqlclient12=4.0.24-2 mysql-client=4.0.24-2
3) apt-get install nagios-mysql.

At this point, you'll see fun stuff like:
Jun  3 15:53:01 jack nagios: Error: Could not insert retention data for
host 'gw' in table 'hostretention'
Jun  3 15:53:13 jack last message repeated 61176 times

The problem here is twofold:
a) nagios will repeatedly connect to the database, and endlessly retry a
failed SQL command.
b) mysql-server will allow as many attempts as possible.  In a later
version of mysql, a patch was included to throttle connections; I think
this one:

mysql-dfsg (4.0.24-3) unstable; urgency=low
...
  * Added upstream patch to fix the per hour connection limit
    (thanks to Rene Konasz). Closes: #285044

 -- Christian Hammers <ch@debian.org>  Fri, 25 Mar 2005 22:23:40 +0100


So, the RC-ness comes from the combination of these two bugs.  Since
this is fixed by later version of mysql-server, you can downgrade this
bug.  However, given that nagios will still happily pound the database
(which may or may not support throttling by default.. what happens w/
postgres?), and there are other ways to trigger this, I still *highly*
recommend fixing this for sarge (whether in 3.1r1, or whatever).


Just out of curiousity, I reinstalled mysql-server 4.0.24-2 to nuke the
woody-style mysql.user table; it still contained the blank user.  So, I
reinstalled mysql-server from the latest, and then downgraded to
4.0.24-2, so that it had the 4.0.24-10 style mysql.user table, as well
as the throttling bug.

The result:

Jun  3 16:07:52 jack nagios: Error: Could not insert retention data for
host 'gw' in table 'hostretention'
Jun  3 16:07:52 jack last message repeated 15 times
Jun  3 16:07:52 jack nagios: Error: Could not connect to MySQL database
'' on host '' using username '' and password 'XXXXXX' for downtime data!
Jun  3 16:07:52 jack nagios: Error: Could not insert retention data for
host 'gw' in table 'hostretention'
Jun  3 16:08:01 jack last message repeated 20896 times
Jun  3 16:08:01 jack nagios: Error: Could not insert retention data for
host 'gw' in table 'hostretention'
Jun  3 16:08:32 jack last message repeated 80890 times

This really does come down to whether or not the database works around
the nagios bug by throttling; the default mysql-server mysql.user table
still allows nagios to connect, which nagios takes to mean that
everything is configured.