View Issue Details

IDProjectCategoryView StatusLast Update
0000027FromDual Performance Monitor for MariaDB and MySQL (fpmmm)fpmmm Agentpublic2014-05-24 13:16
Reportersclaret Assigned Tooli  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformLinuxOSCentosOS Version5.x
Product Version0.9.2 
Target Version0.9.3Fixed in Version0.9.3 
Summary0000027: MySQL is down Trigger does not fire
DescriptionThe "MySQL is down" trigger does not work correctly out of the box.

This seems to be because mpm fails to update FromDual.MySQL.mysql.alive when the server is down.
Steps To Reproduce> service mysqld start

Install & configure mpm

> service zabbix-agent restart

Set up a zabbix 'host' to monitor the machine with Template_FromDual.MySQL.mpm, & Template_FromDual.MySQL.server.

Set up another 'host' to monitor mysqld with Template_FromDual.MySQL.mysql, Template_FromDual.MySQL.process & Template_FromDual.MySQL.myisam.

Data is regularly updated in Monitoring --> Latest data!

Now, run:

> service mysqld stop

In /var/log/zabbix/FromDualMySQLagent.log we see:

25719:2012-05-22 15:46:14.433 - INFO: FromDual Performance Monitor for MySQL (0.9) run started.
25719:2012-05-22 15:46:15.553 - ERR : Cannot read PID file /var/run/mysqld/mysqld.pid. Either file does not exist or I have no read permissions. Are you sure the process is running?
25719:2012-05-22 15:46:15.574 - ERR : DBI connect with database=mysql, host=localhost, port=3306 and user=monitor failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
25719:2012-05-22 15:46:15.575 - DBG : Database connection failed (rc=2400).
25719:2012-05-22 15:46:15.579 - ERR : DBI connect with database=mysql, host=localhost, port=3306 and user=monitor failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
25719:2012-05-22 15:46:15.580 - DBG : Database connection failed (rc=2500).
25719:2012-05-22 15:46:15.580 - INFO: FromDual Performance Monitor for MySQL run finshed (rc=0).

In Monitoring --> Latest Data

The variable 'MySQL is alive' is frozen to the value it had right before we stopped mysqld:

Description Last check Last value Change History
MySQL is alive 22 May 2012 15:29:45 1 - Graph

Hence, zabbix still thinks the server is up, and the trigger does not fire:

MySQL is down {Template_FromDual.MySQL.mysql:FromDual.MySQL.mysql.alive.last(0)}=0

Additional InformationI plan to work around this by creating a custom trigger, that fires when FromDual.MySQL.mysql.alive has not updated in a while.

However, it would probably be better to fix this at the source, though I don't know how much coding this would take (I would have expected FromDual.MySQL.mysql.alive to turn to 0 when mysql shut down).

One way or another, the "mysql is dead" trigger should probably be made to work out of the box.
TagsNo tags attached.

Activities

sclaret

2012-10-18 04:41

reporter   ~0000015

Workarounds, from our experience:

These triggers seem to both be non-functional, due to defective logic in the m.p.m. plugin.

That is problematic; these are really conditions that should generate alerts.

Let's deal with each individually.

-----

"MySQL is down"

Definition of the trigger from m.p.m.:
{Template_FromDual.MySQL.mysql:FromDual.MySQL.mysql.alive.last(0)}=0

Initially, the item updates every 10 seconds as expected:

Wed Oct 17 18:38:41 PDT 2012
MySQL is alive 17 Oct 2012 18:38:31 1
Wed Oct 17 18:39:02 PDT 2012
MySQL is alive 17 Oct 2012 18:39:05 1
MPM Agent is alive 17 Oct 2012 18:41:08 1
[root@neut14 lib]# service mysqld stop
Stopping MySQL: [ OK ]
Wed Oct 17 18:41:50 PDT 2012
MySQL is alive 17 Oct 2012 18:41:19 1
Wed Oct 17 18:42:17 PDT 2012
MySQL is alive 17 Oct 2012 18:41:19 1
Wed Oct 17 18:42:33 PDT 2012
MySQL is alive 17 Oct 2012 18:41:19 1

But once MySQL dies, it stops updating, remaining stuck at the last "good" reading.

Having "mysql server must be up" as a precondition to "set mysql.alive to 0" does not make much sense.

Let's work around this with a new trigger expression.

Configuration->Templates->Template_FromDual.MySQL.mysql->Triggers->MySQL is down
Expression: {Template_FromDual.MySQL.mysql:FromDual.MySQL.mysql.alive.nodata(180)}=1
Event generation: Normal
<Save>

So, we get email when a mysql server has been down for at least 3 minutes.

-----

"MPM Agent is down"

Was the same problem. Fixed by changing trigger expression from:

{Template_FromDual.MySQL.mpm:FromDual.MySQL.check.last(0)}=0

to:

{Template_FromDual.MySQL.mpm:FromDual.MySQL.check.nodata(180)}=1

-----

oli

2014-05-08 20:23

developer   ~0000117

duplicate of 0000138

oli

2014-05-24 13:16

developer   ~0000120

fixed in rev420

Issue History

Date Modified Username Field Change
2012-05-24 02:18 sclaret New Issue
2012-05-24 02:18 sclaret Status new => assigned
2012-05-24 02:18 sclaret Assigned To => oli
2012-05-25 08:40 oli Assigned To oli =>
2012-10-18 04:41 sclaret Note Added: 0000015
2014-05-08 20:23 oli Note Added: 0000117
2014-05-08 20:24 oli Assigned To => oli
2014-05-24 13:16 oli Note Added: 0000120
2014-05-24 13:16 oli Status assigned => resolved
2014-05-24 13:16 oli Resolution open => fixed
2014-05-24 13:16 oli Product Version => 0.9.2
2014-05-24 13:16 oli Fixed in Version => 0.9.3
2014-05-24 13:16 oli Target Version => 0.9.3
2015-05-01 11:07 administrator Category mpm Agent => fpmmm Agent