Monitoring server performance problems

[charles]

Hi

we have been plagued with perforance problems on our zabbix monitoring server. It's no slouch - dual 2.8G xeon with 2G of RAM. Disks are IDE, but sufficient (but this could be the issue).

Load goes from 0 to 4-6 when we start the suckerd (currently running 8 as we had a queue backlog with fewer).

Load appears to mostly be mysql taking cpu, and occasionaly suckerd, but I have noticed a very large number of local connections (127.0.0.1) to services like smtp/pop/imap when zabbix is running. if I shut off zabbix they stop. I checked the host definition for the monitoring server and it isn't configured to check those ports. I even deleted that host, but it doesn't stop.

Any way to check if we have some sort of misconfiguration causing those checks to check localhost instead of the actusal server it should be (because thats whhat it seems is happening).

FYI, We are about to try run sql on another server, mainly because zabbix is crippling all the other services on the current server, but hope it will help zabbix as well. We have had corrupted history tables several times now.

Any thoughts?

thanks
charles

Is zabbix_suckerd running ? Yes
Is zabbix_trapperd running ? No
Number of values stored 229843285
Number of trends stored 26340362
Number of alarms 75270
Number of alerts 1965
Number of triggers (enabled/disabled) 3386(3385/1)
Number of items (active/trapper/not active/not supported) 6636(5721/0/174/741)
Number of users 16
Number of hosts (monitored/not monitored) 296(271/8)

[Alexei]

The hardware seems to be quite adequate to the stats. Be aware that when you start ZABBIX server it starts performing housekeeping procedure which is quite resource hungry. So, CPU (and disk) load can go up for several minutes, and then return to 'normal' values.

Are you sure you have no servers with local hostname, 127.0.0.1 or IP address of your server defined in ZABBIX database? Please, double-check it. It is highly doubtful that ZABBIX does something it is not supposed to do.

Please, let me know about any progress made. Thanks!

[charles]

Hi Alexei

I hope you are well, and I see you started a commercial entity for Zabbix! Excellent!

The hardware seems to be quite adequate to the stats. Be aware that when you start ZABBIX server it starts performing housekeeping procedure which is quite resource hungry. So, CPU (and disk) load can go up for several minutes, and then return to 'normal' values.

Are you sure you have no servers with local hostname, 127.0.0.1 or IP address of your server defined in ZABBIX database? Please, double-check it. It is highly doubtful that ZABBIX does something it is not supposed to do.

Please, let me know about any progress made. Thanks!

I should have mentioned we stopped it because the load was too high in general and causing us pain to use all the other services in the box. So it's not just startup.

I did check for the hostnames, but unless I am not checking correctly, this is not it. As you can see I even deleted it to be sure

mysql> select * from hosts where ip='127.0.0.1' or host like 'dtg000%';
+--------+------------------+-------+-----------+-------+--------+---------------+----------------+
| hostid | host | useip | ip | port | status | disable_until | network_errors |
+--------+------------------+-------+-----------+-------+--------+---------------+----------------+
| 10035 | DTG000 [DELETED] | 1 | 127.0.0.1 | 10005 | 4 | 1114008460 | 3 |
+--------+------------------+-------+-----------+-------+--------+---------------+----------------+
1 row in set (0.00 sec)

I'll let you know what we see once I move the DB to a dedicated P4 2.4ghz 1G RAM with 2x36gb 10krpm scsi drives software raid 1 connected via private gigabit ethernet.

charles

[Alexei]

Hi Charles,

I'm fine, thanks! I hope you're all right too.

I did check for the hostnames, but unless I am not checking correctly, this is not it. As you can see I even deleted it to be sure

mysql> select * from hosts where ip='127.0.0.1' or host like 'dtg000%';
+--------+------------------+-------+-----------+-------+--------+---------------+----------------+
| hostid | host | useip | ip | port | status | disable_until | network_errors |
+--------+------------------+-------+-----------+-------+--------+---------------+----------------+
| 10035 | DTG000 [DELETED] | 1 | 127.0.0.1 | 10005 | 4 | 1114008460 | 3 |
+--------+------------------+-------+-----------+-------+--------+---------------+----------------+
1 row in set (0.00 sec)

The host obviously has another IP address.Have you tried "select * from hosts where ip='<host IP>'"?

I'll let you know what we see once I move the DB to a dedicated P4 2.4ghz 1G RAM with 2x36gb 10krpm scsi drives software raid 1 connected via private gigabit ethernet.

What database format you're using, MyISAM or Innodb? Is MySQL configured and tuned to use most of available memory?

[charles]

Hi Charles,

I'm fine, thanks! I hope you're all right too.

Good thanks.

The host obviously has another IP address.Have you tried "select * from hosts where ip='<host IP>'"?

Yes, and by various possible hostnames (we don't use IP much). Still only that one row.

What database format you're using, MyISAM or Innodb? Is MySQL configured and tuned to use most of available memory?

MyISAM. It's probably not tuned the best. I'd like to hear you recommendations for my size (only getting bigger), and I will try them on both this server and the new one.

Thanks!
charles