Zabbix perfomance tunning

[AndreyHammer]

Hi!
Please, help to adjust correctly server. A configuration and versions of software and hardware:

HP DL380G5: 2*quad xeon E5345, 4Gb RAM, swap 4Gb.
OS: Debian etch, kernel 2.6.23.12
Zabbix: 1.4.5
Postgresql: 8.2.5, pg_xlog mounted on separate disk, zabbix database size about 24Gb.

Zabbix status:

STATUS OF ZABBIX   
Parameter Value 
ZABBIX server is running Yes 
Number of hosts (monitored/not monitored/templates/deleted) 92(76/4/12/0) 
Number of items (monitored/disabled/not supported)[trapper] 3240(3037/10/193)[1] 
Number of triggers (enabled/disabled)[true/unknown/false] 645(290/355)[1/4/285] 
Number of events 39064 
Number of alerts 0

Zabbix queue:

QUEUE OF ITEMS TO BE UPDATED  Overview Details 
Items 5 seconds 10 seconds 30 seconds 1 minute 5 minutes More than 5 minutes 
ZABBIX agent 4 8 21 22 1 0 
ZABBIX agent (active) 0 0 0 0 0 0 
SNMPv1 agent 0 0 0 0 0 0 
SNMPv2 agent 139 116 470 479 31 0 
SNMPv3 agent 0 0 0 0 0 0 
Simple check 0 0 0 0 0 0 
ZABBIX internal 0 0 0 0 0 0 
ZABBIX aggregate 0 0 0 0 0 0 
External check 0 0 0 0 0 0

top:

top - 11:13:04 up 117 days, 22:19,  1 user,  load average: 14.60, 16.47, 15.70
Tasks: 365 total,   2 running, 363 sleeping,   0 stopped,   0 zombie
Cpu0  : 39.1%us, 21.7%sy,  0.0%ni, 39.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 42.9%us, 14.3%sy,  0.0%ni, 42.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  8.7%us,  8.7%sy,  0.0%ni, 82.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  : 21.7%us, 13.0%sy,  0.0%ni, 65.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  : 13.0%us,  0.0%sy,  0.0%ni, 87.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  : 17.4%us,  8.7%sy,  0.0%ni, 73.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  9.5%us,  0.0%sy,  0.0%ni, 90.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  : 17.4%us,  4.3%sy,  0.0%ni, 78.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4147960k total,  3998460k used,   149500k free,    43212k buffers
Swap:  4192924k total,     9668k used,  4183256k free,  3149076k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  802 postgres  20   0  339m 311m 307m R   13  7.7   2:25.07 postmaster
  920 postgres  20   0  343m 313m 307m S   13  7.7   2:18.55 postmaster
  936 postgres  20   0  339m 311m 307m S   13  7.7   2:11.55 postmaster
  804 postgres  20   0  343m 316m 307m S    9  7.8   2:29.69 postmaster
  926 postgres  20   0  343m 314m 307m S    9  7.8   2:27.17 postmaster
  927 postgres  20   0  343m 316m 307m S    9  7.8   2:25.04 postmaster
  939 postgres  20   0  343m 316m 307m S    9  7.8   2:19.32 postmaster
  943 postgres  20   0  343m 315m 307m S    9  7.8   2:23.39 postmaster
  945 postgres  20   0  343m 316m 307m S    9  7.8   2:27.28 postmaster
  946 postgres  20   0  343m 313m 307m S    9  7.7   2:24.34 postmaster
  947 postgres  20   0  343m 313m 307m S    9  7.7   2:20.01 postmaster
  948 postgres  20   0  343m 316m 307m S    9  7.8   2:21.96 postmaster
  954 postgres  20   0  343m 315m 307m S    9  7.8   2:26.39 postmaster
  955 postgres  20   0  339m 311m 307m S    9  7.7   2:26.03 postmaster
  966 postgres  20   0  343m 315m 307m S    9  7.8   2:21.62 postmaster
  980 postgres  20   0  343m 316m 307m S    9  7.8   2:22.78 postmaster
 2003 postgres  20   0  335m 177m 175m S    9  4.4   0:00.72 postmaster
  807 postgres  20   0  343m 316m 307m S    4  7.8   2:30.60 postmaster
  812 postgres  20   0  343m 315m 307m S    4  7.8   2:12.79 postmaster
  832 postgres  20   0  336m 309m 307m S    4  7.7   2:09.92 postmaster
  897 postgres  20   0  343m 316m 307m S    4  7.8   2:21.63 postmaster
  916 postgres  20   0  343m 315m 307m S    4  7.8   2:24.32 postmaster
  918 postgres  20   0  343m 316m 307m S    4  7.8   2:27.74 postmaster
  919 postgres  20   0  343m 316m 307m S    4  7.8   2:23.77 postmaster
  921 postgres  20   0  343m 316m 307m S    4  7.8   2:26.38 postmaster
  922 postgres  20   0  343m 315m 307m S    4  7.8   2:27.38 postmaster
  923 postgres  20   0  343m 316m 307m S    4  7.8   2:24.12 postmaster
  924 postgres  20   0  343m 316m 307m S    4  7.8   2:25.87 postmaster
  925 postgres  20   0  343m 315m 307m S    4  7.8   2:25.48 postmaster
  928 postgres  20   0  343m 315m 307m S    4  7.8   2:15.06 postmaster
  929 postgres  20   0  343m 316m 307m S    4  7.8   2:24.16 postmaster
  930 postgres  20   0  343m 316m 307m S    4  7.8   2:21.19 postmaster
  931 postgres  20   0  343m 316m 307m S    4  7.8   2:18.18 postmaster
  932 postgres  20   0  343m 315m 307m S    4  7.8   2:23.21 postmaster
  935 postgres  20   0  343m 316m 307m S    4  7.8   2:20.42 postmaster
  937 postgres  20   0  343m 316m 307m S    4  7.8   2:09.98 postmaster
  938 postgres  20   0  343m 316m 307m S    4  7.8   2:24.72 postmaster
  940 postgres  20   0  343m 313m 307m S    4  7.7   2:18.32 postmaster
  941 postgres  20   0  344m 316m 307m S    4  7.8   2:22.49 postmaster
  942 postgres  20   0  343m 314m 307m S    4  7.8   2:26.83 postmaster
  952 postgres  20   0  343m 315m 307m S    4  7.8   2:24.88 postmaster
  953 postgres  20   0  343m 316m 307m S    4  7.8   2:20.99 postmaster
  957 postgres  20   0  343m 315m 307m S    4  7.8   2:22.68 postmaster
  960 postgres  20   0  343m 316m 307m S    4  7.8   2:23.07 postmaster
  967 postgres  20   0  343m 316m 307m S    4  7.8   2:25.69 postmaster

In attachmet config files of zabbix & postgresql.

[xs-]

Does the load drop / performance increase when you do a full reindex? (or reindex the big tables like history, trends, etc).
check load, stop zabbix_server, full reindex, start zabbix server, wait until queue is processed, check load again.

Because of the 'relative' insane amount of inserts/deletes the table indexes become useless in time unless you reindex regularly.
I've had the same problem up to the point where it required too much time per week to maintain for it to be cost effective.
For shits and giggles, i tried to run it on a mysql db and got baffled as the performance was great.
It looks like mysql manages the indexes in a way they dont get effected as much as postgres (or oracle for that matter).
You should give it a try, migration is rather easy since the table structures for both are exactly the same. The 3 tables which do differ (seq/trig -> auto_increment) should be empty when you stop zabbix so no data migration there.

--edit--
For the record, stats for our main node
Number of hosts (monitored/not monitored/templates/deleted) 711(515/10/186/0)
Number of items (monitored/disabled/not supported)[trapper] 18330(12920/1689/3721)[156]
Number of triggers (enabled/disabled)[true/unknown/false] 13142(12126/1016)[31/156/11939]

load on db server:
top - 09:33:28 up 4 days, 18:19, 3 users, load average: 2.54, 2.14, 2.06
Tasks: 82 total, 2 running, 80 sleeping, 0 stopped, 0 zombie
Cpu0 : 12.0%us, 0.0%sy, 0.0%ni, 87.7%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu1 : 44.7%us, 6.5%sy, 0.0%ni, 48.1%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
Cpu2 : 10.3%us, 0.3%sy, 0.0%ni, 89.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 64.2%us, 7.2%sy, 0.0%ni, 20.1%id, 0.3%wa, 2.7%hi, 5.5%si, 0.0%st
Mem: 4054572k total, 3727432k used, 327140k free, 191712k buffers
Swap: 1951888k total, 140k used, 1951748k free, 2033564k cached

and the web frontend is fast (bet its really really slow for you )

[Alexei]

It all is about how PostgreSQL manages updates. Every update actually adds a new record (tuple), while old one is marked as "deleted". ZABBIX does lots of updates and this kills PostgreSQL performance very fast.

MySQL is much more efficient is this respect.

[AndreyHammer]

Thanks for recommendations. In the future probably I migrate on mysql

I have found not full enabled autovacuum in a configuration file of postgresql. Now autovacuum enabled and you can see results:

QUEUE OF ITEMS TO BE UPDATED  Overview Details 
Items 5 seconds 10 seconds 30 seconds 1 minute 5 minutes More than 5 minutes 
ZABBIX agent 0 2 0 0 0 0 
ZABBIX agent (active) 0 0 0 0 0 0 
SNMPv1 agent 0 0 0 0 0 0 
SNMPv2 agent 9 16 0 0 0 0 
SNMPv3 agent 0 0 0 0 0 0 
Simple check 0 0 0 0 0 0 
ZABBIX internal 0 0 0 0 0 0 
ZABBIX aggregate 0 0 0 0 0 0 
External check 0 0 0 0 0 0

top

top - 16:41:22 up 119 days,  3:47,  1 user,  load average: 0.38, 1.78, 2.25
Tasks: 364 total,   1 running, 363 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  : 14.3%us,  3.6%sy,  0.0%ni, 82.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  : 10.7%us,  3.6%sy,  0.0%ni, 85.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4147960k total,  3996032k used,   151928k free,    33772k buffers
Swap:  4192924k total,       48k used,  4192876k free,  3502696k cached

[milprog]

I'm using postgresql 8.3 happily, but with some performance tuning as suggested in the postgresql forums. My server is a cheap ML110G4, 3GB RAM, dual core Xeon CPU with smart 200i controller and four 500GB SATA disks in a RAID 10 configuration. This box now has >100GB used on the database partition and manages 48 hosts for more than 8 months now.

Unfortunately Zabbix has some problems with the strict type cast requirements in version 8.3 up. But the implicit casts required by zabbix can easily be readded (see http://people.planetpostgresql.org/p...reSQL-8.3.html ). Given the flexibility and quality of the postgresql database, the possibility of transaction-safe backup/archiving and its superior achitecture, I would never switch back to MySQL again.

Regards,
--Marcel