Zabbix proxy big lag

[porlock]

I receive a 16000 items values over zabbix trapper every 5 minutes on zabbix proxy, and observe a huge lag betwen zabbix server and zabbix proxy, more than 2 hours .. :-(
Both machines that is zabbix server and zabbix proxy have a more than 90% of CPU idle, and a lots of ram free, and there is no network lag between them (ping latency aprox 0.107 ms)

Proxy config:

ProxyMode=0
Server=192.168.0.40
ServerPort=10051
Hostname=proxy_41
ListenPort=10051
SourceIP=192.168.0.41
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=0
DebugLevel=3
PidFile=/var/run/zabbix/zabbix_proxy.pid
DBHost=
DBName=zabbix
DBUser=zabbix
DBPassword=xxxxxxxxxxxxxx
DBSocket=/var/lib/mysql/mysql.sock
ProxyLocalBuffer=0
ProxyOfflineBuffer=24
ConfigFrequency=60
DataSenderFrequency=1
StartPollers=256
StartTrappers=384
StartPingers=200
StartDiscoverers=10
StartHTTPPollers=0
SNMPTrapperFile=/opt/snmpprobe/log/share/zabbix.log
StartSNMPTrapper=1
HousekeepingFrequency=1
CacheSize=128M
StartDBSyncers=40
HistoryTextCacheSize=64M
ExternalScripts=/usr/lib/zabbix/externalscripts
LogSlowQueries=1000

Server config:
LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=0
PidFile=/var/run/zabbix/zabbix_server.pid
DBHost=
DBName=zabbix
DBUser=zabbix
DBPassword=xxxxxxxx
SNMPTrapperFile=/var/log/snmptt/snmptt.log
CacheSize=32M
HistoryCacheSize=64M
TrendCacheSize=32M
HistoryTextCacheSize=64M
AlertScriptsPath=/usr/lib/zabbix/alertscripts
ExternalScripts=/usr/lib/zabbix/externalscripts

Stats from zabbix server and zabbix proxy are fine, utylization is minimal.



Number of records on zabbix proxy db continually grows, it’s seams that the problem is on zabbix proxy, I caught by tcpdump packet transmitted between zabbix proxy and zabbix server and it have a 2 hours delayed clock field.



I use:


Name : zabbix-proxy-pgsql
Arch : x86_64
Version : 2.4.4
Release : 1.el6
Size : 1.3 M
Repo : installed
From repo : zabbix

Name : zabbix-server-pgsql
Arch : x86_64
Version : 2.4.4
Release : 1.el6
Size : 4.3 M
Repo : installed
From repo : zabbix

Installed Packages
Name : postgresql94
Arch : x86_64
Version : 9.4.1
Release : 1PGDG.rhel6
Size : 5.4 M
Repo : installed
From repo : pgdg94

I try to recreate zabbix proxy db but it does nothing.

[ingus.vilnis]

Hi,

Performance tuning may have affect on your issue.

I assume that you don't have any statistics about performance on your proxy_41. Even though you have host proxy_41 added to frontend, looks like you are monitoring the proxy from server therefore it is displaying the same values for both server and proxy. Check your own Latest data screenshot. All the processes with same names have the same values. The values are from server and not from proxy. Set the proxy_41 host to be monitored by itself - proxy_41.

How will you know when the proxy is unreachable, you will ask. Create an item on Zabbix server template with key zabbix[proxy,"proxy_41",lastaccess] which will tell when the proxy contacted Zabbix server for the last time. Create a trigger which fires if the time is more than 120 seconds for example. https://www.zabbix.com/documentation...types/internal

With this done please collect statistics about proxy usage and then report with screenshots of Zabbix proxy performance graphs from proxy template. Based on the graphs we can then try to adjust the proxy config settings.

How many hosts are you monitoring? Your settings would be suitable for thousands of monitored machines.
Few notes on the current settings -
StartPollers=256 - check the graphs to adjust to ~30% usage.
StartTrappers=384 - check the graphs to adjust to ~30% usage.
StartPingers=200 - check the graphs to adjust to ~30% usage.
StartDiscoverers=10 - how many discovery rules do you use? Each rule may be used by 1 discoverer at a time. If you have 1 discovery rule then having 10 discoveres is useless.
CacheSize=128M - check the graphs to adjust to ~99-90% free. (all of the caches)
StartDBSyncers=40 - crazy value. Might cause you lots of problems on DB performance. Each DB syncer is capable of processing ~1000 new values per second. Unless you have 40.000 NVPS, set this back to default 4.

All the same rules apply to Zabbix server configuration as well.

Good luck with tuning!

Best Regards,
Ingus