Critical: After upgrade zabbix proxy isn't updating configs from server
Hello All,
Recently i had upgraded zabbix instances from 2.0.6 to 2.2.12. Upgrade went smoother, no issues at all, followed the guide and all was fine.
Now there seems to be a strange behaviour from existing zabbix proxy instances. It's just not updating the configs received by the server. Its not even adding new configs. I can see in the server logs that configs are sent via json and i can see that proxy recieves these configs, but in the json array 'sending data to server' it still uses old enters to send.
For example:
- i have a n/w devices ( example router) - which had configured with interface port6 to be monitored, so oid for incoming traffic looks like
considering 515 is my SNMP Index number. Now for some reason i wanted it to changed to port7, so SNMP index becomes 516, and so is OID. When i change this from the Zabbix Server UI, it changes and sits in db. And changed config gets sent using
also changed config gets received on the proxy under
- But i fail to understand why config changes are not getting applied?
- In proxy db, i still see older entries and they are gone in server.
- As proxy still has older entries, it still sends data using older index ( as that index does not exists since port changed, it sends no such instance exists in this OID , which is ok)
- Even if i delete that interface from the template, it immediately reflects in server , but its stays in proxy.
- When i add new item entry in template, it reflects in server, but proxy still has entry which got deleted in server, and misses new entry which added in server ( though config is sent by server/received by proxy).
- Untouched configs from server is working via proxy though. That means, proxy seems to be working, but its failing to update the configs passed from server thus monitoring is gone for toss
This is critical as when i update something in server its not getting reflected in proxy ( so is delete / addition of new items/templates) etc - which tosses up existing monitoring environment.
Please help me. Throw me some pointers.
Here are the details.
- Server :
Server config zabbix_server.conf:
Server MySql config:
Proxy config:
Proxy config zabbix_proxy.conf
In Administration ->DM -> Proxies
In Administration->Queue
Please let me know if anything is missing / needed. Any pointers are greatly appreciated.
Thanks.
Recently i had upgraded zabbix instances from 2.0.6 to 2.2.12. Upgrade went smoother, no issues at all, followed the guide and all was fine.
Now there seems to be a strange behaviour from existing zabbix proxy instances. It's just not updating the configs received by the server. Its not even adding new configs. I can see in the server logs that configs are sent via json and i can see that proxy recieves these configs, but in the json array 'sending data to server' it still uses old enters to send.
For example:
- i have a n/w devices ( example router) - which had configured with interface port6 to be monitored, so oid for incoming traffic looks like
Code:
.1.3.6.1.2.1.2.2.1.10.515
Code:
18128:20160510:174112.574 sending configuration data to proxy "zbx-proxy2", datalen 1190436
Code:
12333:20160510:174011.893 Received [{
"globalmacro":{
- In proxy db, i still see older entries and they are gone in server.
- As proxy still has older entries, it still sends data using older index ( as that index does not exists since port changed, it sends no such instance exists in this OID , which is ok)
- Even if i delete that interface from the template, it immediately reflects in server , but its stays in proxy.
- When i add new item entry in template, it reflects in server, but proxy still has entry which got deleted in server, and misses new entry which added in server ( though config is sent by server/received by proxy).
- Untouched configs from server is working via proxy though. That means, proxy seems to be working, but its failing to update the configs passed from server thus monitoring is gone for toss
This is critical as when i update something in server its not getting reflected in proxy ( so is delete / addition of new items/templates) etc - which tosses up existing monitoring environment.
Please help me. Throw me some pointers.
Here are the details.
- Server :
Code:
2.6.32-279.el6.x86_64 , CentOS 6.3, 64G, 24Core MySql: 5.1.73-log, httpd : Apache/2.2.15
Code:
LogFile=/var/log/zabbix/zabbix_server.log LogFileSize=500 DebugLevel=4 PidFile=/var/run/zabbix/zabbix_server.pid DBName=zabbix DBUser=zabbix DBPassword=zabbix StartPollers=130 StartPollersUnreachable=45 StartTrappers=60 StartPingers=90 StartDiscoverers=1 StartSNMPTrapper=1 ListenIP=0.0.0.0 HousekeepingFrequency=2 MaxHousekeeperDelete=300 SenderFrequency=360 CacheSize=1G CacheUpdateFrequency=300 StartDBSyncers=13 HistoryCacheSize=256M TrendCacheSize=1G HistoryTextCacheSize=256M Timeout=30 TrapperTimeout=180 UnreachablePeriod=600 UnavailableDelay=120 AlertScriptsPath=/etc/zabbix/alert.d/ FpingLocation=/usr/local/sbin/fping LogSlowQueries=5 ProxyDataFrequency=180
Code:
[mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql symbolic-links=0 slow_query_log_file=/var/log/mysql/slow-query.log long_query_time = 10 log-queries-not-using-indexes=YES innodb_lock_wait_timeout=500 innodb_locks_unsafe_for_binlog=1 expire_logs_days=5 max_binlog_size=100M innodb_file_per_table innodb_flush_method=O_DIRECT innodb_log_file_size=1G innodb_buffer_pool_size=24G innodb_file_per_table max_allowed_packet = 64M innodb_additional_mem_pool_size = 30M innodb_thread_concurrency = 8 key_buffer_size = 60M max_connections=375 table_cache=4096 query_cache_size = 96M tmp_table_size = 32M thread_cache_size = 64 sort_buffer_size = 12M query_cache_limit=64M thread_cache_size=512 read_buffer_size=2M read_rnd_buffer_size=8M join_buffer_size=8M log-bin=/var/log/mysql/mysql-bin.log binlog-do-db=zabbix server-id=3 [mysqld_safe] log-error=/var/log/mysql/mysqld.log pid-file=/var/run/mysqld/mysqld.pid
Code:
2.6.32-279.el6.x86_64 , CentOS 6.3, 10G Ram, 6core CPU MySqld: 5.1.73
Code:
Server=netmon.internal.com Hostname=zbx-proxy2 LogFile=/var/log/zabbix/zabbix_proxy.log LogFileSize=325 DebugLevel=4 PidFile=/var/run/zabbix/zabbix_proxy.pid DBName=zabbix DBUser=zabbix DBPassword=zabbix ProxyLocalBuffer=4 ProxyOfflineBuffer=12 ConfigFrequency=60 DataSenderFrequency=10 StartPollers=182 StartPollersUnreachable=75 StartTrappers=45 StartPingers=95 StartSNMPTrapper=1 CacheSize=1G StartDBSyncers=45 HistoryCacheSize=1G HistoryTextCacheSize=256M Timeout=30 UnreachablePeriod=75 FpingLocation=/usr/sbin/fping Fping6Location=/usr/sbin/fping6 LogSlowQueries=3000
Code:
Name Mode LastSeen HostCount ItemCount Required performance (vps) zbx-proxy2 Active 10s 976 4781 21.05
Code:
Proxy 5 seconds 10 seconds 30 seconds 1 minute 5 minutes More than 10 minutes zbx-proxy2 75 115 69 366 2 25
Thanks.
Comment