Hello All,
Post upgrading to 4.0.11, we are seeing lot of issues w.r.to. proxies and also on server, i am not sure how do i debug them, hence seeking forum help.
1). Proxies are sending data every second, where as the configuration is 5s ( basically change is configuration and starting doesn't making any effect).
Zabbix server
2). Same case with configuration frequency received, i have set it to 60s, and then changed to 5m, nothing happenes, i.e newer configurations are taking lot of time to get updated. In on of the proxies, where i had just started VMWare monitoring , though item is deleted at server via dashboard ( and config frequency was 60s). proxy was trying to vmware item monitoring - i had to do multiple restarts to get this cleared out.
3). High zabbix queue - not sure what's changed, but i am seeing high queus,
4). Items becoming un supported. Specially network interface get for windows machines. These use to be working fine in 3.x
5). Housekeeping taking the toll on db. Below are the graphs
Internal process graph before upgrade
Internal server process after upgrade
Server performance, before upgrade,
Zabbix server performance after upgrade
Proxy Internal process busy - before upgrade
Proxy internal process busy after upgrade,
Zabbix server config
MySQL config on zabbix db server,
One of the proxy config, i have 10 proxies running BTW, and all of the them carry similar or lesser number of pollers
Post upgrading to 4.0.11, we are seeing lot of issues w.r.to. proxies and also on server, i am not sure how do i debug them, hence seeking forum help.
1). Proxies are sending data every second, where as the configuration is 5s ( basically change is configuration and starting doesn't making any effect).
Code:
[root@ZABBIX-PROXY ~]# date Sat Sep 21 18:21:46 IST 2019 [root@ZABBIX-PROXY ~]# grep -ir sender /var/log/zabbix/zabbix_proxy.log | tail 48654:20190921:182145.474 End of proxy_data_sender():SUCCEED more:0 flags:0x800a 48654:20190921:182145.474 __zbx_zbx_setproctitle() title:'data sender [sent 38 values in 0.075205 sec, idle 1 sec]' 48654:20190921:182146.474 __zbx_zbx_setproctitle() title:'data sender [sent 38 values in 0.075205 sec, sending data]' 48654:20190921:182146.474 In proxy_data_sender() 48654:20190921:182146.478 End of proxy_data_sender():SUCCEED more:0 flags:0x8000 48654:20190921:182146.479 __zbx_zbx_setproctitle() title:'data sender [sent 0 values in 0.004685 sec, idle 1 sec]' 48654:20190921:182147.479 __zbx_zbx_setproctitle() title:'data sender [sent 0 values in 0.004685 sec, sending data]' 48654:20190921:182147.479 In proxy_data_sender() 48654:20190921:182147.483 End of proxy_data_sender():SUCCEED more:0 flags:0x8000 48654:20190921:182147.483 __zbx_zbx_setproctitle() title:'data sender [sent 0 values in 0.003863 sec, idle 1 sec]' [root@NM-ZABBIX-PROXY ~]# [root@ZABBIX-PROXY ~]# grep DataSender /usr/local/etc/zabbix_proxy.conf ### Option: DataSenderFrequency DataSenderFrequency=5 [root@ZABBIX-PROXY ~]# zabbix_proxy -V zabbix_proxy (Zabbix) 4.0.11 Revision 53bb6bc0f0 29 July 2019, compilation time: Sep 4 2019 17:46:18 Copyright (C) 2019 Zabbix SIA License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it according to the license. There is NO WARRANTY, to the extent permitted by law. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/). Compiled with OpenSSL 1.0.2k-fips 26 Jan 2017 Running with OpenSSL 1.0.2k-fips 26 Jan 2017 [root@ZABBIX-PROXY ~]# [root@ZABBIX-PROXY ~]#
Code:
[root@zabbix ~]# zabbix_server -V zabbix_server (Zabbix) 4.0.11 Revision 53bb6bc0f0 29 July 2019, compilation time: Sep 4 2019 19:36:13 Copyright (C) 2019 Zabbix SIA License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it according to the license. There is NO WARRANTY, to the extent permitted by law. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/). Compiled with OpenSSL 1.0.2k-fips 26 Jan 2017 Running with OpenSSL 1.0.2k-fips 26 Jan 2017 [root@zabbix ~]#
2). Same case with configuration frequency received, i have set it to 60s, and then changed to 5m, nothing happenes, i.e newer configurations are taking lot of time to get updated. In on of the proxies, where i had just started VMWare monitoring , though item is deleted at server via dashboard ( and config frequency was 60s). proxy was trying to vmware item monitoring - i had to do multiple restarts to get this cleared out.
3). High zabbix queue - not sure what's changed, but i am seeing high queus,
4). Items becoming un supported. Specially network interface get for windows machines. These use to be working fine in 3.x
Code:
[root@compute-zbx-proxy ~]# zabbix_get -s XX.XX.XX.XX -k net.if.in["Realtek RTL8139C+ Fast Ethernet NIC-McAfee NDIS Light-Weight Filter-0000",bytes] ZBX_NOTSUPPORTED: Cannot obtain network interface information.
Internal process graph before upgrade
Internal server process after upgrade
Server performance, before upgrade,
Zabbix server performance after upgrade
Proxy Internal process busy - before upgrade
Proxy internal process busy after upgrade,
Zabbix server config
Code:
[root@netmonit ~]# grep -v '#' /usr/local/etc/zabbix_server.conf | sed -e '/^$/d' LogFile=/var/log/zabbix/zabbix_server.log LogFileSize=500 DebugLevel=4 PidFile=/var/run/zabbix/zabbix_server.pid DBHost=172.xx.xx.xx DBName=zabbix DBUser=zabbix DBPassword=zabbix StartPollers=225 StartIPMIPollers=1 StartPollersUnreachable=125 StartTrappers=20 StartPingers=100 StartDiscoverers=5 StartSNMPTrapper=1 ListenIP=0.0.0.0 HousekeepingFrequency=1 MaxHousekeeperDelete=250 CacheSize=1G CacheUpdateFrequency=300 StartDBSyncers=8 HistoryCacheSize=256M HistoryIndexCacheSize=256M TrendCacheSize=1G ValueCacheSize=128M Timeout=30 TrapperTimeout=180 UnreachablePeriod=600 UnavailableDelay=180 AlertScriptsPath=/etc/zabbix/alert.d/ FpingLocation=/usr/sbin/fping Fping6Location=/usr/sbin/fping6 LogSlowQueries=300 StartProxyPollers=2 ProxyDataFrequency=180 [root@netmonit ~]#
Code:
[root@new-db ~]# grep -v '#' /etc/my.cnf | sed -e '/^$/d' [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock server-id=996 skip_name_resolve=ON log-bin=mysql-bin binlog-do-db=zabbix expire_logs_days=5 max_binlog_size=100M innodb_flush_log_at_trx_commit=1 sync_binlog=1 innodb_flush_method=O_DIRECT innodb_log_file_size=1G innodb_buffer_pool_size=80G innodb_file_per_table max_allowed_packet = 256M innodb_thread_concurrency = 8 key_buffer_size = 64M max_connections=750 tmp_table_size = 32M thread_cache_size = 64 query_cache_limit=64M thread_cache_size=512 read_buffer_size=2M symbolic-links=0 log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid [client] user=dbuser password=dbuser [root@netmonit-new-db ~]#
Code:
[root@NM-ZABBIX-PROXY ~]# grep -v '#' /usr/local/etc/zabbix_proxy.conf | sed -e '/^$/d' Server=172.xx.xx.xx Hostname=Zabbix-Proxy LogFile=/var/log/zabbix/zabbix_proxy.log LogFileSize=350 DebugLevel=4 PidFile=/var/run/zabbix/zabbix_proxy.pid DBName=zabbix DBUser=zabbix DBPassword=zabbix ProxyLocalBuffer=3 ProxyOfflineBuffer=4 ConfigFrequency=60 DataSenderFrequency=5 StartPollers=165 StartPollersUnreachable=80 StartTrappers=8 StartPingers=60 StartSNMPTrapper=1 HousekeepingFrequency=3 CacheSize=1G StartDBSyncers=8 HistoryCacheSize=1G HistoryIndexCacheSize=1G Timeout=28 UnreachablePeriod=120 UnavailableDelay=90 UnreachableDelay=60 FpingLocation=/usr/sbin/fping LogSlowQueries=300 [root@NM-ZABBIX-PROXY ~]#
Comment