Zabbix Server after update 2 groups of hosts are unreachable
Hi,
i updated my zabbix server, after the update from 3.4 to 4.0.1 some of the hosts are unreachable, 2 complete groups of hosts, they are available on "hosts" page, but there's the trigger which says they are unreachable, version and configuration of proxies of every group is the same, agent are the same.
I tried on the 2 groups hosts to update to latest proxy and latest agents on some hosts of them, but it didn't solved the problem.
I've already tried to delete hosts on server and recreate them, but it didn't work, I also tried to delete proxy from server and recreate it, but nothing.
This is my configuration and server log. (Sensitive names in logs are hidden with XXX and a number) (The host groups with problem are XXX0 and XXX1).
For the update I stopped server, updated with yum and restarted.
The server is an online VPS called by the local proxies (in active mode, every proxy):
/etc/zabbix/zabbix_server.conf (only uncommented data, all the other parameters are default)
/var/log/zabbix/zabbix_server.log
Here are the graphs in the last hour:
i updated my zabbix server, after the update from 3.4 to 4.0.1 some of the hosts are unreachable, 2 complete groups of hosts, they are available on "hosts" page, but there's the trigger which says they are unreachable, version and configuration of proxies of every group is the same, agent are the same.
I tried on the 2 groups hosts to update to latest proxy and latest agents on some hosts of them, but it didn't solved the problem.
I've already tried to delete hosts on server and recreate them, but it didn't work, I also tried to delete proxy from server and recreate it, but nothing.
This is my configuration and server log. (Sensitive names in logs are hidden with XXX and a number) (The host groups with problem are XXX0 and XXX1).
For the update I stopped server, updated with yum and restarted.
The server is an online VPS called by the local proxies (in active mode, every proxy):
| Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz, 2 cores 1.81 GB RAM |
Code:
LogFile=/var/log/zabbix/zabbix_server.log LogFileSize=0 PidFile=/var/run/zabbix/zabbix_server.pid SocketDir=/var/run/zabbix DBName=zabbix DBUser=zabbix DBPassword=XXXXXXXXXXXXXXX StartPollers=10 StartTrappers=10 StartVMwareCollectors=5 SNMPTrapperFile=/var/log/snmptrap/snmptrap.log CacheSize=128M Timeout=4 AlertScriptsPath=/usr/lib/zabbix/alertscripts ExternalScripts=/usr/lib/zabbix/externalscripts LogSlowQueries=3000
Code:
1534:20181108:145919.446 Starting Zabbix Server. Zabbix 4.0.1 (revision 86073). 1534:20181108:145919.446 ****** Enabled features ****** 1534:20181108:145919.446 SNMP monitoring: YES 1534:20181108:145919.446 IPMI monitoring: YES 1534:20181108:145919.446 Web monitoring: YES 1534:20181108:145919.446 VMware monitoring: YES 1534:20181108:145919.446 SMTP authentication: YES 1534:20181108:145919.446 Jabber notifications: YES 1534:20181108:145919.446 Ez Texting notifications: YES 1534:20181108:145919.446 ODBC: YES 1534:20181108:145919.446 SSH2 support: YES 1534:20181108:145919.446 IPv6 support: YES 1534:20181108:145919.446 TLS support: YES 1534:20181108:145919.446 ****************************** 1534:20181108:145919.446 using configuration file: /etc/zabbix/zabbix_server.conf 1534:20181108:145919.450 current database version (mandatory/optional): 04000000/04000000 1534:20181108:145919.450 required mandatory version: 04000000 1534:20181108:145919.942 server #0 started [main process] 1539:20181108:145919.942 server #1 started [configuration syncer #1] 1540:20181108:145919.943 server #2 started [alerter #1] 1541:20181108:145919.943 server #3 started [alerter #2] 1542:20181108:145919.943 server #4 started [alerter #3] 1543:20181108:145919.944 server #5 started [housekeeper #1] 1544:20181108:145919.944 server #6 started [timer #1] 1548:20181108:145919.948 server #9 started [history syncer #1] 1550:20181108:145919.948 server #11 started [history syncer #3] 1552:20181108:145919.948 server #13 started [escalator #1] 1554:20181108:145919.951 server #15 started [self-monitoring #1] 1556:20181108:145919.951 server #17 started [vmware collector #2] 1559:20181108:145919.952 server #18 started [vmware collector #3] 1560:20181108:145919.952 server #19 started [vmware collector #4] 1561:20181108:145919.953 server #20 started [vmware collector #5] 1546:20181108:145919.953 server #7 started [http poller #1] 1563:20181108:145919.953 server #22 started [poller #1] 1567:20181108:145919.953 server #25 started [poller #4] 1568:20181108:145919.954 server #26 started [poller #5] 1547:20181108:145919.961 server #8 started [discoverer #1] 1569:20181108:145919.962 server #27 started [poller #6] 1549:20181108:145919.963 server #10 started [history syncer #2] 1551:20181108:145919.964 server #12 started [history syncer #4] 1553:20181108:145919.964 server #14 started [proxy poller #1] 1570:20181108:145919.964 server #28 started [poller #7] 1555:20181108:145919.966 server #16 started [vmware collector #1] 1571:20181108:145919.966 server #29 started [poller #8] 1564:20181108:145919.967 server #23 started [poller #2] 1573:20181108:145919.969 server #30 started [poller #9] 1575:20181108:145919.969 server #32 started [unreachable poller #1] 1566:20181108:145919.971 server #24 started [poller #3] 1574:20181108:145919.972 server #31 started [poller #10] 1562:20181108:145919.974 server #21 started [task manager #1] 1576:20181108:145919.974 server #33 started [trapper #1] 1584:20181108:145919.975 server #41 started [trapper #9] 1585:20181108:145919.976 server #42 started [trapper #10] 1586:20181108:145919.979 server #43 started [icmp pinger #1] 1587:20181108:145919.979 server #44 started [alert manager #1] 1579:20181108:145919.979 server #36 started [trapper #4] 1588:20181108:145919.980 server #45 started [preprocessing manager #1] 1590:20181108:145919.980 server #47 started [preprocessing worker #2] 1578:20181108:145919.980 server #35 started [trapper #3] 1577:20181108:145919.981 server #34 started [trapper #2] 1582:20181108:145919.984 server #39 started [trapper #7] 1581:20181108:145919.984 server #38 started [trapper #6] 1583:20181108:145919.986 server #40 started [trapper #8] 1580:20181108:145919.986 server #37 started [trapper #5] 1589:20181108:145920.079 server #46 started [preprocessing worker #1] 1591:20181108:145920.079 server #48 started [preprocessing worker #3] 1550:20181108:145928.036 slow query: 3.629924 sec, "select distinct itemid from trends_uint where clock>=1541682000 and (itemid between 30960 and 30974 or itemid between 31083 and 31094 or itemid between 31140 and 31154 or itemid between 36120 and 36134 or itemid between 36180 and 36194 or itemid between 36240 and 36248 or itemid between 36300 and 36314 or itemid between 36360 and 36368 or itemid between 36370 and 36374 or itemid between 40740 and 40754 or itemid between 41400 and 41414 or itemid between 43923 and 43934 or itemid between 59160 and 59173 or itemid in (30428,30434,30486,30492,30493,30544,30550,30551,30603,30614,30660,30661,30666,30667,31080,34630,34688,34694,34746,34752,34753,34804,34810,34811,36251,41058,42665,42669,42670,42672,43860,43861,43863,43867,43920,43945,43956,49341,49347,49350,49361,49850,51404,51407,59048,59054))" 1548:20181108:145929.451 slow query: 5.162465 sec, "select distinct itemid from trends_uint where clock>=1541682000 and (itemid between 30628 and 30648 or itemid between 30928 and 30948 or itemid between 30988 and 30996 or itemid between 31048 and 31068 or itemid between 31108 and 31120 or itemid between 31123 and 31128 or itemid between 36100 and 36109 or itemid between 36138 and 36153 or itemid between 36158 and 36169 or itemid between 36195 and 36201 or itemid between 36206 and 36229 or itemid between 36315 and 36342 or itemid between 36345 and 36349 or itemid between 36375 and 36393 or itemid between 40755 and 40778 or itemid between 51394 and 51398 or itemid between 51600 and 51607 or itemid in (30399,30405,30406,30435,30515,30521,30522,30526,30555,30573,30579,30580,30584,30587,30588,30615,30616,30916,30975,30976,30999,31095,31096,31155,31156,31168,31169,31170,31171,31176,31177,32203,32261,32267,32268,32308,32447,32497,32500,34659,34665,34666,34695,34717,34723,34724,34728,34815,34833,34839,34840,34844,34847,34848,36395,36396,36397,36405,36406,36919,40706,40713,40715,40723,40724,40725,40726,40728,40729,40783,40784,40812,41415,41429,41440,41451,41462,41473,41484,41495,41981,41987,41988,42087,42090,42629,42631,42635,42645,42646,42676,42692,42694,42696,42697,42699,42702,42703,42704,42706,42709,43887,43888,43890,43893,43905,43935,43943,43944,45339,45340,49837,49848,49849,49882,49884,49886,51592,51594,51595,56299,57013,59055))" 1548:20181108:145933.290 slow query: 3.838540 sec, "select itemid,num,value_min,value_avg,value_max from trends_uint where clock=1541682000 and (itemid between 30628 and 30648 or itemid between 30928 and 30948 or itemid between 30988 and 30996 or itemid between 31048 and 31068 or itemid between 31108 and 31120 or itemid between 31123 and 31128 or itemid between 36100 and 36109 or itemid between 36138 and 36153 or itemid between 36158 and 36169 or itemid between 36195 and 36201 or itemid between 36206 and 36229 or itemid between 36315 and 36342 or itemid between 36345 and 36349 or itemid between 36375 and 36393 or itemid between 40755 and 40778 or itemid between 51394 and 51398 or itemid between 51600 and 51607 or itemid in (30399,30405,30406,30435,30515,30521,30522,30526,30555,30573,30579,30580,30584,30587,30588,30615,30616,30916,30975,30976,30999,31095,31096,31155,31156,31168,31169,31170,31171,31176,31177,32203,32261,32267,32268,32308,32447,32497,32500,34659,34665,34666,34695,34717,34723,34724,34728,34815,34833,34839,34840,34844,34847,34848,36395,36396,36397,36405,36406,36919,40706,40713,40715,40723,40724,40725,40726,40728,40729,40783,40784,40812,41415,41429,41440,41451,41462,41473,41484,41495,41981,41987,41988,42087,42090,42629,42631,42635,42645,42646,42676,42692,42694,42696,42697,42699,42702,42703,42704,42706,42709,43887,43888,43890,43893,43905,43935,43943,43944,45339,45340,49837,49848,49849,49882,49884,49886,51592,51594,51595,56299,57013,59055))" 1579:20181108:145938.992 sending configuration data to proxy "Proxy-XXX0" at "XXX.XXX.XXX.XXX", datalen 274142 1549:20181108:150025.731 slow query: 6.832460 sec, "select clock,ns,value from history_uint where itemid=44862 and clock>1541080817 and clock<=1541675057" 1583:20181108:150349.579 sending configuration data to proxy "Proxy-XXX8" at "XXX.XXX.XXX.XXX", datalen 150496 1580:20181108:150349.793 sending configuration data to proxy "Proxy-XXX9" at "XXX.XXX.XXX.XXX", datalen 130413 1582:20181108:150355.152 sending configuration data to proxy "Proxy-XXX10" at "XXX.XXX.XXX.XXX", datalen 140822 1582:20181108:150356.066 sending configuration data to proxy "Proxy-XXX11" at "XXX.XXX.XXX.XXX", datalen 94952 1579:20181108:150409.771 sending configuration data to proxy "Proxy-XXX12" at "XXX.XXX.XXX.XXX", datalen 23706 1582:20181108:150438.778 sending configuration data to proxy "Proxy-XXX13" at "XXX.XXX.XXX.XXX", datalen 342796 1581:20181108:150503.509 sending configuration data to proxy "Proxy-XXX14" at "XXX.XXX.XXX.XXX", datalen 121276 1584:20181108:150508.816 sending configuration data to proxy "Proxy-XXX15" at "XXX.XXX.XXX.XXX", datalen 130215 1582:20181108:150513.809 sending configuration data to proxy "Proxy-XXX16" at "XXX.XXX.XXX.XXX", datalen 63138 1582:20181108:150529.086 sending configuration data to proxy "Proxy-XXX17" at "XXX.XXX.XXX.XXX", datalen 40967 1552:20181108:150840.203 escalation cancelled: trigger id:25509 deleted. 1552:20181108:150846.211 escalation cancelled: trigger id:25545 deleted. 1552:20181108:150849.214 escalation cancelled: trigger id:25521 deleted. 1552:20181108:150855.220 escalation cancelled: trigger id:25343 deleted. 1552:20181108:150904.251 escalation cancelled: trigger id:25533 deleted. 1577:20181108:150920.370 sending configuration data to proxy "Proxy-XXX1" at "XXX.XXX.XXX.XXX", datalen 183144 1576:20181108:151349.255 sending configuration data to proxy "Proxy-XXX1" at "XXX.XXX.XXX.XXX", datalen 187544 1583:20181108:151513.300 sending configuration data to proxy "Proxy-XXX2" at "XXX.XXX.XXX.XXX", datalen 123038 1549:20181108:151631.462 item "XX1_NASSynology:syno.raid.freesizeinperc.[0]" became not supported: Cannot evaluate expression: "Cannot evaluate function "last()": not enough data.". 1552:20181108:151816.501 escalation cancelled: trigger id:28367 deleted. 1585:20181108:151843.295 sending configuration data to proxy "Proxy-XXX1" at "XXX.XXX.XXX.XXX", datalen 210064 1581:20181108:151918.599 Message from XXX.XXX.XXX.XXX is missing header. Message ignored. 1579:20181108:151931.926 sending configuration data to proxy "Proxy-XXX3" at "XXX.XXX.XXX.XXX", datalen 133707 1585:20181108:152008.232 sending configuration data to proxy "Proxy-XXX4" at "XXX.XXX.XXX.XXX", datalen 214601 1580:20181108:152110.132 sending configuration data to proxy "Proxy-XXX3" at "XXX.XXX.XXX.XXX", datalen 395075 1551:20181108:152832.320 item "XXX1_NASSynology:syno.raid.freesizeinperc.[0]" became supported 1543:20181108:152920.295 executing housekeeper 1584:20181108:152920.836 sending configuration data to proxy "XXX0" at "XXX.XXX.XXX.XXX", datalen 274142 1543:20181108:152927.710 slow query: 6.145140 sec, "select itemid,min(clock) from history_uint group by itemid" 1543:20181108:152931.701 slow query: 3.181798 sec, "select itemid,min(clock) from trends_uint group by itemid" 1578:20181108:153004.388 sending configuration data to proxy "Proxy-XXX4" at "XXX.XXX.XXX.XXX", datalen 84321 1543:20181108:153017.720 slow query: 5.423461 sec, "delete from history where itemid=52626 limit 5000" 1543:20181108:153022.850 slow query: 5.127012 sec, "delete from history where itemid=52662 limit 5000" 1543:20181108:153034.590 slow query: 4.742837 sec, "delete from history where itemid=52934 limit 5000" 1543:20181108:153038.972 slow query: 3.139061 sec, "delete from history where itemid=52727 limit 5000" 1543:20181108:153124.131 slow query: 4.423885 sec, "delete from history_uint where itemid=52927 limit 5000" 1543:20181108:153130.298 slow query: 3.273085 sec, "delete from history_uint where itemid=58041 limit 5000" 1543:20181108:153134.629 slow query: 4.324117 sec, "delete from history_uint where itemid=52917 limit 5000" 1543:20181108:153137.850 slow query: 3.218449 sec, "delete from history_uint where itemid=52658 limit 5000" 1543:20181108:153142.155 slow query: 4.303185 sec, "delete from history_uint where itemid=58044 limit 5000" 1543:20181108:153145.470 slow query: 3.314394 sec, "delete from history_uint where itemid=52611 limit 5000" 1543:20181108:153149.001 slow query: 3.530451 sec, "delete from history_uint where itemid=52886 limit 5000" 1543:20181108:153154.906 slow query: 5.901696 sec, "delete from history_uint where itemid=52926 limit 5000" 1543:20181108:153158.449 slow query: 3.542472 sec, "delete from history_uint where itemid=52663 limit 5000" 1584:20181108:153202.526 sending configuration data to proxy "Proxy-XXX5" at "XXX.XXX.XXX.XXX", datalen 100493 1543:20181108:153203.408 slow query: 4.956040 sec, "delete from history_uint where itemid=52919 limit 5000" 1543:20181108:153208.431 slow query: 5.022382 sec, "delete from history_uint where itemid=52634 limit 5000" 1543:20181108:153218.328 slow query: 5.262063 sec, "delete from history_uint where itemid=52600 limit 5000" 1543:20181108:153222.332 slow query: 4.004057 sec, "delete from history_uint where itemid=52666 limit 5000" 1543:20181108:153227.946 slow query: 4.242888 sec, "delete from history_uint where itemid=52887 limit 5000" 1543:20181108:153232.278 slow query: 4.331774 sec, "delete from history_uint where itemid=52929 limit 5000" 1543:20181108:153241.659 slow query: 4.573054 sec, "delete from history_uint where itemid=52607 limit 5000" 1543:20181108:153245.439 slow query: 3.779799 sec, "delete from history_uint where itemid=52637 limit 5000" 1543:20181108:153254.740 slow query: 4.259253 sec, "delete from history_uint where itemid=52918 limit 5000" 1543:20181108:153259.174 slow query: 4.433040 sec, "delete from history_uint where itemid=52928 limit 5000" 1543:20181108:153303.020 slow query: 3.642756 sec, "delete from history_uint where itemid=52254 limit 5000" 1543:20181108:153309.262 slow query: 3.350287 sec, "delete from history_uint where itemid=52659 limit 5000" 1543:20181108:153316.783 slow query: 4.783889 sec, "delete from history_uint where itemid=52922 limit 5000" 1543:20181108:153322.436 slow query: 5.652797 sec, "delete from history_uint where itemid=52791 limit 5000" 1580:20181108:153324.683 sending configuration data to proxy "Proxy-XXX0" at "XXX.XXX.XXX.XXX", datalen 297287 1543:20181108:153330.894 slow query: 5.011620 sec, "delete from history_uint where itemid=52793 limit 5000" 1543:20181108:153337.068 slow query: 6.074304 sec, "delete from history_uint where itemid=52251 limit 5000" 1543:20181108:153343.955 slow query: 4.217305 sec, "delete from history_uint where itemid=52240 limit 5000" 1543:20181108:153348.088 slow query: 4.114544 sec, "delete from history_uint where itemid=52920 limit 5000" 1543:20181108:153351.572 slow query: 3.483200 sec, "delete from history_uint where itemid=52925 limit 5000" 1543:20181108:153356.037 slow query: 4.432989 sec, "delete from history_uint where itemid=52255 limit 5000" 1543:20181108:153359.097 slow query: 3.051513 sec, "delete from history_uint where itemid=52924 limit 5000" 1543:20181108:153403.729 slow query: 4.244115 sec, "delete from history_uint where itemid=53771 limit 5000" 1543:20181108:153408.280 slow query: 4.351539 sec, "delete from history_uint where itemid=52747 limit 5000" 1543:20181108:153413.348 slow query: 4.844688 sec, "delete from history_uint where itemid=52732 limit 5000" 1543:20181108:153417.590 slow query: 3.823012 sec, "delete from history_uint where itemid=52738 limit 5000" 1543:20181108:153421.231 slow query: 3.581991 sec, "delete from history_uint where itemid=52729 limit 5000" 1543:20181108:153425.999 slow query: 4.761925 sec, "delete from history_uint where itemid=52948 limit 5000" 1543:20181108:153438.862 slow query: 4.171129 sec, "delete from history_uint where itemid=52721 limit 5000" 1543:20181108:153442.598 slow query: 3.732901 sec, "delete from history_uint where itemid=52953 limit 5000" 1543:20181108:153449.644 slow query: 4.264514 sec, "delete from history_uint where itemid=52728 limit 5000" 1543:20181108:153457.943 slow query: 5.759012 sec, "delete from history_uint where itemid=52941 limit 5000" 1543:20181108:153501.642 slow query: 3.699273 sec, "delete from history_uint where itemid=52943 limit 5000" 1543:20181108:153505.486 slow query: 3.642924 sec, "delete from history_uint where itemid=52742 limit 5000" 1543:20181108:153509.054 slow query: 3.432503 sec, "delete from history_uint where itemid=52731 limit 5000" 1581:20181108:153518.032 sending configuration data to proxy "Proxy-XXX6" at "XXX.XXX.XXX.XXX", datalen 113880 1543:20181108:153523.718 slow query: 4.012468 sec, "delete from history_uint where itemid=52725 limit 5000" 1543:20181108:153529.166 slow query: 3.433823 sec, "delete from history_uint where itemid=52946 limit 5000" 1543:20181108:153535.936 slow query: 4.580294 sec, "delete from history_uint where itemid=53770 limit 5000" 1543:20181108:153541.860 slow query: 3.232386 sec, "delete from history_uint where itemid=52739 limit 5000" 1576:20181108:153545.921 sending configuration data to proxy "Proxy-XXX7" at "XXX.XXX.XXX.XXX", datalen 41098 1543:20181108:153546.653 slow query: 4.792057 sec, "delete from history_uint where itemid=52743 limit 5000" 1543:20181108:153552.102 slow query: 5.315515 sec, "delete from history_uint where itemid=52724 limit 5000" 1543:20181108:153556.249 slow query: 4.138540 sec, "delete from history_uint where itemid=52730 limit 5000" 1585:20181108:153558.546 sending configuration data to proxy "Proxy-XXX8" at "XXX.XXX.XXX.XXX", datalen 143909 1543:20181108:153601.618 slow query: 5.369442 sec, "delete from history_uint where itemid=52945 limit 5000" 1543:20181108:153606.027 slow query: 4.371158 sec, "delete from history_uint where itemid=53769 limit 5000" 1543:20181108:153609.282 slow query: 3.254914 sec, "delete from history_uint where itemid=52744 limit 5000" 1543:20181108:153613.819 slow query: 4.504190 sec, "delete from history_uint where itemid=52629 limit 5000" 1543:20181108:153619.254 slow query: 3.394871 sec, "delete from history_uint where itemid=52595 limit 5000" 1543:20181108:153623.490 slow query: 4.226176 sec, "delete from history_uint where itemid=52667 limit 5000" 1543:20181108:153632.022 slow query: 4.339933 sec, "delete from history_uint where itemid=52735 limit 5000" 1543:20181108:153637.701 slow query: 3.365671 sec, "delete from history_uint where itemid=52652 limit 5000" 1543:20181108:153641.354 slow query: 3.643753 sec, "delete from history_uint where itemid=52596 limit 5000" 1543:20181108:153645.233 slow query: 3.878974 sec, "delete from history_uint where itemid=52630 limit 5000" 1543:20181108:153649.605 slow query: 4.227379 sec, "delete from history_uint where itemid=53737 limit 5000" 1543:20181108:153654.923 slow query: 5.165721 sec, "delete from history_uint where itemid=52599 limit 5000" 1543:20181108:153702.436 slow query: 4.328566 sec, "delete from history_uint where itemid=52734 limit 5000" 1543:20181108:153708.358 slow query: 3.399818 sec, "delete from history_uint where itemid=53728 limit 5000" 1543:20181108:153720.300 slow query: 6.038584 sec, "delete from history_uint where itemid=52623 limit 5000" 1543:20181108:153727.200 slow query: 4.848048 sec, "delete from history_uint where itemid=52933 limit 5000" 1543:20181108:153734.121 slow query: 3.203812 sec, "delete from history_uint where itemid=52932 limit 5000" 1543:20181108:153738.734 slow query: 4.350720 sec, "delete from history_uint where itemid=52710 limit 5000" 1543:20181108:153747.802 housekeeper [deleted 334935 hist/trends, 1090901 items/triggers, 4 events, 1445 problems, 0 sessions, 0 alarms, 0 audit items in 507.504447 sec, idle for 1 hour(s)] 1582:20181108:153802.878 sending configuration data to proxy "Proxy-XXX0" at "XXX.XXX.XXX.XXX", datalen 297287 1550:20181108:154950.784 item "XXX0_ServerDB:service.discovery" became not supported: Unsupported item key.