Zabbix4 Delayed Item Checks

[stevereaver]

I have Googled the heck out of this problem and I am at my wits end so hopping to get some advice here.

I have just built and prepared a Zabbix4 system using Postgres on a brad new Dell Server (not a VM). It is running on Ubuntu 18.04.01 with ZFS root, 16gb RAM, 2TB RAID10 storage. The performance of the machine is fantastic. Network, Disk and DB performance are all well below 10% utilisation, with plenty of headroom. I have built this system with the intention of replacing our ageing Zabbix 2.2 sever.

In order to facilitate the cutover from Zabbix2.2 to zabbix4 I currently have 2 proxy servers (one for each version of zabbix) and there are 10 agents that connect through these proxies, these agents are version 2.2.14 and are configured to send their data to both proxies. Here is the problem, the agents that feed the zabbix2 proxy/server are working perfectly and in realtime, however the same agents that feed the zabbix4 proxy/server are always delayed (sometime up to 4 or 5 hours!), this results in some data being occasionally missed and dropped (ie: holes in graphs)

I have checked the following;
- Disk I/O, there is next to no disk activity
- Network I/O, also very low
- DB, monitoring with PGADMIN shows around only 50 transactions per second, with occasional spike to 600 or so.
- Have checked the time is synced across the agent, proxy and server (although the proxy and agents use a different NTP source to the server, this is unavoidable as the agents are on customer premises)
- There are no Unsupported Items in the list of monitored items.
- The zabbix server is set for 50 pollers, 10 trappers, 20 pingers, 30 http pollers
- The zabbix data gathering processes show only about 15% busy, see attached image. (we have around 12000 items all up), similar graphs for the proxy show no discernible problems.

Some trouble shooting I have done;
- Tried the proxy in active and passive mode, the problem persists in either mode.
- Turned the number of pollers and trappers up on both the server and proxy, no difference.
- Increased the interval for many of the checks.
- Removed all but one agent from the proxy, however even with only a small amount of items they still get delayed.

So the question is what is the difference between the working proxy and the non working one ??? Well obviously there is a zabbix version difference, as I do not see the same delayed data on zabbix2. However, the data coming from the zabbix4 proxy is coming via a VPN, and the zabbix2 proxy is direct linked. But I also have other Zabbix4 proxies working through a VPN just fine.

Some other important things you will need to know. The zabbix4 server is currently 4.0.0rc3 on Ubuntu 18.04.01, the affected zabbix proxy is 4.0.0 and it is running on Solaris 11.3, also the agents are Zabbix 2.2.14 and are running on Solaris 11.3.

I'm starting to think I have struck some sort of bug, so any assistance is greatly appreciated.

[stevereaver]

To put a little more light on this I have enabled full debug logging in zabbix server. I can see that during the period where there is a hole in the grap data I get the following from the proxy.

15052:20181015:153200.617 recv_getqueue() json.buffer:'{"response":"success","data":[{"itemid":30133,"nextcheck":1539570133},{"itemid": 30134,"nextcheck":1539570134},{"itemid":30135,"nex tcheck":1539570135},{"itemid":23292
,"nextcheck":1539574092},{"itemid":23293,"nextchec k":1539574093},{"itemid":23294,"nextcheck":1539574 094},{"itemid":23295,"nextcheck":1539574095},{"ite mid":23299,"nextcheck":1539574099},{"itemid":49125 ,"nextcheck":15395747
25},{"itemid":30142,"nextcheck":1539574942},{"item id":30143,"nextcheck":1539574943},{"itemid":30144, "nextcheck":1539574944},{"itemid":30145,"nextcheck ":1539574945},{"itemid":30146,"nextcheck":15395749 46},{"itemid":30147,"
nextcheck":1539574947},{"itemid":30148,"nextcheck" :1539574948},{"itemid":22984,"nextcheck":153957498 4},{"itemid":22985,"nextcheck":1539574985},{"itemi d":23313,"nextcheck":1539575313},{"itemid":23314," nextcheck":1539575314
},{"itemid":23315,"nextcheck":1539575315},{"itemid ":23316,"nextcheck":1539575316},{"itemid":23317,"n extcheck":1539575317},{"itemid":23318,"nextcheck": 1539575318},{"itemid":23319,"nextcheck":1539575319 },{"itemid":28942,"ne
xtcheck":1539576142},{"itemid":22959,"nextcheck":1 539576159},{"itemid":22960,"nextcheck":1539576160} ,{"itemid":23309,"nextcheck":1539576509},{"itemid" :23310,"nextcheck":1539576510},{"itemid":23311,"ne xtcheck":1539576511},
{"itemid":23312,"nextcheck":1539576512},{"itemid": 50026,"nextcheck":1539576826},{"itemid":50048,"nex tcheck":1539576848},{"itemid":50049,"nextcheck":15 39576849},{"itemid":50050,"nextcheck":1539576850}, {"itemid":50051,"next
check":1539576851},{"itemid":50052,"nextcheck":153 9576852},{"itemid":50053,"nextcheck":1539576853},{ "itemid":46458,"nextcheck":1539576858},{"itemid":4 6459,"nextcheck":1539576859},{"itemid":25525,"next check":1539576925},{"
itemid":31072,"nextcheck":1539577072},{"itemid":34 728,"nextcheck":1539577128},{"itemid":34729,"nextc heck":1539577129},{"itemid":34730,"nextcheck":1539 577130},{"itemid":34731,"nextcheck":1539577131},{" itemid":34732,"nextch
eck":1539577132},{"itemid":25516,"nextcheck":15395 77276},{"itemid":26957,"nextcheck":1539577277},{"i temid":25517,"nextcheck":1539577277},{"itemid":255 18,"nextcheck":1539577278},{"itemid":25519,"nextch eck":1539577279},{"it
emid":25520,"nextcheck":1539577280},{"itemid":2552 1,"nextcheck":1539577281},{"itemid":22881,"nextche ck":1539577281},{"itemid":26962,"nextcheck":153957 7282},{"itemid":25522,"nextcheck":1539577282},{"it emid":25523,"nextchec
k":1539577283},{"itemid":30113,"nextcheck":1539577 313},{"itemid":30140,"nextcheck":1539577340},{"ite mid":30141,"nextcheck":1539577341},{"itemid":30149 ,"nextcheck":1539577349},{"itemid":30150,"nextchec k":1539577350},{"item
id":30151,"nextcheck":1539577351},{"itemid":30152, "nextcheck":1539577352},{"itemid":30153,"nextcheck ":1539577353},{"itemid":30154,"nextcheck":15395773 54},{"itemid":22968,"nextcheck":1539577368},{"item id":22969,"nextcheck"
:1539577369},{"itemid":31070,"nextcheck":153957737 0},{"itemid":22970,"nextcheck":1539577370},{"itemi d":22971,"nextcheck":1539577371},{"itemid":22972," nextcheck":1539577372},{"itemid":22973,"nextcheck" :1539577373},{"itemid
":22974,"nextcheck":1539577374},{"itemid":22975,"n extcheck":1539577375},{"itemid":22976,"nextcheck": 1539577376},{"itemid":22977,"nextcheck":1539577377 },{"itemid":23277,"nextcheck":1539577377},{"itemid ":22978,"nextcheck":1
539577378},{"itemid":22979,"nextcheck":1539577379} ,{"itemid":22980,"nextcheck":1539577380},{"itemid" :22981,"nextcheck":1539577381},{"itemid":22982,"ne xtcheck":1539577382},{"itemid":22983,"nextcheck":1 539577383},{"itemid":
25524,"nextcheck":1539577524},{"itemid":25526,"nex tcheck":1539577526},{"itemid":26967,"nextcheck":15 39577527},{"itemid":25527,"nextcheck":1539577527}, {"itemid":25528,"nextcheck":1539577528},{"itemid": 23365,"nextcheck":153
9577765},{"itemid":23366,"nextcheck":1539577766},{ "itemid":26969,"nextcheck":1539577769},{"itemid":2 5529,"nextcheck":1539577769},{"itemid":25530,"next check":1539577770},{"itemid":25531,"nextcheck":153 9577771},{"itemid":26
971,"nextcheck":1539577771},{"itemid":25532,"nextc heck":1539577772},{"itemid":25533,"nextcheck":1539 577773},{"itemid":25534,"nextcheck":1539577774},{" itemid":40620,"nextcheck":1539577800},{"itemid":33 840,"nextcheck":15395
77800},{"itemid":33841,"nextcheck":1539577801},{"i temid":26941,"nextcheck":1539577801},{"itemid":406 21,"nextcheck":1539577801},{"itemid":26942,"nextch eck":1539577802},{"itemid":28622,"nextcheck":15395 77802},{"itemid":4062
2,"nextcheck":1539577802},{"itemid":33842,"nextche ck":1539577802},{"itemid":40623,"nextcheck":153957 7803},{"itemid":40624,"nextcheck":1539577804},{"it emid":33844,"nextcheck":1539577804},{"itemid":4999 0,"nextcheck":1539577
810},{"itemid":49991,"nextcheck":1539577811},{"ite mid":37716,"nextcheck":1539577866},{"itemid":30906 ,"nextcheck":1539577866},{"itemid":30876,"nextchec k":1539577866},{"itemid":37717,"nextcheck":1539577 867},{"itemid":37718,
"nextcheck":1539577868},{"itemid":37719,"nextcheck ":1539577869},{"itemid":37720,"nextcheck":15395778 70},{"itemid":37721,"nextcheck":1539577871},{"item id":56771,"nextcheck":1539577871},{"itemid":37722, "nextcheck":153957787
2},{"itemid":37723,"nextcheck":1539577873},{"itemi d":30796,"nextcheck":1539577876},{"itemid":30886," nextcheck":1539577876},{"itemid":31036,"nextcheck" :1539577876},{"itemid":31037,"nextcheck":153957787 7},{"itemid":30797,"n
extcheck":1539577877},{"itemid":30828,"nextcheck": 1539577878},{"itemid":30881,"nextcheck":1539577879 },{"itemid":30800,"nextcheck":1539577880},{"itemid ":30882,"nextcheck":1539577880},{"itemid":37747,"n extcheck":1539577897}
,{"itemid":30178,"nextcheck":1539577908},{"itemid" :30108,"nextcheck":1539577908},{"itemid":31668,"ne xtcheck":1539577908},{"itemid":56158,"nextcheck":1 539577908},{"itemid":40608,"nextcheck":1539577908} ,{"itemid":30169,"nex
tcheck":1539577909},{"itemid":40609,"nextcheck":15 39577909},{"itemid":31670,"nextcheck":1539577910}, {"itemid":30200,"nextcheck":1539577910},{"itemid": 40610,"nextcheck":1539577910},{"itemid":56151,"nex tcheck":1539577911},{
"itemid":40611,"nextcheck":1539577911},{"itemid":3 0201,"nextcheck":1539577911},{"itemid":30171,"next check":1539577911},{"itemid":31672,"nextcheck":153 9577912},{"itemid":37732,"nextcheck":1539577912},{ "itemid":30202,"nextc
heck":1539577912},{"itemid":30203,"nextcheck":1539 577913},{"itemid":37733,"nextcheck":1539577913},{" itemid":30179,"nextcheck":1539577913},{"itemid":30 173,"nextcheck":1539577913},{"itemid":30180,"nextc heck":1539577914},{"i
temid":30174,"nextcheck":1539577914},{"itemid":316 74,"nextcheck":1539577914},{"itemid":30204,"nextch eck":1539577914}],"total":161}'

The item id I am interested in is item 25525 and it appears that the proxy simply does not send it.

[stevereaver]

Additional graph showing proxy usage.

[thesubmitter]

Have you adjusted all the timeouts?

[d0n0van]

I can confirm the issue, adjusted all timeouts to, nothing resolved the issue, all on zabbix 4.0.1, will upgrade to 4.0.2 tomorrow

[d0n0van]

Upgrade to zabbix-proxy-mysql 4.0.2 did not resolve the issue.

This proxy needs to monitor 16 hosts @rate 16VPS

Queue length avg of 300 items and several items above 5 & 10 minutes.

Before upgrade never any issue with a avg queue length of 12 items.

ntp is OK on all servers.



My current proxy-config

class { 'mysql::server': }

class { 'zabbix::proxy':
  zabbix_server_host    => $ZBX_SERVER_ADDR,
  database_type         => 'mysql',
  database_socket       => '/var/lib/mysql/mysql.sock',
  manage_repo           => false,
  tlsaccept             => 'psk',
  tlsconnect            => 'psk',
  tlspskfile            => '/etc/zabbix/zabbix_agentd.psk',
  tlspskidentity        => 'BEDEF',
  startpingers          => 10,
  timeout               => 30,
  historycachesize      => '64M',
  cachesize             => '64M',
}

No performance issues at proxy-server or mysql.
No performance issues at zabbix-server.

[d0n0van]

#setsebool zabbix_run_sudo 1

on all clients resolved the issue.
No real selinux alerts found
but in /var/log/secure

pam_systemd(sudo:session): Failed to create session: Connection timed out

[vso]

What kind of item is that and what kind of Zabbix proxy, passive or active ?

[stevereaver]

#setsebool zabbix_run_sudo 1

on all clients resolved the issue.
No real selinux alerts found
but in /var/log/secure

pam_systemd(sudo:session): Failed to create session: Connection timed out

Unfortunately does not help me in my case as the clients are all Solaris. selinux is a RedHat thing.