The zabbix queue 10m - 500000 item

[BayramGrbz45]

Hello.
Zabbix tail rises after a certain time.
Every 24 hours a queue builds up on my zabbix server and my zabbix proxies and data retrieval stop.
Zabbix server, proxy, postgresql logs are clean.
When I look at the monitoring screens, I can't find any significant results.

https://gist.github.com/BayramGrbz45...2bafbf6afffde5
https://gist.github.com/BayramGrbz45...8989463f874308

771:20221023:103234.152 slow query: 4.998958 sec, "select i.itemid,i.hostid,i.status,i.type,i.value_type,i.k ey_,i.snmp_oid,i.ipmi_sensor,i.delay,i.trapper_hos ts,i.logtimefmt,i.params,ir.state,i.authtype,i.use rname,i.password,i.publickey,i.privatekey,i.flags, i.interfaceid,ir.lastlogsize,ir.mtime,i.history,i. trends,i.inventory_link,i.valuemapid,i.units,ir.er ror,i.jmx_endpoint,i.master_itemid,i.timeout,i.url ,i.query_fields,i.posts,i.status_codes,i.follow_re directs,i.post_type,i.http_proxy,i.headers,i.retri eve_mode,i.request_method,i.output_format,i.ssl_ce rt_file,i.ssl_key_file,i.ssl_key_password,i.verify _peer,i.verify_host,i.allow_traps,i.templateid,nul l from items i inner join hosts h on i.hostid=h.hostid join item_rtdata ir on i.itemid=ir.itemid where h.status in (0,1) and i.flags<>2"

---

zabbix version 6.2.3
2 zabbix servers
2 zabbix web 3 postgresql clusters
5dc * 2 node proxy
10,000 vm (vmware guest)
5000 vps
80 snmp device
20 vm (agent)
10 VMware
----
Sorry for my English.

[Markku]

Just an observation first: you are burning huge amounts of memory for the various caches on the server even though their utilization is very low. Like

CacheSize=12G (under 6% used)
ValueCacheSize=4G (under 3% used)
HistoryIndexCache=2G (under 3% used)
VMwareCacheSize=2G (under 1% used)
TrendCacheSize=2G (under 3% used)

Then, what happens between ~08:35 and ~09:00.

What does your database server statistics say during that time?

The high history write cache utilization means that Zabbix was not able to write data to the database fast enough.

I'm not sure about your graphs: why are the proxy and server history write cache utilizations high at the same time. Are server and proxy using the same database backend?

Markku

[BayramGrbz45]

I set up a postgresql cluster build for the Zabbix server.
https://github.com/vitabaks/postgresql_cluster Type A
For zabbix-proxies it connects to its own postgresql service on the server it is running on.

The high history write cache utilization means that Zabbix was not able to write data to the database fast enough.

Do you have a solution suggestion for this?

I made the necessary adjustments, taking into account your cache suggestions.

[Markku]

I don't know what you are trying to say with the images.

You showed an image with 0 queries in the database. What's that?

Also, you are saying you have 5000 NVPS and then you showed an image of some "sda" disk (on an unknown server) showing 7 writes per second. That doesn't add up.

Markku

[BayramGrbz45]

I added the database statistics as a larger time range.
I'm sorry I forgot to add the sdb disk in my postgresql server.
I added the NVPS value as an image.


Sorry for my English, I didn't explain the situation well.

[Markku]

How is your proxy monitoring configured? I'm asking because of this:



Markku

[BayramGrbz45]

My Zabbix proxies are running in active mode.



The changes I made in the config file are in the link.

https://gist.github.com/BayramGrbz45/65bd6f8b55808d2a442bafbf6afffde5

https://gist.github.com/BayramGrbz45/65bd6f8b55808d2a442bafbf6afffde5

[Markku]

I meant: how are the proxies monitored?

Markku

[BayramGrbz45]

I'm watching via zabbix-server.

[Markku]

At this point I'm pretty sure your best bet is to get a local language-speaking Zabbix partner to help you: https://www.zabbix.com/partners#map_of_partners

Markku

[BayramGrbz45]

I experienced the systemic outage I mentioned again.
Let's try to give you the information you want about Zabbix.
Currently, there is no company that I can get joint support from.
I would be happy if you could help me.
I will get support from a friend who can speak your language.