Queues are full at big setup

[mbuyukkarakas]

Dear all,

I Today we faced a strange problem at office. The Zabbix server is installed with os Centos 7.4 & Zabbix 4.2.4 at vmware platform with a big full SSD storage backend.

You can see in the picture that this VM has enough resources. 2 vcpu, 100 Gb disk for tests and 32 GB Ram.



All the agents are active agents and we plan to collect too much metrics at production environment. When we started with a few agents ( 105 agents ) we faced a strange problem. We noticed that the queues are full.



As you can see , the queues are almost always full.





and my.cnf

max_connections = 512

innodb_autoextend_increment = 256
innodb_buffer_pool_instances = 12
innodb_buffer_pool_size = 10G
innodb_concurrency_tickets = 5000
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
innodb_log_files_in_group = 4
innodb_old_blocks_time = 1000
innodb_open_files = 2048
innodb_stats_on_metadata = OFF

binlog-row-event-max-size = 8192
binlog-format = MIXED
expire_logs_days = 1
join_buffer_size = 262144
max_heap_table_size = 134217728
query_cache_type = 0
query_cache_size = 0
slow-query-log = ON
table_open_cache = 2048
thread_cache_size = 64
tmp_table_size = 134217728
wait_timeout = 86400

and important zabbix.conf parameters.

StartPollers=20
StartTrappers=100
StartDBSyncers=12


We couldnt find a reasonable explanation. Any help will be appreciated.
Thank you. Regards.

[1berto]

1) How are the frequency of the items?
2) Have you made some change recently?
3) Have you considered using some proxies?

Here is another question that may help you:

https://www.zabbix.com/forum/zabbix-for-large-environments/368792-zabbix-3-0-starttrappers-are-getting-stuck-processing-data

[mbuyukkarakas]

well, thank you for your response. I must say almost all items are about 30-60 sec intervals. I know its high.
but I'm open to all ideas.
thank you

[mbuyukkarakas]

well, thank you for your response. I must say almost all items are about 30-60 sec intervals. I know its high.
its possible to use proxies but it must be my last exit from bridge.
anyway I'm open to all ideas.

thanks