queue problem

[perun.84]

Here, you can see difference between zabbix server (first column), first proxy (second column) and third one (third column). Is it possible that it is buffer on proxy?

[mortuletti]

And "LAST CHECK" time goes forward or just stay "05:06"?
In Administration - Proxies, Last seen (age) is 0 or not?
Br, Alex

[perun.84]

Yes, it goes forward. One minute in one minute. In proxies Last seen for both of proxies are 1s.

[perun.84]

Data sender and heartbeat sender on problematic proxy are on 100%. Maybe it could be important information... And when I changed timezone from GMT+1 to BST last check went from 4:x to 5:x. But now, timezone is ok. Could reinstall help? Or it is some timestamp value in server database...

[perun.84]

After few hours there are no delays. I didn't realized what happened, but it is working... I've only disabled buffers on proxy. May it helped.

Now I'm going to solve big queue problem...

[mortuletti]

Great!
Will wait for news about queue.
Br, Alexander

[perun.84]

Unfortunately situation is the same.



Stop drawing graphs and receiving new data takes few hours (since noon till midnight). What do you think, is smaller partition time (now it is one day parititon) will help?

[mortuletti]

Hi!
So on Zabbix Server we have 100% busy History syncer process. On the Proxy Timer process. Bough processes doing approximately the same job, - processing received data and save it do database.

Can you attach pictures of "Value cash effectiveness", "Zabbix cache usage.." and "data gathering processes busy" for bough (Proxy and Server)?

Thank you!
Regards,
Alexander

[perun.84]

Thnaks for answer. Here is zabbix cache graph:



And here is data gathering graph for that period:

[perun.84]

I've just upgraded mariadb from version 5.5 to 10.1.

[mortuletti]

Upgrade it is fine, but don't think this will helps. Let's see.

In regards of graphs.
On Zabbix Server
"StartPollers=128" obviously can be set to 25-50 (extra processes block Configuration cash for extra time). Actually only 1-2 is in use.
"StartDiscoverers=9" - can be set to 1-2. Depends how many discovery rules you have but looks not in use.

On Proxy
"Poller processes" should be extend 2-3 times.

Can you share Proxy and Server log file and conf files one more time?

Looks like can be 2 causes of problem:
1. performance of Server and Proxy database
2. and/or Syncer processes cannot perform their job because History and/or Configuration cache is locked.

To avoid extra locks of caches, as I write above, reduce number of processes. If according to graphs Trappers or Poller processes is almost not in use, reduce it.
As well Hystory Syncer processes should be not more than 4-8. Theoretically each can perform up to 1000 NVPS.

Thank you!
Br, Alexander

[perun.84]

Today, it's working correctly. I've also change HistoryCache from 256M to 1G.

Here is server.conf:

LogFile=/var/log/zabbix/zabbix_server.log

LogFileSize=0

DebugLevel=3

PidFile=/var/run/zabbix/zabbix_server.pid

DBHost=xyz.xyz.com

DBName=xyz

DBUser=xyz

DBPassword=xyz

StartPollers=35

StartDiscoverers=0

SNMPTrapperFile=/var/log/snmptrap/snmptrap.log

HousekeepingFrequency=0

CacheSize=1G

StartDBSyncers=8

HistoryCacheSize=1G

HistoryIndexCacheSize=64M

TrendCacheSize=1G

ValueCacheSize=1G

Timeout=15

AlertScriptsPath=/usr/lib/zabbix/alertscripts

ExternalScripts=/usr/lib/zabbix/externalscripts

LogSlowQueries=3000

And here is proxy.conf:

Server=xxx.xxx.xxx.xxx

Hostname=xyz

LogFile=/var/log/zabbix/zabbix_proxy.log

LogFileSize=0

DebugLevel=3

PidFile=/var/run/zabbix/zabbix_proxy.pid

DBHost=localhost

DBName=xxx

DBUser=xxx

DBPassword=xxx

DBSocket=/var/lib/mysql/mysql.sock

ConfigFrequency=600

StartPollers=1000

StartPollersUnreachable=150

StartDiscoverers=16

SNMPTrapperFile=/var/log/snmptrap/snmptrap.log

CacheSize=1G

StartDBSyncers=24

HistoryCacheSize=512M

HistoryIndexCacheSize=64M

Timeout=30

UnreachableDelay=5

ExternalScripts=/usr/lib/zabbix/externalscripts

LogSlowQueries=3000

In log I cannot see anything interesting. Just slow queries "INSERT INTO history_uint..."

[mortuletti]

Config files looks good.
Only, on the Proxy "StartDBSyncers=24" should be reduced to ~8 or less. This process is like reading information from CD drive, - in overall, one process will works few times faster as few. So try to set it to 8 or even 4. Each should cover up to 0.5 - 1k NVPS. Busy DBSyncers usually indicates low performance of database.
Br, Alexander

[mortuletti]

history_uint - table with history raw data.
Can you please share MySQL configuration file?
Br, Alexander

[mortuletti]

To reduce delay between Zabbix and DB, in place of DNS name can set IP address of database server or define it in local host file (if possible).
As well, check connection between Zabbix - DB. Should be as fast as possible.