Zabbix Server Queue & Config

[dvwyngaa]

Ingus,

I managed to get the slow queries to less than a percent (by removing this line: "log-queries-not-using-indexes = 1" from the my.cnf), but I still have a bunch of items in the "over 10 min" queue.

It works it self out, but I still have the calculated item graph coming out with missing pieces in it.

Any further ideas?

Regards,

Dawid

[ingus.vilnis]

Hi Dawid,

Glad you got that slow queries issue solved.
Regarding queue. Check what are the exact hosts and items (Administration -> Queue -> Details from top right dropdown). Maybe that can give you a clue why are they delayed?

BR
Ingus

[dvwyngaa]

Hi Ingus,

Thanks...here's a screenshot of some of the items....it's from one specific proxy, its the same one that the calculated graph comes from.

I have upgraded that proxy's MySQL form 5.1 to 5.5 a week ago, but the queue, as mentioned, is still the same.



Regards,

Dawid

[ingus.vilnis]

Dawid,

Try increasing Timeout= parameter in your zabbix_server.conf and zabbix_proxy.conf files to somewhat higher. For test try the maximum 30 seconds. Restart server and proxies to apply the changes and check the queue after a while.

BR
Ingus

[dvwyngaa]

Ingus,

Thanks. I have LogSlowQueries set to 1000 and there is no slow queries logged in the zabbix_server.log file for now...

I'll change the TimeOut parameter and feedback a bit later.

Regards,

Dawid

[Renate]

Hello,

I have been reading your post and have/ had the same "challenge".
Under Administration - Queue a lot of items waiting More than 10 minutes

I didn't see in your zabbix_server.conf that DebugLevel was enabled.
If you enable DebugLevel (3 or 4) in de zabbix_server.conf you can see something like this in your zabbix_server.log file (like i did)

23663:20140901:134149.450 Zabbix agent item "XXXXXXX" on host "XXXXXXXX" failed: another network error, wait for 15 seconds

although in zabbix the item very nice said enabled, my graphs where like yours with gaps. I have tried a lot of the solutions mentioned here in the post, but didn't help.

It is not a solution, but i disabled all the items which gave the error in the log file (it was a lot of items) and then looked at my graphs They look a lot better, no gaps.
Again it is not a solution, but maybe it gives you a direction.

I hope that it will help you a little bit. I don't have a solution for my challenge

[ingus.vilnis]

Hello Renate,

I know that in most cases I will be definitely tried to be proven wrong but are you 1000% sure that it is not what the log actually says - network problems? Or maybe you have many users with many open sessions? Or heavy scripts? Or some hyperadvanced router where the admin says that it is not blocking anything but actually is?

Try to increase the Timeout in zabbix_server and proxy.conf, if you have not tried it yet.

And the last thing - check the queue details for the delay time on a specific item. And then check the Latest data for that item. Do the times match and do you really see that the latest data is delayed?

Best Regards,
Ingus

[dvwyngaa]

Hi Renate / ingus,

My Proxy, where all the queue'd items are popping up, has the DebugLevel set to 3, but I'm not getting any error messages as you mentioned. When I run the mysqltuner.pl script, my output seems all OK, see below:

-------- Performance Metrics -------------------------------------------------
[--] Up for: 23h 12m 48s (373K q [4.467 qps], 18K conn, TX: 140M, RX: 72M)
[--] Reads / Writes: 82% / 18%
[--] Total buffers: 582.0M global + 3.1M per thread (200 max threads)
[OK] Maximum possible memory usage: 1.2G (64% of installed RAM)
[OK] Slow queries: 0% (0/373K)
[OK] Highest usage of available connections: 38% (76/200)
[OK] Key buffer size / total MyISAM indexes: 10.0M/97.0K
[OK] Query cache efficiency: 45.0% (113K cached / 252K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 10 sorts)
[OK] Temporary tables created on disk: 0% (0 on disk / 23K total)
[OK] Thread cache hit rate: 99% (76 created / 18K connections)
[OK] Table cache hit rate: 95% (166 open / 173 opened)
[OK] Open file limit used: 4% (49/1K)
[OK] Table locks acquired immediately: 100% (244K immediate / 244K locks)
[OK] InnoDB buffer pool / data size: 512.0M/2.3M
[OK] InnoDB log waits: 0

@ingus, although I have changed the TimeOUT setting to 30s on both server and proxy and left it for about 24 hours, there is no real improvement.

I'll continue to monitor the queue and the logs...

I'm a bit stumped at the moment.

Dawid

[Renate]

Hi Ingus/ Dawid,

The error in the log file i don't have it anymore. In the item i changed the Type from zabbix agent to zabbix agent (active)....
But in my graphs i still have no data, so i still have a challenge.

@Ingus, we use zabbix in combination with an application Sonic (middleware application). We have a few users, no heavy scripts and no routers. Most of the items works just fine. Only with 2 specific items (which monitor multiple queues on Sonic) i have a challenge with.
Indeed i have adjust Timeout and several other options, but unfortunately without success.

@Dawid, good to see that your log file shows no errors. Although it would maybe better if you had an error, so you could see what is wrong...

[ingus.vilnis]

Hi Renate,

Three quick questions:

1. What version of Zabbix are you running?
2. Can you give an exact example of one item that you have in your logs with network errors?
3. Have you analyzed your server performance graphs and checked that Zabbix internal processes are not overloaded?

Best Regards,
Ingus

[Renate]

Hello Ingus,

1. What version of Zabbix are you running?

Zabbix version 2.0.3

2. Can you give an exact example of one item that you have in your logs with network errors?

The error was:

2859:20140905:162326.142 Zabbix agent item "Esb.service.messages.AverageProcessingTime.servic e.cbr.4.1" on host "XXXXX_SonicMC_2013" failed: another network error, wait for 15 seconds
2859:20140905:162350.816 Zabbix agent item "Esb.service.messages.AverageProcessingTime.servic e.cbr.4.1" on host "XXXXX_SonicMC_2013" failed: another network error, wait for 15 seconds
2859:20140905:162425.826 Zabbix agent item "Esb.service.messages.AverageProcessingTime.servic e.cbr.4.1" on host "XXXXX_SonicMC_2013" failed: another network error, wait for 15 seconds

I don't have an error anymore. I changed the type in de Item from zabbix agent to zabbix agent (active), but i still have no data in my graphs.
So no error in the log file (changed the Debuglevel to lvl 4 to be sure) and no data in the graphs.

3. Have you analyzed your server performance graphs and checked that Zabbix internal processes are not overloaded?

The server performance is just fine. The zabbix internal processes has a spike every hour when it's executing housekeeper. Then it is 100%

[ingus.vilnis]

Hello Renate,

1. You really should consider upgrading your Zabbix at least to the latest 2.0.12 or even better to the most recent 2.2.6

2. What is the item key for this item? Maybe it is not getting data back properly?

3. Glad that the server performance is fine. Housekeeper at 100% every hour is normal.

Best Regards,
Ingus

[Renate]

Hello Ingus,

Thanks for your reply.

1. You really should consider upgrading your Zabbix at least to the latest 2.0.12 or even better to the most recent 2.2.6

On a test server i have installed 2.2.5, but have the same issue there.

2. What is the item key for this item? Maybe it is not getting data back properly?
In the application Sonic i can get data, but if a close it all is lost. So that's why i what to transfer it to zabbix like many other item i done this with. What is working fine.

also tried, type of informatie from Numeric (unsigned) to Numeric (float), but same result

key is: esb.service.messages.AverageProcessingTime.service .cbr.4.1

[ingus.vilnis]

Hello Renate,

Now it is getting more clear to me.

I see that you are using your own item key that is passed to UserParameter defined on the zabbix_agentd.conf file on each host, right?
In such case it is possible that it takes some time for host to execute this command and return data. Therefore in the agent configuration there is also Timeout= parameter.
Please try increasing that one, restart zabbix_agentd process on that host and check the graphs and logs again.

Best Regards,
Ingus

[Renate]

Hello Ignus,

Yes, you are correct. We use an appender difined on the application.
The Timeout parameter i will adjust, try different values. I will get back on that.