zabbix issues after adding proxy

[isilvera]

Hello! I'm having issues actually with zabbix performance (I suppose) but let me try to explain the situation

I have a zabbix server and a zabbix proxy installed recently (maybe one week ago). Zabbix server collect most of the data from SNMP devices polling it in different intervals. Most of the devices are linked to templates, and those templates, have snmp traps items.
Since I added zabbix proxy one week ago, I'm getting alerts every day at 23:30. All the fired triggers are snmp traps with nodata function associated and also are triggered hosts that are monitred by zabbix proxy. I could find that at 23:30 a scheduled backup runs to this machine, so it can explain why it always all the triggers are fired at the same time. But the weird thing here is that if I remove all the hosts from zabbix proxy, and delegate it to be monitored by zabbix server, those triggers are not fired.

I checekd the queue while the backup is executed and I could see that zabbix server queue grows a lot during the backup (3k of items queued), and the queued items are all from hosts monitored by zabbix proxy. Zabbix proxy never has more than 5 items on the queue. I tried adding more trappers process to zabbix server(25 actually) to avoid the queue, but nothing helps. Zabbix has in average 300 values per second.

So my questions are:
1 why if the hosts are monitored by zabbix server,triggers are not fired, but if I configure it as monitored by proxy, triggers are fired.
2 why snmp traps are configured with nodata function (traps items will never have data as I can understand). Those traps items were configured as is by default on the template.
3 zabbix-trapper process is who recived the zabbix proxy data?

Please feel free to ask me anything and I will be happy to share more details

Thanks!

[LenR]

What kind of backup? There really isn't much reason to backup a proxy, with proper devops config, you can just re-deploy (rebuild) a proxy as fast as you could restore it. If your backup process is disrupting the Zabbix proxy, it is probably disrupting other services.

One option would be to put all hosts on the proxy in maintenance for the duration of the backup.

[isilvera]

Hello!

Thank you for your answer! The backup is on the zabbix server node, not on zabbix proxy. And also the queue is on the zabbix server, not proxy.
I could see many processes like this during the backup

zabbix 15083 0.0 0.1 7551672 22948 ? S 22:57 0:00 /usr/sbin/zabbix_server: trapper #17 [processed data in 438.240648 sec, waiting for connection]
zabbix 15035 0.0 0.1 7553856 21600 ? S 22:57 0:00 /usr/sbin/zabbix_server: trapper #3 [processed data in 288.240350 sec, waiting for connection]

Can be caused by some kind of network issue?

Thanks!

[isilvera]

More information

vps during the backup time

zabbix server queue during the backup time

Bussy trappers

[isaqueprofeta]

1 why if the hosts are monitored by zabbix server,triggers are not fired, but if I configure it as monitored by proxy, triggers are fired.

It's not about zabbix stack it's about how the configurations are done and how the data is being received.

2 why snmp traps are configured with nodata function (traps items will never have data as I can understand). Those traps items were configured as is by default on the template.

It's not a rule and to be honest, I'd never configured an SNMP trap trigger to use nodata.

And to make it clear an item will aways have data, even if you "do not keep history", Traps receive data from a Trap/Push of an agent (SNMP), Pooling items receive data from a Pool/Get from the server/proxy.

One of golden rules from zabbix top 10: Never use any template as is by default, it was not made for you.

3 zabbix-trapper process is who recived the zabbix proxy data?

Yes.

Humm... So, can have 2 problems here, one is Network congestion and your network are giving the backup flow priority. Another thing is Database slowdown. Since you're under backup, the database goes under the pressure of being riped for backup, and the I/O of disk can't keep up, so it's queueing your processes.

I'd advise you to customize your approach on default nodata, since the delay of the queue to processing items seems to be hitting your trigger evaluation.

[isilvera]

Hello! I will try to update the post with detailed information

It's not about zabbix stack it's about how the configurations are done and how the data is being received.

And how it really works? The items are traps as the following:

snmptrap[.1.3.6.1.2.1.33.2.4]

All the items that are monitored by proxy like the above with triggers using nodata(300)=0 function are fired during the backup, but if the item is monitored by zabbix server, it is not fired. How can it be happening? VMs get in read only status during backup, but as I explained before, if the item is monitored by server, it is not fired.
An example of the trigger:

 {[URL="http://172.16.104.200/zabbix/items.php?form=update&itemid=51172"]HOST:snmptrap[.1.3.6.1.2.1.33.2.4][/URL].[B]nodata([/B]300[B])[/B]}=0

And to make it clear an item will aways have data, even if you "do not keep history", Traps receive data from a Trap/Push of an agent (SNMP), Pooling items receive data from a Pool/Get from the server/proxy.

So, why trigger is being fired? Due the item recives data? If yes, I never see it on latest data and there's no traps on the trap file, and the devices are completely fine, so there is no reason to send a trap.

I know that during the backup (that can take from 2 minutes to 15 minutes), the VM goes to read only state, so probably is not possible to zabbix to insert data on database and I can understand it, but I do not have any explanation of why it only happen with the items monitored by proxy as I mentioned several time ago, so I think the question here is.
Why there is a different behavior on triggers when the Item is monitored by proxy or zabbix server?

Thanks in advance!

[isaqueprofeta]

Everything you asked this last time I've written about in my previous answer, except for:

Why there is a different behavior on triggers when the Item is monitored by proxy or zabbix server?

There's no difference in behavior.
The only dataflow difference is that the time of data gather is at the arrival in proxy, and the time of trigger calculation is the time of data arriving at the server database.
If you're only using the server both could be the same.

I know that during the backup (that can take from 2 minutes to 15 minutes), the VM goes to read only state

Ok, there's your root problem.

[isilvera]

Hi isaqueprofeta! Thanks for your answer and your efforts to help me to resolve the issue!

There's no difference in behavior.
The only dataflow difference is that the time of data gather is at the arrival in proxy, and the time of trigger calculation is the time of data arriving at the server database.
If you're only using the server both could be the same.

I can understand it, but think that if server cannot process the information sent by proxy, why it can process the information collected by itself? I mean, considering that the trigger is being fired due to the data is processed by zabbix 8 minutes before, why do not happen the same when the data is collected by zabbix server?

I will try to make a time line

Host monitored by zabbix proxy
23:00: Zabbix server and proxy are collecting data, including possible traps sent by hosts
23:30 backup begins, zabbix database goes to read only state. Zabbix server cannot write new data. Zabbix proxy is collecting information normally
23:35 triggers are fired due tue there's no data on the items (triggers with nodata(300)=0)
23:38 triggers are resolved and backup finishes.

Host monitored by server
23:00: Zabbix server and proxy are collecting data, including possible traps sent by hosts
23:30 backup begins, zabbix database goes to read only state. Zabbix server cannot write new data.
23:38 backup finishes

Can you please try to explain me a little bit more why can it be happeing? or at least what I can check to understand it.

Thanks!

[isaqueprofeta]

Host monitored by zabbix proxy
23:00: Zabbix server and proxy are collecting data, including possible traps sent by hosts
23:30 backup begins, zabbix database goes to read only state. Zabbix server cannot write new data. Zabbix proxy is collecting information normally
23:35 triggers are fired due tue there's no data on the items (triggers with nodata(300)=0)
23:38 triggers are resolved and backup finishes.

Host monitored by server
23:00: Zabbix server and proxy are collecting data, including possible traps sent by hosts
23:30 backup begins, zabbix database goes to read only state. Zabbix server cannot write new data.
23:38 backup finishes

Can you please try to explain me a little bit more why can it be happeing? or at least what I can check to understand it.

Thanks!

Sorry but in this case, I really have no idea.

The only thing that I can tell is that IMHO in this scenario the second situation is the right one. Since the server has no access to the database, the DB watcher process should hang the zabbix-server until the database is available, so, no triggers until it gets back.

[anuhya@123]

What sort of reinforcement? There truly isn't a lot of motivation to reinforce an intermediary, with legitimate DevOps config, you can just re-convey (revamp) an intermediary as quickly as you could reestablish it. On the off chance that your reinforcement cycle is upsetting the Zabbix intermediary, it is likely disturbing different administrations. One alternative has placed all hosts on the intermediary in support for the span of the reinforcement. telecharger minecraft