1.4.5 process dies

[Alexei]

From NMAP manual:

Filtered means that a firewall, filter, or other network obstacle is blocking the port so that Nmap cannot tell whether it is open or closed.

What "netstat -an|grep 10051" says? Can you "telnet localhost 10051"?

[bbrendon]

From NMAP manual:

Filtered means that a firewall, filter, or other network obstacle is blocking the port so that Nmap cannot tell whether it is open or closed.

What "netstat -an|grep 10051" says? Can you "telnet localhost 10051"?

After I restart zabbix, nmap gives the proper response.

I'll add the netstat -an to the fix script. Hmmm, I'm not sure how to sciprt the "telnet localhost 10051" which is why I used nmap.

[bbrendon]

I need to have all this automated. We've had tons of downtime. With my script, I only have about 20 minutes every few days which is tolerable. How about this for the test?

nmap -v -sT -p 10051 127.0.0.1

The nmap man page says:
Nmap asks the underlying operating system to establish a connection with the target machine and port by issuing the connect() system call

[Alexei]

After I restart zabbix, nmap gives the proper response.

I'll add the netstat -an to the fix script. Hmmm, I'm not sure how to sciprt the "telnet localhost 10051" which is why I used nmap.

Thanks. The fact that ZABBIX restart fixed the problem does not necessary mean that it was a ZABBIX fault. I think that it could also be some kind of kernel or network related issue (limits, kernel-level network flooding protection, whatever).

[bbrendon]

Thanks. The fact that ZABBIX restart fixed the problem does not necessary mean that it was a ZABBIX fault. I think that it could also be some kind of kernel or network related issue (limits, kernel-level network flooding protection, whatever).

Thanks very much for the feedback. I'll see what I can come up with.

[Alexei]

Thank you for the log file.

Short advice, increase number of trappers to 50. ZABBIX works absolutely fine.

At some point your network becomes very slow or unreliable causing large number of timeouts, so all of your five trappers hang in a 5 min (currently hard coded) timeout state.

Case closed.

[bbrendon]

Thank you for the log file.

Short advice, increase number of trappers to 50. ZABBIX works absolutely fine.

At some point your network becomes very slow or unreliable causing large number of timeouts, so all of your five trappers hang in a 5 min (currently hard coded) timeout state.

Case closed.

Shit, I would have never guessed. Thanks for the advice!!!!

[bbrendon]

Ahh, it seems its a good idea to increase MySQL max_connections when increasing trappers!

[xs-]

Just out of curiosity, how much hosts are you monitoring infinity005?

I my setup, i've set the max_trappers low on purpose so there are less concurrent db connections and thus less concurrent updates (after some testing, this decreased the load in the db server quite a bit).

[Alexei]

I my setup, i've set the max_trappers low on purpose so there are less concurrent db connections and thus less concurrent updates (after some testing, this decreased the load in the db server quite a bit).

This is not correct. Number of trappers does not affect database performance in a direct way at all.

[bbrendon]

Just out of curiosity, how much hosts are you monitoring infinity005?

I my setup, i've set the max_trappers low on purpose so there are less concurrent db connections and thus less concurrent updates (after some testing, this decreased the load in the db server quite a bit).

I don't see a measurable performance hit when bumping trappers from 5 to 50.
Number of hosts (monitored/not monitored/templates/deleted) 107(70/10/27/0)

Number of items (monitored/disabled/not supported)[trapper] 345(1590/2483/272)[50]

Number of triggers (enabled/disabled)[true/unknown/false] 929(514/415)[8/82/424]

[xs-]

This is not correct. Number of trappers does not affect database performance in a direct way at all.

the trapper processes are the ones receiving the active agent checks right?
If not, which processes receive it then.

If less processes do concurrent connections and this concurrent queries on the database, it will reduce the load on the database (a little less parallel, a little more sequential). If tuned right, it will reduce the load on the database, and not impact the zabbix performance too much *i think*

[just2blue4u]

i find this thread very interesting!

Made me think about the question:
"is there a way to monitor availibility/utilisation of zabbix trappers"?

[bbrendon]

Okay, looks like things are working smoothly!! This is probably the most stable zabbix has been for me in a long time. Yippie!

Alexei, how can I tell how many of the trappers are hung because of bad network connections?

[Alexei]

Okay, looks like things are working smoothly!! This is probably the most stable zabbix has been for me in a long time. Yippie!

Alexei, how can I tell how many of the trappers are hung because of bad network connections?

Good news indeed!

The processes do not hang forever, it is just for 5 minutes (default) and only in case of serious network related problems.

I do not see an easy way of calculating number of trappers in the "sleep" state.

OS level TCP keep alive settings can decrease number of such trappers and hung network connections very much. Not sure how to setup this in Linux, it is done nicely in HP-UX, for example.