Problem with Proxy in Large Enviroment

[alessandro.silva]

Hello,

I have a enviroment bellow:

1 Zabbix Server + 8 Zabbix Proxies and I have saw this message in ZABBIX SERVER LOG:

10253:20130307:125422.805 Sending configuration data to proxy 'ZPROXY1'. Datalen 1099246
10221:20130307:125804.502 Sending configuration data to proxy 'ZPROXY2'. Datalen 5790775
10221:20130307:125804.504 Error while sending configuration. ZBX_TCP_WRITE() failed: [32] Broken pipe


I can't sincronize queue and the alerts aren't being sent in 2 proxies. The enviroment worked fine, but suddenly stopped working (only to 2 proxies)

What's happening ?

[zabbn00b]

Network problem?

Anything in the system logs?

[tchjts1]

What do your Zabbix proxy logs say?
What OS are these proxies sitting on? Linux? Have they recently been rebooted?

[heaje]

Also, did the proxy configuration change at all or did it literally just stop working with no changes at all?

[tchjts1]

The reason I ask about the OS and being rebooted recently... selinux may be the culprit...

[alessandro.silva]

Help me please

- The OS is CentOS 6.3 (both server and proxy)
- The SELINUX is disabled
- There is nothing firewall in the server (IPTABLES is disabled)
- Sever was ot rebooted

The problem start two weeks ago ans persist.

ZABBIX SERVER Log ERROR:
10221:20130307:125804.504 Error while sending configuration. ZBX_TCP_WRITE() failed: [32] Broken pipe

[alessandro.silva]

Help me please

In thr proxy - Error Log

Error while receiving answer from server [ZBX_TCP_READ() failed: [4] Interrupted system call]

[Alexei]

It is clearly a timeout situation of Zabbix Proxy side. You may consider raising log level to see exactly why it happened. I suspect either performance or locking issues on the proxy database.

[alessandro.silva]

Terrible problem

Alexei,

You're right:

In Zabbix proxy I executed:
# while true; do date; nc -vv -w3 <ZABBIX SERVER IP> 10051; echo; sleep 5; done

Qui Mar 14 22:02:34 BRT 2013
nc: connect to <ZABBIX SERVER IP> port 10051 (tcp) timed out: Operation now in progress

Qui Mar 14 22:02:42 BRT 2013
nc: connect to <ZABBIX SERVER IP> port 10051 (tcp) timed out: Operation now in progress
.

But I changed the database SQLITE to MYSQL and problem persisted. I think isn't performance because my proxies have 4 CPU's and 4Gb memory. Also, My Zabbix server is a VIRTUAL machine with 16 CPU's and 16 GB memory and DATABASE is a Physical machine with 32 CPU's and 128 GB memory.


What do you think ?

[alessandro.silva]

Log

Log attached

[Alexei]

The log file does not have any useful information, where is the timeout error message?

[alessandro.silva]

Response

The timeout message appear (not always) when I test as bellow:

In Zabbix proxy I executed in the shell:
# while true; do date; nc -vv -w3 <ZABBIX SERVER IP> 10051; echo; sleep 5; done


Thu Mar 14 22:45:18 BRT 2013
Connection to Zabbix Server IP 10051 port [tcp/zabbix_trap] succeeded!

Thu Mar 14 22:45:26 BRT 2013
Connection to Zabbix Server IP 10051 port [tcp/zabbix_trap] succeeded!

Thu Mar 14 22:45:34 BRT 2013
Connection to Zabbix Server IP 10051 port [tcp/zabbix_trap] succeeded!

Thu Mar 14 22:45:42 BRT 2013
nc: connect to Zabbix Server IP 10.100.10.232 port 10051 (tcp) timed out: Operation now in progress

Thu Mar 14 22:45:50 BRT 2013
nc: connect to Zabbix Server IP port 10051 (tcp) timed out: Operation now in progress

Thu Mar 14 22:45:58 BRT 2013
nc: connect to Zabbix Server IP port 10051 (tcp) timed out: Operation now in progress

[Alexei]

It does look like you do no have enough trapper processes to handle all requests or (less likely) the connections are blocked on a firewall level for some reason.

[alessandro.silva]

Response

Alexei,

There are a lot of rules that permit conection on the 10051/TCP between the Zabbix Server and proxies.

- Do you think the server and proxy can't be behind the firewall?
- The VM can be a problem ? NIC shared ???