Hello,
We are investigating a recurring issue in a large Zabbix environment running Zabbix 7.4.2, where the Zabbix Proxy data sender becomes blocked for approximately 5 minutes.
We have observed the same behavior in three different Zabbax Proxies, communicating with Zabbix Servers, and have captured the traffic simultaneously at both endpoints.
Environment:
Observed behavior:
In all three incidents, the TCP connection between a Proxy and the Zabbix Server shows the following pattern:
Real example:
One of the incidents showed:
Initial TCP sequence:
4294963497
Payload:
4104 bytes
Server ACK:
305
This is consistent with the sequence number wrapping around:
4294963497 + 4104 = 4294967601
4294967601 - 4294967296 = 305
After this point, the Proxy repeatedly retransmits:
SEQ=305
LEN=1293
We observed 11 retransmissions in the Proxy capture.
However, none of these retransmissions appear in the Server-side capture.
The Server keeps acknowledging:
ACK=305
for approximately 300 seconds and then closes the connection. Repeated incidents
The same general pattern has been observed three times in different proxies with different proxy groups.
In the first two incidents, the missing segment was 1368 bytes. In the third incident it was 1293 bytes.
The common characteristics are:
Could this behavior be caused by Zabbix itself, or is it more likely related to TCP/network infrastructure handling of sequence number wrap-around?
In particular, we would like to know whether there are any known issues in Zabbix 7.4.x related to proxy data sender connections, TCP sequence number wrap-around, or the approximately 300-second timeout.
We are also investigating firewalls, virtual networking and other stateful network components in the path, since the retransmitted packets are clearly visible leaving the Proxy but are not visible at the Server.
We can provide sanitized packet captures and additional details if required.
Any advice on what Zabbix-side logs, configuration parameters or packet-level evidence would be useful to distinguish a Zabbix issue from a network/TCP issue would be appreciated.
We are investigating a recurring issue in a large Zabbix environment running Zabbix 7.4.2, where the Zabbix Proxy data sender becomes blocked for approximately 5 minutes.
We have observed the same behavior in three different Zabbax Proxies, communicating with Zabbix Servers, and have captured the traffic simultaneously at both endpoints.
Environment:
- Zabbix Server: 7.4.2
- Zabbix Proxy: 7.4.2
- Large-scale environment with multiple proxies
- Proxy-to-server communication uses TCP port 10051
Observed behavior:
In all three incidents, the TCP connection between a Proxy and the Zabbix Server shows the following pattern:
- The TCP connection is established normally.
- The initial TCP sequence number is very close to the 32-bit maximum (2^32).
- During the data transfer, the TCP sequence number wraps around.
- The Zabbix Server acknowledges the data received before the wrap-around.
- A TCP segment immediately after the wrap-around is not received by the Server.
- The Proxy retransmits the same segment repeatedly, typically 11 times.
- The retransmissions are visible in the packet capture taken at the Proxy, but are not present in the packet capture taken at the Server.
- The Server keeps the connection waiting for approximately 300 seconds.
- The Server then closes the TCP connection.
- The Proxy establishes a new connection and successfully sends the accumulated data.
Real example:
One of the incidents showed:
Initial TCP sequence:
4294963497
Payload:
4104 bytes
Server ACK:
305
This is consistent with the sequence number wrapping around:
4294963497 + 4104 = 4294967601
4294967601 - 4294967296 = 305
After this point, the Proxy repeatedly retransmits:
SEQ=305
LEN=1293
We observed 11 retransmissions in the Proxy capture.
However, none of these retransmissions appear in the Server-side capture.
The Server keeps acknowledging:
ACK=305
for approximately 300 seconds and then closes the connection. Repeated incidents
The same general pattern has been observed three times in different proxies with different proxy groups.
In the first two incidents, the missing segment was 1368 bytes. In the third incident it was 1293 bytes.
The common characteristics are:
- TCP sequence number close to 2^32
- Sequence number wrap-around
- Segment immediately after the wrap-around is not received by the Server
- Repeated retransmissions from the Proxy
- Retransmissions not visible at the Server
- Connection remains blocked for approximately 300 seconds
- New connection succeeds immediately afterwards
Could this behavior be caused by Zabbix itself, or is it more likely related to TCP/network infrastructure handling of sequence number wrap-around?
In particular, we would like to know whether there are any known issues in Zabbix 7.4.x related to proxy data sender connections, TCP sequence number wrap-around, or the approximately 300-second timeout.
We are also investigating firewalls, virtual networking and other stateful network components in the path, since the retransmitted packets are clearly visible leaving the Proxy but are not visible at the Server.
We can provide sanitized packet captures and additional details if required.
Any advice on what Zabbix-side logs, configuration parameters or packet-level evidence would be useful to distinguish a Zabbix issue from a network/TCP issue would be appreciated.
Comment