Zabbix-Proxy: HA Setup: Sequential redirect responses detected

[lapniklas]

Hello,

i would like to setup two zabbix proxies as a high availability setup. These proxies do run in passive mode. I have also configured agents running behind the proxy that should be able to do active checks. But the agents do not become available in the zabbix-ui when the host is assigned to a proxy group. But when i assign the Host to an individual proxy instead of a proxy group the active checks become available. Both proxies are online in the zabbix-ui. What should i change in order to fix this?

Docker zabbix-proxies running on 172.19.10.61,172.19.10.62.

From the logs i can see that both proxies seem to think that the other proxy is responsible for the agent:

Zabbix Logs:

2024/06/19 07:24:43.971857 Zabbix Agent2 hostname: [ops-zabbix-proxy-1]
2024/06/19 07:24:44.995900 [101] sequential redirect responses detected
2024/06/19 07:24:44.995985 [101] active check configuration update from host [ops-zabbix-proxy-1] started to fail
2024/06/19 07:24:45.017184 [101] sequential redirect responses detected
2024/06/19 07:24:45.017247 [101] sending of heartbeat message for [ops-zabbix-proxy-1] started to fail
2024/06/19 08:50:08.986356 [101] active check configuration update from [172.19.10.61:10051] is working again
2024/06/19 08:50:08.991253 [101] sending of heartbeat message to [172.19.10.61:10051] is working again
2024/06/19 08:50:43.990209 [101] sequential redirect responses detected
2024/06/19 08:50:43.990325 [101] history upload to [172.19.10.61:10051] [ops-zabbix-proxy-1] started to fail
2024/06/19 08:50:44.990428 [101] sequential redirect responses detected
2024/06/19 08:50:44.990498 [101] active check configuration update from host [ops-zabbix-proxy-1] started to fail
2024/06/19 08:51:09.985338 [101] sequential redirect responses detected
2024/06/19 08:51:09.985561 [101] sending of heartbeat message for [ops-zabbix-proxy-1] started to fail

Zabbix proxy Logs:

169:20240619:114351.689 thread started [discoveryworker#1]
169:20240619:114351.690 thread started [discoveryworker#5]
169:20240619:114351.690 thread started [discoveryworker#4]
169:20240619:114351.690 thread started [discoveryworker#3]
169:20240619:114351.690 thread started [discoveryworker#2]
161:20240619:114353.352 received configuration data from server at "<SOME PUBLIC IP>", datalen 24395
162:20240619:114353.567 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
164:20240619:114353.572 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
160:20240619:114356.978 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
162:20240619:114356.985 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
162:20240619:114359.566 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
162:20240619:114359.573 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
163:20240619:114402.976 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
162:20240619:114402.981 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
164:20240619:114403.052 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
163:20240619:114405.565 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
164:20240619:114405.570 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
164:20240619:114408.977 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
162:20240619:114408.986 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
164:20240619:114411.568 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
162:20240619:114411.574 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
163:20240619:114413.725 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
162:20240619:114414.978 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
162:20240619:114414.984 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy

Zabbix_agent2.conf:

Server=172.19.10.61,172.19.10.62
ListenIP=172.19.10.61
ListenPort=10050
ServerActive=172.19.10.61:10051;172.19.10.62:10051
Hostname=ops-zabbix-proxy-1

[Markku]

At least this is incorrect:

"ZBX_HOSTNAME=hetzner-proxy",

on both proxies.

https://www.zabbix.com/documentation/current/en/manual/appendix/config/zabbix_proxy#hostname

A unique, case sensitive proxy name.

Markku

[Markku]

Please provide logs for all three components (agent, proxy1, proxy2) from the same time period.

Markku

[cyber]

Server=172.19.10.61,172.19.10.62

I think it should be

Server=172.19.10.61;172.19.10.62

Based on https://blog.zabbix.com/zabbix-7-0-p...lancing/28173/ I think there is still some issue with docs.. But in case you have 2 servers, separated with comma, it take s them as 2 separate instances and tries to connect to both, confusing proxy HA setup, which tries to redirect, when it is not needed...

[Markku]

Server = for passive items, no semicolon syntax available
(ServerActive has the issue cyber mentions)

[cyber]

I stand corrected... Must be something in the air ... I constantly miss obvious things today ..

[lapniklas]

Logs of Zabbix-Proxy-1 (172.19.10.61):

   164:20240620:131504.061 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   161:20240620:131505.565 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   161:20240620:131505.572 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   161:20240620:131511.567 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   161:20240620:131511.572 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   162:20240620:131513.101 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   164:20240620:131517.568 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   164:20240620:131517.574 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   163:20240620:131523.166 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   163:20240620:131523.567 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   162:20240620:131523.574 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   160:20240620:131529.565 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   164:20240620:131529.571 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   160:20240620:131533.195 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   163:20240620:131535.569 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   161:20240620:131535.575 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   162:20240620:131541.566 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   164:20240620:131541.573 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   164:20240620:131543.255 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   162:20240620:131547.566 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   163:20240620:131547.571 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   162:20240620:131553.336 received configuration data from server at "<SOME PUBLIC IP>", datalen 919
   161:20240620:131553.569 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   163:20240620:131553.576 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   164:20240620:131559.565 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   164:20240620:131559.572 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   161:20240620:131603.419 received configuration data from server at "<SOME PUBLIC IP>", datalen 897
   161:20240620:131605.565 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   164:20240620:131605.570 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   164:20240620:131611.566 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   161:20240620:131611.572 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   163:20240620:131613.530 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   163:20240620:131617.567 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   163:20240620:131617.575 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   162:20240620:131623.570 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   161:20240620:131623.576 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   162:20240620:131623.676 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   161:20240620:131629.565 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   163:20240620:131629.572 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   162:20240620:131633.730 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   164:20240620:131635.567 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   160:20240620:131635.573 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   164:20240620:131641.567 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   163:20240620:131641.572 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   160:20240620:131643.761 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   164:20240620:131647.566 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   160:20240620:131647.573 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   162:20240620:131653.567 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   163:20240620:131653.574 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   161:20240620:131653.882 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   164:20240620:131659.567 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   163:20240620:131659.572 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   163:20240620:131703.918 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   161:20240620:131705.565 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy
   161:20240620:131705.572 cannot send list of active checks to "172.19.10.62": host "ops-zabbix-proxy-2" is monitored by another proxy

Logs of Zabbix-Proxy-2 (172.19.10.62):

   160:20240620:131454.136 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   161:20240620:131504.176 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   163:20240620:131514.215 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   161:20240620:131524.282 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   160:20240620:131534.336 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   159:20240620:131544.379 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   161:20240620:131554.580 received configuration data from server at "<SOME PUBLIC IP>", datalen 919
   163:20240620:131558.019 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   161:20240620:131558.024 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   160:20240620:131604.020 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   162:20240620:131604.024 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   163:20240620:131604.656 received configuration data from server at "<SOME PUBLIC IP>", datalen 12734
   162:20240620:131610.020 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   160:20240620:131610.024 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   159:20240620:131614.746 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   160:20240620:131616.020 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   163:20240620:131616.027 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   160:20240620:131622.020 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   160:20240620:131622.025 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   161:20240620:131624.786 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   162:20240620:131628.019 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   160:20240620:131628.028 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   162:20240620:131634.020 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   160:20240620:131634.026 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   162:20240620:131634.932 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   185:20240620:131637.093 enabling Zabbix agent checks on host "ops-zabbix-proxy-1": interface became available
   159:20240620:131640.020 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   159:20240620:131640.026 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   160:20240620:131644.958 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   159:20240620:131646.019 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   161:20240620:131646.023 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   159:20240620:131652.019 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   161:20240620:131652.024 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   159:20240620:131655.004 received configuration data from server at "<SOME PUBLIC IP>", datalen 487
   161:20240620:131658.021 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   161:20240620:131658.028 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   163:20240620:131704.021 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   159:20240620:131704.026 cannot send list of active checks to "172.19.10.61": host "ops-zabbix-proxy-1" is monitored by another proxy
   162:20240620:131705.050 received configuration data from server at "<SOME PUBLIC IP>", datalen 487

Logs of Zabbix-Agent running on Zabbix-Proxy-1 (172.19.10.61)

2024/06/20 13:15:02.953204 Zabbix Agent 2 stopped. (7.0.0)
2024/06/20 13:15:03.009399 Starting Zabbix Agent 2 (7.0.0)
2024/06/20 13:15:03.011478 OpenSSL library (OpenSSL 3.0.2 15 Mar 2022) initialized
2024/06/20 13:15:03.011610 using configuration file: /etc/zabbix/zabbix_agent2.conf
2024/06/20 13:15:03.011754 using plugin 'Agent' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.011769 using plugin 'Ceph' (built-in) providing following interfaces: exporter, runner, configurator
2024/06/20 13:15:03.011781 using plugin 'Cpu' (built-in) providing following interfaces: exporter, collector, runner
2024/06/20 13:15:03.011792 using plugin 'DNS' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.011802 using plugin 'Docker' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.011816 using plugin 'File' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.011827 using plugin 'Hw' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.011836 using plugin 'Kernel' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.011845 using plugin 'Log' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.011854 using plugin 'MQTT' (built-in) providing following interfaces: watcher, configurator
2024/06/20 13:15:03.011881 using plugin 'Memcached' (built-in) providing following interfaces: exporter, runner, configurator
2024/06/20 13:15:03.011893 using plugin 'Memory' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.011907 using plugin 'Modbus' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.011924 using plugin 'MongoDB' (/usr/sbin/zabbix-agent2-plugin/zabbix-agent2-plugin-mongodb) providing following interfaces: exporter, runner, configurator
2024/06/20 13:15:03.011940 using plugin 'Mysql' (built-in) providing following interfaces: exporter, runner, configurator
2024/06/20 13:15:03.011950 using plugin 'NetIf' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.011965 using plugin 'Oracle' (built-in) providing following interfaces: exporter, runner, configurator
2024/06/20 13:15:03.011999 using plugin 'PostgreSQL' (/usr/sbin/zabbix-agent2-plugin/zabbix-agent2-plugin-postgresql) providing following interfaces: exporter, runner, configurator
2024/06/20 13:15:03.012017 using plugin 'Proc' (built-in) providing following interfaces: exporter, collector
2024/06/20 13:15:03.012026 using plugin 'ProcExporter' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.012035 using plugin 'Redis' (built-in) providing following interfaces: exporter, runner, configurator
2024/06/20 13:15:03.012044 using plugin 'Smart' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.012057 using plugin 'Sw' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.012066 using plugin 'Swap' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.012075 using plugin 'SystemRun' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.012097 using plugin 'Systemd' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.012112 using plugin 'TCP' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.012124 using plugin 'UDP' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.012142 using plugin 'Uname' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.012155 using plugin 'Uptime' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.012167 using plugin 'Users' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.012179 using plugin 'VFSDev' (built-in) providing following interfaces: exporter, collector
2024/06/20 13:15:03.012192 using plugin 'VFSDir' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.012221 using plugin 'VfsFs' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.012240 using plugin 'WebCertificate' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.012253 using plugin 'WebPage' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.012265 using plugin 'ZabbixAsync' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.012280 using plugin 'ZabbixStats' (built-in) providing following interfaces: exporter, configurator
2024/06/20 13:15:03.012291 lowering the plugin ZabbixSync capacity to 1 as the configured capacity 1000 exceeds limits
2024/06/20 13:15:03.012303 using plugin 'ZabbixSync' (built-in) providing following interfaces: exporter
2024/06/20 13:15:03.013150 Plugin communication protocol version is 6.4.0
2024/06/20 13:15:03.013206 Zabbix Agent2 hostname: [ops-zabbix-proxy-1]
2024/06/20 13:15:55.031256 [101] sequential redirect responses detected
2024/06/20 13:15:55.031434 [101] history upload to [172.19.10.62:10051] [ops-zabbix-proxy-1] started to fail
2024/06/20 13:15:58.024719 [101] sequential redirect responses detected
2024/06/20 13:15:58.024779 [101] active check configuration update from host [ops-zabbix-proxy-1] started to fail
2024/06/20 13:16:05.023981 [101] sequential redirect responses detected
2024/06/20 13:16:05.025272 [101] sending of heartbeat message for [ops-zabbix-proxy-1] started to fail​
<EOF>

Zabbix-UI:



I have changed the topology in the mean time this results in the logs being different. I have changed the monitoring proxy for the hosts to proxy1 which resulted in both agents becoming available for active checks (as described before), but when i deleted the hosts in zabbix-ui and restarted the docker containers, the hosts registered again with the proxy-groups (as intended).

[Markku]

Thanks for the excellent logs.

What do you make of this entry in the agent log:

2024/06/20 13:15:55.031434 [101] history upload to [172.19.10.62:10051] [ops-zabbix-proxy-1] started to fail

Why does it say "172.19.10.62" (= ops-zabbix-proxy-2 IP address) and "ops-zabbix-proxy-1"?

Are you sure that your proxies have the correct "Address for active agents" field configured?

Markku

[Markku]

Walking through your setup:

- The agent "ops-zabbix-proxy-1" has "ServerActive=172.19.10.61:10051;172.19.10.62:1005 1", meaning that it will first connect to 172.19.10.61 (ops-zabbix-proxy-1 = itself)
- Based on your screenshot, the agent is actually monitored by ops-zabbix-proxy-2, so ops-zabbix-proxy-1 should give the agent a redirection to 172.19.10.62
- The agent should then connect to 172.19.10.62, and ops-zabbix-proxy-2 should accept the agent connections
- But based on the logs on ops-zabbix-proxy-2, it is not happy with the agent connecting there.

Since I'm a packethead, I would definitely fire up some tcpdumps to collect all that traffic above (as well as the traffic between proxies and Zabbix server), and then see it in Wireshark 4.3.0rc0 (or newer) what's really happening. Or, you could increase logging level of all those Zabbix components and see what you get in the logs.

Markku

[lapniklas]

I have executed all Parts with DebugLevel=4 which resulted in a lot of logs. Too much to be posted here, so i picked the relevant parts:

Zabbix-Agent (running on ops-zabbix-proxy-1 (connecting to 172.19.10.61)):
It looks like the agent contacted 172.19.10.61 once receives a redirect and from than on only talks to connecting to 172.19.10.62. Which is expected since (according to the zabbix-ui) zabbix-agent on 172.19.10.61 is monitored by zabbix-proxy on host 172.19.10.62.

2024/06/21 09:51:26.323505 Zabbix Agent 2 stopped. (7.0.0)
2024/06/21 09:51:26.393613 Starting Zabbix Agent 2 (7.0.0)
2024/06/21 09:51:26.397941 OpenSSL library (OpenSSL 3.0.2 15 Mar 2022) initialized
2024/06/21 09:51:26.398091 default context ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
2024/06/21 09:51:26.398175 psk context ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
2024/06/21 09:51:26.398389 using configuration file: /etc/zabbix/zabbix_agent2.conf
2024/06/21 09:51:26.398798 using plugin 'Agent' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.398838 using plugin 'Ceph' (built-in) providing following interfaces: exporter, runner, configurator
2024/06/21 09:51:26.398854 using plugin 'Cpu' (built-in) providing following interfaces: exporter, collector, runner
2024/06/21 09:51:26.398868 using plugin 'DNS' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.398903 using plugin 'Docker' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.398928 using plugin 'File' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.398948 using plugin 'Hw' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.398991 using plugin 'Kernel' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399009 using plugin 'Log' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.399024 using plugin 'MQTT' (built-in) providing following interfaces: watcher, configurator
2024/06/21 09:51:26.399091 using plugin 'Memcached' (built-in) providing following interfaces: exporter, runner, configurator
2024/06/21 09:51:26.399118 using plugin 'Memory' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399170 using plugin 'Modbus' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.399196 using plugin 'MongoDB' (/usr/sbin/zabbix-agent2-plugin/zabbix-agent2-plugin-mongodb) providing following interfaces: exporter, runner, configurator
2024/06/21 09:51:26.399243 using plugin 'Mysql' (built-in) providing following interfaces: exporter, runner, configurator
2024/06/21 09:51:26.399264 using plugin 'NetIf' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399282 using plugin 'Oracle' (built-in) providing following interfaces: exporter, runner, configurator
2024/06/21 09:51:26.399367 using plugin 'PostgreSQL' (/usr/sbin/zabbix-agent2-plugin/zabbix-agent2-plugin-postgresql) providing following interfaces: exporter, runner, configurator
2024/06/21 09:51:26.399424 using plugin 'Proc' (built-in) providing following interfaces: exporter, collector
2024/06/21 09:51:26.399442 using plugin 'ProcExporter' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399458 using plugin 'Redis' (built-in) providing following interfaces: exporter, runner, configurator
2024/06/21 09:51:26.399504 using plugin 'Smart' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.399528 using plugin 'Sw' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.399543 using plugin 'Swap' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399587 using plugin 'SystemRun' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.399607 using plugin 'Systemd' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399626 using plugin 'TCP' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.399701 using plugin 'UDP' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.399751 using plugin 'Uname' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399766 using plugin 'Uptime' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399780 using plugin 'Users' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.399835 using plugin 'VFSDev' (built-in) providing following interfaces: exporter, collector
2024/06/21 09:51:26.399852 using plugin 'VFSDir' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399880 using plugin 'VfsFs' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399922 using plugin 'WebCertificate' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.399938 using plugin 'WebPage' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.399954 using plugin 'ZabbixAsync' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.399993 using plugin 'ZabbixStats' (built-in) providing following interfaces: exporter, configurator
2024/06/21 09:51:26.400007 lowering the plugin ZabbixSync capacity to 1 as the configured capacity 1000 exceeds limits
2024/06/21 09:51:26.400024 using plugin 'ZabbixSync' (built-in) providing following interfaces: exporter
2024/06/21 09:51:26.400099 [Modbus] Config is valid
2024/06/21 09:51:26.401313 Plugin communication protocol version is 6.4.0
2024/06/21 09:51:26.401398 Zabbix Agent2 hostname: [ops-zabbix-proxy-1]
2024/06/21 09:51:26.401477 starting manager
2024/06/21 09:51:26.401736 [101] starting memory cache
2024/06/21 09:51:26.401768 [101] starting server connector for 172.19.10.61:10051,172.19.10.62:10051
2024/06/21 09:51:26.402270 [0] starting listener for '172.19.10.61:10050'
2024/06/21 09:51:26.402438 listening for control connections on /tmp/agent.sock
2024/06/21 09:51:27.401938 [101] In refreshActiveChecks() from 172.19.10.61:10051,172.19.10.62:10051
2024/06/21 09:51:27.409354 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:27.412801 performing redirect to [172.19.10.62:10051]
2024/06/21 09:51:27.422459 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:27.430638 [101] sequential redirect responses detected
2024/06/21 09:51:27.430704 [101] active check configuration update from host [ops-zabbix-proxy-1] started to fail
2024/06/21 09:51:27.430715 [101] End of refreshActiveChecks() from 172.19.10.62:10051,172.19.10.61:10051,172.19.10.62:10051
2024/06/21 09:51:27.430732 [101] In sendHeartbeatMsg() from 172.19.10.62:10051,172.19.10.61:10051,172.19.10.62:10051
2024/06/21 09:51:27.435412 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:27.438578 performing redirect to [172.19.10.62:10051]
2024/06/21 09:51:27.442813 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:27.446424 [101] sequential redirect responses detected
2024/06/21 09:51:27.446464 [101] sending of heartbeat message for [ops-zabbix-proxy-1] started to fail
2024/06/21 09:51:27.446475 [101] End of sendHeartBeatMsg() from 172.19.10.62:10051,172.19.10.61:10051,172.19.10.62:10051
2024/06/21 09:51:33.401993 [101] In refreshActiveChecks() from 172.19.10.62:10051,172.19.10.61:10051,172.19.10.62:10051
2024/06/21 09:51:33.408229 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:33.410916 performing redirect to [172.19.10.62:10051]
2024/06/21 09:51:33.415217 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:33.417937 [101] End of refreshActiveChecks() from 172.19.10.62:10051,172.19.10.61:10051,172.19.10.62:10051
2024/06/21 09:51:38.353384 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:38.354151 received passive check request: '{"request":"passive checks","data":[{"key":"agent.ping","timeout":3}]}' from '172.19.10.62'
2024/06/21 09:51:38.354705 [1] processing update request (1 requests)
2024/06/21 09:51:38.354900 [1] registering new client
2024/06/21 09:51:38.354983 [1] adding new request for key: 'agent.ping'
2024/06/21 09:51:38.355183 [1] created direct exporter task for plugin 'Agent' itemid:0 key 'agent.ping'
2024/06/21 09:51:38.355385 executing direct exporter task for key 'agent.ping'
2024/06/21 09:51:38.355479 executed direct exporter task for key 'agent.ping[[]]'
2024/06/21 09:51:38.355641 sending passive check response: '{"version":"7.0.0","variant":2,"data":[{"value":"1"}]}' to '172.19.10.62'
2024/06/21 09:51:39.402091 [101] In refreshActiveChecks() from 172.19.10.62:10051,172.19.10.61:10051,172.19.10.62:10051
2024/06/21 09:51:39.411505 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:39.413366 performing redirect to [172.19.10.62:10051]
2024/06/21 09:51:39.417722 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:39.420670 [101] End of refreshActiveChecks() from 172.19.10.62:10051,172.19.10.61:10051,172.19.10.62:10051
2024/06/21 09:51:45.402899 [101] In refreshActiveChecks() from 172.19.10.62:10051,172.19.10.61:10051,172.19.10.62:10051
2024/06/21 09:51:45.408224 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:45.409955 performing redirect to [172.19.10.62:10051]
2024/06/21 09:51:45.413464 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
2024/06/21 09:51:45.424997 [101] End of refreshActiveChecks() from 172.19.10.62:10051,172.19.10.61:10051,172.19.10.62:10051

Zabbix-Proxy 1 only gets one connection from the Zabbix-Agent and responds with a redirect, which is also correct:

{"log":"     1:20240621:095127.393 End of zbx_ipc_service_recv():2\n","stream":"stdout","time":"2024-06-21T09:51:27.394361038Z"}
{"log":"     1:20240621:095127.393 In zbx_ipc_service_recv() timeout:1.000\n","stream":"stdout","time":"2024-06-21T09:51:27.394431239Z"}
{"log":"   179:20240621:095127.403 zbx_setproctitle() title:'trapper #3 [processed data in 0.033413 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.403333238Z"}
{"log":"   178:20240621:095127.405 In zbx_tls_accept()\n","stream":"stdout","time":"2024-06-21T09:51:27.405879368Z"}
{"log":"   178:20240621:095127.406 zbx_psk_server_cb() requested PSK identity \"PSK 001\"\n","stream":"stdout","time":"2024-06-21T09:51:27.406326073Z"}
{"log":"   178:20240621:095127.409 End of zbx_tls_accept():SUCCEED (established TLSv1.3 TLS_CHACHA20_POLY1305_SHA256)\n","stream":"stdout","time":"2024-06-21T09:51:27.411933554Z"}
{"log":"   178:20240621:095127.409 zbx_setproctitle() title:'trapper #2 [processing data]'\n","stream":"stdout","time":"2024-06-21T09:51:27.411971223Z"}
{"log":"   178:20240621:095127.409 In zbx_ipc_async_socket_recv() timeout:0\n","stream":"stdout","time":"2024-06-21T09:51:27.411978736Z"}
{"log":"   178:20240621:095127.409 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.411986187Z"}
{"log":"   178:20240621:095127.411 trapper got '{\"request\":\"active checks\",\"host\":\"ops-zabbix-proxy-1\",\"version\":\"7.0.0\",\"variant\":2,\"session\":\"bdce532d99ca406a690c133d2d8b9838\",\"config_revision\":0,\"host_metadata\":\"zabbix_proxy-linux\",\"ip\":\"172.19.10.61\"}'\n","stream":"stdout","time":"2024-06-21T09:51:27.411994501Z"}
{"log":"   178:20240621:095127.411 In send_list_of_active_checks_json()\n","stream":"stdout","time":"2024-06-21T09:51:27.412004374Z"}
{"log":"   178:20240621:095127.411 In zbx_is_ip4() ip:'172.19.10.61'\n","stream":"stdout","time":"2024-06-21T09:51:27.412010606Z"}
{"log":"   178:20240621:095127.411 End of zbx_is_ip4():SUCCEED\n","stream":"stdout","time":"2024-06-21T09:51:27.412016259Z"}
{"log":"   178:20240621:095127.411 In get_hostid_by_host_or_autoregister() host:'ops-zabbix-proxy-1' metadata:'zabbix_proxy-linux'\n","stream":"stdout","time":"2024-06-21T09:51:27.41203635Z"}
{"log":"   178:20240621:095127.411 End of get_hostid_by_host_or_autoregister():FAIL\n","stream":"stdout","time":"2024-06-21T09:51:27.412043318Z"}
{"log":"   178:20240621:095127.411 cannot send list of active checks to \"172.19.10.61\": host \"ops-zabbix-proxy-1\" is monitored by another proxy\n","stream":"stdout","time":"2024-06-21T09:51:27.412050818Z"}
{"log":"   178:20240621:095127.412 send_list_of_active_checks_json() sending [{\"response\":\"failed\",\"redirect\":{\"revision\":9,\"address\":\"172.19.10.62:10051\"}}]\n","stream":"stdout","time":"2024-06-21T09:51:27.412229184Z"}
{"log":"   178:20240621:095127.412 End of send_list_of_active_checks_json():SUCCEED\n","stream":"stdout","time":"2024-06-21T09:51:27.413124794Z"}
{"log":"   178:20240621:095127.412 zbx_setproctitle() title:'trapper #2 [processed data in 0.002316 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.413150673Z"}
{"log":"   180:20240621:095127.759 zbx_setproctitle() title:'trapper #4 [processed data in 0.028470 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.759343324Z"}
{"log":"   182:20240621:095127.833 End of zbx_ipc_service_recv():2\n","stream":"stdout","time":"2024-06-21T09:51:27.833983682Z"}
{"log":"   182:20240621:095127.833 In zbx_pp_manager_process_finished()\n","stream":"stdout","time":"2024-06-21T09:51:27.834045559Z"}
{"log":"   182:20240621:095127.833 End of zbx_pp_manager_process_finished() values_num:0\n","stream":"stdout","time":"2024-06-21T09:51:27.834057962Z"}
{"log":"   182:20240621:095127.833 In zbx_ipc_service_recv() timeout:0.500\n","stream":"stdout","time":"2024-06-21T09:51:27.83406799Z"}
{"log":"   200:20240621:095127.844 End of zbx_ipc_service_recv():2\n","stream":"stdout","time":"2024-06-21T09:51:27.844895529Z"}
{"log":"   200:20240621:095127.844 In zbx_ipc_service_recv() timeout:1.000\n","stream":"stdout","time":"2024-06-21T09:51:27.844948036Z"}
{"log":"   181:20240621:095127.853 zbx_setproctitle() title:'trapper #5 [processed data in 0.021896 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.854292953Z"} zz0.fhl8oz35xw5zz

Zabbix-Proxy 2 gets serval requests from the Zabbix-agent but it falsely answers that this agent is monitored by a different host:

{"log":"   178:20240621:095126.900 In zbx_recv_response()\n","stream":"stdout","time":"2024-06-21T09:51:26.900574902Z"}
{"log":"   182:20240621:095126.910 End of zbx_ipc_service_recv():2\n","stream":"stdout","time":"2024-06-21T09:51:26.910530417Z"}
{"log":"   182:20240621:095126.910 In zbx_pp_manager_process_finished()\n","stream":"stdout","time":"2024-06-21T09:51:26.910590448Z"}
{"log":"   182:20240621:095126.910 End of zbx_pp_manager_process_finished() values_num:0\n","stream":"stdout","time":"2024-06-21T09:51:26.910602999Z"}
{"log":"   182:20240621:095126.910 In zbx_ipc_service_recv() timeout:0.500\n","stream":"stdout","time":"2024-06-21T09:51:26.910612014Z"}
{"log":"   178:20240621:095126.913 zbx_recv_response() '{\"response\":\"success\"}'\n","stream":"stdout","time":"2024-06-21T09:51:26.914179417Z"}
{"log":"   178:20240621:095126.913 End of zbx_recv_response():SUCCEED\n","stream":"stdout","time":"2024-06-21T09:51:26.914274079Z"}
{"log":"   178:20240621:095126.913 query [txnlev:1] [begin;]\n","stream":"stdout","time":"2024-06-21T09:51:26.914285353Z"}
{"log":"   178:20240621:095126.913 In zbx_pb_update_state() more:0\n","stream":"stdout","time":"2024-06-21T09:51:26.914292773Z"}
{"log":"   178:20240621:095126.913 In zbx_pb_update_state()\n","stream":"stdout","time":"2024-06-21T09:51:26.914299615Z"}
{"log":"   178:20240621:095126.913 query [txnlev:1] [commit;]\n","stream":"stdout","time":"2024-06-21T09:51:26.914307468Z"}
{"log":"   178:20240621:095126.914 End of send_proxy_data()\n","stream":"stdout","time":"2024-06-21T09:51:26.914318242Z"}
{"log":"   178:20240621:095126.914 zbx_setproctitle() title:'trapper #2 [processed data in 0.030557 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:26.914627279Z"}
{"log":"   189:20240621:095127.027 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.027791228Z"}
{"log":"   189:20240621:095127.027 zbx_setproctitle() title:'history syncer #3 [processed 0 values in 0.000686 sec, syncing history]'\n","stream":"stdout","time":"2024-06-21T09:51:27.027927545Z"}
{"log":"   189:20240621:095127.027 In zbx_sync_history_cache() history_num:0\n","stream":"stdout","time":"2024-06-21T09:51:27.027947897Z"}
{"log":"   189:20240621:095127.027 zbx_setproctitle() title:'history syncer #3 [processed 0 values in 0.000091 sec, idle 1 sec]'\n","stream":"stdout","time":"2024-06-21T09:51:27.02797463Z"}
{"log":"   189:20240621:095127.027 In zbx_ipc_async_socket_recv() timeout:1\n","stream":"stdout","time":"2024-06-21T09:51:27.027991806Z"}
{"log":"   190:20240621:095127.028 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.02863866Z"}
{"log":"   190:20240621:095127.028 zbx_setproctitle() title:'history syncer #4 [processed 0 values in 0.000076 sec, syncing history]'\n","stream":"stdout","time":"2024-06-21T09:51:27.028662457Z"}
{"log":"   190:20240621:095127.028 In zbx_sync_history_cache() history_num:0\n","stream":"stdout","time":"2024-06-21T09:51:27.028669255Z"}
{"log":"   190:20240621:095127.028 zbx_setproctitle() title:'history syncer #4 [processed 0 values in 0.000061 sec, idle 1 sec]'\n","stream":"stdout","time":"2024-06-21T09:51:27.028674625Z"}
{"log":"   190:20240621:095127.028 In zbx_ipc_async_socket_recv() timeout:1\n","stream":"stdout","time":"2024-06-21T09:51:27.028679653Z"}
{"log":"   188:20240621:095127.046 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.046441879Z"}
{"log":"   188:20240621:095127.046 zbx_setproctitle() title:'history syncer #2 [processed 0 values in 0.000046 sec, syncing history]'\n","stream":"stdout","time":"2024-06-21T09:51:27.046488605Z"}
{"log":"   188:20240621:095127.046 In zbx_sync_history_cache() history_num:0\n","stream":"stdout","time":"2024-06-21T09:51:27.046496159Z"}
{"log":"   188:20240621:095127.046 zbx_setproctitle() title:'history syncer #2 [processed 0 values in 0.000030 sec, idle 1 sec]'\n","stream":"stdout","time":"2024-06-21T09:51:27.046501919Z"}
{"log":"   188:20240621:095127.046 In zbx_ipc_async_socket_recv() timeout:1\n","stream":"stdout","time":"2024-06-21T09:51:27.046507818Z"}
{"log":"   202:20240621:095127.337 In zbx_ipc_async_socket_recv() timeout:0\n","stream":"stdout","time":"2024-06-21T09:51:27.338204975Z"}
{"log":"   202:20240621:095127.338 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.33825751Z"}
{"log":"   202:20240621:095127.338 In zbx_dc_config_get_poller_items() poller_type:7\n","stream":"stdout","time":"2024-06-21T09:51:27.338395909Z"}
{"log":"   202:20240621:095127.338 End of zbx_dc_config_get_poller_items():0\n","stream":"stdout","time":"2024-06-21T09:51:27.338836612Z"}
{"log":"   202:20240621:095127.338 queue processing_num:0 pending:0\n","stream":"stdout","time":"2024-06-21T09:51:27.339042871Z"}
{"log":"   204:20240621:095127.341 In zbx_ipc_async_socket_recv() timeout:0\n","stream":"stdout","time":"2024-06-21T09:51:27.341458961Z"}
{"log":"   204:20240621:095127.341 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.341481084Z"}
{"log":"   204:20240621:095127.341 In zbx_dc_config_get_poller_items() poller_type:9\n","stream":"stdout","time":"2024-06-21T09:51:27.34175119Z"}
{"log":"   204:20240621:095127.341 End of zbx_dc_config_get_poller_items():0\n","stream":"stdout","time":"2024-06-21T09:51:27.342053121Z"}
{"log":"   204:20240621:095127.342 queue processing_num:0 pending:0\n","stream":"stdout","time":"2024-06-21T09:51:27.342268172Z"}
{"log":"   203:20240621:095127.342 In zbx_ipc_async_socket_recv() timeout:0\n","stream":"stdout","time":"2024-06-21T09:51:27.34256701Z"}
{"log":"   203:20240621:095127.342 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.342581829Z"}
{"log":"   203:20240621:095127.342 In zbx_dc_config_get_poller_items() poller_type:8\n","stream":"stdout","time":"2024-06-21T09:51:27.342763902Z"}
{"log":"   203:20240621:095127.342 End of zbx_dc_config_get_poller_items():0\n","stream":"stdout","time":"2024-06-21T09:51:27.342779703Z"}
{"log":"   203:20240621:095127.342 queue processing_num:0 pending:0\n","stream":"stdout","time":"2024-06-21T09:51:27.342917361Z"}
{"log":"   191:20240621:095127.343 zbx_setproctitle() title:'self-monitoring [processing data]'\n","stream":"stdout","time":"2024-06-21T09:51:27.343593518Z"}
{"log":"   191:20240621:095127.343 In collect_selfmon_stats()\n","stream":"stdout","time":"2024-06-21T09:51:27.343609684Z"}
{"log":"   191:20240621:095127.343 In zbx_timekeeper_collect()\n","stream":"stdout","time":"2024-06-21T09:51:27.343615909Z"}
{"log":"   191:20240621:095127.343 End of zbx_timekeeper_collect()\n","stream":"stdout","time":"2024-06-21T09:51:27.343621318Z"}
{"log":"   191:20240621:095127.343 End of collect_selfmon_stats()\n","stream":"stdout","time":"2024-06-21T09:51:27.343626833Z"}
{"log":"   191:20240621:095127.343 zbx_setproctitle() title:'self-monitoring [processed data in 0.000082 sec, idle 1 sec]'\n","stream":"stdout","time":"2024-06-21T09:51:27.343632744Z"}
{"log":"   203:20240621:095127.356 In zbx_ipc_async_socket_recv() timeout:0\n","stream":"stdout","time":"2024-06-21T09:51:27.35685077Z"}
{"log":"   203:20240621:095127.356 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.356899886Z"}
{"log":"   182:20240621:095127.410 End of zbx_ipc_service_recv():2\n","stream":"stdout","time":"2024-06-21T09:51:27.411428221Z"}
{"log":"   182:20240621:095127.411 In zbx_pp_manager_process_finished()\n","stream":"stdout","time":"2024-06-21T09:51:27.411499854Z"}
{"log":"   182:20240621:095127.411 In zbx_timekeeper_collect()\n","stream":"stdout","time":"2024-06-21T09:51:27.411515849Z"}
{"log":"   182:20240621:095127.411 End of zbx_timekeeper_collect()\n","stream":"stdout","time":"2024-06-21T09:51:27.411528089Z"}
{"log":"   182:20240621:095127.411 End of zbx_pp_manager_process_finished() values_num:0\n","stream":"stdout","time":"2024-06-21T09:51:27.411540116Z"}
{"log":"   182:20240621:095127.411 In zbx_ipc_service_recv() timeout:0.500\n","stream":"stdout","time":"2024-06-21T09:51:27.412006883Z"}
{"log":"   177:20240621:095127.417 zbx_setproctitle() title:'trapper #1 [processed data in 0.034673 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.417579975Z"}
{"log":"   178:20240621:095127.417 In zbx_tls_accept()\n","stream":"stdout","time":"2024-06-21T09:51:27.418062151Z"}
{"log":"   178:20240621:095127.418 zbx_psk_server_cb() requested PSK identity \"PSK 001\"\n","stream":"stdout","time":"2024-06-21T09:51:27.418892384Z"}
{"log":"   178:20240621:095127.422 End of zbx_tls_accept():SUCCEED (established TLSv1.3 TLS_CHACHA20_POLY1305_SHA256)\n","stream":"stdout","time":"2024-06-21T09:51:27.423176328Z"}
{"log":"   178:20240621:095127.422 zbx_setproctitle() title:'trapper #2 [processing data]'\n","stream":"stdout","time":"2024-06-21T09:51:27.423220054Z"}
{"log":"   178:20240621:095127.422 In zbx_ipc_async_socket_recv() timeout:0\n","stream":"stdout","time":"2024-06-21T09:51:27.42323034Z"}
{"log":"   178:20240621:095127.422 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.423239252Z"}
{"log":"   178:20240621:095127.429 trapper got '{\"request\":\"active checks\",\"host\":\"ops-zabbix-proxy-1\",\"version\":\"7.0.0\",\"variant\":2,\"session\":\"bdce532d99ca406a690c133d2d8b9838\",\"config_revision\":0,\"host_metadata\":\"zabbix_proxy-linux\",\"ip\":\"172.19.10.61\"}'\n","stream":"stdout","time":"2024-06-21T09:51:27.429934998Z"}
{"log":"   178:20240621:095127.429 In send_list_of_active_checks_json()\n","stream":"stdout","time":"2024-06-21T09:51:27.429975907Z"}
{"log":"   178:20240621:095127.429 In zbx_is_ip4() ip:'172.19.10.61'\n","stream":"stdout","time":"2024-06-21T09:51:27.430091281Z"}
{"log":"   178:20240621:095127.429 End of zbx_is_ip4():SUCCEED\n","stream":"stdout","time":"2024-06-21T09:51:27.430140883Z"}
{"log":"   178:20240621:095127.429 In get_hostid_by_host_or_autoregister() host:'ops-zabbix-proxy-1' metadata:'zabbix_proxy-linux'\n","stream":"stdout","time":"2024-06-21T09:51:27.430150748Z"}
{"log":"   178:20240621:095127.429 End of get_hostid_by_host_or_autoregister():FAIL\n","stream":"stdout","time":"2024-06-21T09:51:27.430160045Z"}
{"log":"   178:20240621:095127.429 cannot send list of active checks to \"172.19.10.61\": host \"ops-zabbix-proxy-1\" is monitored by another proxy\n","stream":"stdout","time":"2024-06-21T09:51:27.43016904Z"}
{"log":"   178:20240621:095127.429 send_list_of_active_checks_json() sending [{\"response\":\"failed\",\"redirect\":{\"revision\":9,\"address\":\"172.19.10.62:10051\"}}]\n","stream":"stdout","time":"2024-06-21T09:51:27.430178628Z"}
{"log":"   178:20240621:095127.429 End of send_list_of_active_checks_json():SUCCEED\n","stream":"stdout","time":"2024-06-21T09:51:27.430191088Z"}
{"log":"   178:20240621:095127.430 zbx_setproctitle() title:'trapper #2 [processed data in 0.006989 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.430339742Z"}
{"log":"   178:20240621:095127.431 In zbx_tls_accept()\n","stream":"stdout","time":"2024-06-21T09:51:27.432215607Z"}
{"log":"   178:20240621:095127.432 zbx_psk_server_cb() requested PSK identity \"PSK 001\"\n","stream":"stdout","time":"2024-06-21T09:51:27.432786847Z"}
{"log":"   178:20240621:095127.435 End of zbx_tls_accept():SUCCEED (established TLSv1.3 TLS_CHACHA20_POLY1305_SHA256)\n","stream":"stdout","time":"2024-06-21T09:51:27.436058626Z"}
{"log":"   178:20240621:095127.435 zbx_setproctitle() title:'trapper #2 [processing data]'\n","stream":"stdout","time":"2024-06-21T09:51:27.436104608Z"}
{"log":"   178:20240621:095127.435 In zbx_ipc_async_socket_recv() timeout:0\n","stream":"stdout","time":"2024-06-21T09:51:27.436125145Z"}
{"log":"   178:20240621:095127.435 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.436142106Z"}
{"log":"   178:20240621:095127.436 trapper got '{\"request\":\"active check heartbeat\",\"host\":\"ops-zabbix-proxy-1\",\"heartbeat_freq\":60,\"version\":\"7.0.0\",\"variant\":2}'\n","stream":"stdout","time":"2024-06-21T09:51:27.436754938Z"}
{"log":"   178:20240621:095127.436 In zbx_send_response_json()\n","stream":"stdout","time":"2024-06-21T09:51:27.437307359Z"}
{"log":"   178:20240621:095127.436 zbx_send_response_json() '{\"redirect\":{\"revision\":9,\"address\":\"172.19.10.62:10051\"},\"response\":\"failed\"}'\n","stream":"stdout","time":"2024-06-21T09:51:27.43753872Z"}
{"log":"   178:20240621:095127.438 End of zbx_send_response_json():SUCCEED\n","stream":"stdout","time":"2024-06-21T09:51:27.438235625Z"}
{"log":"   178:20240621:095127.438 zbx_setproctitle() title:'trapper #2 [processed data in 0.002142 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.439013688Z"}
{"log":"   177:20240621:095127.439 In zbx_tls_accept()\n","stream":"stdout","time":"2024-06-21T09:51:27.442270558Z"}
{"log":"   177:20240621:095127.440 zbx_psk_server_cb() requested PSK identity \"PSK 001\"\n","stream":"stdout","time":"2024-06-21T09:51:27.442388894Z"}
{"log":"   177:20240621:095127.443 End of zbx_tls_accept():SUCCEED (established TLSv1.3 TLS_CHACHA20_POLY1305_SHA256)\n","stream":"stdout","time":"2024-06-21T09:51:27.443391544Z"}
{"log":"   177:20240621:095127.443 zbx_setproctitle() title:'trapper #1 [processing data]'\n","stream":"stdout","time":"2024-06-21T09:51:27.443416399Z"}
{"log":"   177:20240621:095127.443 In zbx_ipc_async_socket_recv() timeout:0\n","stream":"stdout","time":"2024-06-21T09:51:27.443426076Z"}
{"log":"   177:20240621:095127.443 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.443572811Z"}
{"log":"   177:20240621:095127.444 trapper got '{\"request\":\"active check heartbeat\",\"host\":\"ops-zabbix-proxy-1\",\"heartbeat_freq\":60,\"version\":\"7.0.0\",\"variant\":2}'\n","stream":"stdout","time":"2024-06-21T09:51:27.444970738Z"}
{"log":"   177:20240621:095127.444 In zbx_send_response_json()\n","stream":"stdout","time":"2024-06-21T09:51:27.445026745Z"}
{"log":"   177:20240621:095127.444 zbx_send_response_json() '{\"redirect\":{\"revision\":9,\"address\":\"172.19.10.62:10051\"},\"response\":\"failed\"}'\n","stream":"stdout","time":"2024-06-21T09:51:27.445040364Z"}
{"log":"   177:20240621:095127.445 End of zbx_send_response_json():SUCCEED\n","stream":"stdout","time":"2024-06-21T09:51:27.446098867Z"}
{"log":"   177:20240621:095127.446 zbx_setproctitle() title:'trapper #1 [processed data in 0.002395 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.446435672Z"}
{"log":"   179:20240621:095127.573 zbx_setproctitle() title:'trapper #3 [processed data in 0.031405 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.573782637Z"}
{"log":"     1:20240621:095127.640 End of zbx_ipc_service_recv():2\n","stream":"stdout","time":"2024-06-21T09:51:27.640295114Z"}
{"log":"   180:20240621:095127.640 zbx_setproctitle() title:'trapper #4 [processed data in 0.053581 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.640336915Z"}
{"log":"     1:20240621:095127.640 In zbx_ipc_service_recv() timeout:1.000\n","stream":"stdout","time":"2024-06-21T09:51:27.640345516Z"}
{"log":"   186:20240621:095127.672 End of zbx_ipc_service_recv():2\n","stream":"stdout","time":"2024-06-21T09:51:27.672376077Z"}
{"log":"   186:20240621:095127.672 In zbx_timekeeper_collect()\n","stream":"stdout","time":"2024-06-21T09:51:27.672431085Z"}
{"log":"   186:20240621:095127.672 End of zbx_timekeeper_collect()\n","stream":"stdout","time":"2024-06-21T09:51:27.672485506Z"}
{"log":"   186:20240621:095127.672 In process_results() del_druleids:0\n","stream":"stdout","time":"2024-06-21T09:51:27.672496683Z"}
{"log":"   186:20240621:095127.672 process_results() results=0 checks:0/0 del_druleids=0 incomplete_druleids=0\n","stream":"stdout","time":"2024-06-21T09:51:27.672508756Z"}
{"log":"   186:20240621:095127.672 End of process_results() ret:0\n","stream":"stdout","time":"2024-06-21T09:51:27.672515961Z"}
{"log":"   186:20240621:095127.672 zbx_setproctitle() title:'discovery manager #1 [processing 0 rules, 0 unsaved checks]'\n","stream":"stdout","time":"2024-06-21T09:51:27.672527977Z"}
{"log":"   186:20240621:095127.672 In process_discovery()\n","stream":"stdout","time":"2024-06-21T09:51:27.672535317Z"}
{"log":"   186:20240621:095127.672 End of process_discovery() rule_count:0 nextcheck:-1\n","stream":"stdout","time":"2024-06-21T09:51:27.672568401Z"}
{"log":"   186:20240621:095127.672 In zbx_ipc_service_recv() timeout:5.000\n","stream":"stdout","time":"2024-06-21T09:51:27.67259323Z"}
{"log":"   181:20240621:095127.812 zbx_setproctitle() title:'trapper #5 [processed data in 0.030077 sec, waiting for connection]'\n","stream":"stdout","time":"2024-06-21T09:51:27.812887032Z"}
{"log":"   187:20240621:095127.852 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.852657994Z"}
{"log":"   187:20240621:095127.852 zbx_setproctitle() title:'history syncer #1 [processed 0 values in 0.000029 sec, syncing history]'\n","stream":"stdout","time":"2024-06-21T09:51:27.85271002Z"}
{"log":"   187:20240621:095127.852 In zbx_sync_history_cache() history_num:0\n","stream":"stdout","time":"2024-06-21T09:51:27.852718999Z"}
{"log":"   187:20240621:095127.852 zbx_setproctitle() title:'history syncer #1 [processed 0 values in 0.000039 sec, idle 1 sec]'\n","stream":"stdout","time":"2024-06-21T09:51:27.852724286Z"}
{"log":"   187:20240621:095127.852 In zbx_ipc_async_socket_recv() timeout:1\n","stream":"stdout","time":"2024-06-21T09:51:27.852730307Z"}
{"log":"   200:20240621:095127.900 End of zbx_ipc_service_recv():2\n","stream":"stdout","time":"2024-06-21T09:51:27.900690944Z"}
{"log":"   200:20240621:095127.900 In zbx_ipc_service_recv() timeout:1.000\n","stream":"stdout","time":"2024-06-21T09:51:27.900734223Z"}
{"log":"   182:20240621:095127.912 End of zbx_ipc_service_recv():2\n","stream":"stdout","time":"2024-06-21T09:51:27.912520768Z"}
{"log":"   182:20240621:095127.912 In zbx_pp_manager_process_finished()\n","stream":"stdout","time":"2024-06-21T09:51:27.912583978Z"}
{"log":"   182:20240621:095127.912 End of zbx_pp_manager_process_finished() values_num:0\n","stream":"stdout","time":"2024-06-21T09:51:27.912594224Z"}
{"log":"   182:20240621:095127.912 In zbx_ipc_service_recv() timeout:0.500\n","stream":"stdout","time":"2024-06-21T09:51:27.912600535Z"}
{"log":"   178:20240621:095127.984 In zbx_tls_accept()\n","stream":"stdout","time":"2024-06-21T09:51:27.985121194Z"}
{"log":"   178:20240621:095127.984 zbx_psk_server_cb() requested PSK identity \"psk_003\"\n","stream":"stdout","time":"2024-06-21T09:51:27.985186866Z"}
{"log":"   178:20240621:095127.998 End of zbx_tls_accept():SUCCEED (established TLSv1.3 TLS_CHACHA20_POLY1305_SHA256)\n","stream":"stdout","time":"2024-06-21T09:51:27.999178124Z"}
{"log":"   178:20240621:095127.998 zbx_setproctitle() title:'trapper #2 [processing data]'\n","stream":"stdout","time":"2024-06-21T09:51:27.999255527Z"}
{"log":"   178:20240621:095127.998 In zbx_ipc_async_socket_recv() timeout:0\n","stream":"stdout","time":"2024-06-21T09:51:27.999270148Z"}
{"log":"   178:20240621:095127.998 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:27.99928102Z"}
{"log":"   178:20240621:095128.011 trapper got '{\"request\":\"proxy data\"}'\n","stream":"stdout","time":"2024-06-21T09:51:28.011523425Z"}
{"log":"   178:20240621:095128.011 In send_proxy_data()\n","stream":"stdout","time":"2024-06-21T09:51:28.011557754Z"}
{"log":"   178:20240621:095128.011 In zbx_get_interface_availability_data()\n","stream":"stdout","time":"2024-06-21T09:51:28.0115664Z"}
{"log":"   178:20240621:095128.011 In zbx_dc_get_interfaces_availability()\n","stream":"stdout","time":"2024-06-21T09:51:28.011572002Z"}
{"log":"   178:20240621:095128.011 End of zbx_dc_get_interfaces_availability() interfaces:0\n","stream":"stdout","time":"2024-06-21T09:51:28.011577044Z"}
{"log":"   178:20240621:095128.011 End of zbx_get_interface_availability_data():FAIL\n","stream":"stdout","time":"2024-06-21T09:51:28.011582022Z"}
{"log":"   178:20240621:095128.011 In zbx_pb_history_get_rows() lastid:0\n","stream":"stdout","time":"2024-06-21T09:51:28.011586989Z"}
{"log":"   178:20240621:095128.011 In pb_history_get_db()\n","stream":"stdout","time":"2024-06-21T09:51:28.012247583Z"}
{"log":"   178:20240621:095128.011 In pb_get_lastid() field:'proxy_history.history_lastid'\n","stream":"stdout","time":"2024-06-21T09:51:28.012263859Z"}
{"log":"   178:20240621:095128.011 query [txnlev:0] [select nextid from ids where table_name='proxy_history' and field_name='history_lastid']\n","stream":"stdout","time":"2024-06-21T09:51:28.012269691Z"}
{"log":"   178:20240621:095128.011 End of pb_get_lastid():612\n","stream":"stdout","time":"2024-06-21T09:51:28.012285352Z"}
{"log":"   178:20240621:095128.012 In pb_history_get_rows_db() lastid:612\n","stream":"stdout","time":"2024-06-21T09:51:28.012290426Z"}
{"log":"   178:20240621:095128.012 query [txnlev:0] [select id,itemid,clock,ns,timestamp,source,severity,value,logeventid,state,lastlogsize,mtime,flags from proxy_history where id\u003e612 order by id limit 1000]\n","stream":"stdout","time":"2024-06-21T09:51:28.012295511Z"}
{"log":"   178:20240621:095128.012 End of pb_history_get_rows_db() rows:0\n","stream":"stdout","time":"2024-06-21T09:51:28.01268381Z"}
{"log":"   178:20240621:095128.012 End of pb_history_get_db() lastid:0 records_num:0 size:~45 more:0\n","stream":"stdout","time":"2024-06-21T09:51:28.012695833Z"}
{"log":"   178:20240621:095128.012 End of zbx_pb_history_get_rows() rows:0\n","stream":"stdout","time":"2024-06-21T09:51:28.012701389Z"}
{"log":"   178:20240621:095128.012 In zbx_pb_discovery_get_rows() lastid:0\n","stream":"stdout","time":"2024-06-21T09:51:28.012711937Z"}
{"log":"   178:20240621:095128.012 In pb_get_lastid() field:'proxy_dhistory.dhistory_lastid'\n","stream":"stdout","time":"2024-06-21T09:51:28.012717218Z"}
{"log":"   178:20240621:095128.012 query [txnlev:0] [select nextid from ids where table_name='proxy_dhistory' and field_name='dhistory_lastid']\n","stream":"stdout","time":"2024-06-21T09:51:28.012745259Z"}
{"log":"   178:20240621:095128.012 End of pb_get_lastid():0\n","stream":"stdout","time":"2024-06-21T09:51:28.012768257Z"}
{"log":"   178:20240621:095128.012 In pb_get_rows_db() table:'proxy_dhistory'\n","stream":"stdout","time":"2024-06-21T09:51:28.013011628Z"}
{"log":"   178:20240621:095128.012 query [txnlev:0] [select id,clock,druleid,dcheckid,ip,dns,port,value,status,error from proxy_dhistory where id\u003e0 order by id limit 1000]\n","stream":"stdout","time":"2024-06-21T09:51:28.013020728Z"}
{"log":"   178:20240621:095128.012 End of pb_get_rows_db():0 lastid:0 more:0 size:45\n","stream":"stdout","time":"2024-06-21T09:51:28.013026274Z"}
{"log":"   178:20240621:095128.012 End of zbx_pb_discovery_get_rows() rows:0\n","stream":"stdout","time":"2024-06-21T09:51:28.013492741Z"}
{"log":"   178:20240621:095128.013 In zbx_pb_autoreg_get_rows() lastid:0\n","stream":"stdout","time":"2024-06-21T09:51:28.013523573Z"}
{"log":"   178:20240621:095128.013 In pb_get_lastid() field:'proxy_autoreg_host.autoreg_host_lastid'\n","stream":"stdout","time":"2024-06-21T09:51:28.013530719Z"}
{"log":"   178:20240621:095128.013 query [txnlev:0] [select nextid from ids where table_name='proxy_autoreg_host' and field_name='autoreg_host_lastid']\n","stream":"stdout","time":"2024-06-21T09:51:28.013538024Z"}
{"log":"   178:20240621:095128.013 End of pb_get_lastid():0\n","stream":"stdout","time":"2024-06-21T09:51:28.013763418Z"}
{"log":"   178:20240621:095128.013 In pb_get_rows_db() table:'proxy_autoreg_host'\n","stream":"stdout","time":"2024-06-21T09:51:28.014218983Z"}
{"log":"   178:20240621:095128.014 query [txnlev:0] [select id,clock,host,listen_ip,listen_dns,listen_port,host_metadata,flags,tls_accepted from proxy_autoreg_host where id\u003e0 order by id limit 1000]\n","stream":"stdout","time":"2024-06-21T09:51:28.01424541Z"}
{"log":"   178:20240621:095128.014 End of pb_get_rows_db():0 lastid:0 more:0 size:45\n","stream":"stdout","time":"2024-06-21T09:51:28.014887553Z"}
{"log":"   178:20240621:095128.014 End of zbx_pb_autoreg_get_rows() rows:0\n","stream":"stdout","time":"2024-06-21T09:51:28.014926615Z"}
{"log":"   178:20240621:095128.014 In zbx_ipc_socket_write()\n","stream":"stdout","time":"2024-06-21T09:51:28.014991843Z"}
{"log":"   200:20240621:095128.014 End of zbx_ipc_service_recv():1\n","stream":"stdout","time":"2024-06-21T09:51:28.01500684Z"}
{"log":"   200:20240621:095128.014 In zbx_ipc_client_send() clientid:7\n","stream":"stdout","time":"2024-06-21T09:51:28.015020094Z"}
{"log":"   200:20240621:095128.014 End of zbx_ipc_client_send():SUCCEED\n","stream":"stdout","time":"2024-06-21T09:51:28.015033774Z"}
{"log":"   200:20240621:095128.014 In zbx_ipc_service_recv() timeout:1.000\n","stream":"stdout","time":"2024-06-21T09:51:28.015722022Z"}
{"log":"   178:20240621:095128.014 End of zbx_ipc_socket_write():SUCCEED\n","stream":"stdout","time":"2024-06-21T09:51:28.015757601Z"}
{"log":"   178:20240621:095128.015 In zbx_ipc_socket_read()\n","stream":"stdout","time":"2024-06-21T09:51:28.015772924Z"}
{"log":"   178:20240621:095128.015 End of zbx_ipc_socket_read():SUCCEED\n","stream":"stdout","time":"2024-06-21T09:51:28.015786463Z"}
{"log":"   178:20240621:095128.015 query [txnlev:0] [select t.taskid,t.type,t.clock,t.ttl,r.status,r.parent_taskid,r.info,tr.status,tr.parent_taskid,tr.info,d.data,d.parent_taskid,d.type from task t left join task_remote_command_result r on t.taskid=r.taskid left join task_result tr on t.taskid=tr.taskid left join task_data d on t.taskid=d.taskid where t.status=1 and t.type in (3,8,9) order by t.taskid]\n","stream":"stdout","time":"2024-06-21T09:51:28.01580023Z"}
{"log":"   178:20240621:095128.016 In zbx_recv_response()\n","stream":"stdout","time":"2024-06-21T09:51:28.01673509Z"}
{"log":"   189:20240621:095128.028 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:28.029185376Z"}
{"log":"   189:20240621:095128.028 zbx_setproctitle() title:'history syncer #3 [processed 0 values in 0.000091 sec, syncing history]'\n","stream":"stdout","time":"2024-06-21T09:51:28.029303983Z"}
{"log":"   189:20240621:095128.029 In zbx_sync_history_cache() history_num:0\n","stream":"stdout","time":"2024-06-21T09:51:28.029313151Z"}
{"log":"   189:20240621:095128.029 zbx_setproctitle() title:'history syncer #3 [processed 0 values in 0.000041 sec, idle 1 sec]'\n","stream":"stdout","time":"2024-06-21T09:51:28.029319043Z"}
{"log":"   189:20240621:095128.029 In zbx_ipc_async_socket_recv() timeout:1\n","stream":"stdout","time":"2024-06-21T09:51:28.029325068Z"}
{"log":"   190:20240621:095128.029 End of zbx_ipc_async_socket_recv():0\n","stream":"stdout","time":"2024-06-21T09:51:28.029469128Z"}
{"log":"   190:20240621:095128.029 zbx_setproctitle() title:'history syncer #4 [processed 0 values in 0.000061 sec, syncing history]'\n","stream":"stdout","time":"2024-06-21T09:51:28.02966969Z"}
{"log":"   190:20240621:095128.029 In zbx_sync_history_cache() history_num:0\n","stream":"stdout","time":"2024-06-21T09:51:28.029709782Z"}
{"log":"   190:20240621:095128.029 zbx_setproctitle() title:'history syncer #4 [processed 0 values in 0.000134 sec, idle 1 sec]'\n","stream":"stdout","time":"2024-06-21T09:51:28.029948478Z"}​

The behaviour of zabbix-proxy-2 continues forever on... Meanwhile the zabbix-ui tells me that zabbix-proxy-2 is responsible to monitor this zabbix-agent.

Configuration of zabbix-agent

PidFile=/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
LogFileSize=0
ControlSocket=/tmp/agent.sock
# Include docker networks since local requests proxy1 <=> agent on proxy1 will not be routed through eth0
Server=172.19.10.61,172.19.10.62,172.26.0.1/16,172.27.0.0/16,172.28.0.0/16,172.29.0.0/16
DebugLevel=4
ListenIP=172.19.10.61
ListenPort=10050
ServerActive=172.19.10.61:10051;172.19.10.62:10051
Hostname=ops-zabbix-proxy-1
Include=/etc/zabbix/zabbix_agent2.d/*.conf
Include=/etc/zabbix/zabbix_agent2.d/plugins.d/*.conf

# PSK encryption
TLSConnect=psk
TLSAccept=psk
TLSPSKIdentity=PSK 001
TLSPSKFile=/etc/zabbix/psk_001.psk

HostMetadata=zabbix_proxy-linux​

Configuration of zabbix-proxy-1 (Docker-Envs)

                "ZBX_SERVER_HOST=<SOME PUBLIC IP>",
                "ZBX_PROXYMODE=1",
                "ZBX_TLSACCEPT=psk",
                "ZBX_TLSCONNECT=psk",
                "ZBX_HOSTNAME=hetzner-proxy",
                "ZBX_TLSPSKIDENTITY=psk_003",
                "ZBX_TLSPSKFILE=/var/lib/zabbix/enc/psk.psk",
                "ZBX_DEBUGLEVEL=4",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "ZBX_VERSION=7.0.0",
                "ZBX_SOURCES=https://git.zabbix.com/scm/zbx/zabbix.git",
                "MIBDIRS=/var/lib/mibs/ietf:/var/lib/mibs/iana:/usr/share/snmp/mibs:/var/lib/zabbix/mibs",
                "MIBS=+ALL",
                "NMAP_PRIVILEGED="​

Configuration of zabbix-proxy-2 (Docker-Envs)

                "ZBX_SERVER_HOST=<SOME PUBLIC IP>",
                "ZBX_PROXYMODE=1",
                "ZBX_TLSACCEPT=psk",
                "ZBX_TLSCONNECT=psk",
                "ZBX_HOSTNAME=hetzner-proxy",
                "ZBX_TLSPSKIDENTITY=psk_003",
                "ZBX_TLSPSKFILE=/var/lib/zabbix/enc/psk.psk",
                "ZBX_DEBUGLEVEL=4",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "ZBX_VERSION=7.0.0",
                "ZBX_SOURCES=https://git.zabbix.com/scm/zbx/zabbix.git",
                "MIBDIRS=/var/lib/mibs/ietf:/var/lib/mibs/iana:/usr/share/snmp/mibs:/var/lib/zabbix/mibs",
                "MIBS=+ALL",
                "NMAP_PRIVILEGED="​

[Markku]

At least this is incorrect:

"ZBX_HOSTNAME=hetzner-proxy",

on both proxies.

https://www.zabbix.com/documentation/current/en/manual/appendix/config/zabbix_proxy#hostname

A unique, case sensitive proxy name.

Markku

[lapniklas]

Nice catch! This was the issue... Thank you soooo much for your help!