Hello,
I work with an NGO and we have been using Zabbix for about a year now to monitor our various network devices and locations -
We have a Zabbix Server in AWS, EC2 - recently updated to 6.4, had been running 6.0 so far
We have a Raspberry Pi at each location, that acts as a Proxy for the location, monitors the local devices via PING or SNMP and reports to the server.
Zabbix Server listens to port 8080 (setup in config) and the agents/proxies we have work as ACTIVE and inform the zabbix server on zabbix.OURDOMAIN.com:8080
8080 is one of the supported ports for cloudflare DNS proxy
we use Cloudflare for our domain / dns, I had setup a subdomain, zabbix.OURDOMAIN.com on cloudflare, with A record, with IPV4 pointing to our AWS EC2 Public IP, I had to keep cloudflare proxy disabled - this would show SSL error but the zabbix agents/proxies could connect to the zabbix server
I created another subdomain monitor.OURDOMAIN.com on cloudflare, with A record, pointing to the same IPV4 but this time, I kept the cloudflare proxy enabled, this allowed me to provide the monitor subdomain to people to use and they wouldnt see the SSL error.
the ABOVE ALL WORKS... has been working and we are quite happy with how beneficial Zabbix has been for us.
Now, AWS is starting to charge for IPV4, from 1st Feb 2024 - so I wanted to see if I can run Zabbix on AWS EC2 with a VPC that only has IPV6 - I was able to run Zabbix and it works
on Cloudflare, I kept the same SUB-DOMAINS, but changed the record type from A to AAAA and entered the public IPV6
Some locations work, most dont - many of the locations are in rural areas, so the ISPs dont have IPV6 support
Now, some of our locations have ISPs that have IPV6 enabled, the agents and proxies there can connect to zabbix via the zabbix.OURDOMAIN.com host, but where the ISP does not have IPV6 support, the agents and proxies fail to connect - when I connect the agents/proxies via the PROXIED cloudflare, I get error, upon checking the logs, It says header missing or invalid header and message ignored.
Can I do some configuration / setup on either SERVER or AGENT/PROXY side to fix/avoid these errors, maybe ignore these header changes
Cloudflare changes some of the headers of packets, as per https://developers.cloudflare.com/fundamentals/reference/http-request-headers/#:~:text=Cloudflare%20passes%20all%20HTTP%20reques t,(dot)%20character.
Snippet of ERROR LOG from Proxy - similar issues seen in the Agent
2402:20240130:191045.100 In zbx_recv_response()
2402:20240130:191045.103 Message from zabbix.OURDOMAIN.com is missing header. Message ignored.
2402:20240130:191045.104 End of zbx_recv_response():FAIL
2402:20240130:191045.104 End of zbx_put_data_to_server():FAIL
2402:20240130:191045.104 cannot send proxy data to server at "zabbix.fOURDOMAIN.com":
2402:20240130:191045.104 End of proxy_data_sender():FAIL more:0 flags:0x8000
2402:20240130:191045.105 zbx_setproctitle() title:'data sender [sent 0 values in 0.014359 sec, idle 1 sec]'
THANKS for your time to read and many more thanks if you can guide me with a reply.
If I need to share anything else, let me know ?
My Zabbix Server / Agents / Proxies are all on 6.4 (latest)
My server is on a Ubuntu 22 machine hosted in AWS EC2
My agents/proxies are mostly Raspberry Pi
I have verified that the error occurs ONLY when the agents/proxies try to connect to Zabbix via the cloudflare proxied sub-domain
I know, I am just trying to save 5USD here, but I want to see or understand why Zabbix is ignoring the messages, even when devices are connecting via Cloudflare Proxy ?
ultimately, I have changed almost all other services from IPV4 to IPV6 only EC2 machines and they work fine with CloudFlare Proxy - for Zabbix, even in the past, I had to create an additional subdomain without the proxy, but now the IPV6 problem on ISP side doesnt allow me to connect to IPV6 directly, and zabbix ignores the messages via the proxy.
I work with an NGO and we have been using Zabbix for about a year now to monitor our various network devices and locations -
We have a Zabbix Server in AWS, EC2 - recently updated to 6.4, had been running 6.0 so far
We have a Raspberry Pi at each location, that acts as a Proxy for the location, monitors the local devices via PING or SNMP and reports to the server.
Zabbix Server listens to port 8080 (setup in config) and the agents/proxies we have work as ACTIVE and inform the zabbix server on zabbix.OURDOMAIN.com:8080
8080 is one of the supported ports for cloudflare DNS proxy
we use Cloudflare for our domain / dns, I had setup a subdomain, zabbix.OURDOMAIN.com on cloudflare, with A record, with IPV4 pointing to our AWS EC2 Public IP, I had to keep cloudflare proxy disabled - this would show SSL error but the zabbix agents/proxies could connect to the zabbix server
I created another subdomain monitor.OURDOMAIN.com on cloudflare, with A record, pointing to the same IPV4 but this time, I kept the cloudflare proxy enabled, this allowed me to provide the monitor subdomain to people to use and they wouldnt see the SSL error.
the ABOVE ALL WORKS... has been working and we are quite happy with how beneficial Zabbix has been for us.
Now, AWS is starting to charge for IPV4, from 1st Feb 2024 - so I wanted to see if I can run Zabbix on AWS EC2 with a VPC that only has IPV6 - I was able to run Zabbix and it works
on Cloudflare, I kept the same SUB-DOMAINS, but changed the record type from A to AAAA and entered the public IPV6
Some locations work, most dont - many of the locations are in rural areas, so the ISPs dont have IPV6 support
Now, some of our locations have ISPs that have IPV6 enabled, the agents and proxies there can connect to zabbix via the zabbix.OURDOMAIN.com host, but where the ISP does not have IPV6 support, the agents and proxies fail to connect - when I connect the agents/proxies via the PROXIED cloudflare, I get error, upon checking the logs, It says header missing or invalid header and message ignored.
Can I do some configuration / setup on either SERVER or AGENT/PROXY side to fix/avoid these errors, maybe ignore these header changes
Cloudflare changes some of the headers of packets, as per https://developers.cloudflare.com/fundamentals/reference/http-request-headers/#:~:text=Cloudflare%20passes%20all%20HTTP%20reques t,(dot)%20character.
Snippet of ERROR LOG from Proxy - similar issues seen in the Agent
2402:20240130:191045.100 In zbx_recv_response()
2402:20240130:191045.103 Message from zabbix.OURDOMAIN.com is missing header. Message ignored.
2402:20240130:191045.104 End of zbx_recv_response():FAIL
2402:20240130:191045.104 End of zbx_put_data_to_server():FAIL
2402:20240130:191045.104 cannot send proxy data to server at "zabbix.fOURDOMAIN.com":
2402:20240130:191045.104 End of proxy_data_sender():FAIL more:0 flags:0x8000
2402:20240130:191045.105 zbx_setproctitle() title:'data sender [sent 0 values in 0.014359 sec, idle 1 sec]'
THANKS for your time to read and many more thanks if you can guide me with a reply.
If I need to share anything else, let me know ?
My Zabbix Server / Agents / Proxies are all on 6.4 (latest)
My server is on a Ubuntu 22 machine hosted in AWS EC2
My agents/proxies are mostly Raspberry Pi
I have verified that the error occurs ONLY when the agents/proxies try to connect to Zabbix via the cloudflare proxied sub-domain
I know, I am just trying to save 5USD here, but I want to see or understand why Zabbix is ignoring the messages, even when devices are connecting via Cloudflare Proxy ?
ultimately, I have changed almost all other services from IPV4 to IPV6 only EC2 machines and they work fine with CloudFlare Proxy - for Zabbix, even in the past, I had to create an additional subdomain without the proxy, but now the IPV6 problem on ISP side doesnt allow me to connect to IPV6 directly, and zabbix ignores the messages via the proxy.
Comment