Zabbix slow using local IP

[AvmOps]

Hello,

We're facing an issue in our production environnement. We have 2 Datacenters monitored by 2 Zabbix servers on each side using Zabbix 4.0.17 and Mysql 5.5.43 for Zabbix DB.

The Zabbix servers have only one private IP configured and monitore only Servers on their own networks.

Previously, we used to access to Zabbix Web through the Internet using public IPs configured on the firewall and using NAT to map the public IPs to the the private IPs of the zabbix servers, with HTTPS and certificates configured on the firewall, and everything was working fine.

Recently we decided to disable Zabbix web access through the internet and only access to Zabbix Web through our local network with private IP and HTTPS without certificate (since it was configured on the firewall), so we disabled the public IPs and NAT but didn't changed anything on the zabbix servers since it was already binded on private IP and port 443.

But since, Zabbix web access started to be really slow for some queries, the main Dashboard take few seconds to load (sometimes it never charges) and when i try to update triggers, it often leads to ERR_CONNECTION_RESET.

I cannot see the reason why we are facing this problem since we did no Zabbix or network modification on the server side, and we just stopped accessing to it through the internet.

We tried enabling the public IP and NAT again for one server, and everything works fine when accessing to Zabbix Web through internet, i can load two pages on the same browser, one with the url using the public IP and the other with the private IP and the first one works completely fine while the other one takes long time to display dashboard or some pages.

It seems that the DB take more time to responds when we access Zabbix web through the private IP instead of NAT’d public IP, but i cannot explain why.

I've searched for related problems but cannot find anyone facing the same issue.

Thank's for your time and Help

Regards

[sudoRoman]

Maybe something with apache (nginx) setting? Are you use ip or hostname to web access? Yes, it seems that the DB take more time to responds, but problem reproduction indicates network problems. What traceroute says? Logs? Zabbix server and base on the same machine? What names (ip`s) use in zabbix server config?

[LenR]

What is your configuration? Zabbix server, mysql DB, web frontend all on one server or spread out?

[AvmOps]

Maybe something with apache (nginx) setting? Are you use ip or hostname to web access? Yes, it seems that the DB take more time to responds, but problem reproduction indicates network problems. What traceroute says? Logs? Zabbix server and base on the same machine? What names (ip`s) use in zabbix server config?

We use ip address for the web access (192.168.111.35 & 192.168.112.35 for both Zabbix servers). I think we have identified the problem and it may not be related to Zabbix, I said before that we now use the private IP for Zabbix web access but to do so we use a VPN Ipsec tunnel from our office, and the problems start from here.When i try to access to Zabbix web with local IP from a browser on a server which is on the SAME network everything works fine, and the load problems only happen when I connect to Zabbix web from our office through the Ipsec VPN tunnel using the private IP. Another strange behaviour is that we can't reproduce the problem if we connect to Zabbix web through the VPN Ipsec but from another network, home or 4g modem for example, the problem seems specifically related to our network Office but i can't see why.

This problem is quite complicated to discern, some pages work perfectly. The main dashboard works quite well even if a bit slower and ends up becoming slower and slower until it doesn't load at all after a while.

I can't traceroute the IP since ICMP is not allowed through the VPN, but we checked and there is no other route than the VPN that could explain a conflict or network issue. Also all others ports are opened through the VPN ( HTTP, HTTPS).
There are no error in Apache logs and yes Zabbix servers and DB are on the same machine.

The private IP is binded in the zabbix_server.conf for each Zabbix servers.
Here is a screenshot of our Zabbix web acess with dashboard loading :

[sudoRoman]

It could be a problem with the network settings on the hardware. I had similar problems, it was vpn with the ipsec on my Mikrotik, but another service with a speed problem. It could be firewall settings, for example. Maybe different hardware handles the connection, i dont know. But now we know for sure that it looks not Zabbix problem)