Zabbix UI Performance Issue After Migration - Troubleshooting Steps

[Dion]

Hi there,

After completing a migration of vms to a knew host/enviorement, we noticed that the Zabbix UI is very slow. While Zabbix is still sending out notifications and processing data as usual, the web interface is taking almost 10 seconds to load a single page.

What steps can we take to troubleshoot this issue and what should we check first?

We are running Zabbix on a separate host/VM, with the database (MariaDB) on a different server. The underlying hypervisor is ESXi. What I've Checked and Adjusted So Far:

I’ve already investigated several areas and made the following adjustments, but the issue persists:

1. PHP Configuration: Located in /etc/php/7.3/apache2/php.ini:
   • memory_limit: Increased to 1000M (previously 256M)
   • max_execution_time: Increased to 300 (previously 30)
   • max_input_vars: Increased to 3000 (previously 1000)
   • post_max_size: Increased to 32M (previously 8M)
2. Apache MPM (prefork) Configuration: Located in /etc/apache2/mods-available/mpm_prefork.conf:
   • StartServers: Increased to 10 (previously 5)
   • MinSpareServers: Increased to 10 (previously 5)
   • MaxSpareServers: Increased to 30 (previously 10)
   • MaxRequestWorkers: Increased to 200 (previously 150)
   • MaxConnectionsPerChild: Set to 1000 (previously 0)
3. Database Optimization:
   • Ran mysqlcheck --optimize --all-databases -u root -p < pretty sure that was all okay
   • Updated MariaDB configuration in /etc/mysql/mariadb.conf.d/50-server.cnf with the following settings under [mysqld]:
     • innodb_buffer_pool_size = 1G
     • innodb_log_file_size = 256M
     • innodb_flush_log_at_trx_commit = 2
     • innodb_log_buffer_size = 16M
     • innodb_thread_concurrency = 0
     • innodb_io_capacity = 200
     • query_cache_size = 64M
     • query_cache_limit = 2M

We are monitoring around 40 VM's.

2 CPU's and 8GB memory for DB server and another VM for the frontend with the same specs.
We are running on high-end enterprise servers, so i think 2 vCPU's should be enough (please let me know if not).

The load isn't high when i look with top and this worked before.Even when going to the local webpage of the server/IP it is slow. not just the external URL/domain.

Next Steps:
I would appreciate any advice on additional areas I should focus on or things I might have missed.
Is there anything else we should consider for performance tuning, especially on the database side or in Zabbix's configuration?

Thanks in advance for your help!

[buedi]

When the UI loads, do you see php-fpm processes running on all available CPU cores, but only conuming about 25% of each core? This is what I see with Zabbix 7.2.

I have 2 Zabbix environments with 6.2 (native install) and 6.4 (dockerized) and the UI is blazing fast.
I built a new system with 7.4 on Ubuntu 24.04 dockerized and just the default install takes about 5-8 seconds to just load the default Dashboard. While I am waiting, I can always see php-fpm processes consuming CPU, but only around 25% per core.

I installed v7.2 now on 2 different systems and tried the combinations of Apache+MySQL and Nginx+PostgreSQL, but the outcome is always the same. The behavior does not change whether I upgrade an existing 6.x installation or just spin up a fresh default Zabbix instance.

I have no idea yet how to solve this. I was about to roll this out in a company, wanted to show how blazingly fast the UI is, but somehow with 7.2 is is slow as you describe it, no matter on which system I tried.
I am also interested in how to debug this. I suspect I am waiting for the php-fpm processes to finish, but I do not understand why they consume so much CPU and if they need it, why they do not go beyond 25% per CPU.
On the 6.2 and 6.4 systems, the UI loads quicker than I can release the mouse button :-(

[buedi]

I will answer myself and for anyone who will find this in the future. When switching from the Alpine to the Ubuntu images, loading the default Dashboard right after a new installation improves from ~3.5 Seconds to ~0.6 seconds. It makes no difference whether using the Nginx or Apache version, nor PostgreSQL or MySQL.
I have no idea why the Alpine images are that much slower, but I switched back and forth a few times now and I can reproduce this very reliably. I will stay on Ubuntu now, despite the images being around twice the size, the performance loss is not worth it.

[MRedbourne]

I will answer myself and for anyone who will find this in the future. When switching from the Alpine to the Ubuntu images, loading the default Dashboard right after a new installation improves from ~3.5 Seconds to ~0.6 seconds. It makes no difference whether using the Nginx or Apache version, nor PostgreSQL or MySQL.
I have no idea why the Alpine images are that much slower, but I switched back and forth a few times now and I can reproduce this very reliably. I will stay on Ubuntu now, despite the images being around twice the size, the performance loss is not worth it.

If you can reliably reproduce this issue, I would highly suggest (and request) you submit a bug report to Zabbix.
ZABBIX BUGS AND ISSUES - Issues - ZABBIX SUPPORT