zabbix-server service failed to start

[dango]

Hi there, I am getting this error on my newly deploy zabbix.

zabbix-server.service: Referenced but unset environment variable evaluates to an empty string: MAINPID

OS is Ubuntu 26.04 LTS with zabbix version 7.4.10. This was fresh deployment with mysql and apache. It was working fine the 1st day when I build some templates in the portal. A couple of days later, I noticed browser state the service is not running.

[dango]

So the fix is to change CacheSize to a bigger value in /etc/zabbix/zabbix_server.conf

[hrrapozo]

As a suggestion you should specify which version of Zabbix you are running. Latest version of Zabbix could either mean 7.0.26 if you installed Zabbix 7.0 LTS, or 7.4.10 if you installed Zabbix 7.4 and I have no idea which version you have if you installed the Zabbix 8.0 pre-release.

[dango]

As a suggestion you should specify which version of Zabbix you are running. Latest version of Zabbix could either mean 7.0.26 if you installed Zabbix 7.0 LTS, or 7.4.10 if you installed Zabbix 7.4 and I have no idea which version you have if you installed the Zabbix 8.0 pre-release.

Updated with the version of 7.4.10

[dango]

Does anyone able to help?

[AlexeyBrizgalov]

Hi, if zabbix server is failing to start then logs should provide a clue, by default they are located in /var/log/zabbix/zabbix_server.log
Restart the zabbix server service and then check what you see in logs when it attempts to start.

[cyber]

Based on error in first post, it will never get to starting a Zabbxi server. Error indicates issues with service file. There is something with env var MAINPID, which is (or in this is not) set by systemd. Thre can be some changes in how systemd is configured etc.. Consult your local friendly linux admin maybe, what could be wrong.

[dango]

Why would systemd suddenly change? The system was running fine after the deployment and nothing else was done in CLI on the zabbix-server.config change to enable vmware worker.

[dango]

Hi, if zabbix server is failing to start then logs should provide a clue, by default they are located in /var/log/zabbix/zabbix_server.log
Restart the zabbix server service and then check what you see in logs when it attempts to start.

17169:20260521:222702.119 Starting Zabbix Server. Zabbix 7.4.10 (revision 8f4151333a4).
17169:20260521:222702.119 ****** Enabled features ******
17169:20260521:222702.119 SNMP monitoring: YES
17169:20260521:222702.119 IPMI monitoring: YES
17169:20260521:222702.119 Web monitoring: YES
17169:20260521:222702.119 VMware monitoring: YES
17169:20260521:222702.119 SMTP authentication: YES
17169:20260521:222702.119 ODBC: YES
17169:20260521:222702.119 SSH support: YES
17169:20260521:222702.119 IPv6 support: YES
17169:20260521:222702.119 TLS support: YES
17169:20260521:222702.119 ******************************
17169:20260521:222702.119 using configuration file: /etc/zabbix/zabbix_server.conf
17169:20260521:222702.130 current database version (mandatory/optional): 07040000/07040010
17169:20260521:222702.130 required mandatory version: 07040000
17170:20260521:222702.132 starting HA manager
17170:20260521:222702.134 HA manager started in active mode
17169:20260521:222702.135 server #0 started [main process]
17171:20260521:222702.136 server #1 started [service manager #1]
17172:20260521:222702.136 server #2 started [configuration syncer #1]
17172:20260521:222702.606 __mem_malloc: skipped 0 asked 24 skip_min 18446744073709551615 skip_max 0
17172:20260521:222702.606 === memory statistics for configuration cache ===
17172:20260521:222702.606 min chunk size: 0 bytes
17172:20260521:222702.606 max chunk size: 0 bytes
17172:20260521:222702.606 memory of total size 28395816 bytes fragmented into 322390 chunks
17172:20260521:222702.606 of those, 0 bytes are in 0 free chunks
17172:20260521:222702.606 of those, 28395816 bytes are in 322390 used chunks
17172:20260521:222702.606 of those, 5158224 bytes are used by allocation overhead
17172:20260521:222702.606 ================================
17172:20260521:222702.606 === Backtrace: ===
17172:20260521:222702.608 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x50) [0x647ea89e2410]
17172:20260521:222702.608 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_shmem_realloc+0x403) [0x647ea88e59a3]
17172:20260521:222702.608 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x2b2aca) [0x647ea8854aca]
17172:20260521:222702.608 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_dc_sync_configuration+0x387d) [0x647ea886c3bd]
17172:20260521:222702.608 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x104) [0x647ea8691be4]
17172:20260521:222702.608 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x24) [0x647ea88f0954]
17172:20260521:222702.608 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0xe63bb) [0x647ea86883bb]
17172:20260521:222702.608 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0x106c) [0x647ea8689b3c]
17172:20260521:222702.608 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x37f) [0x647ea867d77f]
17172:20260521:222702.608 2: /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a601) [0x7f3cf5a2a601]
17172:20260521:222702.608 1: /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x88) [0x7f3cf5a2a718]
17172:20260521:222702.608 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x25) [0x647ea8684ab5]
17172:20260521:222702.608 [file:dbconfig.c,line:266] __zbx_shmem_realloc(): out of memory (requested 24 bytes)
17172:20260521:222702.608 [file:dbconfig.c,line:266] __zbx_shmem_realloc(): please increase CacheSize configuration parameter
17169:20260521:222703.134 One child process died (PID:17172,exitcode/signal:1). Exiting ...
17170:20260521:222703.134 HA manager has been paused
17170:20260521:222704.241 HA manager has been stopped
17169:20260521:222704.242 Zabbix Server stopped. Zabbix 7.4.10 (revision 8f4151333a4).

[dango]

Fixed. Had to change the cache size to a bigger. Not sure why I need to set it. It doesn't dynamically increase based on the system size? 28MB makes no sense at all

[dango]

So the fix is to change CacheSize to a bigger value in /etc/zabbix/zabbix_server.conf

[AlexeyBrizgalov]

Fixed. Had to change the cache size to a bigger. Not sure why I need to set it. It doesn't dynamically increase based on the system size? 28MB makes no sense at all

It is normal for config cache usage to increase over time as your deployment gets bigger. Allocated cache does not increase automatically, you need to change it manually in the config, by default it is set to only 32M on zabbix 7.4. Depending on how large your setup is going to be you can immediately set it to much larger value like 1G - 4G, just keep in mind that it shouldn't be larger than the amount of free RAM you currently have on your machine.
You can also periodically check cache usage on zabbix server health dashboard and just double it in config once it goes above 50% usage.

[cyber]

dango commented
Today, 01:35
Why would systemd suddenly change? The system was running fine after the deployment and nothing else was done in CLI on the zabbix-server.config change to enable vmware worker.

I really don't know, there was too little info in your first post to decide. Maybe system updates (a lot of people run automatic ones and do not even notice, when they get installed and then complain later that something is not working...). Also internet says, that there has been changes in ubuntu systemd, what causes such error messages..
I went for interpreting the service startup failure, there was nothing to indicate that it managed to actually get to starting service up and fail with cachesize ..