zabbix server is not running the information may not be current

[Acho]

Hai,

Please somebody could help me?

I have problem with my Zabbix Monitoring is that the zabbix server status is not running and show notification "zabbix server is not running the information may not be current" on the web display.
This is the 3rd time happen to my zabbix. On the previous I must drop and re-ccreate the database to fix it, but it is really unfavorable due to have to setup all all the host and map configuration from the begining.

The zabbix server status show "Activating" not active. But if I stop the MySQL, zabbix server status will show running.

Below is the output of,

root@delame:/home/delame# systemctl status zabbix-server
● zabbix-server.service - Zabbix Server
Loaded: loaded (/lib/systemd/system/zabbix-server.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2019-07-29 18:33:21 WIB; 1s ago
Process: 6458 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
Process: 2624 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 2626 (code=exited, status=1/FAILURE)


root@delame:/home/delame# ps ax | grep zabbix
1105 ? S 0:00 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
1106 ? S 0:48 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
1107 ? S 0:01 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
1108 ? S 0:01 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
1109 ? S 0:01 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
1110 ? S 0:10 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
2700 pts/1 R+ 0:00 grep --color=auto zabbix

And below is the output when I stop MySQL and check the zabbix-server status :

root@delame:/home/delame# systemctl status zabbix-server
● zabbix-server.service - Zabbix Server
Loaded: loaded (/lib/systemd/system/zabbix-server.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-07-29 18:38:15 WIB; 15s ago
Process: 6458 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
Process: 2729 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 2731 (zabbix_server)
Tasks: 1 (limit: 4399)
CGroup: /system.slice/zabbix-server.service
└─2731 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf

Jul 29 18:38:15 delame systemd[1]: Starting Zabbix Server...
Jul 29 18:38:15 delame systemd[1]: zabbix-server.service: Supervising process 2731 which is not our child. We'll most likely not notice when i
Jul 29 18:38:15 delame systemd[1]: Started Zabbix Server.


please any good body i need your help on how to fix this problem

[Atsushi]

Check the error message output to the Zabbix server log.
path:
/var/log/zabbix/zabbix_server.log

[Acho]

Check the error message output to the Zabbix server log.
path:
/var/log/zabbix/zabbix_server.log

this is the output:

10582:20190729:003449.581 Starting Zabbix Server. Zabbix 4.0.9 (revision 97a69d5d5a).
10582:20190729:003449.581 ****** Enabled features ******
10582:20190729:003449.581 SNMP monitoring: YES
10582:20190729:003449.581 IPMI monitoring: YES
10582:20190729:003449.581 Web monitoring: YES
10582:20190729:003449.581 VMware monitoring: YES
10582:20190729:003449.581 SMTP authentication: YES
10582:20190729:003449.581 Jabber notifications: YES
10582:20190729:003449.581 Ez Texting notifications: YES
10582:20190729:003449.581 ODBC: YES
10582:20190729:003449.581 SSH2 support: YES
10582:20190729:003449.581 IPv6 support: YES
10582:20190729:003449.581 TLS support: YES
10582:20190729:003449.581 ******************************
10582:20190729:003449.581 using configuration file: /etc/zabbix/zabbix_server.conf
10582:20190729:003449.583 current database version (mandatory/optional): 04000000/04000003
10582:20190729:003449.583 required mandatory version: 04000000
10582:20190729:003449.780 __mem_malloc: skipped 1 asked 512 skip_min 272 skip_max 272
10582:20190729:003449.780 [file:dbconfig.c,line:94] __zbx_mem_realloc(): out of memory (requested 512 bytes)
10582:20190729:003449.780 [file:dbconfig.c,line:94] __zbx_mem_realloc(): please increase CacheSize configuration parameter
10582:20190729:003449.780 === memory statistics for configuration cache ===
10582:20190729:003449.780 free chunks of size >= 256 bytes: 1
10582:20190729:003449.780 min chunk size: 272 bytes
10582:20190729:003449.780 max chunk size: 272 bytes
10582:20190729:003449.780 memory of total size 8388232 bytes fragmented into 72355 chunks
10582:20190729:003449.780 of those, 272 bytes are in 1 free chunks
10582:20190729:003449.780 of those, 7230296 bytes are in 72354 used chunks
10582:20190729:003449.780 ================================
10582:20190729:003449.780 === Backtrace: ===
10582:20190729:003449.780 11: /usr/sbin/zabbix_server(zbx_backtrace+0x4e) [0x55e21b61474d]
10582:20190729:003449.780 10: /usr/sbin/zabbix_server(__zbx_mem_realloc+0x16b) [0x55e21b60fb46]
10582:20190729:003449.780 9: /usr/sbin/zabbix_server(+0xce9a2) [0x55e21b5d89a2]
10582:20190729:003449.780 8: /usr/sbin/zabbix_server(+0x10c9c1) [0x55e21b6169c1]
10582:20190729:003449.780 7: /usr/sbin/zabbix_server(zbx_binary_heap_insert+0x73) [0x55e21b616f32]
10582:20190729:003449.780 6: /usr/sbin/zabbix_server(+0xda571) [0x55e21b5e4571]
10582:20190729:003449.780 5: /usr/sbin/zabbix_server(DCsync_configuration+0x1146) [0x55e21b5e579c]
10582:20190729:003449.780 4: /usr/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x72e) [0x55e21b54342d]
10582:20190729:003449.780 3: /usr/sbin/zabbix_server(daemon_start+0x2f0) [0x55e21b613e8c]
10582:20190729:003449.780 2: /usr/sbin/zabbix_server(main+0x305) [0x55e21b542ce9]
10582:20190729:003449.780 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f7a1889cb97]
10582:20190729:003449.780 0: /usr/sbin/zabbix_server(_start+0x2a) [0x55e21b541dfa]


Any idea where is the problem sir? Thank you

[Atsushi]

Please take a close look at the log file.

10582:20190729:003449.780 [file:dbconfig.c,line:94] __zbx_mem_realloc(): out of memory (requested 512 bytes)
10582:20190729:003449.780 [file:dbconfig.c,line:94] __zbx_mem_realloc(): please increase CacheSize configuration parameter

Out of memory has occurred.
Did you try to increase CacheSize as shown in the log message?

[Acho]

Please take a close look at the log file.

10582:20190729:003449.780 [file:dbconfig.c,line:94] __zbx_mem_realloc(): out of memory (requested 512 bytes)
10582:20190729:003449.780 [file:dbconfig.c,line:94] __zbx_mem_realloc(): please increase CacheSize configuration parameter

Out of memory has occurred.
Did you try to increase CacheSize as shown in the log message?

Not yet, alright I will try to find how to increase the cache size first.

[Acho]

yeayy, I,ve increased the CacheSize and HistoryCacheSize. And now its working again, alhamdulillah.

Thx for your help Atsushi

[Acho]

hi again A Atsushi and team,
Now i have problem with the zabbix server status on menu Configuration > Host. It colored red and if I point that will show information "get value from agent failed cannot connect to connection refused".

Even though when I check the status of zabbix-server and zabbix-agent, they all running. For SNMP host configuration for monitoring device is just works.
I already try toa dd ufw firewall to allow port 10050 and 10051 as follos, but still no change :

root@devel-com:/home/bangkarni# ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
80/tcp (Nginx HTTP)        ALLOW IN    Anywhere                  
22/tcp (OpenSSH)           ALLOW IN    Anywhere                  
5000                       ALLOW IN    Anywhere                  
5432                       ALLOW IN    Anywhere                  
5002                       ALLOW IN    Anywhere                  
3000                       ALLOW IN    Anywhere                  
10050/tcp                  ALLOW IN    Anywhere                  
80,443/tcp (Apache Full)   ALLOW IN    Anywhere                  
10050/tcp                  ALLOW IN    172.16.x.x               
80                         ALLOW IN    Anywhere                  
10050,10051/tcp            ALLOW IN    Anywhere                  
80/tcp (Nginx HTTP (v6))   ALLOW IN    Anywhere (v6)             
22/tcp (OpenSSH (v6))      ALLOW IN    Anywhere (v6)             
5000 (v6)                  ALLOW IN    Anywhere (v6)             
5432 (v6)                  ALLOW IN    Anywhere (v6)             
5002 (v6)                  ALLOW IN    Anywhere (v6)             
3000 (v6)                  ALLOW IN    Anywhere (v6)             
10050/tcp (v6)             ALLOW IN    Anywhere (v6)             
80,443/tcp (Apache Full (v6)) ALLOW IN    Anywhere (v6)             
80 (v6)                    ALLOW IN    Anywhere (v6)             
10050,10051/tcp (v6)       ALLOW IN    Anywhere (v6)

Is there any advise sir?

thanks.

[Atsushi]

Make sure that the Zabbix agent is ready to accept connections using the IP address and port number set on the host. For example, is the Zabbix agent listening at 127.0.0.1:10050 but specifying another IP address on the host interface?

[Acho]

Thank you very much, yes it's solved mow by following your advice. Before, i set not the IP 127.0.0.1 but refers to server IP 172.16.x.x.
By the way is it possible to configure so that I can use the server's IP instead?

[Acho]

Make sure that the Zabbix agent is ready to accept connections using the IP address and port number set on the host. For example, is the Zabbix agent listening at 127.0.0.1:10050 but specifying another IP address on the host interface?

Hi Kaleem,
I followed this Atsushi's solution to make it solved.