Ad Widget

Collapse

Zabbix service failed.

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • Aaron_Yu
    Junior Member
    • May 2020
    • 26
    #1

    Zabbix service failed.

    Today, Open the zabbix server web page show a error:
    zabbix server is not running the information displayed may not be current.

    so I check the zabbix server on the server, that is failed.

    [root@monitor ~]# systemctl status zabbix-server
    ● zabbix-server.service - Zabbix Server
    Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
    Active: activating (auto-restart) (Result: exit-code) since Tue 2020-06-02 08:25:29 CST; 7s ago
    Process: 8309 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
    Process: 8299 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
    Main PID: 8301 (code=exited, status=0/SUCCESS)

    Jun 02 08:25:29 monitor.sh.dto kill[8309]: -q, --queue <sig> use sigqueue(2) rather than kill(2)
    Jun 02 08:25:29 monitor.sh.dto kill[8309]: -p, --pid print pids without signaling them
    Jun 02 08:25:29 monitor.sh.dto kill[8309]: -l, --list [=<signal>] list signal names, or convert one to a name
    Jun 02 08:25:29 monitor.sh.dto kill[8309]: -L, --table list signal names and numbers
    Jun 02 08:25:29 monitor.sh.dto kill[8309]: -h, --help display this help and exit
    Jun 02 08:25:29 monitor.sh.dto kill[8309]: -V, --version output version information and exit
    Jun 02 08:25:29 monitor.sh.dto kill[8309]: For more details see kill(1).
    Jun 02 08:25:29 monitor.sh.dto systemd[1]: zabbix-server.service: control process exited, code=exited status=1
    Jun 02 08:25:29 monitor.sh.dto systemd[1]: Unit zabbix-server.service entered failed state.
    Jun 02 08:25:29 monitor.sh.dto systemd[1]: zabbix-server.service failed.
    the log of the zabbix server:
    8827:20200602:083208.635 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x37) [0x55b15d40186f]
    8827:20200602:083208.635 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0x9c5) [0x55b15d32d563]
    8827:20200602:083208.635 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](daemon_start+0x305) [0x55b15d3f364d]
    8827:20200602:083208.635 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x312) [0x55b15d32cb9c]
    8827:20200602:083208.635 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff3c94e3555]
    8827:20200602:083208.635 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x39cc9) [0x55b15d32bcc9]
    8822:20200602:083208.640 One child process died (PID:8827,exitcode/signal:1). Exiting ...
    8822:20200602:083208.641 syncing trend data...
    8822:20200602:083208.641 syncing trend data done
    8822:20200602:083208.642 Zabbix Server stopped. Zabbix 4.0.20 (revision 3b651d1b0b).

    What do I do?
    Tags: configuration, service, zabbix, zabbix server
    • tim.mooney
      Senior Member
      • Dec 2012
      • 1427
      #2
      There's not enough of the early messages from the 2nd log snippet, but it looks like one of the zabbix processes (configuration syncer) crashed.

      For now, you should try restart the zabbix-server. If it's an intermittent problem, the server may run for quite some time before it happens again. That doesn't fix the problem, but it might get you back up and running, while you investigate what the real fix is.

      There is a 4.0.21 version available, it has a fix for a server crash, but the symptoms don't look like the crash you experienced. You could try update to 4.0.21 and see if that helps.

      Otherwise, you may need to open a bug report in the Zabbix bug tracker and supply the information they ask for, to help the developers track down the source of the issue.

        Comment

        • Aaron_Yu
          Junior Member
          • May 2020
          • 26
          #3
          Originally posted by tim.mooney
          There's not enough of the early messages from the 2nd log snippet, but it looks like one of the zabbix processes (configuration syncer) crashed.

          For now, you should try restart the zabbix-server. If it's an intermittent problem, the server may run for quite some time before it happens again. That doesn't fix the problem, but it might get you back up and running, while you investigate what the real fix is.

          There is a 4.0.21 version available, it has a fix for a server crash, but the symptoms don't look like the crash you experienced. You could try update to 4.0.21 and see if that helps.

          Otherwise, you may need to open a bug report in the Zabbix bug tracker and supply the information they ask for, to help the developers track down the source of the issue.
          How to open a bug report in the zabbix server?

            Comment

            • mlenis
              #3.1
              mlenis commented
              04-06-2020, 16:01
              Editing a comment
              Good Morning to all. So I came into the office today fully focused on getting this issue resolved. My first step was to set the log levels on the zabbix_agent.conf and zabbix_server.conf to level 4. Nothing out of the ordinary stuck out on the agent logs. However, when looking a the zabbix_server logs, i did notice a couple of things. The connection to the local db was not established because either the db was not running or a bad password. I first checked the status of the db and noticed the db was in a running state. I next proceeded to the zabbix_server.conf file to ensure I hadnt gone in there and messed up the configuration where the password for the db should go. To my surprise there was no password in there. Odd. Once i retyped the password the zabbix_server logs cleared the db connection errors. Now I was seeing Cachesize was set to low errors.(Not the exact error message) I remember reading some threads before about this error. So I un-commented the Cachesize line and removed the 8M and increased to 32M. Restarted the zabbix-server and boom. Back in business.Hope this helps anyone out there.
            • tim.mooney
              Senior Member
              • Dec 2012
              • 1427
              #4
              Loading...
              https://support.zabbix.com/secure/Dashboard.jspa


              They will almost certainly ask you to try 4.0.21 first, though, so be sure you can reproduce the problem on that version before opening a bug report.

                Comment

                • mlenis
                  Junior Member
                  • May 2020
                  • 1
                  #5

                  Good Evening,

                  This is my first first post. So. Hi everyone. Im a network admin working for a Financial Institution of about 100 users. I needed a solution that I can use on a daily basis to monitor the environment. Great Solution. I love it. However, Im on my second installation of Zabbix 5.01 on Ubuntu 20.04 and 18. Everything works great for 2 days and all of a sudden I receive the aforementioned message "zabbix server is not running the information displayed may not be current." I did notice that it also happens when I start adding some Cisco 2960, Cisco 4500, Cisco 3850 Switches. Not sure if its related to the root cause but thought I share that as well. Also followed some recommendations about the Ubuntu Firewall. Checked the initial status of the firewall and it was disabled. I enabled the firewall and allowed ports 80 10050 and 10051. Below are the server and agent log. Both are sunning on the same Virtual Machine. Thank You for your time.


                  *******Server Log*******

                  26834:20200603:195947.013 Starting Zabbix Server. Zabbix 5.0.1 (revision c2a0b03480).
                  26834:20200603:195947.013 ****** Enabled features ******

                  26834:20200603:195947.013 SNMP monitoring: YES

                  26834:20200603:195947.013 IPMI monitoring: YES

                  26834:20200603:195947.013 Web monitoring: YES

                  26834:20200603:195947.013 VMware monitoring: YES

                  26834:20200603:195947.013 SMTP authentication: YES

                  26834:20200603:195947.013 ODBC: YES

                  26834:20200603:195947.013 SSH support: YES

                  26834:20200603:195947.013 IPv6 support: YES

                  26834:20200603:195947.013 TLS support: YES

                  26834:20200603:195947.013 ******************************

                  26834:20200603:195947.013 using configuration file: /etc/zabbix/zabbix_server.conf

                  26834:20200603:195947.025 current database version (mandatory/optional): 05000000/05000000

                  26834:20200603:195947.025 required mandatory version: 05000000

                  26834:20200603:195947.038 server #0 started [main process]

                  26835:20200603:195947.038 server #1 started [configuration syncer #1]

                  26835:20200603:195947.222 __mem_malloc: skipped 9 asked 32056 skip_min 616 skip_max 27544

                  26835:20200603:195947.223 [file:dbconfig.c,line:95] __zbx_mem_realloc(): out of memory (requested 32056 bytes)

                  26835:20200603:195947.223 [file:dbconfig.c,line:95] __zbx_mem_realloc(): please increase CacheSize configuration parameter

                  26835:20200603:195947.223 === memory statistics for configuration cache ===

                  26835:20200603:195947.223 free chunks of size 24 bytes: 88

                  26835:20200603:195947.223 free chunks of size >= 256 bytes: 9

                  26835:20200603:195947.223 min chunk size: 24 bytes

                  26835:20200603:195947.223 max chunk size: 27544 bytes

                  26835:20200603:195947.223 memory of total size 8388232 bytes fragmented into 72246 chunks

                  26835:20200603:195947.223 of those, 110864 bytes are in 97 free chunks

                  26835:20200603:195947.223 of those, 7121448 bytes are in 72149 used chunks

                  26835:20200603:195947.223 ================================

                  26835:20200603:195947.223 === Backtrace: ===

                  26835:20200603:195947.223 15: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x52) [0x560facf23646]

                  26835:20200603:195947.223 14: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_mem_realloc+0x164) [0x560facf1e137]

                  26835:20200603:195947.223 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1a22c5) [0x560facee22c5]

                  26835:20200603:195947.223 12: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_reserve+0xc3) [0x560facf29a24]

                  26835:20200603:195947.223 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert_ext+0xf3) [0x560facf29ca4]

                  26835:20200603:195947.223 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert+0x31) [0x560facf29baf]

                  26835:20200603:195947.223 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCfind_id+0xb4) [0x560facee2d15]

                  26835:20200603:195947.223 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1aadbc) [0x560faceeadbc]

                  26835:20200603:195947.223 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0xff4) [0x560faceefaa3]

                  26835:20200603:195947.223 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x11a) [0x560facd97170]

                  26835:20200603:195947.223 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x3b) [0x560facf3342b]

                  26835:20200603:195947.223 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0x9e8) [0x560facd881bc]

                  26835:20200603:195947.223 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](daemon_start+0x303) [0x560facf23188]

                  26835:20200603:195947.223 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x309) [0x560facd877be]

                  26835:20200603:195947.223 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f7a503b20b3]

                  26835:20200603:195947.223 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x2e) [0x560facd867ae]

                  26834:20200603:195947.226 One child process died (PID:26835,exitcode/signal:1). Exiting ...

                  26834:20200603:195947.228 syncing trend data...

                  26834:20200603:195947.228 syncing trend data done

                  26834:20200603:195947.228 Zabbix Server stopped. Zabbix 5.0.1 (revision c2a0b03480).


                  *******Agent******* 91534:20200603:172853.198 Zabbix Agent stopped. Zabbix 5.0.1 (revision c2a0b03480).
                  919:20200603:172906.630 Starting Zabbix Agent [Zabbix server]. Zabbix 5.0.1 (revision c2a0b03480).

                  919:20200603:172906.632 **** Enabled features ****

                  919:20200603:172906.632 IPv6 support: YES

                  919:20200603:172906.632 TLS support: YES

                  919:20200603:172906.632 **************************

                  919:20200603:172906.632 using configuration file: /etc/zabbix/zabbix_agentd.conf

                  919:20200603:172906.632 agent #0 started [main process]

                  922:20200603:172906.633 agent #1 started [collector]

                  924:20200603:172906.633 agent #3 started[listener #2]

                  926:20200603:172906.633 agent #5 started [active checks #1]

                  923:20200603:172906.634 agent #2 started[listener #1]

                  925:20200603:172906.634 agent #4 started[listener #3]

                  926:20200603:172906.643 active check configuration update from [127.0.0.1:10051] started to fail (cannot connect to [[127.0.0.1]:10051]: [111] Connection refused)


                    Comment

                    • Aaron_Yu
                      Junior Member
                      • May 2020
                      • 26
                      #6
                      Originally posted by tim.mooney
                      https://support.zabbix.com/secure/Dashboard.jspa

                      They will almost certainly ask you to try 4.0.21 first, though, so be sure you can reproduce the problem on that version before opening a bug report.

                      I have upgrade it to 4.4.
                      it's OK now.

                      Thanks your help.

                        Comment

                        • Aaron_Yu
                          Junior Member
                          • May 2020
                          • 26
                          #7
                          Originally posted by mlenis
                          Good Evening,

                          This is my first first post. So. Hi everyone. Im a network admin working for a Financial Institution of about 100 users. I needed a solution that I can use on a daily basis to monitor the environment. Great Solution. I love it. However, Im on my second installation of Zabbix 5.01 on Ubuntu 20.04 and 18. Everything works great for 2 days and all of a sudden I receive the aforementioned message "zabbix server is not running the information displayed may not be current." I did notice that it also happens when I start adding some Cisco 2960, Cisco 4500, Cisco 3850 Switches. Not sure if its related to the root cause but thought I share that as well. Also followed some recommendations about the Ubuntu Firewall. Checked the initial status of the firewall and it was disabled. I enabled the firewall and allowed ports 80 10050 and 10051. Below are the server and agent log. Both are sunning on the same Virtual Machine. Thank You for your time.


                          *******Server Log*******

                          26834:20200603:195947.013 Starting Zabbix Server. Zabbix 5.0.1 (revision c2a0b03480).
                          26834:20200603:195947.013 ****** Enabled features ******

                          26834:20200603:195947.013 SNMP monitoring: YES

                          26834:20200603:195947.013 IPMI monitoring: YES

                          26834:20200603:195947.013 Web monitoring: YES

                          26834:20200603:195947.013 VMware monitoring: YES

                          26834:20200603:195947.013 SMTP authentication: YES

                          26834:20200603:195947.013 ODBC: YES

                          26834:20200603:195947.013 SSH support: YES

                          26834:20200603:195947.013 IPv6 support: YES

                          26834:20200603:195947.013 TLS support: YES

                          26834:20200603:195947.013 ******************************

                          26834:20200603:195947.013 using configuration file: /etc/zabbix/zabbix_server.conf

                          26834:20200603:195947.025 current database version (mandatory/optional): 05000000/05000000

                          26834:20200603:195947.025 required mandatory version: 05000000

                          26834:20200603:195947.038 server #0 started [main process]

                          26835:20200603:195947.038 server #1 started [configuration syncer #1]

                          26835:20200603:195947.222 __mem_malloc: skipped 9 asked 32056 skip_min 616 skip_max 27544

                          26835:20200603:195947.223 [file:dbconfig.c,line:95] __zbx_mem_realloc(): out of memory (requested 32056 bytes)

                          26835:20200603:195947.223 [file:dbconfig.c,line:95] __zbx_mem_realloc(): please increase CacheSize configuration parameter

                          26835:20200603:195947.223 === memory statistics for configuration cache ===

                          26835:20200603:195947.223 free chunks of size 24 bytes: 88

                          26835:20200603:195947.223 free chunks of size >= 256 bytes: 9

                          26835:20200603:195947.223 min chunk size: 24 bytes

                          26835:20200603:195947.223 max chunk size: 27544 bytes

                          26835:20200603:195947.223 memory of total size 8388232 bytes fragmented into 72246 chunks

                          26835:20200603:195947.223 of those, 110864 bytes are in 97 free chunks

                          26835:20200603:195947.223 of those, 7121448 bytes are in 72149 used chunks

                          26835:20200603:195947.223 ================================

                          26835:20200603:195947.223 === Backtrace: ===

                          26835:20200603:195947.223 15: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x52) [0x560facf23646]

                          26835:20200603:195947.223 14: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_mem_realloc+0x164) [0x560facf1e137]

                          26835:20200603:195947.223 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1a22c5) [0x560facee22c5]

                          26835:20200603:195947.223 12: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_reserve+0xc3) [0x560facf29a24]

                          26835:20200603:195947.223 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert_ext+0xf3) [0x560facf29ca4]

                          26835:20200603:195947.223 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert+0x31) [0x560facf29baf]

                          26835:20200603:195947.223 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCfind_id+0xb4) [0x560facee2d15]

                          26835:20200603:195947.223 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1aadbc) [0x560faceeadbc]

                          26835:20200603:195947.223 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0xff4) [0x560faceefaa3]

                          26835:20200603:195947.223 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x11a) [0x560facd97170]

                          26835:20200603:195947.223 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x3b) [0x560facf3342b]

                          26835:20200603:195947.223 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0x9e8) [0x560facd881bc]

                          26835:20200603:195947.223 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](daemon_start+0x303) [0x560facf23188]

                          26835:20200603:195947.223 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x309) [0x560facd877be]

                          26835:20200603:195947.223 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f7a503b20b3]

                          26835:20200603:195947.223 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x2e) [0x560facd867ae]

                          26834:20200603:195947.226 One child process died (PID:26835,exitcode/signal:1). Exiting ...

                          26834:20200603:195947.228 syncing trend data...

                          26834:20200603:195947.228 syncing trend data done

                          26834:20200603:195947.228 Zabbix Server stopped. Zabbix 5.0.1 (revision c2a0b03480).


                          *******Agent******* 91534:20200603:172853.198 Zabbix Agent stopped. Zabbix 5.0.1 (revision c2a0b03480).
                          919:20200603:172906.630 Starting Zabbix Agent [Zabbix server]. Zabbix 5.0.1 (revision c2a0b03480).

                          919:20200603:172906.632 **** Enabled features ****

                          919:20200603:172906.632 IPv6 support: YES

                          919:20200603:172906.632 TLS support: YES

                          919:20200603:172906.632 **************************

                          919:20200603:172906.632 using configuration file: /etc/zabbix/zabbix_agentd.conf

                          919:20200603:172906.632 agent #0 started [main process]

                          922:20200603:172906.633 agent #1 started [collector]

                          924:20200603:172906.633 agent #3 started[listener #2]

                          926:20200603:172906.633 agent #5 started [active checks #1]

                          923:20200603:172906.634 agent #2 started[listener #1]

                          925:20200603:172906.634 agent #4 started[listener #3]

                          926:20200603:172906.643 active check configuration update from [127.0.0.1:10051] started to fail (cannot connect to [[127.0.0.1]:10051]: [111] Connection refused)

                          Please check the CacheSize option in the zabbix_server.conf file.
                          The default value of the option is 8M, you can set it to 2G or other value, that depends your memory.


                          The issue is out of memory

                          26835:20200603:195947.223 [file:dbconfig.c,line:95] __zbx_mem_realloc(): out of memory (requested 32056 bytes)

                            Comment

                            • lokha
                              Junior Member
                              • Jun 2020
                              • 15
                              #8
                              Originally posted by Aaron_Yu

                              Please check the CacheSize option in the zabbix_server.conf file.
                              The default value of the option is 8M, you can set it to 2G or other value, that depends your memory.


                              The issue is out of memory

                              26835:20200603:195947.223 [file:dbconfig.c,line:95] __zbx_mem_realloc(): out of memory (requested 32056 bytes)
                              Yes, increasing the CacheSize helped me. Cheers thanks!

                                Comment

                                • hannah131517
                                  Junior Member
                                  • Nov 2024
                                  • 3
                                  #9
                                  hi, how did you check the log? i have an issue that i added the hosts in zabbix proxy and agents stop sending data. when i check the proxy, it is still on

                                    Comment

                                    Previous template Next
                                    Working...