Ad Widget

Collapse

Zabbix Server is not running. Zabbix Server is auto restart

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • iwidjaya
    Junior Member
    • Apr 2016
    • 16

    #1

    Zabbix Server is not running. Zabbix Server is auto restart

    Folks,

    I'm running into issues where the frontend tells 'Zabbix Server is not running....'. When I check on the zabbix_server.log, it shows below:

    13974:20170119:103654.271 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13972:20170119:103654.272 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13979:20170119:103654.273 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13975:20170119:103654.273 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13980:20170119:103654.274 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13978:20170119:103654.275 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13977:20170119:103654.275 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13981:20170119:103654.276 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13983:20170119:103654.276 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13985:20170119:103654.277 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13989:20170119:103654.277 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13986:20170119:103654.277 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13982:20170119:103654.278 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13990:20170119:103654.279 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13988:20170119:103654.280 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13984:20170119:103654.280 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13992:20170119:103654.281 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13976:20170119:103654.281 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13991:20170119:103654.281 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13994:20170119:103654.282 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13973:20170119:103654.282 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13970:20170119:103654.283 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13969:20170119:103654.283 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13995:20170119:103654.284 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13971:20170119:103654.284 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13987:20170119:103654.284 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13993:20170119:103654.285 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13963:20170119:103654.285 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13963:20170119:103656.291 syncing history data...
    13963:20170119:103656.297 syncing history data done
    13963:20170119:103656.297 syncing trends data...
    13963:20170119:103656.576 syncing trends data done
    13963:20170119:103656.577 Zabbix Server stopped. Zabbix 3.0.2 (revision 59540).
    14019:20170119:103706.825 Starting Zabbix Server. Zabbix 3.0.2 (revision 59540).
    14019:20170119:103706.825 ****** Enabled features ******
    14019:20170119:103706.825 SNMP monitoring: YES
    14019:20170119:103706.825 IPMI monitoring: YES
    14019:20170119:103706.825 Web monitoring: YES
    14019:20170119:103706.825 VMware monitoring: YES
    14019:20170119:103706.825 SMTP authentication: YES
    14019:20170119:103706.825 Jabber notifications: YES
    14019:20170119:103706.825 Ez Texting notifications: YES
    14019:20170119:103706.826 ODBC: YES
    14019:20170119:103706.826 SSH2 support: YES
    14019:20170119:103706.826 IPv6 support: YES
    14019:20170119:103706.826 TLS support: YES
    14019:20170119:103706.826 ******************************
    14019:20170119:103706.826 using configuration file: /etc/zabbix/zabbix_server.conf
    14019:20170119:103706.844 current database version (mandatory/optional): 03000000/03000000
    14019:20170119:103706.844 required mandatory version: 03000000
    14019:20170119:103706.922 server #0 started [main process]
    14025:20170119:103706.923 server #1 started [configuration syncer #1]
    14026:20170119:103706.923 server #2 started [db watchdog #1]
    14027:20170119:103706.924 server #3 started [poller #1]
    14028:20170119:103706.925 server #4 started [poller #2]
    14029:20170119:103706.926 server #5 started [poller #3]
    14030:20170119:103706.926 server #6 started [poller #4]
    14031:20170119:103706.927 server #7 started [poller #5]
    14032:20170119:103706.928 server #8 started [unreachable poller #1]
    14033:20170119:103706.929 server #9 started [trapper #1]
    14039:20170119:103706.936 server #15 started [alerter #1]
    14041:20170119:103706.937 server #17 started [timer #1]
    14042:20170119:103706.938 server #18 started [http poller #1]
    14035:20170119:103706.939 server #11 started [trapper #3]
    14044:20170119:103706.940 server #20 started [history syncer #1]
    14046:20170119:103706.942 server #22 started [history syncer #3]
    14038:20170119:103706.943 server #14 started [icmp pinger #1]
    14034:20170119:103706.944 server #10 started [trapper #2]
    14048:20170119:103706.944 server #24 started [escalator #1]
    14037:20170119:103706.946 server #13 started [trapper #5]
    14036:20170119:103706.950 server #12 started [trapper #4]
    14045:20170119:103706.953 server #21 started [history syncer #2]
    14050:20170119:103706.953 server #26 started [proxy poller #1]
    14047:20170119:103706.954 server #23 started [history syncer #4]
    14040:20170119:103706.955 server #16 started [housekeeper #1]
    14051:20170119:103706.956 server #27 started [self-monitoring #1]
    14049:20170119:103706.969 server #25 started [snmp trapper #1]
    14043:20170119:103707.009 server #19 started [discoverer #1]

    It keeps doing these (looks like auto restart) whenever it stops.
    I looked through the previous mail thread about this and doesn't seems to be matched with any uses cases with I have experienced.

    Any idea what went wrong and how to deal with this?
  • batchenr
    Senior Member
    • Sep 2016
    • 440

    #2
    Originally posted by iwidjaya
    Folks,

    I'm running into issues where the frontend tells 'Zabbix Server is not running....'. When I check on the zabbix_server.log, it shows below:

    13974:20170119:103654.271 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13972:20170119:103654.272 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13979:20170119:103654.273 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13975:20170119:103654.273 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13980:20170119:103654.274 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13978:20170119:103654.275 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13977:20170119:103654.275 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13981:20170119:103654.276 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13983:20170119:103654.276 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13985:20170119:103654.277 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13989:20170119:103654.277 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13986:20170119:103654.277 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13982:20170119:103654.278 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13990:20170119:103654.279 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13988:20170119:103654.280 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13984:20170119:103654.280 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13992:20170119:103654.281 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13976:20170119:103654.281 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13991:20170119:103654.281 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13994:20170119:103654.282 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13973:20170119:103654.282 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13970:20170119:103654.283 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13969:20170119:103654.283 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13995:20170119:103654.284 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13971:20170119:103654.284 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13987:20170119:103654.284 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13993:20170119:103654.285 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13963:20170119:103654.285 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...
    13963:20170119:103656.291 syncing history data...
    13963:20170119:103656.297 syncing history data done
    13963:20170119:103656.297 syncing trends data...
    13963:20170119:103656.576 syncing trends data done
    13963:20170119:103656.577 Zabbix Server stopped. Zabbix 3.0.2 (revision 59540).
    14019:20170119:103706.825 Starting Zabbix Server. Zabbix 3.0.2 (revision 59540).
    14019:20170119:103706.825 ****** Enabled features ******
    14019:20170119:103706.825 SNMP monitoring: YES
    14019:20170119:103706.825 IPMI monitoring: YES
    14019:20170119:103706.825 Web monitoring: YES
    14019:20170119:103706.825 VMware monitoring: YES
    14019:20170119:103706.825 SMTP authentication: YES
    14019:20170119:103706.825 Jabber notifications: YES
    14019:20170119:103706.825 Ez Texting notifications: YES
    14019:20170119:103706.826 ODBC: YES
    14019:20170119:103706.826 SSH2 support: YES
    14019:20170119:103706.826 IPv6 support: YES
    14019:20170119:103706.826 TLS support: YES
    14019:20170119:103706.826 ******************************
    14019:20170119:103706.826 using configuration file: /etc/zabbix/zabbix_server.conf
    14019:20170119:103706.844 current database version (mandatory/optional): 03000000/03000000
    14019:20170119:103706.844 required mandatory version: 03000000
    14019:20170119:103706.922 server #0 started [main process]
    14025:20170119:103706.923 server #1 started [configuration syncer #1]
    14026:20170119:103706.923 server #2 started [db watchdog #1]
    14027:20170119:103706.924 server #3 started [poller #1]
    14028:20170119:103706.925 server #4 started [poller #2]
    14029:20170119:103706.926 server #5 started [poller #3]
    14030:20170119:103706.926 server #6 started [poller #4]
    14031:20170119:103706.927 server #7 started [poller #5]
    14032:20170119:103706.928 server #8 started [unreachable poller #1]
    14033:20170119:103706.929 server #9 started [trapper #1]
    14039:20170119:103706.936 server #15 started [alerter #1]
    14041:20170119:103706.937 server #17 started [timer #1]
    14042:20170119:103706.938 server #18 started [http poller #1]
    14035:20170119:103706.939 server #11 started [trapper #3]
    14044:20170119:103706.940 server #20 started [history syncer #1]
    14046:20170119:103706.942 server #22 started [history syncer #3]
    14038:20170119:103706.943 server #14 started [icmp pinger #1]
    14034:20170119:103706.944 server #10 started [trapper #2]
    14048:20170119:103706.944 server #24 started [escalator #1]
    14037:20170119:103706.946 server #13 started [trapper #5]
    14036:20170119:103706.950 server #12 started [trapper #4]
    14045:20170119:103706.953 server #21 started [history syncer #2]
    14050:20170119:103706.953 server #26 started [proxy poller #1]
    14047:20170119:103706.954 server #23 started [history syncer #4]
    14040:20170119:103706.955 server #16 started [housekeeper #1]
    14051:20170119:103706.956 server #27 started [self-monitoring #1]
    14049:20170119:103706.969 server #25 started [snmp trapper #1]
    14043:20170119:103707.009 server #19 started [discoverer #1]

    It keeps doing these (looks like auto restart) whenever it stops.
    I looked through the previous mail thread about this and doesn't seems to be matched with any uses cases with I have experienced.

    Any idea what went wrong and how to deal with this?
    yes it does look like a restart,
    there is no script that could have do it ?

    try to start zabbix like this :

    stop it first (/etc/init.d/zabbix stop | systemctl stop zabbix_agentd)
    and then

    chown -R zabbix:zabbix /var/log/zabbix
    chown -R zabbix:zabbix /var/run/zabbix
    chmod -R 775 /var/log/zabbix/
    chmod -R 775 /var/run/zabbix/
    -------------------------------------> or your own file path

    /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf

    see if you get some errors.

    if not then make zabbix_server.conf debug level 4 and post

    Comment

    • iwidjaya
      Junior Member
      • Apr 2016
      • 16

      #3
      same error and also after I change the permissions, looks like now I'm not able to access the webUI - got HTTP 500 Internal Server error.

      I am unable to attach the file here, it says quota has been reached.

      Comment

      • iwidjaya
        Junior Member
        • Apr 2016
        • 16

        #4
        I extract some lines from the log from where it was before completely stopped.

        See attached. Not sure whether this is useful since it's very limited to upload files in this forum.
        Attached Files

        Comment

        • iwidjaya
          Junior Member
          • Apr 2016
          • 16

          #5
          in the /var/log/messages, I found below:

          Jan 20 09:18:41 zabbix_server_hostname systemd: Starting Zabbix Server...
          Jan 20 09:18:41 zabbix_server_hostname systemd: PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
          Jan 20 09:20:11 zabbix_server_hostname systemd: zabbix-server.service start operation timed out. Terminating.
          Jan 20 09:20:13 zabbix_server_hostname systemd: Failed to start Zabbix Server.
          Jan 20 09:20:13 zabbix_server_hostname systemd: Unit zabbix-server.service entered failed state.
          Jan 20 09:20:13 zabbix_server_hostname systemd: zabbix-server.service failed.
          @Jan 20 09:20:23 zabbix_server_hostname systemd: zabbix-server.service holdoff time over, scheduling restart.
          Jan 20 09:20:23 zabbix_server_hostname systemd: Starting Zabbix Server...
          Jan 20 09:20:23 zabbix_server_hostname systemd: PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
          Jan 20 09:21:53 zabbix_server_hostname systemd: zabbix-server.service start operation timed out. Terminating.
          Jan 20 09:21:55 zabbix_server_hostname systemd: Failed to start Zabbix Server.
          Jan 20 09:21:55 zabbix_server_hostname systemd: Unit zabbix-server.service entered failed state.
          Jan 20 09:21:55 zabbix_server_hostname systemd: zabbix-server.service failed.

          BTW, I'm starting the zabbix server using: service zabbix-server start

          Comment

          • Pada
            Senior Member
            • Apr 2012
            • 236

            #6
            Make sure that your /etc/zabbix/zabbix_server.conf file's PidFile is matching your init.d/systemd service script of "/run/zabbix/zabbix_server.pid"

            To me it looks like the one is configured for /var/run/zabbix and the other one just /run/zabbix

            One they're matching and its still not working, then check that the folder exists and that the permissions on the folder and files are correct.

            Comment

            • batchenr
              Senior Member
              • Sep 2016
              • 440

              #7
              Originally posted by Pada
              Make sure that your /etc/zabbix/zabbix_server.conf file's PidFile is matching your init.d/systemd service script of "/run/zabbix/zabbix_server.pid"

              To me it looks like the one is configured for /var/run/zabbix and the other one just /run/zabbix

              One they're matching and its still not working, then check that the folder exists and that the permissions on the folder and files are correct.
              check what Pada told you, if it helps this is my settings :

              #cat /etc/zabbix/zabbix_server.conf | grep -i pid
              PidFile=/var/run/zabbix/zabbix_server.pid

              -rw-rw-r-- 1 zabbix zabbix /var/run/zabbix/zabbix_server.pid

              Comment

              • iwidjaya
                Junior Member
                • Apr 2016
                • 16

                #8
                Issue Resolved

                Thanks guys.

                I resolved the issue.

                The /usr/lib/system/system/zabbix-server.service contained default *.pid which was not matched with /etc/zabbix/zabbix_server.conf.

                After I matched it and then restart again, it is now working fine.

                Comment

                • maas187
                  Junior Member
                  • Dec 2015
                  • 5

                  #9
                  Zabbix 3 - Fails to start on Centos7

                  Hey guys.
                  Everytime I try to start zabbix it fails.

                  [root@ulenmon01 zabbix]# systemctl restart zabbix-server.service
                  Warning: zabbix-server.service changed on disk. Run 'systemctl daemon-reload' to reload units.
                  Job for zabbix-server.service failed because a timeout was exceeded. See "systemctl status zabbix-server.service" and "journalctl -xe" for details.


                  However : when I use the command /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf

                  It works just fine..



                  - I matched the PID in the /usr/lib/systemd/system/zabbix-server.service and conf file.

                  Still the same...

                  one thing I noticed, when I create the pid file manually - assuming I can restart zabbixserver and see if it can pick it up. It disappears without me touching.

                  Anyone has seen this before.

                  Thanks

                  Comment

                  • sancho
                    Senior Member
                    Zabbix Certified SpecialistZabbix Certified Professional
                    • Mar 2015
                    • 295

                    #10
                    Hello everyone,
                    I encountered the same problem, also had different routes from the PID, but although I modified them to match the .conf and zabbix-server.service the problem was repeated.

                    I have solved it by deleting in zabbix-server.service the line "TimeoutSec = infinity"

                    After doing so it has restarted correctly.

                    A greeting.

                    Comment

                    • maas187
                      Junior Member
                      • Dec 2015
                      • 5

                      #11
                      I think its fixed now.

                      I just ran a yum update and it installed a new version that has this fixed

                      Thanks for the help.

                      Comment

                      Working...