Announcement

Collapse
No announcement yet.

Zabbix server is not running: the iformation may not be current

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Zabbix server is not running: the iformation may not be current

    Problem:
    ++++++++
    • Zabbix Webfrontend displays this message:
      Zabbix server is not running: the iformation may not be current
    • "Zabbix server is running" No is displayed in the webfrontend as well.
    • Webfrontend is accessible and displays new problems.
    • Zabbix-server log contains this message: 1704:20171119:031406.248 failed to accept an incoming connection: connection rejected, getpeername() failed: [107] Transport endpoint is not connected




    Config:
    +++++++
    CentOS Linux release 7.4.1708 (Core)

    zabbix-agent.x86_64 3.4.3-1.el7 @zabbix
    zabbix-get.x86_64 3.4.3-1.el7 @zabbix
    zabbix-java-gateway.x86_64 3.4.3-1.el7 @zabbix
    zabbix-release.noarch 3.4-2.el7 @zabbix
    zabbix-sender.x86_64 3.4.3-1.el7 @zabbix
    zabbix-server-mysql.x86_64 3.4.3-1.el7 @zabbix
    zabbix-web.noarch 3.4.3-1.el7 @zabbix
    zabbix-web-mysql.noarch 3.4.3-1.el7 @zabbix

    SELinux and Firewall are testwise disabled

    echo $HOSTNAME
    chdcvzabbix01.our-OLD-domain.name.domainending


    Zabbix Processes:
    +++++++++++++++++
    Code:
    ps -ef | grep -i zab
    zabbix    1138     1  0 11:05 ?        00:00:00 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
    zabbix    1144  1138  0 11:05 ?        00:00:00 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
    zabbix    1147  1138  0 11:05 ?        00:00:00 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
    zabbix    1149  1138  0 11:05 ?        00:00:00 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
    zabbix    1150  1138  0 11:05 ?        00:00:00 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
    zabbix    1151  1138  0 11:05 ?        00:00:00 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
    zabbix    1466     1  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix    1664  1466  0 11:05 ?        00:00:01 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.501025 sec, idle 60 sec]
    zabbix    1665  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: alerter #1 started
    zabbix    1666  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: housekeeper [startup idle for 30 minutes]
    zabbix    1667  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: timer #1 [processed 0 triggers, 0 events in 0.000016 sec, 0 maintenances in 0.012494 sec, idle 30 sec]
    zabbix    1668  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000682 sec, idle 5 sec]
    zabbix    1669  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000622 sec, idle 60 sec]
    zabbix    1670  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: history syncer #1 [synced 16 items in 0.001925 sec, idle 1 sec]
    zabbix    1672  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000007 sec, idle 1 sec]
    zabbix    1674  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: history syncer #3 [synced 0 items in 0.000054 sec, idle 1 sec]
    zabbix    1676  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000006 sec, idle 1 sec]
    zabbix    1678  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000839 sec, idle 3 sec]
    zabbix    1679  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000003 sec, idle 5 sec]
    zabbix    1680  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: self-monitoring [processed data in 0.000006 sec, idle 1 sec]
    zabbix    1681  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000001 sec, idle 5 sec]
    zabbix    1682  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: vmware collector #2 [updated 0, removed 0 VMware services in 0.000002 sec, idle 5 sec]
    zabbix    1684  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: task manager [processed 0 task(s) in 0.000424 sec, idle 5 sec]
    zabbix    1686  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: poller #1 [got 5 values in 0.001862 sec, idle 1 sec]
    zabbix    1688  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: poller #2 [got 5 values in 0.001888 sec, idle 1 sec]
    zabbix    1689  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: poller #3 [got 6 values in 0.001923 sec, idle 1 sec]
    zabbix    1690  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000005 sec, idle 5 sec]
    zabbix    1691  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: trapper #1 [processing data]
    zabbix    1692  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: trapper #2 [processing data]
    zabbix    1693  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: trapper #3 [processing data]
    zabbix    1695  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: icmp pinger #1 [got 0 values in 0.000048 sec, idle 1 sec]
    zabbix    1697  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: icmp pinger #2 [got 0 values in 0.000042 sec, idle 1 sec]
    zabbix    1698  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: alert manager #1 [sent 0, failed 0 alerts, idle 5.009370 sec during 5.009372 sec]
    zabbix    1699  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 15 values, idle 5.004293 sec during 5.004424 sec]
    zabbix    1700  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: preprocessing worker #1 started
    zabbix    1701  1466  0 11:05 ?        00:00:00 /usr/sbin/zabbix_server: preprocessing worker #2 started
    root      2086  1770  0 11:16 pts/0    00:00:00 grep --color=auto -i zab
    Open Ports from User Zabbix
    +++++++++++++++++++++++++++
    Code:
     lsof -a -i -u zabbix
    COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
    zabbix_ag 1138 zabbix    4u  IPv4  16236      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1138 zabbix    5u  IPv6  16237      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1144 zabbix    4u  IPv4  16236      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1144 zabbix    5u  IPv6  16237      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1147 zabbix    4u  IPv4  16236      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1147 zabbix    5u  IPv6  16237      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1149 zabbix    4u  IPv4  16236      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1149 zabbix    5u  IPv6  16237      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1150 zabbix    4u  IPv4  16236      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1150 zabbix    5u  IPv6  16237      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1151 zabbix    4u  IPv4  16236      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_ag 1151 zabbix    5u  IPv6  16237      0t0  TCP *:zabbix-agent (LISTEN)
    zabbix_se 1466 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1466 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1664 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1664 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1665 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1665 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1666 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1666 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1667 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1667 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1668 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1668 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1669 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1669 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1670 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1670 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1672 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1672 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1674 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1674 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1676 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1676 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1678 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1678 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1679 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1679 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1680 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1680 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1681 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1681 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1682 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1682 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1684 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1684 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1686 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1686 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1688 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1688 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1689 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1689 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1690 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1690 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1691 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1691 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1691 zabbix    8u  IPv4  25696      0t0  TCP our-domani:zabbix-trapper->10.189.15.21:49422 (CLOSE_WAIT)
    zabbix_se 1691 zabbix    9u  IPv4  23202      0t0  UDP our-domani:37547->servername2.our-domani:domain
    zabbix_se 1692 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1692 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1692 zabbix    8u  IPv4  23193      0t0  TCP our-domani:zabbix-trapper->10.189.15.22:60518 (CLOSE_WAIT)
    zabbix_se 1692 zabbix    9u  IPv4  23201      0t0  UDP our-domani:53823->servername2.our-domani:domain
    zabbix_se 1693 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1693 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1693 zabbix    8u  IPv4  23197      0t0  TCP our-domani:zabbix-trapper->10.171.75.11:50300 (CLOSE_WAIT)
    zabbix_se 1693 zabbix    9u  IPv4  23198      0t0  UDP our-domani:37602->servername1.our-domani:domain
    zabbix_se 1693 zabbix   10u  IPv4  23200      0t0  UDP our-domani:45733->servername2.our-domani:domain
    zabbix_se 1695 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1695 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1697 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1697 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1698 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1698 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1699 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1699 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1700 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1700 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1701 zabbix    5u  IPv4  18198      0t0  TCP *:zabbix-trapper (LISTEN)
    zabbix_se 1701 zabbix    6u  IPv6  18199      0t0  TCP *:zabbix-trapper (LISTEN)

    Zabbix-Agent (systemctl)
    ++++++++++++++++++++++++
    Code:
    systemctl status zabbix-agent
    ● zabbix-agent.service - Zabbix Agent
       Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
       Active: active (running) since Tue 2017-11-21 11:26:39 CET; 13s ago
      Process: 2221 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=0/SUCCESS)
      Process: 2394 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)
     Main PID: 2396 (zabbix_agentd)
       CGroup: /system.slice/zabbix-agent.service
               ├─2396 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
               ├─2397 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
               ├─2398 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
               ├─2399 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
               ├─2400 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
               └─2401 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
    
    Nov 21 11:26:39 chdcvzabbix01our-domani systemd[1]: Starting Zabbix Agent...
    Nov 21 11:26:39 chdcvzabbix01our-domani systemd[1]: Started Zabbix Agent.
    After a while:

    Code:
    systemctl status zabbix-agent
    ● zabbix-agent.service - Zabbix Agent
       Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
       Active: active (running) since Tue 2017-11-21 12:56:55 CET; 28min ago
      Process: 4120 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=0/SUCCESS)
      Process: 4123 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)
     Main PID: 4125 (zabbix_agentd)
       CGroup: /system.slice/zabbix-agent.service
               ├─4125 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
               ├─4126 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
               ├─4127 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
               ├─4128 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
               ├─4129 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
               └─4130 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
    
    Nov 21 12:56:55 chdcvzabbix01.ch.viamat.com systemd[1]: Starting Zabbix Agent...
    Nov 21 12:56:55 chdcvzabbix01.ch.viamat.com systemd[1]: zabbix-agent.service: Supervising process 4125 which is not our child. We'll most likely not notice when it exits.
    Nov 21 12:56:55 chdcvzabbix01.ch.viamat.com systemd[1]: Started Zabbix Agent.


    Zabbix-Agent (journalctl)
    ++++++++++++++++++++++++
    Code:
    journalctl -u zabbix-agent
    -- Logs begin at Tue 2017-11-21 11:28:30 CET, end at Tue 2017-11-21 11:29:16 CET. --
    Nov 21 11:28:40 chdcvzabbix01.our-domani systemd[1]: Starting Zabbix Agent...
    Nov 21 11:28:40 chdcvzabbix01.our-domani systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start.
    Nov 21 11:28:40 chdcvzabbix01.our-domani systemd[1]: Started Zabbix Agent.
    
    ----> ls -l /run/zabbix/zabbix_agentd.pid
    -rw-rw-r-- 1 zabbix zabbix 4 Nov 21 11:28 /run/zabbix/zabbix_agentd.pid


    Zabbix-Server (systemctl)
    +++++++++++++++++++++++++
    Code:
     systemctl status zabbix-server
    ● zabbix-server.service - Zabbix Server
       Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
       Active: active (running) since Tue 2017-11-21 11:28:41 CET; 2min 5s ago
      Process: 1070 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
     Main PID: 1512 (zabbix_server)
       CGroup: /system.slice/zabbix-server.service
               ├─1512 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
               ├─1668 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.523490 sec, idle 60 sec]
               ├─1669 /usr/sbin/zabbix_server: alerter #1 started
               ├─1670 /usr/sbin/zabbix_server: housekeeper [startup idle for 30 minutes]
               ├─1671 /usr/sbin/zabbix_server: timer #1 [processed 0 triggers, 0 events in 0.000082 sec, 0 maintenances in 0.000000 sec, idle 30 sec]
               ├─1673 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000674 sec, idle 5 sec]
               ├─1675 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000513 sec, idle 60 sec]
               ├─1676 /usr/sbin/zabbix_server: history syncer #1 [synced 3 items in 0.003086 sec, idle 1 sec]
               ├─1677 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000001 sec, idle 1 sec]
               ├─1679 /usr/sbin/zabbix_server: history syncer #3 [synced 0 items in 0.000002 sec, idle 1 sec]
               ├─1682 /usr/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000001 sec, idle 1 sec]
               ├─1684 /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000687 sec, idle 3 sec]
               ├─1685 /usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000003 sec, idle 5 sec]
               ├─1686 /usr/sbin/zabbix_server: self-monitoring [processed data in 0.000006 sec, idle 1 sec]
               ├─1687 /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000002 sec, idle 5 sec]
               ├─1688 /usr/sbin/zabbix_server: vmware collector #2 [updated 0, removed 0 VMware services in 0.000004 sec, idle 5 sec]
               ├─1689 /usr/sbin/zabbix_server: task manager [processed 0 task(s) in 0.000475 sec, idle 5 sec]
               ├─1691 /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.000689 sec, idle 1 sec]
               ├─1692 /usr/sbin/zabbix_server: poller #2 [got 1 values in 0.000678 sec, idle 1 sec]
               ├─1693 /usr/sbin/zabbix_server: poller #3 [got 1 values in 0.000561 sec, idle 1 sec]
               ├─1695 /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000004 sec, idle 5 sec]
               ├─1696 /usr/sbin/zabbix_server: trapper #1 [processing data]
               ├─1697 /usr/sbin/zabbix_server: trapper #2 [processing data]
               ├─1698 /usr/sbin/zabbix_server: trapper #3 [processing data]
               ├─1699 /usr/sbin/zabbix_server: icmp pinger #1 [got 0 values in 0.000112 sec, idle 1 sec]
               ├─1700 /usr/sbin/zabbix_server: icmp pinger #2 [got 0 values in 0.000113 sec, idle 1 sec]
               ├─1701 /usr/sbin/zabbix_server: alert manager #1 [sent 0, failed 0 alerts, idle 5.009563 sec during 5.009565 sec]
               ├─1702 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 22 values, idle 5.004756 sec during 5.004926 sec]
               ├─1703 /usr/sbin/zabbix_server: preprocessing worker #1 started
               └─1704 /usr/sbin/zabbix_server: preprocessing worker #2 started
    
    Nov 21 11:28:40 chdcvzabbix01.our-domani systemd[1]: Starting Zabbix Server...
    Nov 21 11:28:41 chdcvzabbix01.our-domani systemd[1]: Started Zabbix Server.

    Zabbix-Server (journalctl)
    +++++++++++++++++++++++++
    Code:
     journalctl -u zabbix-server
    -- Logs begin at Tue 2017-11-21 11:28:30 CET, end at Tue 2017-11-21 11:29:16 CET. --
    Nov 21 11:28:40 chdcvzabbix01.our-domani systemd[1]: Starting Zabbix Server...
    Nov 21 11:28:41 chdcvzabbix01.our-domani systemd[1]: Started Zabbix Server.




    Zabbix Server Conf
    +++++++++++++++++++
    Code:
    grep ^[^#] /etc/zabbix/zabbix_server.conf
    LogFile=/var/log/zabbix/zabbix_server.log
    LogFileSize=128
    DebugLevel=3
    PidFile=/var/run/zabbix/zabbix_server.pid
    SocketDir=/var/run/zabbix
    DBHost=localhost
    DBName=zabbixdb
    DBUser=zabbixuser
    DBPassword=OurPassword
    StartPollers=3
    StartIPMIPollers=0
    StartPreprocessors=2
    StartPollersUnreachable=1
    StartTrappers=3
    StartPingers=2
    StartDiscoverers=1
    StartHTTPPollers=1
    StartTimers=1
    StartEscalators=1
    StartAlerters=1
    StartJavaPollers=0
    StartVMwareCollectors=2
    VMwareFrequency=600
    VMwarePerfFrequency=300
    VMwareCacheSize=48M
    VMwareTimeout=10
    SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
    CacheSize=128M
    CacheUpdateFrequency=60
    StartDBSyncers=4
    HistoryCacheSize=48M
    HistoryIndexCacheSize=24M
    TrendCacheSize=24M
    ValueCacheSize=24M
    Timeout=4
    AlertScriptsPath=/usr/lib/zabbix/alertscripts
    ExternalScripts=/usr/lib/zabbix/externalscripts
    LogSlowQueries=3000

    Zabbix Agent Conf
    +++++++++++++++++++
    Code:
    PidFile=/var/run/zabbix/zabbix_agentd.pid
    LogFile=/var/log/zabbix/zabbix_agentd.log
    LogFileSize=128
    DebugLevel=4
    Server=10.170.12.240 (This was 127.0.0.1 before,but this way the agent did not work. So I changed it with the ip)
    ServerActive=10.170.12.240 (This was 127.0.0.1 before,but this way the agent did not work. So I changed it with the ip)
    HostnameItem=system.hostname
    Include=/etc/zabbix/zabbix_agentd.d/*.conf

    Zabbix-Server Log: /var/log/zabbix/zabbix_server
    +++++++++++++++++++++++++++++++++++++++++++++++++
    Code:
    1704:20171119:031349.818 failed to accept an incoming connection: connection rejected, getpeername() failed: [107] Transport endpoint is not connected


    Zabbix-Server Log: /var/log/zabbix/zabbix_agentd.log
    +++++++++++++++++++++++++++++++++++++++++++++++++
    Code:
    8684:20171121:105617.859 Starting Zabbix Agent [chdcvzabbix01.our-olddomain]. Zabbix 3.4.3 (revision 73588).
      8684:20171121:105617.859 **** Enabled features ****
      8684:20171121:105617.859 IPv6 support:          YES
      8684:20171121:105617.859 TLS support:           YES
      8684:20171121:105617.859 **************************
      8684:20171121:105617.859 using configuration file: /etc/zabbix/zabbix_agentd.conf
      8684:20171121:105617.859 In zbx_load_modules()
      8684:20171121:105617.859 End of zbx_load_modules():SUCCEED
      8684:20171121:105617.859 In init_collector_data()
      8684:20171121:105617.859 In zbx_dshm_create() size:0
      8684:20171121:105617.859 End of zbx_dshm_create():SUCCEED shmid:-1
      8684:20171121:105617.859 End of init_collector_data()
      8684:20171121:105617.859 agent #0 started [main process]
      8686:20171121:105617.862 agent #2 started[listener #1]
      8686:20171121:105617.862 In zbx_tls_init_child()
      8686:20171121:105617.863 OpenSSL library (version OpenSSL 1.0.1e-fips 11 Feb 2013) initialized
      8686:20171121:105617.863 End of zbx_tls_init_child()
      8686:20171121:105617.863 __zbx_zbx_setproctitle() title:'listener #1 [waiting for connection]'
      8688:20171121:105617.863 agent #4 started[listener #3]
      8688:20171121:105617.864 In zbx_tls_init_child()
      8688:20171121:105617.865 OpenSSL library (version OpenSSL 1.0.1e-fips 11 Feb 2013) initialized
      8688:20171121:105617.865 End of zbx_tls_init_child()
      8688:20171121:105617.865 __zbx_zbx_setproctitle() title:'listener #3 [waiting for connection]'
      8689:20171121:105617.867 agent #5 started [active checks #1]
      8689:20171121:105617.867 In zbx_tls_init_child()
      8689:20171121:105617.869 OpenSSL library (version OpenSSL 1.0.1e-fips 11 Feb 2013) initialized
      8689:20171121:105617.869 End of zbx_tls_init_child()
      8689:20171121:105617.869 In init_active_metrics()
      8689:20171121:105617.869 buffer: first allocation for 100 elements
      8689:20171121:105617.869 End of init_active_metrics()
      8689:20171121:105617.869 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
      8689:20171121:105617.869 End of send_buffer():SUCCEED
      8689:20171121:105617.869 __zbx_zbx_setproctitle() title:'active checks #1 [getting list of active checks]'
      8689:20171121:105617.869 In refresh_active_checks() host:'127.0.0.1' port:10051
      8687:20171121:105617.869 agent #3 started[listener #2]
      8687:20171121:105617.869 In zbx_tls_init_child()
      8687:20171121:105617.871 OpenSSL library (version OpenSSL 1.0.1e-fips 11 Feb 2013) initialized
      8685:20171121:105617.871 agent #1 started [collector]
      8687:20171121:105617.871 End of zbx_tls_init_child()
      8689:20171121:105617.871 active check configuration update from [127.0.0.1:10051] started to fail (cannot connect to [[127.0.0.1]:10051]: [111] Connection refused)
      8685:20171121:105617.872 In init_cpu_collector()
      8689:20171121:105617.872 End of refresh_active_checks():FAIL
      8685:20171121:105617.872 End of init_cpu_collector():SUCCEED
      8687:20171121:105617.872 __zbx_zbx_setproctitle() title:'listener #2 [waiting for connection]'
      8689:20171121:105617.872 __zbx_zbx_setproctitle() title:'active checks #1 [processing active checks]'
      8689:20171121:105617.872 In process_active_checks() server:'127.0.0.1' port:10051
      8685:20171121:105617.872 __zbx_zbx_setproctitle() title:'collector [processing data]'
      8689:20171121:105617.872 End of process_active_checks()
      8685:20171121:105617.872 In update_cpustats()
      8689:20171121:105617.872 In get_min_nextcheck()
      8685:20171121:105617.872 End of update_cpustats()
      8689:20171121:105617.873 End of get_min_nextcheck():-1
      8685:20171121:105617.873 __zbx_zbx_setproctitle() title:'collector [idle 1 sec]'
      8689:20171121:105617.873 __zbx_zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
      8689:20171121:105618.873 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
      8689:20171121:105618.873 End of send_buffer():SUCCEED
      8685:20171121:105618.873 __zbx_zbx_setproctitle() title:'collector [processing data]'
      8689:20171121:105618.873 __zbx_zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
      8685:20171121:105618.873 In update_cpustats()
      8685:20171121:105618.873 End of update_cpustats()
      8685:20171121:105618.873 __zbx_zbx_setproctitle() title:'collector [idle 1 sec]'
      8689:20171121:105619.873 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
      8689:20171121:105619.873 End of send_buffer():SUCCEED
      8689:20171121:105619.873 __zbx_zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
      8685:20171121:105619.873 __zbx_zbx_setproctitle() title:'collector [processing data]'
      8685:20171121:105619.873 In update_cpustats()
      8685:20171121:105619.874 End of update_cpustats()
      8685:20171121:105619.874 __zbx_zbx_setproctitle() title:'collector [idle 1 sec]'
      8689:20171121:105620.874 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
      8689:20171121:105620.874 End of send_buffer():SUCCEED
      8689:20171121:105620.874 __zbx_zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
      8685:20171121:105620.874 __zbx_zbx_setproctitle() title:'collector [processing data]'
      8685:20171121:105620.874 In update_cpustats()
      8685:20171121:105620.874 End of update_cpustats()
      8685:20171121:105620.874 __zbx_zbx_setproctitle() title:'collector [idle 1 sec]'
      8689:20171121:105621.874 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
      8689:20171121:105621.874 End of send_buffer():SUCCEED
      8689:20171121:105621.874 __zbx_zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
      8685:20171121:105621.874 __zbx_zbx_setproctitle() title:'collector [processing data]'
      8685:20171121:105621.874 In update_cpustats()
      8685:20171121:105621.874 End of update_cpustats()
      8685:20171121:105621.874 __zbx_zbx_setproctitle() title:'collector [idle 1 sec]'
      8689:20171121:105622.874 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
      8689:20171121:105622.874 End of send_buffer():SUCCEED
      8689:20171121:105622.874 __zbx_zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
      8685:20171121:105622.875 __zbx_zbx_setproctitle() title:'collector [processing data]'
      8685:20171121:105622.875 In update_cpustats()
      8685:20171121:105622.875 End of update_cpustats()
      8685:20171121:105622.875 __zbx_zbx_setproctitle() title:'collector [idle 1 sec]'
      8689:20171121:105623.874 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
      8689:20171121:105623.874 End of send_buffer():SUCCEED
      8689:20171121:105623.875 __zbx_zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
      8685:20171121:105623.875 __zbx_zbx_setproctitle() title:'collector [processing data]'
      8685:20171121:105623.875 In update_cpustats()
      8685:20171121:105623.875 End of update_cpustats()
      8685:20171121:105623.875 __zbx_zbx_setproctitle() title:'collector [idle 1 sec]'
      8689:20171121:105624.875 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
      8689:20171121:105624.875 End of send_buffer():SUCCEED
      8689:20171121:105624.875 __zbx_zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
      8685:20171121:105624.876 __zbx_zbx_setproctitle() title:'collector [processing data]'
      8685:20171121:105624.876 In update_cpustats()

    /etc/zabbix/web/zabbix.conf.php
    +++++++++++++++++++++++++++++++
    Code:
      <?php
    // Zabbix GUI configuration file.
    global $DB;
    
    $DB['TYPE']     = 'MYSQL';
    $DB['SERVER']   = 'localhost';
    $DB['PORT']     = '0';
    $DB['DATABASE'] = 'zabbixdb';
    $DB['USER']     = 'zabbixuser';
    $DB['PASSWORD'] = 'OurPW';
    
    // Schema name. Used for IBM DB2 and PostgreSQL.
    $DB['SCHEMA'] = '';
    
    $ZBX_SERVER      = 'localhost';
    $ZBX_SERVER_PORT = '10051';
    $ZBX_SERVER_NAME = 'Zabbix Monitoring';
    
    $IMAGE_FORMAT_DEFAULT = IMAGE_FORMAT_PNG;



    /etc/httpd/conf/httpd.conf (not all)
    ++++++++++++++++++++++++++++++++++++
    Code:
    EnableSendfile on
    IncludeOptional conf.d/*.conf
    <VirtualHost *:443>
            SSLEngine on
            SSLCertificateFile /etc/pki/tls/certs/domain.crt
            SSLCertificateKeyFile /etc/pki/tls/private/domain.key
            <Directory /usr/share/zabbix/>
            AllowOverride All
            </Directory>
            DocumentRoot /usr/share/zabbix/
            ServerName our-other-domain.name.com
    </VirtualHost>
    NameVirtualHost *:80
    <VirtualHost *:80>
    DocumentRoot /usr/share/zabbix/
    ServerName out-other-domain
        <IfModule mod_rewrite.c>
          RewriteEngine On
          RewriteCond %{HTTPS} off
          RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
        </IfModule>
    </VirtualHost>

    #2
    You mention in Zabbix Agent Conf:

    Code:
    Server=10.170.12.240 (This was 127.0.0.1 before,but this way the agent did not work. So I changed it with the ip)
    So perhaps try setting the same IP for frontend, /etc/zabbix/web/zabbix.conf.php:

    Code:
    $ZBX_SERVER      = '10.170.12.240';

    Comment


      #3
      Dear dimir

      Thanks a lot for you replay!

      I had "localhost" in both configs and changed it recently to 10.170.12.240' (but not in both configs -thanks for the info). I changed it now in both, but it still doesn’t work.

      Regards

      DiViNe

      Comment


        #4
        What do you get if you perform from the command-line:

        Code:
        telnet 10.170.12.240 10051
        or this one:

        Code:
        telnet 127.0.0.1 10051
        also this one please:

        Code:
        /sbin/ifconfig

        Comment


          #5
          here we go:

          Code:
          [[email protected] ~]#telnet 10.170.12.240 10051
          Trying 10.170.12.240...
          Connected to 10.170.12.240.
          Escape character is '^]'.
          
          
          [[email protected] ~]#telnet 127.0.0.1 10051
          Trying 127.0.0.1...
          Connected to 127.0.0.1.
          Escape character is '^]'.
          
          
          [[email protected] ~]#/sbin/ifconfig
          ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
                  inet 10.170.12.240  netmask 255.255.255.0  broadcast 10.170.12.255
                  inet6 fe80::250:56ff:fe87:a8fd  prefixlen 64  scopeid 0x20<link>
                  ether 00:50:56:87:a8:fd  txqueuelen 1000  (Ethernet)
                  RX packets 90611516  bytes 7293763950 (6.7 GiB)
                  RX errors 0  dropped 2044  overruns 0  frame 0
                  TX packets 96339415  bytes 10642285259 (9.9 GiB)
                  TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
          
          lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
                  inet 127.0.0.1  netmask 255.0.0.0
                  inet6 ::1  prefixlen 128  scopeid 0x10<host>
                  loop  txqueuelen 1  (Local Loopback)
                  RX packets 4171639  bytes 248051024 (236.5 MiB)
                  RX errors 0  dropped 0  overruns 0  frame 0
                  TX packets 4171639  bytes 248051024 (236.5 MiB)
                  TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

          I tried it also with NC:

          Code:
          [[email protected] ~]#nc -v 10.170.12.240 10051
          Ncat: Version 6.40 ( http://nmap.org/ncat )
          Ncat: Connected to 10.170.12.240:10051.
          
          [[email protected] ~]#nc -v 10.170.12.240 10051
          Ncat: Version 6.40 ( http://nmap.org/ncat )
          Ncat: Connected to 10.170.12.240:10051.

          Comment


            #6
            Do you have SELinux/grsecurity enabled? Could they be causing an issue?

            Comment


              #7
              No:

              Code:
              grep ^[^#] /etc/sysconfig/selinux
              SELINUX=disabled
              SELINUXTYPE=targeted
              I've also disabled testwise the firewall....

              Comment


                #8
                Maybe your Zabbix server trapper process is so busy all the time it cannot handle requests from frontend. In that case you could try increasing the number of trapper processes (you have StartTrappers=3).

                Also make sure the frontend you are using uses the configuration file you specified. As it looks like frontend is has Zabbix server/port misconfigured.

                Comment


                  #9
                  Changed Trapper and looking for frontend-config

                  Thank you for your input!

                  Originally posted by dimir View Post
                  Maybe your Zabbix server trapper process is so busy all the time it cannot handle requests from frontend. In that case you could try increasing the number of trapper processes (you have StartTrappers=3).
                  I increased it up to 500 (and restarted the server) but it didn't work.

                  Also make sure the frontend you are using uses the configuration file you specified. As it looks like frontend is has Zabbix server/port misconfigured.
                  Do you mean this file?
                  /etc/zabbix/web/zabbix.conf.php
                  And what should I change?

                  Thanks you

                  Comment


                    #10
                    I've changed it testwise to 10050 but it didn't work, so I switchted it back to 10051

                    Code:
                    [[email protected] ~]#vi /etc/zabbix/web/zabbix.conf.php
                    <?php
                    // Zabbix GUI configuration file.
                    global $DB;
                    
                    $DB['TYPE']     = 'MYSQL';
                    $DB['SERVER']   = 'localhost';
                    $DB['PORT']     = '0';
                    $DB['DATABASE'] = 'zabbixdb';
                    $DB['USER']     = 'zabbixuser';
                    $DB['PASSWORD'] = 'MyCensoredPassword';
                    
                    // Schema name. Used for IBM DB2 and PostgreSQL.
                    $DB['SCHEMA'] = '';
                    
                    $ZBX_SERVER      = '10.170.12.240';
                    $ZBX_SERVER_PORT = '10050';
                    $ZBX_SERVER_NAME = 'Zabbix Monitoring';
                    
                    $IMAGE_FORMAT_DEFAULT = IMAGE_FORMAT_PNG;

                    Comment


                      #11
                      According to the netstat zabbix server is listening on port 10051. You can connect to it with nc. If frontend is giving "Zabbix server is not running" it means it cannot connect to the port, the only reason I see that might be here is that frontend is using different config file. Try to change the DB settings in this file and see if you get a DB error in frontend.

                      Comment


                        #12
                        Which file?

                        I've started over and set up a new server (new OS and new zabbix installation). I haven't got this message and the frontend said "zabbix server is running" = Yes. Everything is fine, right? Yeah, but not for long:
                        After aprox 15 mins I'll get again the "Zabbix server is not running: the information displayed may not be current" message. Now ther's also an "zabbix server is running" = No. And from this moment on, it stays like that :-/

                        Strange was, that the frontend get's new messages. For ex. when I restart the server, then the "Zabbix server has just been restarted" message appears in the frontend.
                        Now, 1h later, are just the old dashboard messages like "Zabbix agent on Zabbix server is unreachable for 5 minutes" visible. No new "Zabbix server has been restarted" messages appear any more


                        Originally posted by dimir View Post
                        According to the netstat zabbix server is listening on port 10051. You can connect to it with nc.
                        Yup, that looks good:

                        Code:
                        netstat -tulpn
                        Active Internet connections (only servers)
                        Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
                        tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
                        tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1132/sshd
                        tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1651/master
                        tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      1148/zabbix_agentd
                        tcp      129      0 0.0.0.0:10051           0.0.0.0:*               LISTEN      1250/zabbix_server
                        tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      1591/mysqld
                        tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
                        tcp6       0      0 :::80                   :::*                    LISTEN      1141/httpd
                        tcp6       0      0 :::22                   :::*                    LISTEN      1132/sshd
                        tcp6       0      0 ::1:25                  :::*                    LISTEN      1651/master
                        tcp6       0      0 :::10050                :::*                    LISTEN      1148/zabbix_agentd
                        tcp6       0      0 :::10051                :::*                    LISTEN      1250/zabbix_server


                        Originally posted by dimir View Post
                        If frontend is giving "Zabbix server is not running" it means it cannot connect to the port, the only reason I see that might be here is that frontend is using different config file. Try to change the DB settings in this file and see if you get a DB error in frontend.

                        Code:
                        /etc/zabbix/zabbix_server.conf
                        DBHost=localhost
                        to
                        Code:
                        DBHost=10.170.12.240
                        = no sucess

                        Code:
                        /etc/zabbix/web/zabbix.conf.php
                        $DB['SERVER']   = 'localhost';
                        to
                        Code:
                        $DB['SERVER']   = '10.170.12.240';
                        Frontend wasn't available anymore


                        Could you please point out which file I could/should change?
                        Any other ideas other ideas?

                        Comment


                          #13
                          re:Zabbix server is not running: the information may not be current

                          I ran into a similar issue on my newly installed Zabbix server. 3.4 (Ubuntu 16.04).

                          I found that when I added a Cisco switch using a template it added hundreds of items to the host. When I removed the host, I was able to start the service.

                          What the actual fix was that my /usr/share/zabbix/conf/zabbix_server.conf file needed to be tuned. I did not have enough memory allocated to my server.

                          I modified the file /etc/zabbix/zabbix_server.conf and modified the CacheSize to the maximum memory my server would support (8G in my case).

                          I also modified the HistoryIndexCacheSize and TrendCacheSize.

                          Please note that when modifying the Cache sizes, all must be below or equal the CacheSize. Found that the hard way.

                          Hope this helps.

                          Comment

                          Working...
                          X