Stability issue betwwen zabbix_server and RHEL 7

[jusavard]

Hi,
I'm having problem with zabbix-server under Red Hat Enterprise Linux 7. After some time ( i'd say 24h maybe less maybe more) systemd flag the service as "failed"

[root@AnonZabbixSrv ~]# systemctl status zabbix-server
zabbix-server.service - Zabbix Server
   Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled)
   Active: failed (Result: start-limit) since Wed 2015-10-14 09:48:02 EDT; 15min ago
  Process: 58200 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
 Main PID: 58202 (code=exited, status=1/FAILURE)

Oct 14 09:48:02 AnonZabbixSrv systemd[1]: zabbix-server.service: main process exited, code=exited, status=1/FAILURE
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: Unit zabbix-server.service entered failed state.
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: zabbix-server.service holdoff time over, scheduling restart.
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: Stopping Zabbix Server...
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: Starting Zabbix Server...
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: zabbix-server.service start request repeated too quickly, refusing to start.
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: Failed to start Zabbix Server.
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: Unit zabbix-server.service entered failed state.

Trying to restart zabbix-server fail s long the main zabbix-server is not killed :

[root@AnonZabbixSrv ~]# systemctl restart zabbix-server
[root@AnonZabbixSrv ~]# systemctl status zabbix-server
zabbix-server.service - Zabbix Server
   Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled)
   Active: failed (Result: start-limit) since Wed 2015-10-14 09:48:02 EDT; 44s ago
  Process: 58200 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
 Main PID: 58202 (code=exited, status=1/FAILURE)

Oct 14 09:48:02 AnonZabbixSrv systemd[1]: zabbix-server.service: main process exited, code=exited, status=1/FAILURE
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: Unit zabbix-server.service entered failed state.
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: zabbix-server.service holdoff time over, scheduling restart.
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: Stopping Zabbix Server...
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: Starting Zabbix Server...
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: zabbix-server.service start request repeated too quickly, refusing to start.
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: Failed to start Zabbix Server.
Oct 14 09:48:02 AnonZabbixSrv systemd[1]: Unit zabbix-server.service entered failed state.
[root@AnonZabbixSrv ~]# ps -A |grep zabbix
 2375 ?        00:00:00 zabbix_agentd
 2378 ?        00:00:16 zabbix_agentd
 2379 ?        00:00:13 zabbix_agentd
 2380 ?        00:00:15 zabbix_agentd
 2381 ?        00:00:14 zabbix_agentd
 7849 ?        00:00:00 zabbix_server
[root@AnonZabbixSrv ~]# kill -9 7849
[root@AnonZabbixSrv ~]# systemctl start zabbix-server
[root@AnonZabbixSrv ~]# systemctl status zabbix-server
zabbix-server.service - Zabbix Server
   Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled)
   Active: active (running) since Wed 2015-10-14 10:04:13 EDT; 7s ago
  Process: 60196 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
 Main PID: 60198 (zabbix_server)
   CGroup: /system.slice/zabbix-server.service
           ├─60198 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
           ├─60203 /usr/sbin/zabbix_server: configuration syncer [waiting 60 sec for processes]
           ├─60204 /usr/sbin/zabbix_server: db watchdog [synced alerts config in 0.025212 sec, idle 60 sec]
           ├─60205 /usr/sbin/zabbix_server: poller #1 [got 136 values in 3.704853 sec, getting values]
           ├─60207 /usr/sbin/zabbix_server: poller #2 [got 98 values in 3.901976 sec, getting values]
           ├─60208 /usr/sbin/zabbix_server: poller #3 [got 57 values in 3.878049 sec, getting values]
           ├─60209 /usr/sbin/zabbix_server: poller #4 [got 23 values in 5.271251 sec, getting values]
           ├─60210 /usr/sbin/zabbix_server: poller #5 [got 62 values in 4.595845 sec, getting values]
           ├─60211 /usr/sbin/zabbix_server: unreachable poller #1 [got 1 values in 0.003109 sec, getting values]
           ├─60212 /usr/sbin/zabbix_server: trapper #1 [processed data in 0.006886 sec, waiting for connection]
           ├─60213 /usr/sbin/zabbix_server: trapper #2 [processed data in 0.000000 sec, waiting for connection]
           ├─60214 /usr/sbin/zabbix_server: trapper #3 [processed data in 0.000000 sec, waiting for connection]
           ├─60215 /usr/sbin/zabbix_server: trapper #4 [processed data in 0.000000 sec, waiting for connection]
           ├─60216 /usr/sbin/zabbix_server: trapper #5 [processed data in 0.000000 sec, waiting for connection]
           ├─60217 /usr/sbin/zabbix_server: icmp pinger #1 [pinging hosts]
           ├─60218 /usr/sbin/zabbix_server: alerter [sent alerts: 0 success, 0 fail in 0.000953 sec, idle 30 sec]
           ├─60219 /usr/sbin/zabbix_server: housekeeper [startup idle for 30 minutes]
           ├─60220 /usr/sbin/zabbix_server: timer #1 [processed 0 triggers, 0 events in 0.000000 sec, 0 maintenances in 0.000000 s...
           ├─60221 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000000 sec, getting values]
           ├─60222 /usr/sbin/zabbix_server: discoverer #1 [processed 1 rules in 0.006757 sec, idle 10 sec]
           ├─60223 /usr/sbin/zabbix_server: history syncer #1 [synced 2 items in 0.078872 sec, idle 5 sec]
           ├─60224 /usr/sbin/zabbix_server: history syncer #2 [synced 430 items in 0.721895 sec, idle 5 sec]
           ├─60225 /usr/sbin/zabbix_server: history syncer #3 [synced 1 items in 0.042046 sec, idle 5 sec]
           ├─60226 /usr/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000001 sec, idle 5 sec]
           ├─60227 /usr/sbin/zabbix_server: escalator [processed 0 escalations in 0.000845 sec, idle 3 sec]
           ├─60228 /usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000003 sec, idle 5 sec]
           ├─60229 /usr/sbin/zabbix_server: self-monitoring [processed data in 0.000003 sec, idle 1 sec]
           ├─60449 sh -c /usr/sbin/fping -C3 2>&1 </tmp/zabbix_server_60217.pinger;/usr/sbin/fping6 -C3 2>&1 </tmp/zabbix_server_6...
           └─60452 /usr/sbin/fping -C3

Oct 14 10:04:13 AnonZabbixSrv systemd[1]: Starting Zabbix Server...
Oct 14 10:04:13 AnonZabbixSrv systemd[1]: PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 14 10:04:13 AnonZabbixSrv systemd[1]: Started Zabbix Server.

I'm seeing a lot of weird restart in /var/log/message :

[root@AnonZabbixSrv ~]# grep zabbix-server /var/log/messages | grep "Oct 13"
Oct 13 10:35:50 AnonZabbixSrv systemd:Cannot add dependency job for unit zabbix-server-mysql.service, ignoring: Unit zabbix-server-mysql.service failed to load: No such file or directory.
Oct 13 11:55:25 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 49544 which is not our child. We'll most likely not notice when it exits.
Oct 13 11:57:43 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 11:57:43 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 56458 which is not our child. We'll most likely not notice when it exits.
Oct 13 11:59:50 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:03:40 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:08:00 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:11:08 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:11:08 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 7990 which is not our child. We'll most likely not notice when it exits.
Oct 13 12:15:01 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:15:01 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 20002 which is not our child. We'll most likely not notice when it exits.
Oct 13 12:18:52 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:18:52 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 31427 which is not our child. We'll most likely not notice when it exits.
Oct 13 12:22:39 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:22:39 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 42826 which is not our child. We'll most likely not notice when it exits.
Oct 13 12:26:37 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:30:22 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:30:22 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 926 which is not our child. We'll most likely not notice when it exits.
Oct 13 12:34:15 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:34:15 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 12682 which is not our child. We'll most likely not notice when it exits.
Oct 13 12:38:09 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:41:36 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:45:42 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:45:42 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 47380 which is not our child. We'll most likely not notice when it exits.
Oct 13 12:49:40 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:49:40 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 59903 which is not our child. We'll most likely not notice when it exits.
Oct 13 12:53:26 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:53:26 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 6258 which is not our child. We'll most likely not notice when it exits.
Oct 13 12:57:41 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 12:57:41 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 19182 which is not our child. We'll most likely not notice when it exits.
Oct 13 13:01:31 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 13:05:15 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 13:09:04 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 13:12:47 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 13:16:38 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 13:19:52 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 13:19:52 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 63208 which is not our child. We'll most likely not notice when it exits.
Oct 13 13:23:41 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 13:27:44 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 13:31:21 AnonZabbixSrv systemd:zabbix-server.service holdoff time over, scheduling restart.
Oct 13 13:31:21 AnonZabbixSrv systemd:zabbix-server.service: Supervising process 34252 which is not our child. We'll most likely not notice when it exits.

I'm not sure but I think the problem is systemd related.
Here my unit file :

[root@lgp002 ~]# cat /usr/lib/systemd/system/zabbix-server.service
[Unit]
Description=Zabbix Server
After=syslog.target
After=network.target

[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_server.conf"
EnvironmentFile=-/etc/sysconfig/zabbix-server
Type=forking
Restart=always
PIDFile=/run/zabbix/zabbix_server.pid
KillMode=process
ExecStart=/usr/sbin/zabbix_server -c $CONFFILE

[Install]
WantedBy=multi-user.target

And my config file (DB password censored...) :

[root@AnonZabbixSrv ~]# grep -v ^# /etc/zabbix/zabbix_server.conf
LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=0
PidFile=/run/zabbix/zabbix_server.pid
DBHost=127.0.0.1
DBName=zabbix
DBUser=zabbix
DBPassword=*********
DBPort=3308
SNMPTrapperFile=/var/log/snmptt/snmptt.log
AlertScriptsPath=/usr/lib/zabbix/alertscripts
ExternalScripts=/usr/lib/zabbix/externalscripts

Before you ask yeah I know it's localhost host and yes it is port 3308... It's a stunnel to the real database on the database server. Conection the the database is working.

Has anyone ever had such problem with RHEL 7 or even with systemd in general ?

[jusavard]

Forgot to mention :

[root@AnonZabbixSrv~]# grep zabbix_server /var/log/messages
Oct 13 12:03:40 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 12:08:00 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 12:26:38 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 12:41:36 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 13:01:31 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 13:05:15 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 13:09:04 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 13:12:47 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 13:16:38 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 13:46:37 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 13:56:39 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 13:58:48 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 14:13:14 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 14:42:42 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 14:46:38 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 14:53:48 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 15:01:28 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 15:05:28 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 15:08:50 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 15:12:02 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 15:15:58 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 15:31:10 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Oct 13 16:31:23 AnonZabbixSrv systemd:PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.

[Atsushi]

I think PidFile setting is wrong in zabbix_server.conf.

I installed package from Zabbix SIA's repository.
Then, PidFile setting is below.

PidFile=/var/run/zabbix/zabbix_server.pid

[jusavard]

Hi,
so there is a diffrence between the unit file "PidFile" and the zabbix_server.conf ? Systemd should be using a diffrent PidFile then zabbix_server ?

zabbix_server.conf :
PidFile=/var/run/zabbix/zabbix_server.pid

Unit file :
PIDFile=/run/zabbix/zabbix_server.pid

[jusavard]

Interesting update :
Problem is NOT systemd related :

[root@AnonZabbixSrv~]# while true; do date; ps -ef | grep zabbix_server | wc -l; sleep 5; done
Fri Oct 16 10:39:56 EDT 2015
29
Fri Oct 16 10:43:38 EDT 2015
28
Fri Oct 16 10:43:43 EDT 2015
1
Fri Oct 16 10:43:48 EDT 2015
1
Fri Oct 16 10:43:53 EDT 2015
[root@AnonZabbixSrv~]# ps -ef | grep zabbix_server
root     28590  8158  0 10:50 pts/0    00:00:00 grep --color=auto zabbix_server

Without systemd to restart zabbix_server this one lasted only 4 minutes before failling.

[Atsushi]

Please check zabbix_server's log.

ex. /var/log/zabbix/zabbix_server.log

[jusavard]

I think i found it !!!

error is loud and clear in zabbix_server.log :

[root@AnonZabbixSrv zabbix]# egrep "__mem_malloc|mem_realloc" /var/log/zabbix/zabbix_server.log | head
 57335:20151002:224229.531 __mem_malloc: skipped 4 asked 44208 skip_min 5096 skip_max 13544
 57335:20151002:224229.531 [file:dbconfig.c,line:446] zbx_mem_realloc(): out of memory (requested 44208 bytes)
 57335:20151002:224229.531 [file:dbconfig.c,line:446] zbx_mem_realloc(): please increase CacheSize configuration parameter
 57335:20151002:224229.533 11: /usr/sbin/zabbix_server: poller #1 [got 104 values in 4.894004 sec, getting values](__zbx_mem_realloc+0x441) [0x464e11]
 63234:20151006:114946.030 __mem_malloc: skipped 0 asked 136 skip_min 4294967295 skip_max 0
  5351:20151006:115647.378 __mem_malloc: skipped 1 asked 19648 skip_min 5824 skip_max 5824
  5351:20151006:115647.378 [file:dbconfig.c,line:446] zbx_mem_realloc(): out of memory (requested 19648 bytes)
  5351:20151006:115647.378 [file:dbconfig.c,line:446] zbx_mem_realloc(): please increase CacheSize configuration parameter
  5351:20151006:115647.380 11: /usr/sbin/zabbix_server: poller #2 [got 66 values in 5.568508 sec, getting values](__zbx_mem_realloc+0x441) [0x464e11]
 23405:20151006:121531.982 __mem_malloc: skipped 1 asked 19648 skip_min 4032 skip_max 4032
[root@AnonZabbixSrv zabbix]# egrep "__mem_malloc|mem_realloc" /var/log/zabbix/zabbix_server.log | wc -l
2687

So I raised the CacheSize to 64Mb and I will see if it's enough. Default is (i think) 8Mb.
Next time I will start my investigation by this log file.

[nazmul81]

Dear jusavard
How are you? I am facing a problem during configuration of zabbix 3.0 in my rhel7.2
I have configured all the steps of zabbix 3.0 but when i tried to start the zabbix-server there is an error message shown that is

“Job for zabbix-server.service failed because the control process exited with error code. See “systemctl status zabbix-server.service” and “journalctl -xe” for details.”

systemctl status zabbix-server -l
● zabbix-server.service – Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2016-04-17 20:18:03 BDT; 909ms ago
Process: 17817 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=127)
Apr 17 20:18:03 rhel7 systemd[1]: Failed to start Zabbix Server.
Apr 17 20:18:03 rhel7 systemd[1]: Unit zabbix-server.service entered failed state.
Apr 17 20:18:03 rhel7 systemd[1]: zabbix-server.service failed

can anyone help me to find out the problem,thnx in advance

[jusavard]

Hi,
Please post zabbix_server's log and "journalctl -u zabbix-server".

Zabbix_server log should be there : /var/log/zabbix/zabbix_server.log

[nazmul81]

Hi,

Thnx a lot..at least you replied me...I am very much new in zabbix..

I did not find the zabbix-server's log at the specified location

the output is:

[root@rhel7 ~]# cd /var/log/zabbix/
[root@rhel7 zabbix]# ls
[root@rhel7 zabbix]#

"journalctl -u zabbix-server" the output is too big....plz help me


root@rhel7 ~]# journalctl -u zabbix-server
-- Logs begin at Wed 2016-04-20 15:43:37 BDT, end at Wed 2016-04-20 22:07:37 BDT. --
Apr 20 21:36:59 rhel7 systemd[1]: Starting Zabbix Server...
Apr 20 21:37:01 rhel7 zabbix_server[1332]: /usr/sbin/zabbix_server: error while loading
Apr 20 21:37:01 rhel7 systemd[1]: zabbix-server.service: control process exited, code=e
Apr 20 21:37:01 rhel7 systemd[1]: Failed to start Zabbix Server.
Apr 20 21:37:01 rhel7 systemd[1]: Unit zabbix-server.service entered failed state.
Apr 20 21:37:01 rhel7 systemd[1]: zabbix-server.service failed.
Apr 20 21:37:11 rhel7 systemd[1]: zabbix-server.service holdoff time over, scheduling r
Apr 20 21:37:11 rhel7 systemd[1]: Starting Zabbix Server...
Apr 20 21:37:11 rhel7 zabbix_server[2930]: /usr/sbin/zabbix_server: error while loading
Apr 20 21:37:11 rhel7 systemd[1]: zabbix-server.service: control process exited, code=e
Apr 20 21:37:11 rhel7 systemd[1]: Failed to start Zabbix Server.
Apr 20 21:37:11 rhel7 systemd[1]: Unit zabbix-server.service entered failed state.
Apr 20 21:37:11 rhel7 systemd[1]: zabbix-server.service failed.
Apr 20 21:37:22 rhel7 systemd[1]: zabbix-server.service holdoff time over, scheduling r
Apr 20 21:37:22 rhel7 systemd[1]: Starting Zabbix Server...
Apr 20 21:37:22 rhel7 zabbix_server[3136]: /usr/sbin/zabbix_server: error while loading
Apr 20 21:37:22 rhel7 systemd[1]: zabbix-server.service: control process exited, code=e
Apr 20 21:37:22 rhel7 systemd[1]: Failed to start Zabbix Server.
Apr 20 21:37:22 rhel7 systemd[1]: Unit zabbix-server.service entered failed state.
Apr 20 21:37:22 rhel7 systemd[1]: zabbix-server.service failed.
Apr 20 21:37:32 rhel7 systemd[1]: zabbix-server.service holdoff time over, scheduling r
Apr 20 21:37:32 rhel7 systemd[1]: Starting Zabbix Server...
Apr 20 21:37:32 rhel7 zabbix_server[3138]: /usr/sbin/zabbix_server: error while loading
Apr 20 21:37:32 rhel7 systemd[1]: zabbix-server.service: control process exited, code=e
Apr 20 21:37:32 rhel7 systemd[1]: Failed to start Zabbix Server.
Apr 20 21:37:32 rhel7 systemd[1]: Unit zabbix-server.service entered failed state.
Apr 20 21:37:32 rhel7 systemd[1]: zabbix-server.service failed.
Apr 20 21:37:42 rhel7 systemd[1]: zabbix-server.service holdoff time over, scheduling r
Apr 20 21:37:42 rhel7 systemd[1]: Starting Zabbix Server...
Apr 20 21:37:42 rhel7 zabbix_server[3140]: /usr/sbin/zabbix_server: error while loading
Apr 20 21:37:42 rhel7 systemd[1]: zabbix-server.service: control process exited, code=e
Apr 20 21:37:42 rhel7 systemd[1]: Failed to start Zabbix Server.
Apr 20 21:37:42 rhel7 systemd[1]: Unit zabbix-server.service entered failed state.
Apr 20 21:37:42 rhel7 systemd[1]: zabbix-server.service failed.
Apr 20 21:37:52 rhel7 systemd[1]: zabbix-server.service holdoff time over, scheduling r
Apr 20 21:37:52 rhel7 systemd[1]: Starting Zabbix Server...
Apr 20 21:37:52 rhel7 zabbix_server[3155]: /usr/sbin/zabbix_server: error while loading
Apr 20 21:37:52 rhel7 systemd[1]: zabbix-server.service: control process exited, code=e
Apr 20 21:37:52 rhel7 systemd[1]: Failed to start Zabbix Server.
Apr 20 21:37:52 rhel7 systemd[1]: Unit zabbix-server.service entered failed state.
Apr 20 21:37:52 rhel7 systemd[1]: zabbix-server.service failed.
Apr 20 21:38:03 rhel7 systemd[1]: zabbix-server.service holdoff time over, scheduling r
Apr 20 21:38:03 rhel7 systemd[1]: Starting Zabbix Server...
Apr 20 21:38:03 rhel7 zabbix_server[3157]: /usr/sbin/zabbix_server: error while loading
Apr 20 21:38:03 rhel7 systemd[1]: zabbix-server.service: control process exited, code=e
Apr 20 21:38:03 rhel7 systemd[1]: Failed to start Zabbix Server.
Apr 20 21:38:03 rhel7 systemd[1]: Unit zabbix-server.service entered failed state.
Apr 20 21:38:03 rhel7 systemd[1]: zabbix-server.service failed.
Apr 20 21:38:13 rhel7 systemd[1]: zabbix-server.service holdoff time over, scheduling r
Apr 20 21:38:13 rhel7 systemd[1]: Starting Zabbix Server...
Apr 20 21:38:13 rhel7 zabbix_server[3159]: /usr/sbin/zabbix_server: error while loading
Apr 20 21:38:13 rhel7 systemd[1]: zabbix-server.service: control process exited, code=e
Apr 20 21:38:13 rhel7 systemd[1]: Failed to start Zabbix Server.
Apr 20 21:38:13 rhel7 systemd[1]: Unit zabbix-server.service entered failed state.
Apr 20 21:38:13 rhel7 systemd[1]: zabbix-server.service failed.

[jusavard]

Hi,
your Log file should be stated in your conf file :

[root@RHEL7 ~]# grep "LogFile=" /etc/zabbix/zabbix_server.conf

About journalctl, only post last 50 lines. Should be enough.

[root@RHEL7 ~]# journalctl -u httpd | tail -50

[wojtekk]

try to execute

/usr/sbin/zabbix_server -V

In your comment i can see partial errors

"Apr 20 21:37:42 rhel7 zabbix_server[3140]: /usr/sbin/zabbix_server: error while loading"

[nazmul81]

Dear Jusavard,

please check.....

journalctl -u httpd | tail -50
-- Logs begin at Thu 2016-04-21 16:44:30 BDT, end at Thu 2016-04-21 22:44:51 BDT. --
Apr 21 22:37:52 rhel7 systemd[1]: Starting The Apache HTTP Server...
Apr 21 22:37:58 rhel7 httpd[1351]: AH00557: httpd: apr_sockaddr_info_get() failed for rhel7
Apr 21 22:37:58 rhel7 httpd[1351]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
Apr 21 22:37:59 rhel7 systemd[1]: Started The Apache HTTP Server.


[root@rhel7 ~]# grep "Logfile=" /etc/zabbix/zabbix_server.conf
[root@rhel7 ~]#


here is my conf file output for your info...


# This is a configuration file for Zabbix server daemon
# To get more information about Zabbix, visit http://www.zabbix.com

############ GENERAL PARAMETERS #################

### Option: ListenPort
# Listen port for trapper.
#
# Mandatory: no
# Range: 1024-32767
# Default:
# ListenPort=10051

### Option: SourceIP
# Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=

### Option: LogType
# Specifies where log messages are written to:
# system - syslog
# file - file specified with LogFile parameter
# console - standard output
#
# Mandatory: no
# Default:
# LogType=file

### Option: LogFile
# Log file name for LogType 'file' parameter.
#
# Mandatory: no
# Default:
# LogFile=

LogFile=/var/log/zabbix/zabbix_server.log

### Option: LogFileSize
# Maximum size of log file in MB.
# 0 - disable automatic log rotation.
#
# Mandatory: no
# Range: 0-1024
# Default:
# LogFileSize=1

LogFileSize=0

[jusavard]

Hi,
According to your configuration file you log file should be there :

LogFile=/var/log/zabbix/zabbix_server.log

Please post the content of this file. if it's too big please post only the last 100 lines.

If possible as wojtekk said please try to execute "/usr/sbin/zabbix_server -V" and post the result.

[nazmul81]

Hi Jusavard and woj,

i am not able to find the log file in my server:

here is the ouput:

[root@rhel7 ~]# cd /var/log/zabbix/
[root@rhel7 zabbix]# ls
[root@rhel7 zabbix]#

the output of "/usr/sbin/zabbix_server -V" is :

[root@rhel7 ~]# /usr/sbin/zabbix_server -V
/usr/sbin/zabbix_server: error while loading shared libraries: libiksemel.so.3: cannot open shared object file: No such file or directory


sorry to bother you a lot.......but i m grateful that you people are responding me...