Ad Widget

**prism** · 14-01-2010, 04:19

Just wanted to provide some additional information.

/etc/zabbix/zabbix_agentd.conf contains:

Code:

Hostname=Zabbix Server
Server=127.0.0.1

/etc/zabbix/zabbix_server.conf contains:

Code:

DBName=zabbix
DBUser=zabbix
DBPassword=secret
DBSocket=/var/lib/mysql/mysql.sock
FpingLocation=/usr/sbin/fping

/etc/services contains:

Code:

zabbix-agent    10050/tcp                       #Zabbix Agent
zabbix-agent    10050/udp                       #Zabbix Agent
zabbix-trapper  10051/tcp                       #Zabbix Trapper
zabbix-trapper  10051/udp                       #Zabbix Trapper

/etc/init.d/zabbix_agentd contains:

Code:

# base zabbix dir
# BASEDIR=/opt/zabbix
BASEDIR=/usr/local
# pid file (as of 1.0 beta 10)
PIDFILE=/var/tmp/zabbix_agentd.pid
# binary file
ZABBIX_AGENTD=$BASEDIR/sbin/zabbix_agentd

/etc/init.d/zabbix_server contains:

Code:

# base zabbix dir
#BASEDIR=/opt/zabbix
BASEDIR=/usr/local
# PID file
PIDFILE=/var/tmp/zabbix_server.pid
# binary file
ZABBIX_SUCKERD=$BASEDIR/sbin/zabbix_server

/tmp/zabbix_agentd.log contains:

Code:

10154:20100113:174037.408 zabbix_agentd started. Zabbix 1.8 (revision 8565).
 10155:20100113:174037.410 zabbix_agentd collector started
 10156:20100113:174037.413 zabbix_agentd listener started
 10157:20100113:174037.413 zabbix_agentd listener started
 10158:20100113:174037.415 zabbix_agentd listener started
 10159:20100113:174037.415 zabbix_agentd active check started [127.0.0.1:10051]
 10154:20100113:174933.180 One child process died (PID:10155). Exiting ...
 10154:20100113:174933.180 Can't find shared memory for collector. [No such file or directory]
 11223:20100113:174953.960 zabbix_agentd started. Zabbix 1.8 (revision 8565).
 11224:20100113:174953.961 zabbix_agentd collector started
 11225:20100113:174953.964 zabbix_agentd listener started
 11226:20100113:174953.964 zabbix_agentd listener started
 11227:20100113:174953.965 zabbix_agentd listener started
 11228:20100113:174953.965 zabbix_agentd active check started [127.0.0.1:10051]
 11223:20100113:180727.424 One child process died (PID:11224). Exiting ...
 11223:20100113:180729.426 Zabbix Agent stopped. Zabbix 1.8 (revision 8565).
 13086:20100113:180741.572 zabbix_agentd started. Zabbix 1.8 (revision 8565).
 13087:20100113:180741.573 zabbix_agentd collector started
 13088:20100113:180741.611 zabbix_agentd listener started
 13089:20100113:180741.611 zabbix_agentd listener started
 13091:20100113:180741.612 zabbix_agentd active check started [127.0.0.1:10051]
 13090:20100113:180741.613 zabbix_agentd listener started
 13086:20100113:181046.544 One child process died (PID:13087). Exiting ...
 13086:20100113:181048.546 Zabbix Agent stopped. Zabbix 1.8 (revision 8565).
 13571:20100113:181059.199 zabbix_agentd started. Zabbix 1.8 (revision 8565).
 13572:20100113:181059.200 zabbix_agentd collector started
 13573:20100113:181059.233 zabbix_agentd listener started
 13574:20100113:181059.233 zabbix_agentd listener started
 13575:20100113:181059.234 zabbix_agentd listener started
 13576:20100113:181059.234 zabbix_agentd active check started [127.0.0.1:10051]

/tmp/zabbix_server.log contains:

Code:

13114:20100113:181053.308 One child process died (PID:13137). Exiting ...
 13114:20100113:181055.364 Syncing history data...
 13114:20100113:181055.364 Syncing trends data...
 13114:20100113:181055.381 Syncing trends data...done.
 13114:20100113:181055.382 Zabbix Server stopped. Zabbix 1.8 (revision 8565).
 13586:20100113:181103.759 Starting zabbix_server. Zabbix 1.8 (revision 8565).
 13586:20100113:181103.760 **** Enabled features ****
 13586:20100113:181103.760 SNMP monitoring:       YES
 13586:20100113:181103.760 IPMI monitoring:        NO
 13586:20100113:181103.760 WEB monitoring:         NO
 13586:20100113:181103.760 Jabber notifications:  YES
 13586:20100113:181103.760 ODBC:                   NO
 13586:20100113:181103.760 SSH2 support:           NO
 13586:20100113:181103.760 IPv6 support:           NO
 13586:20100113:181103.760 **************************
 13588:20100113:181103.836 server #1 started [DB Cache]
 13590:20100113:181103.885 server #2 started [Poller. SNMP:YES]
 13591:20100113:181103.922 server #3 started [Poller. SNMP:YES]
 13595:20100113:181103.962 server #5 started [Poller. SNMP:YES]
 13598:20100113:181103.982 server #7 started [Trapper]
 13600:20100113:181103.983 server #8 started [Trapper]
 13602:20100113:181103.984 server #9 started [Trapper]
 13592:20100113:181104.021 server #4 started [Poller. SNMP:YES]
 13597:20100113:181104.044 server #6 started [Poller. SNMP:YES]
 13604:20100113:181104.045 server #10 started [Trapper]
 13608:20100113:181104.046 server #11 started [Trapper]
 13610:20100113:181104.047 server #12 started [ICMP pinger]
 13612:20100113:181104.048 server #13 started [Alerter]
 13614:20100113:181104.050 server #14 started [Housekeeper]
 13614:20100113:181104.050 Executing housekeeper
 13616:20100113:181104.054 server #15 started [Timer]
 13618:20100113:181104.100 server #16 started [Poller for unreachable hosts. SNMP:YES]
 13619:20100113:181104.102 server #17 started [Node watcher. Node ID:0]
 13622:20100113:181104.142 server #18 started [Discoverer. SNMP:YES]
 13624:20100113:181104.144 server #19 started [DB Syncer]
 13626:20100113:181104.147 server #20 started [Escalator]
 13586:20100113:181104.148 server #0 started [Watchdog]
 13614:20100113:181110.222 Deleted 0 records from history and trends
 13597:20100113:181719.648 Item [Zabbix Server:kernel.maxproc] error: Not supported by Zabbix Agent
 13591:20100113:181721.605 Item [Zabbix Server:net.if.in[eth1,bytes]] error: Not supported by Zabbix Agent
 13597:20100113:181724.667 Item [Zabbix Server:net.if.out[eth1,bytes]] error: Not supported by Zabbix Agent
 13592:20100113:181742.194 Item [Zabbix Server:sensor[temp1]] error: Not supported by Zabbix Agent
 13595:20100113:181743.754 Item [Zabbix Server:sensor[temp2]] error: Not supported by Zabbix Agent
 13597:20100113:181744.065 Item [Zabbix Server:sensor[temp3]] error: Not supported by Zabbix Agent
 13595:20100113:181803.971 Item [Zabbix Server:vfs.file.cksum[/etc/inetd.conf]] error: Not supported by Zabbix Agent
 13595:20100113:181808.975 Item [Zabbix Server:vfs.file.cksum[/vmlinuz]] error: Not supported by Zabbix Agent
 13597:20100113:182719.024 Item [Zabbix Server:kernel.maxproc] error: Not supported by Zabbix Agent
 13591:20100113:182721.027 Item [Zabbix Server:net.if.in[eth1,bytes]] error: Not supported by Zabbix Agent
 13597:20100113:182724.815 Item [Zabbix Server:net.if.out[eth1,bytes]] error: Not supported by Zabbix Agent
 13592:20100113:182742.711 Item [Zabbix Server:sensor[temp1]] error: Not supported by Zabbix Agent
 13595:20100113:182743.721 Item [Zabbix Server:sensor[temp2]] error: Not supported by Zabbix Agent
 13597:20100113:182744.907 Item [Zabbix Server:sensor[temp3]] error: Not supported by Zabbix Agent
 13595:20100113:182803.753 Item [Zabbix Server:vfs.file.cksum[/etc/inetd.conf]] error: Not supported by Zabbix Agent
 13595:20100113:182808.756 Item [Zabbix Server:vfs.file.cksum[/vmlinuz]] error: Not supported by Zabbix Agent
 13597:20100113:183719.245 Item [Zabbix Server:kernel.maxproc] error: Not supported by Zabbix Agent
 13591:20100113:183721.934 Item [Zabbix Server:net.if.in[eth1,bytes]] error: Not supported by Zabbix Agent
 13597:20100113:183724.941 Item [Zabbix Server:net.if.out[eth1,bytes]] error: Not supported by Zabbix Agent
 13592:20100113:183742.240 Item [Zabbix Server:sensor[temp1]] error: Not supported by Zabbix Agent
 13595:20100113:183743.966 Item [Zabbix Server:sensor[temp2]] error: Not supported by Zabbix Agent
 13597:20100113:183744.981 Item [Zabbix Server:sensor[temp3]] error: Not supported by Zabbix Agent
 13595:20100113:183803.022 Item [Zabbix Server:vfs.file.cksum[/etc/inetd.conf]] error: Not supported by Zabbix Agent
 13595:20100113:183808.046 Item [Zabbix Server:vfs.file.cksum[/vmlinuz]] error: Not supported by Zabbix Agent

All graphs within Monitoring > Graphs are displayed fine. I just don't understand why the services are showing up as being down in the Zabbix Dashboard when they are in fact running

**Firm** · 14-01-2010, 09:58

Go to 'Monitoring->Latest Data' and check if 'SSH server is running' value equals to '1'.

P.S. And there's one strange error an zabbix_agentd.log: '10154:20100113:174933.180 Can't find shared memory for collector. [No such file or directory]'

**prism** · 14-01-2010, 19:50

Hi Firm, thank you for responding

Originally posted by Firm

...P.S. And there's one strange error an zabbix_agentd.log: '10154:20100113:174933.180 Can't find shared memory for collector. [No such file or directory]'

I was able to get rid of that error by following the instructions at: http://ilostmynotes.blogspot.com/200...ring-tips.html

Basically, you execute this command to delete shared memory objects:

Code:

[root@zabbix_server]# ipcrm
[root@zabbix_server]# /sbin/service zabbix_agentd start
/etc/init.d/zabbix_agentd start: zabbix_agentd started
[root@zabbix_server]# /sbin/service zabbix_server start
/etc/init.d/zabbix_server start: zabbix_server started

The error is now gone from zabbix_agentd.log:

Code:

13571:20100114:100957.946 One child process died (PID:13572). Exiting ...
13571:20100114:100959.948 Zabbix Agent stopped. Zabbix 1.8 (revision 8565).
18384:20100114:101241.698 zabbix_agentd started. Zabbix 1.8 (revision 8565).
18385:20100114:101241.699 zabbix_agentd collector started
18386:20100114:101241.720 zabbix_agentd listener started
18387:20100114:101241.720 zabbix_agentd listener started
18388:20100114:101241.720 zabbix_agentd listener started
18389:20100114:101241.721 zabbix_agentd active check started [127.0.0.1:10051]

Originally posted by Firm

Go to 'Monitoring->Latest Data' and check if 'SSH server is running' value equals to '1'.

If I go to Monitoring > Latest Data, I get:

Code:

[b]SSH server is running 	14 Jan 2010 10:32:32 	Down (0) 	- 	Graph[/b]

I'm still not sure what's going on

**Firm** · 18-01-2010, 12:04

Check if this works for you:

$ id
zabbix
$ telnet 127.0.0.1 22
$ nc 127.0.0.1 22
SSH-2.0-OpenSSH_4.3
<Ctrl-C pressed>
$ zabbix_get -s127.0.0.1 -k'net.tcp.service[ssh]'
1
$

**prism** · 19-01-2010, 19:53

Hi Firm, thanks again for responding.

Originally posted by Firm

Check if this works for you:

$ id
zabbix
$ telnet 127.0.0.1 22
$ nc 127.0.0.1 22
SSH-2.0-OpenSSH_4.3
<Ctrl-C pressed>
$ zabbix_get -s127.0.0.1 -k'net.tcp.service[ssh]'
1
$

This is what I'm getting for the above commands:

Code:

[root@zabbix_server]# id zabbix
uid=43469(zabbix) gid=43470(zabbix) groups=43470(zabbix)

Code:

[root@zabbix_server]# telnet 127.0.0.1 22
Trying 127.0.0.1...
Connected to root@zabbix_server (127.0.0.1).
Escape character is '^]'.
SSH-2.0-OpenSSH_3.9p1
exit
Protocol mismatch.
Connection closed by foreign host.

Code:

[root@zabbix_server]# nc 127.0.0.1 22
SSH-2.0-OpenSSH_3.9p1
 punt!

Code:

[root@zabbix_server]# zabbix_get -s127.0.0.1 -k'net.tcp.service[ssh]'
0
[root@zabbix_server]#

Both telnet and nc are able to connect to ssh running on the machine but for some reason zabbix_get is giving me a 0 (e.g. service not running).

Could the fact that I'm also running Nagios and Cacti on the machine be a problem?

Here's the list of all services running on the machine:

Code:

[root@zabbix_server]# /sbin/service --status-all
NetworkManager is stopped
acpid (pid 15614) is running...
anacron is stopped
atd (pid 15807) is running...
auditd is stopped
Configured Mount Points:
------------------------

Active Mount Points:
--------------------
hcid is stopped
sdpd is stopped
hidd is stopped
capi not installed - No such file or directory (2)
cpuspeed is stopped
crond (pid 25503) is running...
cupsd (pid 5877) is running...
cups-config-daemon (pid 15840) is running...
dc_client is stopped
dc_server is stopped
Device not specified in /etc/sysconfig/diskdump
Usage: /etc/init.d/firstboot {start|stop}
gpm (pid 15749) is running...
hald (pid 15851) is running...
httpd (pid 14835 5906 5905 5904 5903 5901 5900 5899 5898) is running...
ibmsphalt is stopped
ipmi_msghandler module not loaded.
ipmi_si module not loaded.
ipmi_devintf module not loaded.
/dev/ipmi0 does not exist.
Firewall is stopped.
irattach is stopped
irqbalance is stopped
lisa is stopped
mdadm is stopped
mdmpd is stopped
dbus-daemon-1 (pid 27715 15827) is running...
/etc/init.d/microcode_ctl: reading microcode status is not yet supported
mysqld (pid 17793) is running...
nagios (pid 31415) is running...
Server address not specified in /etc/sysconfig/netdump
Configured NFS mountpoints:
/scratch
Active NFS mountpoints:
/scratch
netplugd is stopped
Configured devices:
lo eth0
Currently active devices:
lo eth0
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
rpc.statd (pid 15324) is running...
nscd is stopped
ntpd (pid 15710) is running...
afsd is stopped
Stopped
portmap (pid 15314) is running...
Process accounting is enabled.
rdisc is stopped
rdisc is stopped
rpc.idmapd (pid 15354) is running...
saslauthd is stopped
sendmail (pid 15738 15729) is running...
smartd (pid 15604) is running...
snmpd (pid 5922) is running...
snmptrapd is stopped
spamd is stopped
sshd (pid 15677) is running...
syslogd (pid 15289) is running...
klogd (pid 15293) is running...
CIM server is not running
Xvnc is stopped
wdaemon is stopped
winbindd is stopped
wpa_supplicant is stopped
xfs (pid 15788) is running...
xinetd (pid 17318) is running...
ypbind is stopped
Nightly yum update is disabled.
usage: /etc/init.d/zabbix_agentd (start|stop|restart|help)

start      - start zabbix_agentd
stop       - stop zabbix_agentd
restart    - restart zabbix_agentd if running by sending a SIGHUP or start if not running
help       - this screen

usage: /etc/init.d/zabbix_server (start|stop|restart|help)

start      - start zabbix_server
stop       - stop zabbix_server
restart    - restart zabbix_server if running by sending a SIGHUP or start if not running
help       - this screen

[root@zabbix_server]#

Notice that the above also reports that sshd is running. I'm still stumped

**prism** · 19-01-2010, 22:35

Well, I thought the problem was only with the Zabbix Server but I've added a new host for monitoring and I'm seeing the same results. Running services are still reported as down

Attached Files

**Firm** · 20-01-2010, 13:48

Try this:

$ strace -f zabbix_get -s127.0.0.1 -k'net.tcp.service[ssh]' 2>&1|grep connect
connect(3, {sa_family=AF_INET, sin_port=htons(10050), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
$

What is the printed line in your case?

P.S. Do you have selinux enabled?

**prism** · 21-01-2010, 19:39

Hi Firm. Sorry about the late reply.

Originally posted by Firm

Try this:

$ strace -f zabbix_get -s127.0.0.1 -k'net.tcp.service[ssh]' 2>&1|grep connect
connect(3, {sa_family=AF_INET, sin_port=htons(10050), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
$

What is the printed line in your case?

I'm getting the same output:

Code:

[root@zabbix_server]# strace -f zabbix_get -s127.0.0.1 -k'net.tcp.service[ssh]' 2>&1|grep connect
connect(3, {sa_family=AF_INET, sin_port=htons(10050), sin_addr=inet_addr("127.0.0.1")}, 16) = 0

Originally posted by Firm

P.S. Do you have selinux enabled?

No, it's definitely off:

Code:

[root@zabbix_server]# /usr/sbin/sestatus
SELinux status:         disabled

**prism** · 21-01-2010, 21:40

It's interesting to note that for some services, Zabbix does detect them to be running.

For instance, I can successfully check if SMTP and HTTP are running:

Code:

[root@zabbix_server]# zabbix_get -s127.0.0.1 -k'net.tcp.service[http]'
1
[root@zabbix_server]# zabbix_get -s127.0.0.1 -k'net.tcp.service[smtp]'
1

Going to Monitoring > Latest data confirms this fact (see screenshot).

Could this be an issue with Template_Linux or RHEL4 (the distro Zabbix is running on)?

I also noticed the following entry in /var/log/syslog which looks suspicious:

Code:

sshd[26174]: Bad protocol version identification '0' from ::ffff:127.0.0.1

In any case, Zabbix is still reporting SSH to be down even when it is running.

Attached Files

**prism** · 21-01-2010, 21:54

Some additional info about the services that should be reported as running:

SSH:

Code:

[root@zabbix_server]# cat /etc/services | grep ssh
ssh             22/tcp                          # SSH Remote Login Protocol
ssh             22/udp                          # SSH Remote Login Protocol
x11-ssh-offset  6010/tcp                        # SSH X11 forwarding offset

FTP:

Code:

[root@zabbix_server]# cat /etc/services | grep ftp
ftp-data        20/tcp
ftp-data        20/udp
# 21 is registered to ftp, but also used by fsp
ftp             21/tcp
ftp             21/udp          fsp fspd
tftp            69/tcp
tftp            69/udp
sftp            115/tcp
sftp            115/udp
tftp-mcast      1758/tcp
tftp-mcast      1758/udp
mtftp           1759/udp
venus-se        2431/udp                        # udp sftp side effect
codasrv-se      2433/udp                        # udp sftp side effectQ

SSH access:

Code:

[root@zabbix_server]# cat /etc/hosts.allow
#
# hosts.allow   This file describes the names of the hosts which are
#               allowed to use the local INET services, as decided
#               by the '/usr/sbin/tcpd' server.
#
sshd:        137.78. 137.79. 137.228. 128.149. 127.0.0.1 : banners /etc/banners
sshdfwd-X11: 137.78. 137.79. 137.228. 128.149. 127.0.0.1
portmap:     137.78. 137.79. 137.228. 128.149.
sendmail: ALL
snmpd: ALL
zabbix_agentd: ALL
zabbix_server: ALL
#ndo2db: ALL
[root@zabbix_server]#

**p_a_l_o** · 24-01-2010, 16:45

Hello,

I am trying to solve exactly same problem. Did you manage to find solution?

Thanks

Pavel

**prism** · 25-01-2010, 18:36

Originally posted by p_a_l_o

Hello,

I am trying to solve exactly same problem. Did you manage to find solution?

Thanks

Pavel

Unfortunately I have not

**chani** · 09-02-2010, 22:41

i have the same problem ....

**live2** · 04-03-2010, 19:42

i have the same problem with the Message "SSH server is down on {HOSTNAME}" on 5 Server all daemons are running.

If anybody find a solution?

Regards

Ad Widget

Services reported as 'down'

Services reported as 'down'

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment