Hosts not getting data through snmp (host icon is not red/green)

[zabbixfk]

Hello,

I have got into a strange situation, where added hosts ( monitored by proxy and through snmp) aren't getting any data.
I checked in Configuration-> hosts , hosts icons (for snmp) is neither green not red.
But i am able to get data through snmpwalk/snmpget from the device.
I did enabled Debuglevel to 4 ( max) but since lot of devices are added to server, i am not able to figure out much in the logs.
Any pointer are greatly helpful.

Thanks

[aib]

Please, keep in mind, that proxy server getting configuration data after some period of time, described in zabbix_proxy.conf as ConfigFrequency

### Option: ConfigFrequency
#       How often proxy retrieves configuration data from Zabbix Server in seconds.
#       For a proxy in the passive mode this parameter will be ignored.
#
# Mandatory: no
# Range: 1-3600*24*7
# Default:
ConfigFrequency=60

So, when you configure host, the configuration data of Host and Items has to wait until the next synchronization time.
Only after that proxy will start to collect data from device.
Then, after one or two intervals of checks, data will be sent back to the server.

If you want to check data quickly - switch to server (by removing flag Monitored by Proxy), check data, and then return Host back to Monitoring by proxy.

PS Also, zabbix server and zabbix proxy can have a different Timeout= value. Keep in mind, that SNMP request take a long time to answer (sometimes) and use something like Timeout=30

[zabbixfk]

Hosts not getting data through snmp (host icon is not red/green)

Thanks for the reply.

I just made sure that, all the settings are correct.
1). Timeout=30
2). ConfigFrequency=60
3). I am able to do snmpwalk from my proxy machine to the router which is sending snmp traps.
4). Changed snmp from proxy to master, still no data is being received by zabbix

I am not sure how to debug this issue.
I have added lot of devices till now using snmp v2, all of them are working other than devices which are added recently.
Attached is an template used to add devices.

Any pointers are greatly helpful.

Thanks

[zabbixfk]

Hosts not getting data through snmp - zabbix_proxy

Hi,

Any pointers?
I am able to get the data if i configure no_proxy on to the added host(router), but adding via proxy results no data fetch. Below are my zabbix_proxy settings.

Server=master-zbx-server.tinternal.com
Hostname=Zabbix-proxy
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=10
DebugLevel=4
PidFile=/var/run/zabbix/zabbix_proxy.pid
DBName=zabbix
DBUser=zabbix
DBPassword=zabbix
ConfigFrequency=600
DataSenderFrequency=5
StartPollers=65
StartPollersUnreachable=65
StartTrappers=55
StartPingers=65
StartSNMPTrapper=1
ListenIP=192.168.1.67
CacheSize=1G
StartDBSyncers=25
HistoryCacheSize=1G
HistoryTextCacheSize=768M
Timeout=30
UnreachablePeriod=60
FpingLocation=/usr/local/sbin/fping
LogSlowQueries=300000

Whatever the added device till now works fine, but addition of new devices resulting in blank data.

RAM : 32G, Free: 17G, , OS: Centos 6.3 ( on VM), CPU : 8 Core Intel Xenon E7 - 2820

Do i need to make any additional settings on proxy server for SNMP data?
Any pointers are greatly helpful.

Thanks

[aib]

It's hard to say without checking log-files.
It can be a network issue - Zabbix Server can have the connection to SNMP device and Zabbix Proxy cannot.
It can be a security issue - only Zabbix Server can be allowed / configured to access to SNMP device.
It can be a time problem - may be you didn't wait long enough to see the data. Because the data in "Latest Data" tab will appear only after "ConfigFrequency + Data Interval" period of time.

[tchjts1]

Did you compile your proxy server with--net-snmp ?

[zabbixfk]

Hosts not getting data through snmp - zabbix_proxy

Thank you all for the reply.

It can be a network issue - Zabbix Server can have the connection to SNMP device and Zabbix Proxy cannot.

- Nope. I tested connection ( ping/traceroute/snmpwalk) from both zabbix server and from zabbix proxy, i am able to do snmpwalk from both the machines to the SNMP enabled router - however, data is not getting shown in zabbix UI ( nothing being added to database.

It can be a security issue - only Zabbix Server can be allowed / configured to access to SNMP device.

- Again this is not the case as tested with both zabbix server and from proxy.

It can be a time problem - may be you didn't wait long enough to see the data. Because the data in "Latest Data" tab will appear only after "ConfigFrequency + Data Interval" period of time.

- I waited more than a week ( ConfigFrequency=300s + Data Interval=300s = 10min max) - still no luck

Did you compile your proxy server with--net-snmp ?

- Yes, proxy is compiled with SNMP. I am able to see the data getting pulled for already added devices ( close to a month back).
The problem is newly added devices aren't getting data.
- One more point i observed today is , Under Administration->Queue , i see that Proxy server has more than 1K items to be updated for more than 10 mins ( last column in red).

- There is no load on zabbix proxy server, its a VM with more than 65% Ram free, CPU load is negligible ( 0.9 out of 8 core machine ), disk space is fine.
- Now i am sensing, some issue from the proxy side.
- Do i need to change any settings for snmp data pull from zabbix proxy end?
- Close to 300 devices are sending data to proxy is this is big number? SHould i add more number of proxies?

Any pointers are greatly helpful.

Thanks

[aib]

Could you check LOG file on your router?
Maybe it has some error messages which can help to understand your issue.

Did you try to start snmpwalk from zabbix user? Maybe there is some sudo restrictions for snmp request (just a guess)

The last chance is to turn Debug on proxy server, stop all devices exclude only freshly added one and restart proxy for debugging.
Then check the big log file to collect more information about snmp requests and replies.

[tchjts1]

- Close to 300 devices are sending data to proxy is this is big number? SHould i add more number of proxies?

Thanks

Can you take a look at this post https://www.zabbix.com/forum/showthread.php?t=47781 and compare how well your setup is running?

[zabbixfk]

Hosts not getting data through snmp - zabbix_proxy

Thank you all for the reply.

Yes I am able to do snmpwalk from the zabbix proxy server to the device.
Initially i suspected the issue may be from router end, then i added diff divices ( make/location) to zabbix via proxy , still no use.
It just doesn't show data, surprisingly, snmp icon is niether red nor green ( its just grey - the one which shows on Configuration->Hosts ).
Looks like zabbix proxy is a bottleneck here.
As of now i am not in a position to disable all the added device and keep only new one, so checking the logs will be an issue.
I did tried tailing the logs and grepping for newly added device for more than an hour ( opened multiple terminals, did tail -f /var/log/zabbix/zabbix_proxy.log | grep 'router1_ip .... for about 10 diff ip's) just to see if anything is visible in logs - BLANK, just nothing. So looks like, zabbix_proxy is not even trying to look into these newly added devices.

From the zabbix_server performance side, below are the screenshots.
Zabbix Data Gathering Process : Please check this link
Zabbix Internal Process : Please check this link

So you are sure that i don't need to do any snmp based setting in zabbix_proxy?

Any pointers are greatly helpful.

Thanks.

[zabbixei]

Hi Everyone,

i never used proxy in zabbix but if you are trying to monitor your switches thru snmp so make sure you have the right templates.

if you need working templates for cisco switches let me know.

i have been trying to make calculated items map but i was not able. i did have look to zabbix documentation but i did not understand those expressions. i know it is not the right place to ask for help but maybe you guys know about it. can someone help me with it and tell me how can i monitor each single port with calculated items in maps?


thanks!

John

[zabbixfk]

Hosts not getting data through snmp - zabbix_proxy

@ zabbixei,

Hi, I am not using built in templates, what i have done is, added required port info as SNMP item.
Since i don't need to monitor all the port/data, i only check for

a). CPU :

 
Key: enterprises.2636.3.1.13.1.8.9.1.0.0
OID: .1.3.6.1.4.1.2636.3.1.13.1.8.9.1.0.0

b). Memory Utilisation:

Key: enterprises.2636.3.1.13.1.11.9.1.0.0
OID: .1.3.6.1.4.1.2636.3.1.13.1.11.9.1.0.0

c). Traffic Details:

Input Traffic Key: ifInOctets[INTERFACE_NUM]
Input Traffic OID: .1.3.6.1.2.1.2.2.1.10.INTERFACE_NUM
Output Traffic Key: ifOutOctets[INTERFACE_NUM]
Output Traffic OID: .1.3.6.1.2.1.2.2.1.16.INTERFACE_NUM

d). Operational Status:

Operational Status Key: ifOperStatus[INTERFACE_NUM]
Operational Status OID: .1.3.6.1.2.1.2.2.1.8.INTERFACE_NUM

e). Ping Status:

 Type: Simple check
Key: icmpping[,5,,,]

Where INTERFACE_NUM obtained from the perticular interface, you can get it through snmpwalk to your router. For example, you have a port with Name, port1, on snmpwalk if you sniff for ifName, you will get data something similar to this

IF-MIB::ifName.2 = STRING: port1

So "2" is your INTERFACE_NUM.

Then set the triggers accordingly. You don't necessarily need zabbix_proxy for these kind of monitoring, you can use zabbix_server only, i used proxy for load balancing.

Let me know in case of any more data required.

Thanks

[zabbixei]

@ zabbixei,

Hi, I am not using built in templates, what i have done is, added required port info as SNMP item.
Since i don't need to monitor all the port/data, i only check for

a). CPU :

 
Key: enterprises.2636.3.1.13.1.8.9.1.0.0
OID: .1.3.6.1.4.1.2636.3.1.13.1.8.9.1.0.0

b). Memory Utilisation:

Key: enterprises.2636.3.1.13.1.11.9.1.0.0
OID: .1.3.6.1.4.1.2636.3.1.13.1.11.9.1.0.0

c). Traffic Details:

Input Traffic Key: ifInOctets[INTERFACE_NUM]
Input Traffic OID: .1.3.6.1.2.1.2.2.1.10.INTERFACE_NUM
Output Traffic Key: ifOutOctets[INTERFACE_NUM]
Output Traffic OID: .1.3.6.1.2.1.2.2.1.16.INTERFACE_NUM

d). Operational Status:

Operational Status Key: ifOperStatus[INTERFACE_NUM]
Operational Status OID: .1.3.6.1.2.1.2.2.1.8.INTERFACE_NUM

e). Ping Status:

 Type: Simple check
Key: icmpping[,5,,,]

Where INTERFACE_NUM obtained from the perticular interface, you can get it through snmpwalk to your router. For example, you have a port with Name, port1, on snmpwalk if you sniff for ifName, you will get data something similar to this

IF-MIB::ifName.2 = STRING: port1

So "2" is your INTERFACE_NUM.

Then set the triggers accordingly. You don't necessarily need zabbix_proxy for these kind of monitoring, you can use zabbix_server only, i used proxy for load balancing.

Let me know in case of any more data required.

Thanks

well everything looks find to me what you have done but i will send you my template that i have modified have look to it and compare it with what you have done probably there is some missing or otherwise it should work fine.

[tchjts1]

From the zabbix_server performance side, below are the screenshots.
Zabbix Data Gathering Process : Please check this link
Zabbix Internal Process : Please check this link

So you are sure that i don't need to do any snmp based setting in zabbix_proxy?

Any pointers are greatly helpful.

Thanks.

Your performance metrics look fine. Although your unreachable pollers are running at a steady 20% or so. Do you have some hosts that are always unreachable?

One other thing - Are you certain these new devices are not using SNMPv3?
If they are, that requires different settings to be input into Zabbix frontend.

[zabbixfk]

Hosts not getting data through snmp - zabbix_proxy

Thanks for the reply.

I am sure that all added devices are using snmp v3, the question is, i am able to do snmpwalk but through zabbix_proxy, i am not able to get data.

Can anybody share internal working for snmp based data fetch through zabbix_proxy?

Any more settings need to be changed on zabbix_proxy ( as the system is not on load, ram free, cpu load, etc are all okay).


Thanks.