Unusable SNMPv3 performance

[marcherren]

Hi,

I was wondering if anyone uses Zabbix with "a lot" of SNMPv3 items.

In my setup I have ruffly 150 network devices and I monitor only some key items which is around 30 items per switch.

Number of hosts (monitored/not monitored/templates) 206 152 / 1 / 53
Number of items (monitored/disabled/not supported) 4456 4452 / 1 / 3
Number of triggers (enabled/disabled)[problem/unknown/ok] 2147 2147 / 0 [1 / 0 / 2146]
Number of users (online) 2 1
Required server performance, new values per second 44.89 -

Using SNMPv2 everything works perfect, the update queue is empty and no errors in the log.

As soons as I switch all items to SNMPv3 after 30-60 minutes the queue starts to fill up and the server log drops a lot of error messages:

7822:20120522:092136.629 SNMP item [pwstatus.["1"]] on host [FAEGFE06] failed: first network error, wait for 15 seconds
7833:20120522:092139.802 NODE 1: Received history_uint from node 2 for node 2 datalen 2083
7812:20120522:092140.869 SNMP item [fanstatus.["1"]] on host [CBEGFE03] failed: first network error, wait for 15 seconds
7831:20120522:092141.129 resuming SNMP checks on host [ITCOFE13]: connection restored
7831:20120522:092141.160 resuming SNMP checks on host [FAEGFE11]: connection restored
7830:20120522:092145.039 resuming SNMP checks on host [SDEGFE16]: connection restored
7830:20120522:092148.078 resuming SNMP checks on host [FAEGFE01]: connection restored
7814:20120522:092149.361 SNMP item [cpuUtil1Min] on host [STGEGFE02] failed: first network error, wait for 15 seconds
7833:20120522:092149.835 NODE 1: Received history_uint from node 2 for node 2 datalen 1927
7827:20120522:092150.400 SNMP item [.1.3.6.1.2.1.2.2.1.10.1] on host [TKCOFE13] failed: first network error, wait for 15 seconds
7830:20120522:092151.114 resuming SNMP checks on host [FAEGFE06]: connection restored

The server itself has no load at all and I've stress tested one of the switches an could succesfully retrieve 1000 SNMPv3 values within seconds.

To my zabbix does not perform at all with SNMPv3 so I was wondering if anyone is using SNMPv3 in a large scale?

BTW I've also opened ZBX-5028 for this.

[ericgearhart]

Are any of your SNMP username/passwords incorrect?

See the bug report I just filed here... my symptoms are similar to your, but my problem was incorrect SNMPv3 username/password

https://support.zabbix.com/browse/ZBX-5414

[marcherren]

Hi,

I've checked all of my snmp settings and none is wrong and I can access all of them trough snmpwalk directly from my zabbix server.

Also when I get the Network error message, the host itself is reachable

22658:20120808:083919.772 SNMP item [ifOutOctets[GigabitEthernet3/14]] on host [ITCOFE02] failed: first network error, wait for 15 seconds

snmpwalk -v3 -u xxx -Axxx -Xxxx -l authPriv -O n 10.0.248.14 SNMPv2-MIB::sysName.0
.1.3.6.1.2.1.1.5.0 = STRING: ITCOFE02

How many snmpv3 items you observe in total?

[humez]

Hi guys,
Did you resolve this problem? I have the same symptoms as you.
Kind regards,

[ericgearhart]

Have you tested with Zabbix 2.2? I haven't tested the SNMP performance on 2.2 yet but I was closely watching the change logs and various bugs filed against previous versions, and 2.2 has a bunch of SNMP related performance fixes.

Try it out! I'm planning on moving to 2.2 soon also, I'll try to post my experiences here when I do

[neo32]

Similar Trouble 3.2.6

Have you tested with Zabbix 2.2? I haven't tested the SNMP performance on 2.2 yet but I was closely watching the change logs and various bugs filed against previous versions, and 2.2 has a bunch of SNMP related performance fixes.

Try it out! I'm planning on moving to 2.2 soon also, I'll try to post my experiences here when I do

Hello! I have a similar problem. My ZBX version is 3.2.6.
SomeOne know, how resolve this trouble ?

[kloczek]

The server itself has no load at all and I've stress tested one of the switches an could succesfully retrieve 1000 SNMPv3 values within seconds.

To my zabbix does not perform at all with SNMPv3 so I was wondering if anyone is using SNMPv3 in a large scale?

BTW I've also opened ZBX-5028 for this.

Those timeouts have nothing to do with zabbix and you can reproduce them by sequentially executing snmpwalk/snmpget commands.
They are related to weak BMCs on which is running snmpd on monitored devices and/or some locking issues on obtaining data by snmpd on monitored devices.