selective snmp-v3 problem, I give up.

[cvsyazycuj]

Hello everyone.
Zabbix server 2.2.5 on debian 7.6 i386 from precompiled deb package.
I' m trying to configure snmp monitoring of my switches. Some switches that have snmp v3 and are configured to use it - work partially.
Here's my particular problem.
I have Cisco SG300-52 switch configured:

......
snmp-server server
snmp-server engineID local 800000090368bc0c7a27dd
snmp-server location mainOffice
snmp-server view ZABBIX iso included
snmp-server group ZABBIX v3 priv read ZABBIX
encrypted snmp-server user zabbix ZABBIX v3 auth sha ******encpasshere*** priv ******encpasshere***
........

I took care of engineID so it's unique in my administrative domain + clock is synced with the same ntp server as zabbix server.
I made a host on zabbix server and attached the following template to it:
sg300-52_zabbix_template.xml
I'm able to ping, snmpget, snmpwalk, snmpbulkget, snmpbulkwalk from zabbix server to switch and everything works flawlessly. Btw

root@zabbixserver01:~# dpkg -l snmp
............
ii  snmp                                             5.4.3~dfsg-2.8                i386                          SNMP (Simple Network Management Protocol) applications
.......

But when it comes to zabbix i get the following result:

6064:20140826:121638.108 SNMP agent item "ifOutDiscards.[86]" on host "sw-cisco-sg300-52-sob" failed: first network error, wait for 15 seconds
  6069:20140826:121654.127 resuming SNMP agent checks on host "sw-cisco-sg300-52-sob": connection restored
  6065:20140826:121707.804 SNMP agent item "ifOutDiscards.[80]" on host "sw-cisco-sg300-52-sob" failed: first network error, wait for 15 seconds
  6069:20140826:121722.295 resuming SNMP agent checks on host "sw-cisco-sg300-52-sob": connection restored
  6066:20140826:121737.216 SNMP agent item "ifOperStatus.[69]" on host "sw-cisco-sg300-52-sob" failed: first network error, wait for 15 seconds
  6069:20140826:121752.460 resuming SNMP agent checks on host "sw-cisco-sg300-52-sob": connection restored
  6065:20140826:121807.390 SNMP agent item "ifOperStatus.[64]" on host "sw-cisco-sg300-52-sob" failed: first network error, wait for 15 seconds
  6069:20140826:121822.643 resuming SNMP agent checks on host "sw-cisco-sg300-52-sob": connection restored
  6065:20140826:121837.632 SNMP agent item "ifOutDiscards.[93]" on host "sw-cisco-sg300-52-sob" failed: first network error, wait for 15 seconds
  6069:20140826:121852.833 resuming SNMP agent checks on host "sw-cisco-sg300-52-sob": connection restored

and graph

This graph shows that sometimes zabbix gets snmp values, but most of the time it does not. there are even some periods of several hours when zabbix can actually draw continious graphs that last for 1 or 2 hours, but then everything stops. I consider this behaviour as not a configuration error but problem with zabbix itself. I even had a problem when zabbix didnt send community string on snmpv1 swithes and i had to remove and insert host back to resolve this problem. I have some other snmpv3 switches that work "normally" but in some rare cases they get same errors as well.
This particular switch that i show you as example of problem worked for a week or so and then suddenly i started getting those errors.
What i tryed to do:
I changed engineID and recreated users - this helped for 15 minutes.
I removed host and inserted it back again - this helped for 5-10 minutes
I powered off all other snmpv3 swithes - nothing
I played with pollers quantity - nothing, only encreased cpuload
I dont think its server performance issue

root@zabbixserver01:~# uptime
 12:35:29 up 21:43,  1 user,  load average: 0.46, 0.29, 0.26

I dont know what to do.

[ingus.vilnis]

Hi,

Please check the value of Timeout= parameter in your zabbix_server.conf file. If you have it at the default 3 seconds, please increase it to somewhat higher, maybe even maximum 30 seconds, restart Zabbix server and check your SNMP graphs and logs again.

Best Regards,
Ingus

[cvsyazycuj]

Unfortunately setting Timeout paramter on 30 seconds didn't help, it was set on 10 seconds before.
What i noticed is - recreating engineID and users on switch fixes problem for several hours, but then, somehow, housekeeper process starts to burn cpu for 100% and i have to restart service. After restart i get back at situation described on first post.

[tchjts1]

Housekeeper utilizes 100% internal process when it runs, if you look at the internal metrics. It should not be using 100% CPU.

I have the default settings for housekeeper and it runs for maybe 10 minutes per hour. My Zabbix server/DB servers are on VM's and I monitor over 1,000 hosts. I am pointing out that I am not running on huge, high powered servers.

If you are actually consuming 100% CPU while housekeeper runs, there is something wrong there.

Would you describe your Zabbix infrastructure? Zabbix server and DB server on separate servers? How much memory do you have on your DB server?

Can you look at the last paragraph of this post https://www.zabbix.com/forum/showthread.php?t=41219 and share with us similar graphs for what you see in a 24 hour period?

[tchjts1]

Also, I just noticed this post. See if you have similarities. Maybe the patch will fix your problem? https://www.zabbix.com/forum/showthread.php?t=45164