Ad Widget

Collapse

Error doing snmp_open() or +40000 items in queue

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • sducoin
    Junior Member
    • Dec 2011
    • 9
    #1

    Error doing snmp_open() or +40000 items in queue

    Hi all,
    I've been using zabbix for a couple of years now, monitoring both servers with zabbix agents and snmp switches/appliances.
    All worked fine until the last two weeks. The queue length started to grow and no item was being updated. I finally found that setting the value SourceIP=zabbix_server_ip in zabbix_server.conf solved the problem... except for snmp monitoring. All the snmp items became unsupported - I found lots of "Error doing snmp_open()" in the zabbix_server log file.
    I suspect a network issue but I have no clue
    Any help will be appreciated

    My config :
    zabbix 1.8.9
    ubuntu maverick
    postgresql 8.4

    Cheers.
    Tags: None
    • sducoin
      Junior Member
      • Dec 2011
      • 9
      #2
      disabling option SourceIP in zabbix_server.conf doesnt solve the problem

      If I disable option SourceIP I got 40000+ item in queue
      If I enable it, all the snmp items become unsupported
      regards,

      Sebastian

        Comment

        • dima_dm
          Senior Member
          • Dec 2009
          • 2697
          #3
          disable option SourceIP
          Use
          zabbix[process,<type>,<mode>,<state>]
          zabbix[wcache,<cache>,<mode>]
          zabbix[rcache,<cache>,<mode>]
          to collect statistics and adjust the values wcache, rcache, process in zabbix_server.conf.
          http://www.zabbix.com/documentation/...l/config/items
          Last edited by dima_dm; 07-12-2011, 15:48. Reason: add zabbix_server.conf

            Comment

            • sducoin
              Junior Member
              • Dec 2011
              • 9
              #4
              disabled option SourceIP in zabbix_server.conf :

              here are the values in the attached files.
              We can see zabbix suddenly stopped updating data.
              Attached Files

                Comment

                • dima_dm
                  Senior Member
                  • Dec 2009
                  • 2697
                  #5
                  Is running process zabbix_server in this time?

                    Comment

                    • sducoin
                      Junior Member
                      • Dec 2011
                      • 9
                      #6
                      yes it is

                      I got about 100 zabbix_server processes and 28 startPoller in zabbix_server.conf

                      regards,

                      Sebastian

                        Comment

                        • dima_dm
                          Senior Member
                          • Dec 2009
                          • 2697
                          #7
                          Can you check database server performance and error log?

                            Comment

                            • sducoin
                              Junior Member
                              • Dec 2011
                              • 9
                              #8
                              postgresql + pgouncer

                              I use postgresql 8.4 and pgbouncer as a connections pooler and it works just fine :

                              got auto-vacuum and analyse logs for posgtresql :

                              Code:
                              2011-12-05 14:54:27 CET LOG:  automatic analyze of table "zabbix.public.history" system usage: CPU 0.34s/0.43u sec elapsed 30.13 sec
2011-12-05 15:54:55 CET LOG:  automatic analyze of table "zabbix.public.history_uint" system usage: CPU 0.44s/0.57u sec elapsed 56.97 sec
2011-12-06 01:24:43 CET LOG:  automatic analyze of table "zabbix.public.history" system usage: CPU 0.26s/0.53u sec elapsed 31.87 sec
2011-12-06 03:33:00 CET LOG:  automatic analyze of table "zabbix.public.history_uint" system usage: CPU 0.47s/0.58u sec elapsed 45.47 sec
2011-12-06 16:48:11 CET LOG:  automatic analyze of table "zabbix.public.history" system usage: CPU 0.38s/0.54u sec elapsed 36.00 sec
2011-12-06 17:59:27 CET LOG:  automatic analyze of table "zabbix.public.history_uint" system usage: CPU 0.45s/0.51u sec elapsed 50.07 sec
2011-12-07 19:57:39 CET LOG:  automatic analyze of table "zabbix.public.history" system usage: CPU 0.36s/0.49u sec elapsed 34.65 sec
2011-12-07 20:26:52 CET LOG:  automatic analyze of table "zabbix.public.history_uint" system usage: CPU 0.44s/0.49u sec elapsed 47.36 sec
                              and some connections stats for pgbouncer :

                              Code:
                              2011-12-08 13:19:28.387 1289 LOG Stats: 1384 req/s, in 86546 b/s, out 129232 b/s,query 875 us
[...]
2011-12-08 13:20:36.423 1289 LOG C-0xc1aff0: zabbix/[email protected]:36543 login attempt: db=zabbix user=zabbix
2011-12-08 13:20:36.439 1289 LOG C-0xc242e0: zabbix/[email protected]:36545 login attempt: db=zabbix user=zabbix
2011-12-08 13:20:36.453 1289 LOG C-0xbf9f80: zabbix/[email protected]:36547 login attempt: db=zabbix user=zabbix
2011-12-08 13:20:36.501 1289 LOG C-0xc26280: zabbix/[email protected]:36549 login attempt: db=zabbix user=zabbix
2011-12-08 13:20:37.421 1289 LOG C-0xc1aff0: zabbix/[email protected]:36543 closing because: client close request (age=0)
2011-12-08 13:20:37.440 1289 LOG C-0xc1e010: zabbix/[email protected]:36541 closing because: client close request (age=1)
2011-12-08 13:20:37.489 1289 LOG C-0xbf9f80: zabbix/[email protected]:36547 closing because: client close request (age=1)
2011-12-08 13:20:37.847 1289 LOG C-0xc242e0: zabbix/[email protected]:36545 closing because: client close request (age=1)
2011-12-08 13:20:38.020 1289 LOG C-0xc26280: zabbix/[email protected]:36549 closing because: client close request (age=1)
2011-12-08 13:20:43.270 1289 LOG C-0xc26280: zabbix/[email protected]:36562 login attempt: db=zabbix user=zabbix
2011-12-08 13:20:43.565 1289 LOG C-0xc26280: zabbix/[email protected]:36562 closing because: client close request (age=0)
2011-12-08 13:20:43.853 1289 LOG C-0xc26280: zabbix/[email protected]:36564 login attempt: db=zabbix user=zabbix
2011-12-08 13:20:43.949 1289 LOG C-0xc26280: zabbix/[email protected]:36564 closing because: client close request (age=0)
2011-12-08 13:20:50.934 1289 LOG C-0xc26280: zabbix/[email protected]:36571 login attempt: db=zabbix user=zabbix
                              Last edited by sducoin; 08-12-2011, 17:01.

                                Comment

                                • sducoin
                                  Junior Member
                                  • Dec 2011
                                  • 9
                                  #9
                                  internal postgresql statistics :

                                  I use postbix to monitor my postgresql server
                                  we can see values not being updated after setting the SourceIP value... except for postbix values - zabbix trapper is not impacted by this issue.

                                  regards

                                  Sebastian
                                  Attached Files

                                    Comment

                                    • dima_dm
                                      Senior Member
                                      • Dec 2009
                                      • 2697
                                      #10
                                      Check Timeout option.
                                      /etc/zabbix/zabbix_server.conf
                                      Code:
                                      ### Option: Timeout
#       Specifies how long we wait for agent, SNMP device or external check (in seconds).
#
# Mandatory: no
# Range: 1-30
# Default:
Timeout=30
                                      And zabbix_agentd.conf Timeout option.
                                      Last edited by dima_dm; 08-12-2011, 17:23.

                                        Comment

                                        • sducoin
                                          Junior Member
                                          • Dec 2011
                                          • 9
                                          #11
                                          already set

                                          here is my zabbix_server.conf :

                                          Code:
                                          StartPollers=28
StartPollersUnreachable=16
StartTrappers=16
StartPingers=12
StartDiscoverers=6
StartDBSyncers=16
#SourceIP=10.4.1.34
HousekeepingFrequency=22
MaxHousekeeperDelete=2000
SenderFrequency=30
DebugLevel=3
Timeout=30
UnreachablePeriod=300
UnavailableDelay=120
PidFile=/var/run/zabbix/zabbix_server.pid
LogFile=/var/log/zabbix-server/zabbix_server.log
AlertScriptsPath=/etc/zabbix/alert.d/
ExternalScripts=/home/zabbix/bin
DBHost=127.0.0.1
DBPort=6432
DBName=zabbix
DBUser=zabbix
DBPassword=xxxx
CacheSize=384M
TrendCacheSize=128M
CacheUpdateFrequency=300
LogSlowQueries=5000

                                            Comment

                                            • sducoin
                                              Junior Member
                                              • Dec 2011
                                              • 9
                                              #12
                                              Dirty ipam dns

                                              Hi,
                                              The problem came from our dns server (ipam 3.5) who was in a dirty state. Still responding but...
                                              Anyway after a cold reboot of the ipam dns server and SourceIP commented, zabbix worked fine and good as the old good time.

                                              Personal reminder : think about ipam fine monitoring.

                                              Thank you dima_dm for your help.
                                              have a good day

                                              cheers

                                                Comment

                                                Previous template Next
                                                Working...