Ad Widget

Collapse

Getting snmp traps working with zabbix 2.2.12

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • zabbixfk
    Senior Member
    • Jun 2013
    • 256

    #1

    Getting snmp traps working with zabbix 2.2.12

    Hello All,

    I use to run zabbix 2.0.6, and recently upgraded to 2.2.12. I wasn't using snmptraps, but only snmpget/walk so no issues till now. Looks like traps are more reliable than snmpget so decided to implement.
    A quick search leads to this tutorial , but i am having tough time getting things work. I choose perl option, and below is the error i am facing when i am running snmptrapd start,
    below is the /etc/snmp/snmptrapd.conf
    Code:
    # Example configuration file for snmptrapd
    #
    # No traps are handled by default, you must edit this file!
    #
    # authCommunity   log,execute,net public
    # traphandle SNMPv2-MIB::coldStart    /usr/bin/bin/my_great_script cold
    authCommunity log,execute,net public
    authCommunity log,execute,net string1 
    authCommunity log,execute,net string2 
    authCommunity log,execute,net string3 
    perl do  /usr/bin/zabbix_trap_receiver.pl;
    Error:
    Code:
    Starting snmptrapd: Bareword found where operator expected at (eval 2) line 1, near "/usr/bin"
    	(Missing operator before bin?)
                                                               [  OK  ]
    Also, /var/log/messages
    Code:
    May  3 15:50:43 Zabbix-Test1 snmptrapd[119488]: /etc/snmp/snmptrapd.conf: line 11: Warning: Unknown token: /usr/bin/perl.
    May  3 15:50:43 Zabbix-Test1 snmptrapd[119489]: NET-SNMP version 5.5
    I did tried making per as /usr/bin/perl but that time error msg was diff
    Code:
    [loginuser@Zabbix-Test1 snmp]# /etc/init.d/snmptrapd restart ; tailf /var/log/messages
    Stopping snmptrapd:                                        [  OK  ]
    Starting snmptrapd:                                        [  OK  ]
    But /var/log/messages has some other line to show
    Code:
    May  3 16:02:43 Zabbix-Test1 snmptrapd[119642]: /etc/snmp/snmptrapd.conf: line 11: Warning: Unknown token: /usr/bin/perl.
    May  3 16:02:43 Zabbix-Test1 snmptrapd[119643]: NET-SNMP version 5.5
    Trying to send example trap from this guide landed it in /var/log/messages instead of /tmp/zabbix_trapper.tmp

    Should i start working on snmptt instead of perl script?
    Also There are not examples provided in manuals on how to create example items for traps. Say port goes down, fan stopped working, latency is high etc.

    Any pointers to get this working is really helpful.

    Thanks
  • zabbixfk
    Senior Member
    • Jun 2013
    • 256

    #2
    Getting snmp traps working with zabbix 2.2.12

    Hello All,

    Now that i am able to get the example snmp trap working using the perl script provided, how to create items on the zabbix UI? Can somebody share examples on that please.
    - Say a trap sent from a router when the port goes down, how to capture that in zabbix? It may come sit in zabbix_server.log or zabbix_proxy.log, but how to get the deamon understand that?
    - I have couple of n/w devices which are vary in make and wants to send traps for events change, how to set it up in zabbix? Like for cpu usage i can add
    Code:
    Key=enterprises.2636.3.1.13.1.8.9.1.0.0
    OID=.1.3.6.1.4.1.2636.3.1.13.1.8.9.1.0.0
    for snmpget/walk or type SNMPv2 agent under items.
    If the similar thing to be added for traps, how to do it.
    Can somebody share examples of their traps working?

    Any pointers are greatly helpful.

    Thanks

    Comment

    • zabbixfk
      Senior Member
      • Jun 2013
      • 256

      #3
      Getting snmp traps working with zabbix 2.2.12

      *bump* , can somebody reply please.

      Comment

      • Linwood
        Senior Member
        • Dec 2013
        • 398

        #4
        I think the path to the trapper needs to be in quotes in

        perl do /usr/bin/zabbix_trap_receiver.pl;

        And then set up an item in SNMP that has a parameter that matches the trap, though I am stuck even before that on my attempts so I cannot give you a working example. I've copied practically every example I can find without success even in getting test traps to go, so should you find a good working example, I would also appreciate if you would post followup.

        I do know the docs recommend using snmptt first, my guess is that it is easier to debug (and indeed I can see where the failure in my case is, just not why, but I do not want to pollute this thread with my issue).

        Comment

        • zabbixfk
          Senior Member
          • Jun 2013
          • 256

          #5
          Getting snmp traps working with zabbix 2.2.12

          Thanks for the reply. I am able to get test trap working. It is landing under /var/log/messages and also under /tmp/zabbix_trap.tmp, and i can see the formulated trap under zabbix_server.log.
          Yes, that quote for the script path was missing hence i was not able to get it work, now its working, and i am able to get test trap landing under the file.

          Check the examples from this link

          I am stuck in creating items. Though i created an item using type snmp trap, and key as snmp.fallback, it started showing some log.
          I am trying to figure out how to add items and alerts for individual traps from different devices.

          Thanks.

          Comment

          • Linwood
            Senior Member
            • Dec 2013
            • 398

            #6
            Good to hear. I worked through some of my issues and think I am mostly where you are, except I can't get the embedded perl working (long, convoluted story, but the package install didn't include it, and I couldn't get it to compile in at least one run at it).

            But I'm also where you are, trying to figure out how to identify what to look for and set up triggers. I know how to, but it's a bit daunting to try to identify each trap for each device that might be of interest, so we are just starting from the reverse side -- what do we know we want to know, like firewall logins, and going back from there.

            I'm curious -- are you using numeric or symbolic trap identifiers? I found I can generate either with snmptt, and symbolic are a lot more readable in the item key definition, but I wonder what penalty is being paid for the conversion.

            Comment

            • zabbixfk
              Senior Member
              • Jun 2013
              • 256

              #7
              Getting snmp traps working with zabbix 2.2.12

              @Linwood - Not sure if i understand what you said but - if you have difficulty just install net-snmp, net-snmp-devel, net-snmp-libs, net-snmp-perl net-snmp-utils , and then download your zabbix version ( not rpm, but tar.gz file) - it will have zabbix_trap_receiver.pl inside misc/snmptrap folder which you can use.

              - I am not sure of symbolic/numeric trap identifiers, so can't comment on that sorry.

              - Please share example if you have any w.r.to. setting up individual items on zabbix for snmp trap based.

              Thanks

              Comment

              • Linwood
                Senior Member
                • Dec 2013
                • 398

                #8
                I'm working slowly through it.

                I figured out the perl issue, I was basically trying too hard, and trying to build net-snmp, thinking that the packaged version did not include the embedded perl, but it does. There are some issues with late versions of perl and compiling net-snmp, but now I see I do not need to do that at all.

                I did get a real trap working, but I have a long way to go to get it fully correct. If it helps, here is what I did.

                I created a discovery for interfaces and defined an item prototype for the traps as follows:

                snmptrap["(IF-MIB::linkDown|IF-MIB::linkUp).*ifIndex\.{#SNMPINDEX}"]

                This creates an item for one trap for each interface, and the item sees only the linkDown and linkUp types.

                Then I created a trigger prototype:

                {Template SNMP trap Generic:snmptrap["(IF-MIB::linkDown|IF-MIB::linkUp).*ifIndex\.{#SNMPINDEX}"].str(linkDown)}=1

                This all seems to work. But... it requires looking at the trap log that zabbix gets to see the exact format. The symbolic vs numeric aspect is controlled by the -O parameter driven into snmptrapd. I'm using -OS, which tries to use symbolic names more. This goes (for mine) in /etc/snmp/snmptrapd.conf as the TRAPDOPTS.

                The perl script then writes this to the trap file (this is an example:

                Code:
                13:13:41 2016/05/12 ZBXTRAP 10.108.2.1
                PDU INFO:
                  requestid                      2
                  messageid                      0
                  notificationtype               TRAP
                  receivedfrom                   UDP: [10.108.2.1]:162->[10.10.10.33]:162
                  version                        1
                  transactionid                  42
                  community                      redacted
                  errorindex                     0
                  errorstatus                    0
                VARBINDS:
                  DISMAN-EVENT-MIB::sysUpTimeInstance type=67 value=Timeticks: (730518000) 84 days, 13:13:00.00
                  SNMPv2-MIB::snmpTrapOID.0      type=6  value=OID: IF-MIB::linkUp
                  IF-MIB::ifIndex.9              type=2  value=INTEGER: 9
                  IF-MIB::ifAdminStatus.9        type=2  value=INTEGER: 1
                  IF-MIB::ifOperStatus.9         type=2  value=INTEGER: 1
                What I'm doing, which seems a bit backwards, is look at a real trap like this, and write the regex in the item definition to match it. Then write the trigger to also pull from it to tell up vs down.

                This bothers me as it looks rather fragile. For example, if the format of this trap in the log changes even slightly, the regex will not find it, and you can get radically different formats depending on the options, and depending on which MIB's are loaded for translation. And on that note I'm struggling with translation, I can translate a trap OID with snmptranslate just fine, but snmptrapd yields a completely different translation. That's bad, because what I had hoped to do was build triggers (or prototypes) based on MIB's, not actually have to get a trap and see how it comes out. Some traps may not be that easy to generate on a device, yet it seems I cannot (reliably) write a trap's regex until I see how it actually processes them and how they come out.

                There's got to be a simpler approach to all this.
                Last edited by Linwood; 12-05-2016, 23:07.

                Comment

                • herta
                  Senior Member
                  • Sep 2011
                  • 101

                  #9
                  Looks like traps are more reliable than snmpget so decided to implement.
                  What makes you think that?

                  Haven't looked into using traps with Zabbix, because snmp_traps are more prone to getting lost. They consist of a single UDP datagram sent by a device.
                  If a device sends an snmp_trap and zabbix didn't receive it, zabbix has no way of knowing it missed a trap.
                  If you run an snmp_get and don't get an answer you will at least know.
                  Last edited by herta; 13-05-2016, 12:45. Reason: correct confusing typo

                  Comment

                  • zabbixfk
                    Senior Member
                    • Jun 2013
                    • 256

                    #10
                    Getting snmp traps working with zabbix 2.2.12

                    @herta - yes , traps consists single udp datagram and its prone to get lost. But when the device does not give you what you want using snmpget, and its MIB says you can get details using only trap , i don't have any option other than looking to get snmptrap working on zabbix , also when there aren't much documentation available on getting this work makes me think again on using this.

                    @linwood - thanks for the detailed explanation. This should help , let me try something similar to this and get back.

                    Comment

                    • zabbixfk
                      Senior Member
                      • Jun 2013
                      • 256

                      #11
                      Getting snmp traps working with zabbix 2.2.12

                      @linwood - could you share your snmptrapd.conf (not able to locate trapdopts),
                      also discovery rule, item prototype with details, i am somehow not abel to get this rule working. Below is what i have configured
                      Code:
                      Discovery Rule : 
                      Name - Network Interfaces
                      Type -  SNMP V2 Agent
                      OID - .1.3.6.1.2.1.31.1.1.1.1
                      community string - public
                      Port  - blank
                      Update Interval - 300
                      Keep lost resources period (in days)  - 30
                      Item Prototype
                      Code:
                      Name - Port Status
                      Type - SNMP Trap
                      Key - snmptrap["(IF-MIB::linkDown|IF-MIB::linkUp).*ifIndex\.{#SNMPINDEX}"]
                      Type of information - text
                      History storage period (in days) - 180
                      Applications - Port Status
                      Thanks

                      Comment

                      • zabbixfk
                        Senior Member
                        • Jun 2013
                        • 256

                        #12
                        Getting snmp traps working with zabbix 2.2.12

                        @linwood - some success lately !!!
                        So i removed all those discovery type and all. Kept only one item. I wanted to check if things work or not. so below is my config.
                        Code:
                        Name - general
                        Type - SNMP Trap
                        Key - snmptrap["SNMPv2-MIB::authenticationFailure"]
                        Type of information - Log
                        History storage period (in days) - 365
                        Log time format - blank
                        Applications - Port Status
                        and one more item with general.fallback as name, snmptrap type with snmptrap.fallback as key created. These two items tied to example template. And i tied template to one of the n/w switches.
                        Now from that switch, if i run authentication spoof trap, i am able to see it lands in /var/log/messages and so in /var/log/zabbix/zabbix_server.log, also showing up in LatestData->UI on that n/w switches.

                        Changes made :
                        1). Added the community string entry on /etc/snmp/snmpdtrapd.conf
                        2). Made switch send this community string in trap ( check your switch vendor config, in juniper you need to keep SNMP-Group Name as community string name).
                        3). ran :- tcpdump -vv udp -n port "162" to actually see if the traps are hitting my server or not.

                        So next task would be
                        1). Getting switch send traps when there is a actual authentication failure
                        2). Creating an trigger whenever 'authenticationFailure' trap arives, send notification as email or something.
                        3). In that email, mention IP address of the switch, and username used to login - future

                        And all of this gets successfull, i am going to write a detailed entry on how to get snmptrap working

                        @linwood - still need your help in getting discoveries working , pls share how its being done on your end.

                        Thanks.

                        Comment

                        • Linwood
                          Senior Member
                          • Dec 2013
                          • 398

                          #13
                          The config settings for net-snmp are baffling to me in some ways, but I think I have figured out some of it.

                          The catch I have been having is that I do not think all the pieces of snmp work quite the same way with the defaults. I THINK I need to have default min search directories in two different places to make it work, also added mibs.

                          What I've been doing is:

                          /etc/default/snmptrapd
                          Code:
                          TRAPDRUN=yes
                          TRAPDOPTS='-OS -t -M/var/lib/mibs/cisco:/usr/share/snmp/mibs:/var/lib/mibs/iana:/var/lib/mibs/ietf -mCISCO-IPSEC-FLOW-MONITOR-MIB -p /run/snmptrapd.pid'
                          And in /etc/snmp/snmp.conf
                          Code:
                          mibdirs /usr/share/snmp/mibs:/var/lib/mibs/iana:/var/lib/mibs/ietf:/var/lib/mibs/cisco
                          mibs +CISCO-IPSEC-FLOW-MONITOR-MIB
                          Now I THOUGHT that snmp.conf was supposed to be a general default file, so the mibsdir there should be applying to snmptrapd, but it does not seem to, though the mibs entry there does appear to. So I'm not at all sure how these interact, but the above two together seem to work.

                          One thing that drove me nuts was that putting quotes around the mib search string, e.g. after the -M option, BREAKS the option, though it looks like it works (it prints that string saying it is using it, and prints it in quotes -- which are the problem).

                          I also rewrote the perl script to delete most of the lines it prints. Those are interesting and all that, but when they are going to be added to a trigger message, it's too much, so I built a list of regex strings to search and delete, and will add to that list as I go, deleting things like uptime (which often appears twice or three times in some traps), since it's pretty pointless when the timestamp is also there. So most of my traps are now down to 1-3 lines.


                          As to the discovery, not quite sure what you are asking. It looks just like an interface discovery (or could even be added as an item in any template that has a LLD for interface discovery). Each discovered interface then makes an item and trigger as below.

                          But that said, I am not sure those are particularly useful -- having a trigger for link up/down for each interface is too much. It was more of an exercise for me to understand it.

                          All I'm trapping right now are VPN logins, console logins, and anything in IOS Error level or over (or critical for ASA's as they are too chatty).

                          Comment

                          • Linwood
                            Senior Member
                            • Dec 2013
                            • 398

                            #14
                            My apology, I had an error in the snmp trap regex above. I was cleaning up and changed a posix form to a more common form, but that won't work. Here is the correct form:

                            Code:
                            snmptrap["(IF-MIB::linkDown|IF-MIB::linkUp)((.|[[:space:]])*)ifIndex\.{#SNMPINDEX}"]

                            Comment

                            • zabbixfk
                              Senior Member
                              • Jun 2013
                              • 256

                              #15
                              Getting snmp traps working with zabbix 2.2.12

                              @Linwood - Thanks for the reply.

                              I am still stuck at figuring out why my router isn't sending traps when login attempt failed

                              Could you share your part of script and trigger example , just to check what is the change it would make to trigger message. As of now i am not able to generate any trigger message so couldn't actually check how does it look.

                              Thanks

                              Comment

                              Working...