Ad Widget

Collapse

How I got host specific snmp traps & triggers working using modified snmptrap.sh

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • shadowk
    Junior Member
    • Jun 2006
    • 23

    #1

    How I got host specific snmp traps & triggers working using modified snmptrap.sh

    After reading several threads and tinkering a bit I had not found solutions to the following two problems:

    1. How to make a reasonable trigger based on traps that doesn't either get stuck ON or possibly miss some traps during some "nodata" period.

    2. How to send incoming traps directly to the correct zabbix host (if configured) without knowing what the OIDs will be while sending traps from unconfigured hosts to a default trap host for examination.

    I wanted to be able to add a standardized item and trigger to an arbitrary number of hosts and have the snmptrap.sh script forward traps to these items based on the sender hostname. I also wanted to be able to create a trigger which would go off if *any* trap came in on a host in addition to triggers for specific traps.

    Below are the basic steps I went through to get this working:

    In Zabbix:

    1. Create a Default_Trapper host (no IP needed).

    2. Add the following item:
    name: SNMP_Trapper_Item
    type: ZABBIX trapper
    key: snmptraps
    Type of information: Character

    3. Add the following trigger for any received trap (note the expression is true *unless* a trap is received, hence the comparison with 0).
    name: {HOSTNAME} Trap Received
    expr: {Default_Trapper:snmptraps.str(NO_NEW_DATA)}=0

    4. Add any additional triggers for specific traps (these expressions compare with 1 since they will be true when the trap comes in)
    name: {HOSTNAME} Specific Trap Received
    expression: {Default_Trapper:snmptraps.str(word_to_match)}=1

    (For workaround notes on matching more than just one word see this thread)

    5. Add the same items and triggers from above to any other hosts that you will be getting traps from. Any host with the above item will be able to have it's own trap history log and it's own triggers. (I used a template with the items & triggers and then applied the template to a number of hosts to make things a bit easier)

    snmptrapd:

    1. Get snmptrapd itself working (see the first part of this thread for instructions on getting that going)

    2. Edit the included snmptrap.sh script as follows:
    replace:
    Code:
    HOST="snmptraps";
    with:
    Code:
    DEFAULTHOST="Default_Trapper";
    NODATASTRING="NO_NEW_DATA"
    replace:
    Code:
    $ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $HOST $KEY "$str"
    with:
    Code:
    result=`$ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $hostname $KEY "$str"`
    
    echo result is: $result
    
    if [ "$result" = "OK" ]; then
      $ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $hostname $KEY "$NODATASTRING"
    else
      $ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $DEFAULTHOST $KEY "$str"
      $ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $DEFAULTHOST $KEY "$NODATASTRING"
    fi
    Basically I modified the script to send an extra trap to zabbix after each real trap to reset the item to the "NO_NEW_DATA" state. It also now tries to send the traps to specific hosts matching the trap source and then falls back to the default host if that doesn't work.

    Hopefully this is clear enough and helpful for others trying to use zabbix to log and alert on snmptraps.

    Andrew
    Last edited by shadowk; 04-12-2006, 16:39.
  • crayons
    Junior Member
    • Oct 2006
    • 21

    #2
    Nice work created a little patch for this. Even though I know it is only a few lines.
    Attached Files

    Comment

    • shadowk
      Junior Member
      • Jun 2006
      • 23

      #3
      After using the above solution for some time I have noticed that occasionally the ZABBIX_SENDER doesn't appear to return the "OK" result even when it does in fact post the trap info to the specified host. This can cause a duplicate trap to show up in the DEFAULTHOST. To help reduce the chances of this happening I modified my last code sample above by adding an additional check for the OK response on the second call to zabbix_sender:
      Code:
      result=`$ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $hostname $KEY "$str"`
      
      echo result is: $result
      
      if [ "$result" = "OK" ]; then
        $ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $hostname $KEY "$NODATASTRING"
      else
        result=`$ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $hostname $KEY "$NODATASTRING"`
      fi
      
      if [ "$result" != "OK" ]; then
        $ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $DEFAULTHOST $KEY "$str"
        $ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $DEFAULTHOST $KEY "$NODATASTRING"
      fi
      Now two calls to the ZABBIX_SENDER have to fail to return the OK response before the entries are added to the DEFAULTHOST. I'm still getting the occasional duplicate message on DEFAULTHOST but not as often.

      Andrew
      Last edited by shadowk; 22-01-2007, 19:01.

      Comment

      • crayons
        Junior Member
        • Oct 2006
        • 21

        #4
        Here is a little perl script I wrote based on what you did, but took it a step further. It checks the host table to match up the ip for the correct hostname in zabbix and falls back on dns/ip if it can't find it. We don't use dns on most of our servers here. That is why I wrote this. Perhaps this should be included in a zabbix release as the original snmptrap.sh isn't very good.

        Improve it, use it, dont use. Do what you want with it. But thought I would share with the rest of the zabbix community.
        Attached Files

        Comment

        • shadowk
          Junior Member
          • Jun 2006
          • 23

          #5
          Very nice, I will definitely play around with your script. I have the same hostnames not in DNS issue here but had been adding them to the hosts file on my zabbix server. I'm looking at expanding monitoring to a much larger group of devices soon though so querying the database should work much better.

          Thanks,

          Andrew

          Comment

          • husse
            Junior Member
            • Jan 2008
            • 15

            #6
            Shadowk,

            Seems like your explanation is very helpful - and the most detailed document I have found up to now. I've spent hours on this but progress is irritatingly slow.

            Maybe for experienced people your explanation is good enough but I still don't understand what exactly you have done when modifying snmptrap.sh.

            Here is my scenario..

            I have an SnmpAgent sending snmp traps when an alarm occurs in my product system. The goal is to be able to monitor alarms via Zabbix Web interface.


            (My SnmpAgent) --> snmpd --> snmptrap.sh --> zabbix_sender --> zabbix

            - SnmpAgent is the one creating snmp traps and sending them to the snmpd.
            - snmpd is the opne receiving and logging SNMP trap messages. AND I want it to forward it to the snmptrap.sh.
            - snmptrap.sh will convert snmp traps into zabbix traps which can be handled by zabbix.
            - zabbix_sender is used BY the snmptrap.sh for sending generated "zabbix traps" to the Zabbix.
            - Zabbix refers to the Zabbix_server. (AM I RIGHT WITH THIS? DOES ZABBIX REALLY REFERS TO THE ZABBIX_SERVER OR IS IT ANOTHER THING HERE? )


            For snmpd, i need to generate a snmpd.conf file (for access configuration) using "snmpconf". I did it.. Now I am able to send snmp traps from my SnmpAgent to the snmpd and I can see it in the log file of snmpd (/var/log/snmpd.log). By the way, I moved the snmpd.conf to the location /etc/snmp/snmpd.conf.

            As far as I understand I need to add this line
            - traphandle default /bin/bash /home/zabbix/bin/snmptrap.sh -
            into the snmpd.conf that I have generated previously (/etc/snmp/snmpd.conf). AM I RIGHT OR DO I NEED TO ADD THIS LINE INTO THE /var/net-snmp/snmpd.conf..??


            The I need to do the following...
            >> Copy misc/snmptrap/snmptrap.sh to ~zabbix/bin
            DONE!

            Next...
            Edit snmptrap.sh to configure some basic parameters
             Add special host and trapper (type "string") item to ZABBIX. See snmptrap.sh for the item's key.

            This step is making me crazy really.. I have no idea about what it is talking about.. What exactly am I supposed to do here..? Or is it sufficient to leave the snmptrap.sh as it is? Which snmptrap.sh will be modified, ~zabbix/bin/snmptrap.sh or another one at another location..?

            I guess if I manage to edit the snmptrap.sh file properly then snmp traps coming to snmpd will be able to forwarded to the zabbix... But I am not able to get it working..

            You said in your post that:
            In Zabbix:

            1. Create a Default_Trapper host (no IP needed)
            >>> HOW? WHERE? IS THERE ANY TUTORIAL EXPLAINING THIS ISSUE? AND WHY DO WE NEED THIS?

            2. Add the following item:
            name: SNMP_Trapper_Item
            type: ZABBIX trapper
            key: snmptraps
            Type of information: Character
            >>> WHERE TO ADD THIS ITEM??? AIM WITH THIS???

            It will really help me if Shadowk or any other person can give me some information about how to do this.. I am stucked..

            Thanks & Regards

            hs

            Comment

            • shadowk
              Junior Member
              • Jun 2006
              • 23

              #7
              Originally posted by husse
              - Zabbix refers to the Zabbix_server. (AM I RIGHT WITH THIS? DOES ZABBIX REALLY REFERS TO THE ZABBIX_SERVER OR IS IT ANOTHER THING HERE? )
              Yes this refers to the zabbix server, it sounds like you understand this correctly.

              Originally posted by husse
              As far as I understand I need to add this line
              - traphandle default /bin/bash /home/zabbix/bin/snmptrap.sh -
              into the snmpd.conf that I have generated previously (/etc/snmp/snmpd.conf). AM I RIGHT OR DO I NEED TO ADD THIS LINE INTO THE /var/net-snmp/snmpd.conf..??
              You need to add this into whichever conf file your snmptrapd is reading from. In my case I specify the .conf file when I start snmptrapd with the -c /usr/local/etc/snmptrapd.conf flag.

              Originally posted by husse
              Edit snmptrap.sh to configure some basic parameters
               Add special host and trapper (type "string") item to ZABBIX. See snmptrap.sh for the item's key.

              This step is making me crazy really.. I have no idea about what it is talking about.. What exactly am I supposed to do here..? Or is it sufficient to leave the snmptrap.sh as it is? Which snmptrap.sh will be modified, ~zabbix/bin/snmptrap.sh or another one at another location..?
              You need to either whichever snmptrap.sh file is being called from within your snmptrapd.conf file using the instructions above in my original post, use the patch file provided by another poster, or replace it completely with the perl script that was also provided above by another poster.

              Originally posted by husse
              1. Create a Default_Trapper host (no IP needed)
              >>> HOW? WHERE? IS THERE ANY TUTORIAL EXPLAINING THIS ISSUE? AND WHY DO WE NEED THIS?
              Do this from the zabbix web interface under configuration>hosts>add host

              Originally posted by husse
              2. Add the following item:
              name: SNMP_Trapper_Item
              type: ZABBIX trapper
              key: snmptraps
              Type of information: Character
              >>> WHERE TO ADD THIS ITEM??? AIM WITH THIS???
              Do this from the zabbix web interface under configuration>items (under the host you created above)

              Feel free to reply if you have further questions and good luck!

              Andrew

              Comment

              • husse
                Junior Member
                • Jan 2008
                • 15

                #8
                Hello,

                Thanks for the answer.. It was really helpful. But still I am not able to see the result either on the browser or in the log file (/var/log/zabbix/zabbix_server.log)..

                I have edited the /etc/snmp/snmpd.conf file and added the line
                - traphandle default /bin/bash /home/zabbix/snmptrap.sh -
                into the file.
                You can see the content of my snmpd.conf file in the attachment. I just deleted some lines to make its size smaller than 19.5 KB. But they were commented lines so that it shouldn't matter. Does the snmpd.conf file seem OK to you?

                I am not using snmptrapd, using snmpd.. As I know so far it shouldn't matter which one I use in the system. Because when I write "man snmpd" I am forwarded to the man page of snmptrapd. Am I right? Or do I have to use the snmptrapd instead?

                I added the Default_Trapper host with the following values:
                Name: Default_Trapper
                IP Address: 0.0.0.0 >> If I delete it, it complains. Is it OK to write 0.0.0.0 or do I need to write sth else?
                Connect to: IP Address >> WHY? Does it matter? Alternatively, I can set it to "DNS Name" too but in Andrew's post it is said that "NO IP NEEDED". I assume that this parameter will be ignored by the Zabbix in that case.
                Status: Monitored
                Use Profile: Non-checked

                Then I added the item and triggers exactly as you specified.. (With the same values Andrew has given.) And modified the snmptrap.sh as Andrew specified in 2 posts.

                Now..
                (My SnmpAgent) -> snmpd -> snmptrap.sh -> Zabbiz_sender -> Zabbix_server

                The big question...

                My SnmpAgent sends the snmp trap to the snmpd. I see it in the log file (/var/log/snmpd.log). However, I see nothing in the /var/log/zabbix/zabbix_server.log. How can I learn what the problem is? Isn't there any place, log file or sth like that which can show the progress..? Currently I am not able to find any mistake (but certainly there is at least one).. When I send the trap from my SnmpAgent I want to see sth reasonable in the log file of zabbix_server and on the Web browser. But I see nothing.. As I see the trap info is not even coming to the snmptrap.sh..
                I am not good in scripting language so that I don't understand so much from the snmptrap.sh's content actually.. Would you have any recommendations about what I need to do and check now? I am stucked again and not able to find any alternative solution..

                Thanks & Regards

                hs
                Attached Files

                Comment

                • husse
                  Junior Member
                  • Jan 2008
                  • 15

                  #9
                  One more try.. Not successful, but still gives more information...

                  When I start the snmpd, I see the following info in the log file /var/log/snmpd.log.

                  /etc/snmp/snmpd.conf: line 463: Warning: Unknown token: traphandle.
                  /etc/snmp/snmpd.conf: line 464: Warning: Unknown token: logging.
                  /usr/local/share/snmp/snmpd.conf: line 49: Warning: Unknown token: traphandle.
                  NET-SNMP version 5.4



                  And here is the content of /usr/local/share/snmp/snmpd.conf file:
                  ################################################## #########################
                  #
                  # snmpd.conf
                  #
                  # - created by the snmpconf configuration program
                  #

                  ################################################## #########################
                  # SECTION: Access Control Setup
                  #
                  # This section defines who is allowed to talk to your running
                  # snmp agent.

                  # rocommunity: a SNMPv1/SNMPv2c read-only access community name
                  # arguments: community [default|hostname|network/bits] [oid]

                  rocommunity public

                  # rwcommunity: a SNMPv1/SNMPv2c read-write access community name
                  # arguments: community [default|hostname|network/bits] [oid]

                  rwcommunity public


                  ################################################## #########################
                  # SECTION: Trap Destinations
                  #
                  # Here we define who the agent will send traps to.

                  # trap2sink: A SNMPv2c trap receiver
                  # arguments: host [community] [portnum]

                  trap2sink localhost

                  # trapcommunity: Default trap sink community to use
                  # arguments: community-string

                  trapcommunity public

                  # authtrapenable: Should we send traps when authentication failures occur
                  # arguments: 1 | 2 (1 = yes, 2 = no)

                  authtrapenable 1

                  traphandle default /bin/bash /home/zabbix/bin/snmptrap.sh



                  As an anecdote, I start the snmpd with the following command:
                  /usr/sbin/snmpd -c /usr/local/share/snmp/snmpd.conf


                  I guess the snmpd doesn't like the traphandle parameter... But what am I supposed to do in this case? It is said in the manual that I just need to add that line to the snmpd.conf file. Any ideas...???

                  Thanks in advance!

                  Comment

                  • shadowk
                    Junior Member
                    • Jun 2006
                    • 23

                    #10
                    Husse,

                    You might try sending some info to a log file. For example, add the following line to snmptrap.sh just after the first result= statement:

                    echo $ZABBIX_SENDER $ZABBIX_SERVER $ZABBIX_PORT $hostname $KEY "$str" >> /tmp/sending.log

                    Then look in the log file after you send a trap. If it's empty, then the trap probably isn't going to the snmptrap.sh script, otherwise you should see the command used to send the message to zabbix. You can then try running that command directly from the prompt to see what you get.

                    Andrew

                    Comment

                    • shadowk
                      Junior Member
                      • Jun 2006
                      • 23

                      #11
                      Husse,

                      Can you try running snmptrapd instead? From my man pages:

                      snmpd is an SNMP agent which binds to a port and awaits requests from SNMP management software.

                      snmptrapd is an SNMP application that receives and logs SNMP TRAP and INFORM messages.

                      Andrew

                      Comment

                      • husse
                        Junior Member
                        • Jan 2008
                        • 15

                        #12
                        Thanks a lot for the answer!

                        I tried.. But some problems I have..

                        First.. Here is the content of my snmptrapd.conf file:
                        #------------
                        # no access control mechanism will be used.
                        # if this parameter is not set to "yes",
                        #then all traps coming to the snmptrapd is dropped.
                        disableAuthorization yes

                        traphandle default /bin/bash /home/zabbix/snmptrap.sh
                        #------------



                        There is no "snmptrapd.log" in the system. Is it normal?
                        In order to be able to see what is happening I write the following command when running the snmptrapd:
                        /usr/sbin/snmptrapd -f -Le -F "%02.2h:%02.2j TRAP%w.%q from %A\n" -c /etc/snmp/snmptrapd.conf



                        I've sent an example trap, and here is the result:
                        NET-SNMP version 5.4
                        07:25 TRAP0.0 from 192.168.28.212
                        /home/zabbix/snmptrap.sh: line 64: unexpected EOF while looking for matching ``'
                        /home/zabbix/snmptrap.sh: line 71: syntax error: unexpected end of file
                        /home/zabbix/snmptrap.sh: line 64: unexpected EOF while looking for matching ``'
                        /home/zabbix/snmptrap.sh: line 71: syntax error: unexpected end of file


                        I commented out the line
                        <<traphandle default /bin/bash /home/zabbix/snmptrap.sh>>
                        and tried again.. And here is the result:
                        NET-SNMP version 5.4
                        07:27 TRAP0.0 from 192.168.28.212



                        As I see the line <<traphandle default /bin/bash /home/zabbix/snmptrap.sh>>
                        is the one creating the problem. The config file doesn't like it.. But this is the line that I must add to the config file in order to realize the communication with snmptrap.sh.

                        Anecdote: My snmptrap.sh is located at: /home/zabbix/snmptrap.sh

                        Any recommendations? Shouldn't it be accepted because the manual just says that I just need to add that line into the config file..

                        Thanks & Regards

                        hs

                        Comment

                        • shadowk
                          Junior Member
                          • Jun 2006
                          • 23

                          #13
                          snmptrapd usually logs to /var/log/messages by default unless you specify another log file.

                          Also, you might try starting it with only the -c flag to see if that works better. It sounds like the snmptrap.sh script is running but generating errors. Either the script has typos or the input to the script is not as expected.

                          You could also try running the script directly to see if you get errors (./snmptrap.sh for example).

                          Andrew

                          Comment

                          • husse
                            Junior Member
                            • Jan 2008
                            • 15

                            #14
                            Sorry for overwhelming with mails.. But please forget my previous mail..

                            I've done as you said.. I am using snmptrapd instead of snmpd. And I added the "echo ..." line to the snmptrap.sh..

                            And I sent a V2 trap from my SnmpAgent to the snmprtapd.. The screen output:
                            /usr/sbin/snmptrapd -f -Le -F "%02.2h:%02.2j TRAP%w.%q from %A\n" -c /etc/snmp/snmptrapd.conf
                            NET-SNMP version 5.4
                            11:32 TRAP0.0 from 0.0.0.0
                            result is: sent: 0; failed: 1; total: 1
                            sent: 0; failed: 1; total: 1
                            sent: 0; failed: 1; total: 1
                            result is: sent: 0; failed: 1; total: 1
                            sent: 0; failed: 1; total: 1
                            sent: 0; failed: 1; total: 1


                            And the content of /tmp/sending.log:
                            /usr/local/sbin/zabbix_sender zabbix 10001 <UNKNOWN> snmptraps <UNKNOWN>
                            /usr/local/sbin/zabbix_sender zabbix 10001 <UNKNOWN> snmptraps <UNKNOWN>


                            Just wondering... Does it change sth to run snmptrapd, zabbix_server and zabbix_agentd as root or a normal user?

                            I think you will see the problem better than I do.. I still don't see anything in the file zabbix_server.log. Would you have any suggestions about the problem at this point?

                            Thanks a lot again! Your answers helped me a lot!

                            Regards,

                            hs

                            Comment

                            • shadowk
                              Junior Member
                              • Jun 2006
                              • 23

                              #15
                              Try starting snmptrapd like this instead:

                              /usr/sbin/snmptrapd -c /etc/snmp/snmptrapd.conf

                              It seems like your formatting options may be breaking the snmptrap.sh script.

                              Comment

                              Working...