Trouble getting snmp listener working in Zabbix 3.4 on Ubuntu

[surfrock66]

I'm setting up a brand new Zabbix instance, version 3.4.11, on Ubuntu Server 18.04 and Zabbix itself appears to be working, but I'm having issues getting the SNMP listener to work. On one hand I think it's running and I'm receiving test traps, but i'm not getting any data including discovery off the first host I am looking at. I've followed a bunch of configuration guides, but I'm stumped. I've updated all templates/items with our community name.

The core issue is, I have a cisco catalyst at 10.1.1.1 (this is our first SNMP device, added as a test) and I have set up "Template Module Cisco Inventory SNMPv2" on it, but in the zabbix logs I get the following:

 temporarily disabling SNMP agent checks on host "10.1.1.1": host unavailable

I can ping the device. The firewall is off on the zabbix server:

# ufw status
Status: inactive

In the GUI, I have a RED SNMP box next to the host with hover-text "Timeout while connecting to 10.1.1.1:161."

When I run this:

snmptrap -v 1 -c public 127.0.0.1 '.1.3.6.1.6.3.1.1.5.4' '0.0.0.0' 6 33 '55' .1.3.6.1.6.3.1.1.5.4 s "eth0"

My trap file shows this:

08:36:53 2018/07/18 ZBXTRAP 127.0.0.1
PDU INFO:
  version                        0
  errorstatus                    0
  receivedfrom                   UDP: [127.0.0.1]:54995->[127.0.0.1]:162
  requestid                      0
  messageid                      0
  transactionid                  1
  errorindex                     0
  community                      public
  notificationtype               TRAP
VARBINDS:
  iso.3.6.1.2.1.1.3.0            type=67 value=Timeticks: (55) 0:00:00.55
  iso.3.6.1.6.3.1.1.4.1.0        type=6  value=OID: iso.3.6.1.6.3.1.1.5.4.0.33
  iso.3.6.1.6.3.1.1.5.4          type=4  value=STRING: "eth0"
  iso.3.6.1.6.3.18.1.4.0         type=4  value=STRING: "public"
  iso.3.6.1.6.3.1.1.4.3.0        type=6  value=OID: iso.3.6.1.6.3.1.1.5.4

That makes me think things are working, but I can't actually monitor my device.

Below is a dump of relevant configs and scripts.

/etc/zabbix/zabbix_server.conf

LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=0
PidFile=/var/run/zabbix/zabbix_server.pid
SocketDir=/var/run/zabbix
DBName=zabbix
DBUser=zabbix
DBPassword=##REDACTED##
SNMPTrapperFile=/var/log/zabbix_traps.log
StartSNMPTrapper=1
Timeout=4
AlertScriptsPath=/usr/lib/zabbix/alertscripts
ExternalScripts=/usr/lib/zabbix/externalscripts
FpingLocation=/usr/bin/fping
Fping6Location=/usr/bin/fping6
LogSlowQueries=3000

/etc/snmp/snmptrapd.conf

disableAuthorization yes
traphandle default snmptt
perl do "/etc/zabbix/zabbix_trap_receiver/zabbix_trap_receiver.pl"

/etc/zabbix/zabbix_trap_receiver/zabbix_trap_receiver.pl

$SNMPTrapperFile = '/var/log/zabbix_traps.log';
$DateTimeFormat = '%H:%M:%S %Y/%m/%d';
use Fcntl qw(O_WRONLY O_APPEND O_CREAT);
use POSIX qw(strftime);
sub zabbix_receiver
{
        my (%pdu_info) = %{$_[0]};
        my (@varbinds) = @{$_[1]};
        # open the output file
        unless (sysopen(OUTPUT_FILE, $SNMPTrapperFile, O_WRONLY|O_APPEND|O_CREAT, 0666))
        {
                print STDERR "Cannot open [$SNMPTrapperFile]: $!\n";
                return NETSNMPTRAPD_HANDLER_FAIL;
        }
        # get the host name
        my $hostname = $pdu_info{'receivedfrom'} || 'unknown';
        if ($hostname ne 'unknown') {
                $hostname =~ /\[(.*?)\].*/;                    # format: "UDP: [127.0.0.1]:41070->[127.0.0.1]"
                $hostname = $1 || 'unknown';
        }
        # print trap header
        #       timestamp must be placed at the beggining of the first line (can be omitted)
        #       the first line must include the header "ZBXTRAP [IP/DNS address] "
        #              * IP/DNS address is the used to find the corresponding SNMP trap items
        #              * this header will be cut during processing (will not appear in the item value)
        printf OUTPUT_FILE "%s ZBXTRAP %s\n", strftime($DateTimeFormat, localtime), $hostname;
        # print the PDU info
        print OUTPUT_FILE "PDU INFO:\n";
        foreach my $key(keys(%pdu_info))
        {
                printf OUTPUT_FILE "  %-30s %s\n", $key, $pdu_info{$key};
        }
        # print the variable bindings:
        print OUTPUT_FILE "VARBINDS:\n";
        foreach my $x (@varbinds)
        {
                printf OUTPUT_FILE "  %-30s type=%-2d value=%s\n", $x->[0], $x->[2], $x->[1];
        }
        close (OUTPUT_FILE);
        return NETSNMPTRAPD_HANDLER_OK;
}
NetSNMP::TrapReceiver::register("all", \&zabbix_receiver) or
        die "failed to register Zabbix SNMP trap receiver\n";
print STDOUT "Loaded Zabbix SNMP trap receiver\n";

/etc/snmp/snmptt.conf

EVENT coldStart .1.3.6.1.6.3.1.1.5.1 "Status Events" Normal
FORMAT Device reinitialized (coldStart)
EVENT warmStart .1.3.6.1.6.3.1.1.5.2 "Status Events" Normal
FORMAT Device reinitialized (warmStart)
EVENT linkDown .1.3.6.1.6.3.1.1.5.3 "Status Events" Normal
FORMAT Link down on interface $1.  Admin state: $2.  Operational state: $3
EVENT linkUp .1.3.6.1.6.3.1.1.5.4 "Status Events" Normal
FORMAT Link up on interface $1.  Admin state: $2.  Operational state: $3
EVENT authenticationFailure .1.3.6.1.6.3.1.1.5.5 "Status Events" Normal
FORMAT SNMP athentication failure
EVENT general .* "General event" Normal
FORMAT ZBXTRAP $aA $ar

/etc/snmp/snmptt.ini

[General]
snmptt_system_name =
mode = standalone
multiple_event = 1
dns_enable = 0
strip_domain = 0
strip_domain_list = <<END
domain.com
END
resolve_value_ip_addresses = 0
net_snmp_perl_enable = 0
net_snmp_perl_cache_enable = 1
net_snmp_perl_best_guess = 0
translate_log_trap_oid = 0
translate_value_oids = 1
translate_enterprise_oid_format = 1
translate_trap_oid_format = 1
translate_varname_oid_format = 1
translate_integers = 1
wildcard_expansion_separator = " "
allow_unsafe_regex = 0
remove_backslash_from_quotes = 0
dynamic_nodes = 0
description_mode = 0
description_clean = 1
threads_enable = 0
threads_max = 10
date_time_format = %H:%M:%S %Y/%m/%d
[DaemonMode]
daemon_fork = 1
daemon_uid = snmptt
pid_file = /var/run/snmptt.pid
spool_directory = /var/spool/snmptt/
sleep = 5
use_trap_time = 1
keep_unlogged_traps = 1
duplicate_trap_window = 0
[Logging]
stdout_enable = 0
log_enable = 1
log_file = /var/log/zabbix_traps.log
log_system_enable = 0
log_system_file = /var/log/snmptt/snmpttsystem.log
unknown_trap_log_enable = 0
unknown_trap_log_file = /var/log/snmptt/snmpttunknown.log
statistics_interval = 0
syslog_enable = 1
syslog_facility = local0
syslog_level_debug = <<END
END
syslog_level_info = <<END
END
syslog_level_notice = <<END
END
syslog_level_warning = <<END
END
syslog_level_err = <<END
END
syslog_level_crit = <<END
END
syslog_level_alert = <<END
END
syslog_level = warning
syslog_system_enable = 1
syslog_system_facility = local0
syslog_system_level = warning
[SQL]
db_translate_enterprise = 0
db_unknown_trap_format = '$-*'
sql_custom_columns = <<END
END
sql_custom_columns_unknown = <<END
END
mysql_dbi_enable = 0
mysql_dbi_host = localhost
mysql_dbi_port = 3306
mysql_dbi_database = snmptt
mysql_dbi_table = snmptt
mysql_dbi_table_unknown = snmptt_unknown
mysql_dbi_table_statistics =
mysql_dbi_username = snmpttuser
mysql_dbi_password = password
mysql_ping_on_insert = 1
mysql_ping_interval = 300
postgresql_dbi_enable = 0
postgresql_dbi_module = 0
postgresql_dbi_hostport_enable = 0
postgresql_dbi_host = localhost
postgresql_dbi_port = 5432
postgresql_dbi_database = snmptt
postgresql_dbi_table_unknown = snmptt_unknown
postgresql_dbi_table_statistics =
postgresql_dbi_table = snmptt
postgresql_dbi_username = snmpttuser
postgresql_dbi_password = password
postgresql_ping_on_insert = 1
postgresql_ping_interval = 300
dbd_odbc_enable = 0
dbd_odbc_dsn = snmptt
dbd_odbc_table = snmptt
dbd_odbc_table_unknown = snmptt_unknown
dbd_odbc_table_statistics =
dbd_odbc_username = snmptt
dbd_odbc_password = password
dbd_odbc_ping_on_insert = 1
dbd_odbc_ping_interval = 300
[Exec]
exec_enable = 1
pre_exec_enable = 1
unknown_trap_exec =
unknown_trap_exec_format =
exec_escape = 1
[Debugging]
DEBUGGING = 0
DEBUGGING_FILE =
DEBUGGING_FILE_HANDLER =
[TrapFiles]
snmptt_conf_files = <<END
/etc/snmp/snmptt.conf
END

Here are the running processes, with their associated user accounts:

ps -ef | grep snmp
root        636      1  0 Jul17 ?        00:00:00 /usr/sbin/snmptrapd -Lsd -f
Debian-+    662      1  0 Jul17 ?        00:00:15 /usr/sbin/snmpd -Lsd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f
root        932      1  0 Jul17 ?        00:00:00 /usr/bin/perl /usr/sbin/snmptt --daemon
snmptt      933    932  0 Jul17 ?        00:00:00 /usr/bin/perl /usr/sbin/snmptt --daemon
zabbix     1061    989  0 Jul17 ?        00:00:02 /usr/sbin/zabbix_server: snmp trapper [processed data in 0.000010 sec, idle 1 sec]
root       5881   5734  0 10:02 pts/0    00:00:00 grep --color=auto snmp

Any thoughts? There are a ton of things in the logs like this:

Cannot find module 
Cannot adopt OID in UCD-SNMP-MIB: 
Cannot adopt OID in NET-SNMP-AGENT-MIB:

[khemrajk2003]

Hi Surrock66, I am by no means an expert in Zabbix, but I just happen to come across this post. You may have already resolved this issue since the post is about a month old. I am currently setting up SNMP traps on our Zabbix Proxy. Where you able to resolve this? Can you send a test trap from your CISCO device to your Zabbix server and see the traps in the log; can you query your CISCO device from your Zabbix server? The RED SNMP indicates a communication issue between your Zabbix server and the CISCO device, are there any firewalls (not the local one on your Zabbix server) that may be dropping the request? Did you define the Zabbix server as a trap destination on the CISCO device and or allow snmp queries from your Zabbix server. One thing I notice, with your snmptrapd.conf file (this would not fix your issue, but I was just wondering), I was under the impression that you either use snmptt to handle the traps or you use the Zabbix Perl script to handle the traps. I literally setup a new Zabbix Proxy to replace (soon) our current one that is used to handle snmp devices and I just used the Zabbix trap Perl script (referenced it in the snmptrapd.conf file) and the traps are working. It looks like your issue is just the communication between your Zabbix server and the CISCO device.

[khemrajk2003]

Sorry about the type with your name, I meant surfrock66.

[surfrock66]

Sort of. It came down to the fact that the community string isn't being respected. Our organization uses a non-standard community string for most devices, but select devices use the default "public." If we set the global macro with the off-standard community string, then override it on the host itself, it doesn't get respected. I debugged this using snmpwalk;

snmpwalk -v 2c -c public -O bn -C i 10.1.2.2 .1.3.6.1.2.1.1.5.0

snmpwalk -v 2c -c <<other community>> -O bn -C i 10.1.2.2 .1.3.6.1.2.1.1.5.0

I just deleted and recreated the hosts a bunch of times then it worked I've heard of people using MEGA old versions (1.8) looking in the DB and seeing that the effective community string is correct, but I didn't do that.