Help - SNMP Trap

[mfortes]

Guys,

My proxy are receiveng a lot of traps from my devices. That's OK! But, when created a item to show this traps, my zabbix server dont show anything =/
SNMPTrap is enable on the proxy/server side. Path to the .temp are OK too...

Anyone knows whats i'm missing?

[tim.mooney]

We're doing SNMP trap receiving on our main Zabbix server, not a proxy. I'm not aware of anything that would be different between the server and proxy for this setup.

You proxy is running the Zabbix snmp trapper process, correct? Permissions on the file are such that the Zabbix server user can read the file, correct?

If the item you created is an snmptrap[] item, try also creating an snmptrap.fallback item. That will help decide if the problem is that the data is getting to your server and it's just not matching your time, or whether the data is never making it to your server.

[mfortes]

I already tryed whit snmptrap.fallback too...
The data still in my file on /tmp/snmptrap... but my zabbix server dont show on item...
In both snmptrap the process is running and have permissions.

[tim.mooney]

By any chance, is the Zabbix server being started via systemd?

Depending on how the service is configured, systemd may be creating a "chroot" /tmp. Read up on "systemd PrivateTmp" in your favorite web search engine. If PrivateTmp is enabled for Zabbix server, then even though the snmptrapd may be writing the file to /tmp/snmptrap, the Zabbix server is not actually looking directly in /tmp, it's looking in a "private" directory under /tmp.

[mfortes]

Maybe not,
I have others proxys where my server can read and show trap logs perfectly.

[tim.mooney]

So to summarize, the traps are making it to your proxy and snmptrapd is writing them to the file you've configured, but Zabbix isn't displaying them. You've ruled out all of the common things that I can think of that might cause Zabbix to not get the traps.

Have you ever used "strace" to attach to a running process and view what system calls are happening and their results? I recommend you find the pid for the Zabbix snmp trap polling process and then attach to it for a little while. Pay particular attention to the stat() and access() calls and their results. In fact, since you mainly care about the file and file-descriptor related system calls for this particular case, I recommend including '-e trace=file,desc' to limit the output to calls related to files and descriptors:

$ ps -ef | grep 'snmp trap'
zabbix    4047  4029  0  2019 ?        00:10:16 /usr/local/sbin/zabbix_server: snmp trapper [processed data in 0.000009 sec, idle 1 sec]

$ sudo strace -f -s 256 -e trace=file,desc -p 4047
strace: Process 4047 attached
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=117, ...}) = 0
stat("/var/lib/zabbix/snmptrapd.tmp", {st_mode=S_IFREG|0644, st_size=96861472, ...}) = 0
access("/var/lib/zabbix/snmptrapd.tmp", R_OK) = 0
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=117, ...}) = 0
stat("/var/lib/zabbix/snmptrapd.tmp", {st_mode=S_IFREG|0644, st_size=96861472, ...}) = 0
access("/var/lib/zabbix/snmptrapd.tmp", R_OK) = 0
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=117, ...}) = 0
stat("/var/lib/zabbix/snmptrapd.tmp", {st_mode=S_IFREG|0644, st_size=96861472, ...}) = 0
access("/var/lib/zabbix/snmptrapd.tmp", R_OK) = 0
^Cstrace: Process 4047 detached

(note you can just control-C to stop strace and detach).

You can ignore the stat for /etc/resolv.conf, it's the stat and access and any other calls (open()) related to your snmptrapd temp file that are of interest.

Are stat() and access() returning 0 (success)?

If you're running this when an snmp trap is written to the temp file, do you see any open() system call?

[mfortes]

I dont know if i'm doing something wrong, but when i use strace, nothing happens...

[root@mabrlx04 ~]# ps -ef | grep 'snmp trap'
root      3276  1591  0 18:36 pts/0    00:00:00 grep --color=auto snmp trap
zabbix    8783  8721  0 Fev03 ?        00:01:28 /usr/sbin/zabbix_proxy: snmp trapper [processed data in 0.002110 sec, idle 1 sec]
[root@mabrlx04 ~]# sudo strace -f -s 256 -e trace=file,desc -p 8721
strace: Process 8721 attached

Only this...

[tim.mooney]

You attached to the parent pid (8721), not the snmp trapper (8783). Try the exact same command, but with the trapper's pid

[Vicko1]

I am new here, Please i want to setup Zabbix so as to capture snmp from an EMS but i do not know how to go about it. I have downloaded the Appliance version. I would be grateful if someone can help me with step by step guide on how to setup the snmp trap on the appliance

[gert.derouck]

Check this wiki page: https://zabbix.org/wiki/Start_with_SNMP_traps_in_Zabbix

[mfortes]

tim.mooney sorry, my bad...

This is my server:

[root@zabbix ~]# sudo strace -f -s 256 -e trace=file,desc -p 27522
strace: Process 27522 attached
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=280, ...}) = 0
stat("/var/log/zabbix/zabbix_server.log", {st_mode=S_IFREG|0664, st_size=1017159, ...}) = 0
stat("/tmp/zabbix_traps.tmp", {st_mode=S_IFREG|0644, st_size=1766605013, ...}) = 0
access("/tmp/zabbix_traps.tmp", R_OK)   = 0
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=280, ...}) = 0
stat("/var/log/zabbix/zabbix_server.log", {st_mode=S_IFREG|0664, st_size=1017280, ...}) = 0
stat("/tmp/zabbix_traps.tmp", {st_mode=S_IFREG|0644, st_size=1766605013, ...}) = 0
access("/tmp/zabbix_traps.tmp", R_OK)   = 0
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=280, ...}) = 0
stat("/var/log/zabbix/zabbix_server.log", {st_mode=S_IFREG|0664, st_size=1017389, ...}) = 0
stat("/tmp/zabbix_traps.tmp", {st_mode=S_IFREG|0644, st_size=1766605013, ...}) = 0
access("/tmp/zabbix_traps.tmp", R_OK)   = 0
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=280, ...}) = 0
stat("/var/log/zabbix/zabbix_server.log", {st_mode=S_IFREG|0664, st_size=1017495, ...}) = 0
stat("/tmp/zabbix_traps.tmp", {st_mode=S_IFREG|0644, st_size=1766605013, ...}) = 0
access("/tmp/zabbix_traps.tmp", R_OK)   = 0

This is my proxy:

[root@mabrlx04 ~]# sudo strace -f -s 256 -e trace=file,desc -p 8783
strace: Process 8783 attached
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=251, ...}) = 0
stat("/tmp/zabbix_traps.tmp", {st_mode=S_IFREG|0644, st_size=527531339, ...}) = 0
access("/tmp/zabbix_traps.tmp", R_OK)   = 0
lseek(7, 527527798, SEEK_SET)           = 527527798
read(7, "10:17:05 2020/02/06 ZBXTRAP 172.30.200.175\nPDU INFO:\n  notificationtype               TRAP\n  version                        0\n  receivedfrom                   UDP: [172.30.200.175]:64768->[172.30.200.16]:162\n  errorstatus                    0\n  messageid  "..., 64064) = 3541
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
open("/var/log/zabbix/zabbix_proxy.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 8
fstat(8, {st_mode=S_IFREG|0664, st_size=550906450, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2c10efe000
write(8, "  8783:20200206:101706.177 unmatched trap received from \"172.30.200.93\": 10:17:04 2020/02/06 PDU INFO:\n  notificationtype               TRAP\n  version                        1\n  receivedfrom                   UDP: [172.30.200.93]:49328->[172.30.200.16]:162"..., 1522) = 1522
close(8)                                = 0
open("/var/log/zabbix/zabbix_proxy.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 8
fstat(8, {st_mode=S_IFREG|0664, st_size=550907972, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2c10efe000
write(8, "  8783:20200206:101706.181 unmatched trap received from \"172.30.200.175\": 10:17:05 2020/02/06 PDU INFO:\n  notificationtype               TRAP\n  version                        0\n  receivedfrom                   UDP: [172.30.200.175]:64768->[172.30.200.16]:1"..., 1821) = 1821
close(8)                                = 0
stat("/tmp/zabbix_traps.tmp", {st_mode=S_IFREG|0644, st_size=527531339, ...}) = 0
access("/tmp/zabbix_traps.tmp", R_OK)   = 0

[tim.mooney]

tim.mooney sorry, my bad...
This is my proxy:

[root@mabrlx04 ~]# sudo strace -f -s 256 -e trace=file,desc -p 8783
strace: Process 8783 attached
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=251, ...}) = 0
stat("/tmp/zabbix_traps.tmp", {st_mode=S_IFREG|0644, st_size=527531339, ...}) = 0
access("/tmp/zabbix_traps.tmp", R_OK) = 0
lseek(7, 527527798, SEEK_SET) = 527527798
read(7, "10:17:05 2020/02/06 ZBXTRAP 172.30.200.175\nPDU INFO:\n notificationtype TRAP\n version 0\n receivedfrom UDP: [172.30.200.175]:64768->[172.30.200.16]:162\n errorstatus 0\n messageid "..., 64064) = 3541
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
open("/var/log/zabbix/zabbix_proxy.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 8
fstat(8, {st_mode=S_IFREG|0664, st_size=550906450, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2c10efe000
write(8, " 8783:20200206:101706.177 unmatched trap received from \"172.30.200.93\": 10:17:04 2020/02/06 PDU INFO:\n notificationtype TRAP\n version 1\n receivedfrom UDP: [172.30.200.93]:49328->[172.30.200.16]:162"..., 1522) = 1522
close(8) = 0
open("/var/log/zabbix/zabbix_proxy.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 8
fstat(8, {st_mode=S_IFREG|0664, st_size=550907972, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2c10efe000
write(8, " 8783:20200206:101706.181 unmatched trap received from \"172.30.200.175\": 10:17:05 2020/02/06 PDU INFO:\n notificationtype TRAP\n version 0\n receivedfrom UDP: [172.30.200.175]:64768->[172.30.200.16]:1"..., 1821) = 1821
close(8) = 0
stat("/tmp/zabbix_traps.tmp", {st_mode=S_IFREG|0644, st_size=527531339, ...}) = 0
access("/tmp/zabbix_traps.tmp", R_OK) = 0

No worries! The output you captured in your latest response is very useful.

The 'strace' from your proxy shows what I was hoping to see. You can see the stat() and access() calls:

stat("/tmp/zabbix_traps.tmp", {st_mode=S_IFREG|0644, st_size=527531339, ...}) = 0
access("/tmp/zabbix_traps.tmp", R_OK) = 0

That shows two things: that the Zabbix snmp trap poller on the proxy is trying to access your trap file, and that it's successful (the = 0 at the end of each line is the return code)

In addition, you let it run long enough to actually capture what happens when an SNMP trap comes in.

lseek(7, 527527798, SEEK_SET) = 527527798
read(7, "10:17:05 2020/02/06 ZBXTRAP 172.30.200.175\nPDU INFO:\n notificationtype TRAP\n version 0\n receivedfrom UDP: [172.30.200.175]:64768->[172.30.200.16]:162\n errorstatus 0\n messageid "..., 64064) = 3541

That is the Zabbix snmp trap poller process re-positioning where it's going to read from on the open file descriptor #7 (which must be associated with your /tmp/zabbix_traps.tmp file already -- I thought the poller might re-open the file every time it detects a change, but it looks like it just keeps it open), and then reading 3541 bytes of data. You can see the first part of what it read in the buffer. It's a trap from IP 172.30.200.175.

You can ignore the 3 calls to poll() that happen next, but after that it gets interesting again:

open("/var/log/zabbix/zabbix_proxy.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 8
fstat(8, {st_mode=S_IFREG|0664, st_size=550906450, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2c10efe000
write(8, " 8783:20200206:101706.177 unmatched trap received from \"172.30.200.93\": 10:17:04 2020/02/06 PDU INFO:\n notificationtype TRAP\n version 1\n receivedfrom UDP: [172.30.200.93]:49328->[172.30.200.16]:162"..., 1522) = 1522
close(8) = 0
open("/var/log/zabbix/zabbix_proxy.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 8
fstat(8, {st_mode=S_IFREG|0664, st_size=550907972, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2c10efe000
write(8, " 8783:20200206:101706.181 unmatched trap received from \"172.30.200.175\": 10:17:05 2020/02/06 PDU INFO:\n notificationtype TRAP\n version 0\n receivedfrom UDP: [172.30.200.175]:64768->[172.30.200.16]:1"..., 1821) = 1821
close(8) = 0

That's the snmp trap poller opening your /var/log/zabbix/zabbix_proxy.log on your proxy and then writing a message to it, closing it, and then opening it and writing another message to it. The 2nd message it writes is basically "unmatched trap received" from the host we saw earlier. The first log message it wrote was about a different host, that doesn't appear in your strace output. It must have happened very shortly before your strace command started up.

This proves a couple of things:

1) the Zabbix snmp polling process is looking at the correct file, it can access it (no permissions issue), and it's periodically reading trap information from it
2) it's logging to /var/log/zabbix/zabbix_proxy.log on your proxy that these are "unmatched trap". If you look in that log file, you'll probably find a similar message for every trap you've received?

This might imply that the snmptrap[] regex you're using isn't matching any of the actual traps. Can you test with a really simple regex, to see if that causes it to match, to rule out whether it's an issue with the regex?

Also, there's some comments in the snmp trap setup documentation about items being ignored if they aren't associated with an SNMP interface. I'm talking about item #5 in the workflow list. Can you verify that the hosts that are sending these traps have been configured with an snmp interface in their host definition in Zabbix?

[mfortes]

Uow!! Amazing explanation !!!
I'll investigate that... Thank'u <3