Ad Widget

Collapse

SNMPv3 Timeout after rebooting the monitored device

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • josoko
    Junior Member
    • Feb 2021
    • 25
    #1

    SNMPv3 Timeout after rebooting the monitored device

    Hello

    We have following issue:
    We are monitoring FortiAnlyzer 6.2.6 by using SNMPv3, nativ Zabbix ICMP tests and checking system ressourcec by external scripts.
    If we do a restart of the FortiAnlyzer, Zabbix is not able anymore to get SNMP values back from the device. The other monitors are still working. The same behavior occurs in Zabbix 4.4 and Zabbix 5.0.
    We are monitoring the devices by using the method "PrivAuth".

    In Zabbix 5.0 we are getting back this in "sudo systemctl status zabbix-server":
    - Feb 17 15:57:51 vsv-0590 systemd[1]: zabbix-server.service: Supervising process 139939 which is not our child. We'll most likely not notice when it exits.

    The SNMPB status is:
    - Timeout while connecting to "faz-slan.inf.seclab.ch:161".

    After we are doing a "sudo systemctl restart zabbix-server", the FortiAnalyzer can be monitored again by SNMPv3.


    But we can see that the L3 connection between the Zabbix Server and the FortiAnalyzer is still working:
    [scsadmin@vsv-0590 ~]$ sudo tcpdump -nnnvi any 'host 10.25.5.170'
    dropped privs to tcpdump
    tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
    13:44:16.104483 IP (tos 0x0, ttl 64, id 37468, offset 0, flags [DF], proto ICMP (1), length 84)
    10.25.5.90 > 10.25.5.170: ICMP echo request, id 53622, seq 1, length 64
    13:44:16.104736 IP (tos 0x0, ttl 64, id 12567, offset 0, flags [none], proto ICMP (1), length 84)
    10.25.5.170 > 10.25.5.90: ICMP echo reply, id 53622, seq 1, length 64
    13:44:17.105532 IP (tos 0x0, ttl 64, id 37981, offset 0, flags [DF], proto ICMP (1), length 84)
    10.25.5.90 > 10.25.5.170: ICMP echo request, id 53622, seq 3, length 64
    13:44:17.105827 IP (tos 0x0, ttl 64, id 12681, offset 0, flags [none], proto ICMP (1), length 84)
    10.25.5.170 > 10.25.5.90: ICMP echo reply, id 53622, seq 3, length 64
    13:44:18.106554 IP (tos 0x0, ttl 64, id 38765, offset 0, flags [DF], proto ICMP (1), length 84)
    10.25.5.90 > 10.25.5.170: ICMP echo request, id 53622, seq 5, length 64
    13:44:18.106777 IP (tos 0x0, ttl 64, id 12840, offset 0, flags [none], proto ICMP (1), length 84)
    10.25.5.170 > 10.25.5.90: ICMP echo reply, id 53622, seq 5, length 64
    13:44:21.113178 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.25.5.90 tell 10.25.5.170, length 46
    13:44:21.113200 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.25.5.90 is-at 00:50:56:a8:e4:8a, length 28
    13:44:21.492475 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.25.5.170 tell 10.25.5.90, length 28
    13:44:21.492664 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.25.5.170 is-at 00:0c:29:1a:cb:68, length 46
    13:44:25.558394 IP (tos 0x0, ttl 64, id 54724, offset 0, flags [DF], proto UDP (17), length 92)
    10.25.5.90.33236 > 10.25.5.170.161: { SNMPv3 { F=r } { USM B=0 T=0 U="" } { ScopedPDU E= C="" { GetRequest(14) R=655575187 } } }
    13:44:25.558674 IP (tos 0x0, ttl 64, id 22564, offset 0, flags [DF], proto UDP (17), length 153)
    10.25.5.170.161 > 10.25.5.90.33236: { SNMPv3 { F= } { USM B=0 T=10356 U="" } { ScopedPDU E=_80_00_30_44_04_46_41_5a_2d_56_4d_30_30_30_30_30 _37_32_34_31_38 C="" { Report(32) R=655575187 .1.3.6.1.6.3.15.1.1.4.0=3000 } } }
    13:44:25.558743 IP (tos 0x0, ttl 64, id 54725, offset 0, flags [DF], proto UDP (17), length 188)
    10.25.5.90.33236 > 10.25.5.170.161: { SNMPv3 { F=apr } { USM B=0 T=64217 U="ausername" } { ScopedPDU [!scoped PDU]84_88_2c_1a_04_80_cd_a0_a5_35_d8_72_8c_10_09_b2_92 _1b_87_8c_78_54_88_7d_1a_4e_8c_c8_f3_a8_1e_7c_f9_e 0_1d_8c_60_eb_f6_b4_a1_b6_c9_bb_1d_0e_4e_7a_b2_c5_ a7_e3_26_0f_f7_39_b1_e1_a1_98_4c_98} }
    13:44:25.558890 IP (tos 0x0, ttl 64, id 22565, offset 0, flags [DF], proto UDP (17), length 173)
    10.25.5.170.161 > 10.25.5.90.33236: { SNMPv3 { F=a } { USM B=0 T=10360 U="ausername" } { ScopedPDU E=_80_00_30_44_04_46_41_5a_2d_56_4d_30_30_30_30_30 _37_32_34_31_38 C="" { Report(29) R=0 .1.3.6.1.6.3.15.1.1.2.0=416 } } }

    We would appreciate much, if somebody could send us a hint or explain to us, why we must restart the Zabbix Service, to be able to collect the FortiAnlyzer SNMPv3 values again.Hello

    We have following issue:
    We are monitoring FortiAnlyzer 6.2.6 by using SNMPv3, nativ Zabbix ICMP tests and checking system ressourcec by external scripts.
    If we do a restart of the FortiAnlyzer, Zabbix is not able anymore to get SNMP values back from the device. The other monitors are still working. The same behavior occurs in Zabbix 4.4 and Zabbix 5.0.
    We are monitoring the devices by using the method "PrivAuth".

    In Zabbix 5.0 we are getting back this in "sudo systemctl status zabbix-server":
    - Feb 17 15:57:51 vsv-0590 systemd[1]: zabbix-server.service: Supervising process 139939 which is not our child. We'll most likely not notice when it exits.

    The SNMPB status is:
    - Timeout while connecting to "faz-slan.inf.seclab.ch:161".

    After we are doing a "sudo systemctl restart zabbix-server", the FortiAnalyzer can be monitored again by SNMPv3.


    But we can see that the L3 connection between the Zabbix Server and the FortiAnalyzer is still working:
    [scsadmin@vsv-0590 ~]$ sudo tcpdump -nnnvi any 'host 10.25.5.170'
    dropped privs to tcpdump
    tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
    13:44:16.104483 IP (tos 0x0, ttl 64, id 37468, offset 0, flags [DF], proto ICMP (1), length 84)
    10.25.5.90 > 10.25.5.170: ICMP echo request, id 53622, seq 1, length 64
    13:44:16.104736 IP (tos 0x0, ttl 64, id 12567, offset 0, flags [none], proto ICMP (1), length 84)
    10.25.5.170 > 10.25.5.90: ICMP echo reply, id 53622, seq 1, length 64
    13:44:17.105532 IP (tos 0x0, ttl 64, id 37981, offset 0, flags [DF], proto ICMP (1), length 84)
    10.25.5.90 > 10.25.5.170: ICMP echo request, id 53622, seq 3, length 64
    13:44:17.105827 IP (tos 0x0, ttl 64, id 12681, offset 0, flags [none], proto ICMP (1), length 84)
    10.25.5.170 > 10.25.5.90: ICMP echo reply, id 53622, seq 3, length 64
    13:44:18.106554 IP (tos 0x0, ttl 64, id 38765, offset 0, flags [DF], proto ICMP (1), length 84)
    10.25.5.90 > 10.25.5.170: ICMP echo request, id 53622, seq 5, length 64
    13:44:18.106777 IP (tos 0x0, ttl 64, id 12840, offset 0, flags [none], proto ICMP (1), length 84)
    10.25.5.170 > 10.25.5.90: ICMP echo reply, id 53622, seq 5, length 64
    13:44:21.113178 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.25.5.90 tell 10.25.5.170, length 46
    13:44:21.113200 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.25.5.90 is-at 00:50:56:a8:e4:8a, length 28
    13:44:21.492475 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.25.5.170 tell 10.25.5.90, length 28
    13:44:21.492664 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.25.5.170 is-at 00:0c:29:1a:cb:68, length 46
    13:44:25.558394 IP (tos 0x0, ttl 64, id 54724, offset 0, flags [DF], proto UDP (17), length 92)
    10.25.5.90.33236 > 10.25.5.170.161: { SNMPv3 { F=r } { USM B=0 T=0 U="" } { ScopedPDU E= C="" { GetRequest(14) R=655575187 } } }
    13:44:25.558674 IP (tos 0x0, ttl 64, id 22564, offset 0, flags [DF], proto UDP (17), length 153)
    10.25.5.170.161 > 10.25.5.90.33236: { SNMPv3 { F= } { USM B=0 T=10356 U="" } { ScopedPDU E=_80_00_30_44_04_46_41_5a_2d_56_4d_30_30_30_30_30 _37_32_34_31_38 C="" { Report(32) R=655575187 .1.3.6.1.6.3.15.1.1.4.0=3000 } } }
    13:44:25.558743 IP (tos 0x0, ttl 64, id 54725, offset 0, flags [DF], proto UDP (17), length 188)
    10.25.5.90.33236 > 10.25.5.170.161: { SNMPv3 { F=apr } { USM B=0 T=64217 U="ausername" } { ScopedPDU [!scoped PDU]84_88_2c_1a_04_80_cd_a0_a5_35_d8_72_8c_10_09_b2_92 _1b_87_8c_78_54_88_7d_1a_4e_8c_c8_f3_a8_1e_7c_f9_e 0_1d_8c_60_eb_f6_b4_a1_b6_c9_bb_1d_0e_4e_7a_b2_c5_ a7_e3_26_0f_f7_39_b1_e1_a1_98_4c_98} }
    13:44:25.558890 IP (tos 0x0, ttl 64, id 22565, offset 0, flags [DF], proto UDP (17), length 173)
    10.25.5.170.161 > 10.25.5.90.33236: { SNMPv3 { F=a } { USM B=0 T=10360 U="ausername" } { ScopedPDU E=_80_00_30_44_04_46_41_5a_2d_56_4d_30_30_30_30_30 _37_32_34_31_38 C="" { Report(29) R=0 .1.3.6.1.6.3.15.1.1.2.0=416 } } }

    We would appreciate much, if somebody could send us a hint or explain to us, why we must restart the Zabbix Service, to be able to collect the FortiAnlyzer SNMPv3 values again.
    Last edited by vitalijs.m; 23-02-2021, 14:07.
    Tags: v3 faz timeout snmp
    • josoko
      Junior Member
      • Feb 2021
      • 25
      #2
      We found the issue. Fortinet FAZ/FMG is not initializing the boots field with the epochal unix timestamp. Fortinet FOS initialize the boots filed with the epochal unix timestamp on each reboot.

      That defeats the purpose of this attribute:

      Code:
      Engine boots value is the number of times authoritative SNMP engine has been started, booted, executed, initialized, or assumed any other state that can be called “booted”.
Engine time is the number of seconds since the last time authoritative SNMP engine has been ”booted”.
These two values, together, are used for timeliness check.
      By not updating the engineBoots these devices make it impossible for a client - which is keeping track of these numbers - to know the agent was rebooted.


      Checking the EnginID Values by using the snmpget:

      Code:
      AFTER Reboot FOS

 snmpget -v3 -DALL -l authPriv -u <USER> -a SHA -A <AUTHPRIV> -x AES -X <PRIVPASS> <FOS IP> .1.3.6.1.2.1.1.1.0 2>&1 | grep -A1 "lcd_set_enginetime" -A2 -B2
trace: sc_get_openssl_hashfn(): scapi.c, 630:
trace: set_enginetime(): lcd_time.c, 393:
lcd_set_enginetime: engineID 80 00 1F 88 80 3F 70 B8 36 CD AE 2F 60 00 00 00
00 : boots=1, time=0
trace: snmp_call_callbacks(): callback.c, 358:
--
trace: sc_get_openssl_hashfn(): scapi.c, 630:
trace: set_enginetime(): lcd_time.c, 393:
lcd_set_enginetime: engineID 80 00 30 44 04 46 47 56 4D 31 56 54 4D 31 39 30
30 30 34 30 34 : boots=0, time=0
trace: usm_check_secLevel(): snmpusm.c, 3576:
--
trace: sc_get_openssl_hashfn(): scapi.c, 630:
trace: set_enginetime(): lcd_time.c, 393:
lcd_set_enginetime: engineID 80 00 30 44 04 46 47 56 4D 31 56 54 4D 31 39 30
30 30 34 30 34 : boots=1613737393, time=280
trace: usm_process_in_msg(): snmpusm.c, 2791:
--
trace: sc_get_openssl_hashfn(): scapi.c, 630:
trace: set_enginetime(): lcd_time.c, 393:
lcd_set_enginetime: engineID 80 00 30 44 04 46 47 56 4D 31 56 54 4D 31 39 30
30 30 34 30 34 : boots=0, time=0
trace: usm_get_user_from_list(): snmpusm.c, 3699:
--
trace: sc_get_openssl_hashfn(): scapi.c, 630:
trace: set_enginetime(): lcd_time.c, 393:
lcd_set_enginetime: engineID 80 00 30 44 04 46 47 56 4D 31 56 54 4D 31 39 30
30 30 34 30 34 : boots=1613737393, time=280
trace: sc_get_privtype(): scapi.c, 352:
      Code:
      AFTER Reboot FAZ
 snmpget -v3 -DALL -l authPriv -u <USER> -a SHA -A <AUTHPRIV> -x AES -X <PRIVPASS> <FAZ-IP> .1.3.6.1.2.1.1.1.0 2>&1 | grep -A1 "lcd_set_enginetime" -A2 -B2
trace: sc_get_openssl_hashfn(): scapi.c, 630:
trace: set_enginetime(): lcd_time.c, 393:
lcd_set_enginetime: engineID 80 00 1F 88 80 7C 1C A2 64 0C AF 2F 60 00 00 00
00 : boots=1, time=0
trace: snmp_call_callbacks(): callback.c, 358:
--
trace: sc_get_openssl_hashfn(): scapi.c, 630:
trace: set_enginetime(): lcd_time.c, 393:
lcd_set_enginetime: engineID 80 00 30 44 04 46 41 5A 2D 56 4D 30 30 30 30 30
37 32 34 31 38 : boots=0, time=0
trace: usm_check_secLevel(): snmpusm.c, 3576:
--
trace: sc_get_openssl_hashfn(): scapi.c, 630:
trace: set_enginetime(): lcd_time.c, 393:
lcd_set_enginetime: engineID 80 00 30 44 04 46 41 5A 2D 56 4D 30 30 30 30 30
37 32 34 31 38 : boots=0, time=252
trace: usm_process_in_msg(): snmpusm.c, 2791:
--
trace: sc_get_openssl_hashfn(): scapi.c, 630:
trace: set_enginetime(): lcd_time.c, 393:
lcd_set_enginetime: engineID 80 00 30 44 04 46 41 5A 2D 56 4D 30 30 30 30 30
37 32 34 31 38 : boots=0, time=0
trace: usm_get_user_from_list(): snmpusm.c, 3699:
--
trace: sc_get_openssl_hashfn(): scapi.c, 630:
trace: set_enginetime(): lcd_time.c, 393:
lcd_set_enginetime: engineID 80 00 30 44 04 46 41 5A 2D 56 4D 30 30 30 30 30
37 32 34 31 38 : boots=0, time=253
trace: sc_get_privtype(): scapi.c, 352:
      Summary:
      • FOS initializes boots to the current epoch timestamp => changed from 1613729187 (Fri Feb 19 2021 10:06:27 GMT+0000) to 1613737393 (Fri Feb 19 2021 12:23:13 GMT+0000)
      • FAZ doesn't initialize boots to the current epoch timestamp => it's still 0
      Last edited by josoko; 20-02-2021, 00:02.

        Comment

        Previous template Next
        Working...