Ad Widget

Collapse

Appliance 6.0.3 not saving properly values of ILO SNMP requests

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Tpe
    Junior Member
    • May 2022
    • 14

    #1

    Appliance 6.0.3 not saving properly values of ILO SNMP requests

    I'm new with Zabbix and its first time we are testing it out. We are looking if it's something what we could use in future in our environments.
    I downloaded appliance to minimize any possible issues while testing, but I can't get it work with HPE ILO SNMP.
    Values wont save to the zabbix even SNMP requests and receives are fine. At the same time SNMP from other host (pfsense) work and save just fine.

    To reproduce:
    Download and install Zabbix 6.0 LTS 6.0.3 Apr 04, 2022 Open virtualization format (.ovf)
    Update Centos Stream release 8 to latest versions and security patches
    Update HPE ILO4 to latest version and security patches (iLO Firmware Version 2.80 Jan 25 2022)
    From ILO Management - SNMP Settings set parameter Read Community and check that Enable -switch is set on Agentless Management
    From Zabbix configuration - Hosts press Create Host and set following:
    - Host Name: (we use same as in ILO hostname)
    - Templates: HP iLO SNMP
    - Groups: (select/create proper group)
    - Interfaces: select SNMP and give ILO4 IP-address and port (161).
    SNMP version SNMPv2 (if you use v3, need to set credentials)
    SNMP community (leave to {$SNMP_COMMUNITY})
    - Select Macros and add macro {$SNMP_COMMUNITY} with same value which you did set earlier in ILO Read Community
    Press ADD

    It will create host with Items 15, Triggers 9, Graphs 0, Discovery 13 and starts updating the item values which you can check under Monitoring - Latest Data:
    Click image for larger version  Name:	Screenshot 2022-05-03 121523.jpg Views:	0 Size:	60.7 KB ID:	444077
    Now let it run like 5minutes or so and you see, some values start dropping, in this case its System:Temperature status which has not updated at all in last +5min
    Click image for larger version  Name:	Screenshot 2022-05-03 121948.jpg Views:	0 Size:	59.6 KB ID:	444078
    When you check the configuration of that item, you will see that it should update every 50 seconds as set in Custom intervals.
    Even if you take out custom interval, it should update every 1m (1minute).
    Click image for larger version  Name:	Screenshot 2022-05-03 122114.jpg Views:	0 Size:	43.2 KB ID:	444079
    When you check communications between zabbix and ilo4, you can see that zabbix is requesting items and ILO is returning the values to zabbix correcly.
    Code:
    12:24:22.480184 IP zabbixhostname.45432 > ilo4hostnamesnmp: C="yourSNMPsecret" GetRequest(32) E:232.6.2.6.1.0
    12:24:22.483982 IP ilo4hostname.snmp > zabbixhostname.45432: C="yourSNMPsecret" GetResponse(33) E:232.6.2.6.1.0=2
    12:24:26.488680 IP zabbixhostname.46956 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(32) E:232.2.2.4.2.0
    12:24:26.493608 IP ilo4hostname.snmp > zabbixhostname.46956: C="yourSNMPsecret" GetResponse(52) E:232.2.2.4.2.0="ProLiant DL360p"
    12:24:27.493216 IP zabbixhostname.50949 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(32) E:232.2.2.2.1.0
    12:24:27.497498 IP ilo4hostname.snmp > zabbixhostname.50949: C="yourSNMPsecret" GetResponse(48) E:232.2.2.2.1.0=" "
    12:24:31.502696 IP zabbixhostname.36415 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(31) E:232.6.1.3.0
    12:24:31.510035 IP ilo4hostname.snmp > zabbixhostname.36415: C="yourSNMPsecret" GetResponse(32) E:232.6.1.3.0=2
    12:24:32.504910 IP zabbixhostname.57033 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(28) system.sysUpTime.0
    12:24:32.513777 IP ilo4hostname.snmp > zabbixhostname.57033: C="yourSNMPsecret" GetResponse(32) system.sysUpTime.0=22145132
    12:25:02.546221 IP zabbixhostname.52754 > ilo4hostnames.nmp: C="yourSNMPsecret" GetRequest(28) system.sysUpTime.0
    12:25:02.551593 IP ilo4hostname.snmp > zabbixhostname.52754: C="yourSNMPsecret" GetResponse(32) system.sysUpTime.0=22148136
    When you go to configuration - Hosts - discovery of the host, checkmark every line, click enable and execute now, it will discover over 100 new items.
    Every item will update only once, when you add em by discovery. After that around 7-10 of total 122 works properly and rest of the items wont save at all.
    Which values it will update/save seems to be somewhat random.

    Click image for larger version  Name:	Screenshot 2022-05-03 130554.jpg Views:	0 Size:	139.5 KB ID:	444080

    Network traffic between zabbix and ilo4 shows that all values are requested and responded correctly every minute (or what ever interval has been set).
    Code:
    13:14:58.404877 IP zabbixhostname.58337 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.24
    13:14:58.407597 IP ilo4hostname.snmp > zabbixhostname.58337: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.24=2
    13:14:59.409060 IP zabbixhostname.46104 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.25
    13:14:59.412014 IP ilo4hostname.snmp > zabbixhostname.46104: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.25=2
    13:14:59.412521 IP zabbixhostname.38874 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.4.0.7
    13:14:59.415154 IP ilo4hostname.snmp > zabbixhostname.38874: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.4.0.7=24
    13:15:00.417663 IP zabbixhostname.56760 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.4.0.8
    13:15:00.420562 IP ilo4hostname.snmp > zabbixhostname.56760: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.4.0.8=24
    13:15:00.421045 IP zabbixhostname.48469 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.26
    13:15:00.423892 IP ilo4hostname.snmp > zabbixhostname.48469: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.26=2
    13:15:01.435728 IP zabbixhostname.41829 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.4.0.9
    13:15:01.440339 IP ilo4hostname.snmp > zabbixhostname.41829: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.4.0.9=27
    13:15:01.440713 IP zabbixhostname.39000 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.27
    13:15:01.445747 IP ilo4hostname.snmp > zabbixhostname.39000: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.27=2
    13:15:02.447314 IP zabbixhostname.34027 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(28) system.sysUpTime.0
    13:15:02.449826 IP ilo4hostname.snmp > zabbixhostname.34027: C="yourSNMPsecret" GetResponse(32) system.sysUpTime.0=22448126
    13:15:02.450371 IP zabbixhostname.43617 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.28
    13:15:02.453205 IP ilo4hostname.snmp > zabbixhostname.43617: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.28=2
    13:15:02.453532 IP zabbixhostname.46364 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.4.0.10
    13:15:02.456465 IP ilo4hostname.snmp > zabbixhostname.46364: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.4.0.10=24
    13:15:03.458846 IP zabbixhostname.51768 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.29
    13:15:03.463345 IP ilo4hostname.snmp > zabbixhostname.51768: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.29=2
    13:15:03.463929 IP zabbixhostname.53485 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.4.0.11
    13:15:03.469896 IP ilo4hostname.snmp > zabbixhostname.53485: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.4.0.11=23
    13:15:04.472197 IP zabbixhostname.58828 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.30
    13:15:04.475581 IP ilo4hostname.snmp > zabbixhostname.58828: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.30=2
    13:15:05.477716 IP zabbixhostname.45920 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.31
    13:15:05.482015 IP ilo4hostname.snmp > zabbixhostname.45920: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.31=2
    13:15:06.484063 IP zabbixhostname.47022 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.32
    13:15:06.489790 IP ilo4hostname.snmp > zabbixhostname.47022: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.32=2
    13:15:07.491108 IP zabbixhostname.56456 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.33
    13:15:07.494492 IP ilo4hostname.snmp > zabbixhostname.56456: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.33=2
    13:15:08.496696 IP zabbixhostname.39577 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.34
    13:15:08.499903 IP ilo4hostname.snmp > zabbixhostname.39577: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.34=2
    13:15:09.501954 IP zabbixhostname.54787 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.36
    13:15:09.504917 IP ilo4hostname.snmp > zabbixhostname.54787: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.36=2
    13:15:10.506382 IP zabbixhostname.35648 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.37
    13:15:10.509989 IP ilo4hostname.snmp > zabbixhostname.35648: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.37=2
    13:15:11.543022 IP zabbixhostname.40986 > ilo4hostname.snmp: C="yourSNMPsecret" GetRequest(35) E:232.6.2.6.8.1.6.0.38
    13:15:11.545914 IP ilo4hostname.snmp > zabbixhostname.40986: C="yourSNMPsecret" GetResponse(36) E:232.6.2.6.8.1.6.0.38=2
    Zabbix own debug log tells that it receives values via SNMP properly but somehow just wont save em to the database.
    Code:
    End of zbx_snmp_open_session()
    In zbx_snmp_process_standard()
    In zbx_snmp_translate() OID:'1.3.6.1.4.1.232.6.2.6.8.1.4.0.32'
    End of zbx_snmp_translate() oid_translated:'1.3.6.1.4.1.232.6.2.6.8.1.4.0.32'
    In zbx_snmp_get_values() num:1 level:0
    zbx_snmp_get_values() snmp_synch_response() status:0 s_snmp_errno:0 errstat:0 mapping_num:1
    In zbx_snmp_set_result() type:2
    End of zbx_snmp_set_result():SUCCEED
    End of zbx_snmp_get_values():SUCCEED
    End of zbx_snmp_process_standard():SUCCEED
    In zbx_snmp_close_session()
    End of zbx_snmp_close_session()
    End of get_values_snmp()
    Request&respond are identical (except OID value) between items which do update and which do not update, when looked directly from network traffic or zabbix debug log.

    Also it has no effect if you increase StartPollers, StartPollersUnreachable, StartPreprocessors or Timeout from zabbix_server.conf.
    I have tried to disable every single item on the zabbix except two. The one which did update properly from the ILO and the one which did not. Result was that now the working one stopped working as well.
    So we did end up having two enabled items in whole zabbix and it wont save vales from em. Even they are requested from the ILO properly, responded from ILO to zabbix properly and even zabbix log reports that SNMP request was success.
    Testing different templates, HPE ProLiant DL360 SNMP, HPE ProLiant DL380 SNMP and HP iLO SNMP and SNMPv3 instead of SNMPv2 either did not have any effect. Behavior was same on all.
    When you restart the zabbix-server service, most of the items get updated but only one time. After that, again only 10 or less are working.

    Any ideas?
    Last edited by Tpe; 03-05-2022, 13:21.
  • Tpe
    Junior Member
    • May 2022
    • 14

    #2
    I'm not sure if this is relevant, but after set debuglevel to max and comparing two different temperature sensors, other one did save values on zabbix and other one did not, even both got received properly via SNMP, there is small difference.

    Working item:
    80696:20220503:154921.038 In substitute_key_macros_impl() data:'sensor.temp.value[cpqHeTemperatureCelsius.System.0.24]'
    80696:20220503:154921.038 End of zbx_snmp_translate() oid_translated:'1.3.6.1.4.1.232.6.2.6.8.1.4.0.24'
    80696:20220503:154921.147 zbx_snmp_get_values() snmp_synch_response() status:0 s_snmp_errno:0 errstat:0 mapping_num:1
    Code:
    80671:20220503:154921.149 zbx_ipc_socket_read() code:2 size:68 data:2d ab 00 00 00 00 00 00 | 00 01 d1 24 71 62 9d 19 | d7 08 01 03 00 00 00 32 | 33 00 01 00 00 00 00 00 | 00 00 01 03 00 00 00 32 | 34 00 95 24 71 62 0a c4 | b7 04 01 00 00 00 14 03 | 00 00 00 36 68 00 00 01 | 00 00 00 00
    80671:20220503:154921.149 End of zbx_ipc_socket_read():SUCCEED
    80671:20220503:154921.150 worker_preprocess_value(): 23
    80671:20220503:154921.150 worker_preprocess_value: SUCCEED 23
    80671:20220503:154921.150 In zbx_ipc_socket_write()
    80671:20220503:154921.150 End of zbx_ipc_socket_write():SUCCEED
    Not working item:
    80698:20220503:154922.041 In substitute_key_macros_impl() data:'sensor.temp.value[cpqHeTemperatureCelsius.System.0.25]'
    80698:20220503:154922.042 End of zbx_snmp_translate() oid_translated:'1.3.6.1.4.1.232.6.2.6.8.1.4.0.25'
    80698:20220503:154922.046 zbx_snmp_get_values() snmp_synch_response() status:0 s_snmp_errno:0 errstat:0 mapping_num:1
    Code:
    80671:20220503:154922.047 zbx_ipc_socket_read() code:2 size:68 data:2e ab 00 00 00 00 00 00 | 00 01 d2 24 71 62 c8 61 | c4 02 01 03 00 00 00 32 | 33 00 01 00 00 00 00 00 | 00 00 01 03 00 00 00 32 | 33 00 e2 23 71 62 15 ab | e1 2c 01 00 00 00 14 03 | 00 00 00 36 68 00 00 01 | 00 00 00 00
    80671:20220503:154922.047 End of zbx_ipc_socket_read():SUCCEED
    80671:20220503:154922.047 worker_preprocess_value(): 23
    80671:20220503:154922.047 worker_preprocess_value: SUCCEED
    80671:20220503:154922.047 In zbx_ipc_socket_write()
    80671:20220503:154922.047 End of zbx_ipc_socket_write():SUCCEED
    So worker_preprocess_value in working one is "SUCCEED 23" and not working one is "SUCCEED"?
    Also the not working one is missing preprocessor_enqueue_dependent() -task from preprocessor_add_result()
    Code:
    80669:20220503:154921.150 In preprocessor_add_result()
    80669:20220503:154921.150 In preprocessor_enqueue_dependent() itemid: 43821
    80669:20220503:154921.150 End of preprocessor_enqueue_dependent()
    80669:20220503:154921.150 In preprocessor_assign_tasks()
    vs
    Code:
    80669:20220503:154922.048 In preprocessor_add_result()
    80669:20220503:154922.048 In preprocessor_assign_tasks()
    Same as whole:
    Working item:
    Code:
    80696:20220503:154921.148 zbx_setproctitle() title:'poller #2 [got 1 values in 0.110868 sec, idle 1 sec]'
    80696:20220503:154921.148 In zbx_ipc_async_socket_recv() timeout:1
    80669:20220503:154921.148 zbx_ipc_service_recv() code:2 size:81 data:2d ab 00 00 00 00 00 00 | 18 29 00 00 00 00 00 00 | 00 04 00 00 00 00 00 01 | d1 24 71 62 9d 19 d7 08 | 01 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 03 00 00 | 00 32 33 00 00 00 00 00 | 08 00 00 00 00 00 00 00 | 00
    80669:20220503:154921.148 End of zbx_ipc_service_recv():1
    80669:20220503:154921.148 In preprocessor_add_request()
    80669:20220503:154921.148 In preprocessor_sync_configuration()
    80669:20220503:154921.148 In DCconfig_get_preprocessable_items()
    80669:20220503:154921.149 End of DCconfig_get_preprocessable_items() items:15
    80669:20220503:154921.149 End of preprocessor_sync_configuration() item config size: 15, history cache size: 2
    80669:20220503:154921.149 In preprocessor_enqueue() itemid: 43821
    80669:20220503:154921.149 End of preprocessor_enqueue()
    80669:20220503:154921.149 In preprocessor_assign_tasks()
    80669:20220503:154921.149 In preprocessor_get_next_task()
    80669:20220503:154921.149 End of preprocessor_get_next_task()
    80669:20220503:154921.149 In zbx_ipc_client_send() clientid:3
    80669:20220503:154921.149 End of zbx_ipc_client_send():SUCCEED
    80669:20220503:154921.149 In preprocessor_get_next_task()
    80669:20220503:154921.149 End of preprocessor_get_next_task()
    80669:20220503:154921.149 End of preprocessor_assign_tasks()
    80669:20220503:154921.149 End of preprocessor_add_request()
    80669:20220503:154921.149 In zbx_ipc_service_recv() timeout:1.000
    80671:20220503:154921.149 zbx_ipc_socket_read() code:2 size:68 data:2d ab 00 00 00 00 00 00 | 00 01 d1 24 71 62 9d 19 | d7 08 01 03 00 00 00 32 | 33 00 01 00 00 00 00 00 | 00 00 01 03 00 00 00 32 | 34 00 95 24 71 62 0a c4 | b7 04 01 00 00 00 14 03 | 00 00 00 36 68 00 00 01 | 00 00 00 00
    80671:20220503:154921.149 End of zbx_ipc_socket_read():SUCCEED
    80671:20220503:154921.150 worker_preprocess_value(): 23
    80671:20220503:154921.150 worker_preprocess_value: SUCCEED 23
    80671:20220503:154921.150 In zbx_ipc_socket_write()
    80671:20220503:154921.150 End of zbx_ipc_socket_write():SUCCEED
    80671:20220503:154921.150 In zbx_ipc_socket_read()
    80669:20220503:154921.150 zbx_ipc_service_recv() code:3 size:36 data:01 03 00 00 00 32 33 00 | 01 00 00 00 00 00 00 00 | 01 03 00 00 00 32 33 00 | d1 24 71 62 9d 19 d7 08 | 00 00 00 00
    80669:20220503:154921.150 End of zbx_ipc_service_recv():1
    80669:20220503:154921.150 In preprocessor_add_result()
    80669:20220503:154921.150 In preprocessor_enqueue_dependent() itemid: 43821
    80669:20220503:154921.150 End of preprocessor_enqueue_dependent()
    80669:20220503:154921.150 In preprocessor_assign_tasks()
    80669:20220503:154921.150 In preprocessor_get_next_task()
    80669:20220503:154921.150 End of preprocessor_get_next_task()
    80669:20220503:154921.150 End of preprocessor_assign_tasks()
    80669:20220503:154921.150 End of preprocessor_add_result()
    Not working item:
    Code:
    80698:20220503:154922.046 zbx_setproctitle() title:'poller #4 [got 1 values in 0.005209 sec, idle 5 sec]'
    80698:20220503:154922.046 In zbx_ipc_async_socket_recv() timeout:5
    80669:20220503:154922.046 zbx_ipc_service_recv() code:2 size:81 data:2e ab 00 00 00 00 00 00 | 18 29 00 00 00 00 00 00 | 00 04 00 00 00 00 00 01 | d2 24 71 62 c8 61 c4 02 | 01 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 03 00 00 | 00 32 33 00 00 00 00 00 | 08 00 00 00 00 00 00 00 | 00
    80669:20220503:154922.046 End of zbx_ipc_service_recv():1
    80669:20220503:154922.046 In preprocessor_add_request()
    80669:20220503:154922.046 In preprocessor_sync_configuration()
    80669:20220503:154922.046 In DCconfig_get_preprocessable_items()
    80669:20220503:154922.047 End of DCconfig_get_preprocessable_items() items:15
    80669:20220503:154922.047 End of preprocessor_sync_configuration() item config size: 15, history cache size: 2
    80669:20220503:154922.047 In preprocessor_enqueue() itemid: 43822
    80669:20220503:154922.047 End of preprocessor_enqueue()
    80669:20220503:154922.047 In preprocessor_assign_tasks()
    80669:20220503:154922.047 In preprocessor_get_next_task()
    80669:20220503:154922.047 End of preprocessor_get_next_task()
    80669:20220503:154922.047 In zbx_ipc_client_send() clientid:3
    80669:20220503:154922.047 End of zbx_ipc_client_send():SUCCEED
    80669:20220503:154922.047 In preprocessor_get_next_task()
    80669:20220503:154922.047 End of preprocessor_get_next_task()
    80671:20220503:154922.047 zbx_ipc_socket_read() code:2 size:68 data:2e ab 00 00 00 00 00 00 | 00 01 d2 24 71 62 c8 61 | c4 02 01 03 00 00 00 32 | 33 00 01 00 00 00 00 00 | 00 00 01 03 00 00 00 32 | 33 00 e2 23 71 62 15 ab | e1 2c 01 00 00 00 14 03 | 00 00 00 36 68 00 00 01 | 00 00 00 00
    80671:20220503:154922.047 End of zbx_ipc_socket_read():SUCCEED
    80669:20220503:154922.047 End of preprocessor_assign_tasks()
    80669:20220503:154922.047 End of preprocessor_add_request()
    80669:20220503:154922.047 In zbx_ipc_service_recv() timeout:1.000
    80671:20220503:154922.047 worker_preprocess_value(): 23
    80671:20220503:154922.047 worker_preprocess_value: SUCCEED
    80671:20220503:154922.047 In zbx_ipc_socket_write()
    80671:20220503:154922.047 End of zbx_ipc_socket_write():SUCCEED
    80669:20220503:154922.047 zbx_ipc_service_recv() code:3 size:29 data:00 01 00 00 00 00 00 00 | 00 01 03 00 00 00 32 33 | 00 e2 23 71 62 15 ab e1 | 2c 00 00 00 00
    80671:20220503:154922.047 In zbx_ipc_socket_read()
    80669:20220503:154922.048 End of zbx_ipc_service_recv():1
    80669:20220503:154922.048 In preprocessor_add_result()
    80669:20220503:154922.048 In preprocessor_assign_tasks()
    80669:20220503:154922.048 In preprocessor_get_next_task()
    80669:20220503:154922.048 End of preprocessor_get_next_task()
    80669:20220503:154922.048 End of preprocessor_assign_tasks()
    80669:20220503:154922.048 End of preprocessor_add_result()
    btw. maybe this post suits better under troubleshooting and problems :/
    Last edited by Tpe; 03-05-2022, 17:00.

    Comment

    • tim.mooney
      Senior Member
      • Dec 2012
      • 1427

      #3
      Really good job explaining the issue and providing relevant information about the environment and what you've debugged so far.

      I don't have HP ILO in my environment, so I'm not familiar with this particular template. Some of the templates have undergone many revisions and updates over the recent past, especially in the 5.x and 6.x timeframe when the Zabbix developers have greatly improved many of the templates they've developed. That does mean, though, that saying the name of the template sometimes isn't enough to let people know "which version" is being used. With that in mind, was this template pre-installed with the Zabbix appliance, or is it one that you had to download separately? If you downloaded it separately, where did you download it from?

      Final question: for the items that are either pre-defined in the template or that are created dynamically as part of discovery, what is being set for "history storage period"?

      Comment

      • Tpe
        Junior Member
        • May 2022
        • 14

        #4
        Thank you! These are templates which come with the appliance. HP iLO SNMP template description states as follow:
        Template Server HP iLO

        MIBs used:
        CPQHLTH-MIB
        SNMPv2-MIB
        CPQSINFO-MIB
        CPQIDA-MIB

        Template tooling version used: 0.41

        I have not changed any template/item configurations and 0.25 Temp sensor looks like this (temp 0.24 is identical). History storage period seems to be 7d.
        Click image for larger version  Name:	Screenshot 2022-05-04 121949.jpg Views:	0 Size:	70.6 KB ID:	444112
        For the pre-defined items, seems to be set for 1 or 2 weeks, depending of the item.

        Downloaded new 6.04 and 6.2 beta1 versions as well. This time I did not update the OS. Started appliance, set correct timezone to OS with "timedatectl set-timezone Europe/Helsinki" and on zabbix GUI conf changed timezone, theme to dark. Same problem on both versions.

        6.2 beta1 had this new "Execute now" button on latest data. It seems that at least ICMP and Uptime -items do update/save properly every time when I select em and press execute now, but other items do not.
        When I select any of these 7-10 of the 122 items (which do update properly at the time) and press execute now, they seems to update properly in a few seconds.

        Click image for larger version  Name:	Screenshot 2022-05-04 142025.jpg Views:	0 Size:	251.9 KB ID:	444117
        Last edited by Tpe; 04-05-2022, 13:25.

        Comment

        • tim.mooney
          Senior Member
          • Dec 2012
          • 1427

          #5
          Originally posted by Tpe
          For the pre-defined items, seems to be set for 1 or 2 weeks, depending of the item.
          Ok, that confirms it's not what I was guessing: that the template had been designed so that the values weren't being stored.

          I'm guessing you don't have support, but you can still report problems with the product through the free bugtracker support.zabbix.com You've done a thorough job of collecting debug information, perhaps a bug report is in order.

          Comment

          • Tpe
            Junior Member
            • May 2022
            • 14

            #6
            Originally posted by tim.mooney
            I'm guessing you don't have support, but you can still report problems with the product through the free bugtracker support.zabbix.com You've done a thorough job of collecting debug information, perhaps a bug report is in order.
            No I don't have support, I was more like evaluating/testing the product for possible future use.

            Comment

            • Markku
              Senior Member
              Zabbix Certified SpecialistZabbix Certified ProfessionalZabbix Certified Expert
              • Sep 2018
              • 1781

              #7
              I didn't really follow all the details of this troubleshooting but just to be sure: if the item has preprocessing rule set with a "discard unchanged with heartbeat" setting then the discarded values are not shown in Latest data list.

              Markku

              Comment

              • Tpe
                Junior Member
                • May 2022
                • 14

                #8
                Originally posted by Markku
                if the item has preprocessing rule set with a "discard unchanged with heartbeat" setting then the discarded values are not shown in Latest data list.
                This really could be something to do with preprocessing. So it wont save the value if it's same as the previous one?
                Effect can be seen in the dashboard as well, not only in latest data. There is gaps when no data saved.
                Ofc I can fill the gaps if I change missing data setting as "connected", but that doesn't give proper graph e.g. if temperature has gone suddenly up after long period of stable temps.
                If this my issue is with the preprocessing, do you know if there is an option to cache the previous data on preprocessing and when there happens the change, it will save that previous value on time when received and the new changed value as well. Then the graphs would be correct as well.

                EDIT: I see from online manual that there is an history cache. Gonna try to play with it.

                Click image for larger version  Name:	Screenshot 2022-05-10 140858.jpg Views:	0 Size:	186.5 KB ID:	444333
                Last edited by Tpe; 10-05-2022, 13:41.

                Comment

                • Markku
                  Senior Member
                  Zabbix Certified SpecialistZabbix Certified ProfessionalZabbix Certified Expert
                  • Sep 2018
                  • 1781

                  #9
                  The idea of the "discard unchanged" rule is that if the value was the same as before, then it is not saved at all, to save CPU cycles and table space. If that does not suit your needs, then you can remove that rule from the template and then you will have all the data values saved.

                  Markku

                  Comment

                  • Tpe
                    Junior Member
                    • May 2022
                    • 14

                    #10
                    Originally posted by Markku
                    The idea of the "discard unchanged" rule is that if the value was the same as before, then it is not saved at all, to save CPU cycles and table space. If that does not suit your needs, then you can remove that rule from the template and then you will have all the data values saved.
                    Thank you! I did remove preprocessing from the temperature item prototype's and it indeed fixed my issue with saving.

                    Comment

                    Working...