Ad Widget

Collapse

HTTP agent timeout (iDrac redfish)

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Karolis.Bi
    Junior Member
    • Oct 2021
    • 2

    #1

    HTTP agent timeout (iDrac redfish)

    Hello,

    I am using DELL PowerEdge R740 by HTTP template for servers monitoring.


    Made various modifications and added additional items and discovery rules.
    Everything works great expect timeouts.

    Each discovery rule (type HTTP agent) has timeout setting and it maximum value is 60s. Zabbix does not allow setting timeout value to 5m for example.
    Error is: Invalid parameter "/timeout": value must be one of 1-60.

    60 seconds is not enough in order to get response (all data) from iDrac via redfish.
    Some parts like {$API.URL}/redfish/v1/Systems/System.Embedded.1/Storage can have 20+ elements and getting data from all these elements requires more time.
    iDrac redfish api some times can be really slow, especially if it's not latest generation server.

    I have modified discovery rules in order to collect lest data, but 60s timeout is still not enough.

    Can anyone please advise how to overcome this limitation?

    Zabbix version 6.4.1

    Thanks!



    ​​​
  • Karolis.Bi
    Junior Member
    • Oct 2021
    • 2

    #2
    Anyone from Zabbix team maybe could comment on my issue please?

    Comment

  • omgiafs
    Junior Member
    • Dec 2017
    • 12

    #3
    Hi!
    Timeout happens because JavaScript in LLD preprocessing doing next thing: gets list of URLs of all sensor's in Members property of main Sensors page '/redfish/v1/Chassis/System.Embedded.1/Sensors' (by the way, not complete list, but first 50 elements) and tries sequentially get these webpage's content for LLD purposes. I've found that LLD gets about 12-13 pages on average before timeout happens. It's terrible, because chassis can have more than 70 sensors so you have no chances to complete this LLD one time at least.
    But here is a good news: looks like template developers didn't read Redfish specification that good to write a good LLD code. Sad but true. Especially specification's part about queries, like this - http://redfish.dmtf.org/schemas/DSP0...ery-parameters.
    Here's the thing: you don't need to sequentially get ton's of webpages, you can write query which will give you all information you need to use in LLD instead.

    An approximate algorithm for obtaining LLD JSON for sensors:
    1. On the page '/redfish/v1/Chassis/System.Embedded.1/Sensors' we get the number of sensors in [email protected] property. Based on this number, we do this: if the number of sensors is more than 50, then we make the step size 30-40. This is done in order to avoid accidentally exceeding the maximum page size that the web server can serve. 50 - this is maximum collection length Redfish API can get to you. This limit i saw in another Redfish device and IDK it's a part of spec or not, in my case in property [email protected] i got a link like '/redfish/v1/Chassis/System.Embedded.1/Sensors&$skip=50'.

    2. In the do...while loop we do the following: if this is the first iteration, then 'skip' is not used in the query. skip=0 is prohibited by the standard.
    Then make a request /redfish/v1/Chassis/System.Embedded.1/Sensors?$expand=.($levels=1)[email protected],Id, N ame,ReadingType,Status/State&$top={STEP SIZE}&$skip={COUNTER}.
    At each iteration, add elements of the Members array to the common list declared outside the loop.

    Of course, you can use query filter to get rid of sensors in Absent state et cetera, but I advise you NOT to use the filter in requests to the device, because this seems to cause noticeable delays in responses, which can have a bad effect on the final detection result, especially with a large number of iterations. The best option is to use filtering directly in the preprocessing code, after receiving the complete list with information about the sensors.

    3. Next, select from the resulting list elements (sensors) with the desired type (ReadingType): voltage, rotary, temperature; and only those sensors that are physically present (Status.State != 'Absent')

    That's it, you are amazing!
    In 3-4 requests you received information on hundreds of sensors!

    This is why it is so important to read specifications.
    Last edited by omgiafs; 03-05-2024, 20:24.

    Comment

    • hanspeter
      Junior Member
      • Nov 2018
      • 8

      #4
      can you provide a code sample with this stepping technique?

      Comment

      • N1029676
        Junior Member
        • Jun 2024
        • 1

        #5
        I have solved this temporarily to run a proof of concept with about 1.2k servers. The problem I've run into was less that the API was taking more than 10 seconds to respond, but rather the preprocessor was taking more than 10 seconds, even when I set script execution to timeout after 60seconds. It doesn't appear I can adjust that timeout. So to fix this I've used the `$expand` parameter mentioned by omgiafs so instead of sending a query for every entity, it sends one. I've modified some of the preprocessor scripts to then iterate over _that_ response.

        Disclaimer: This is not a good solution and I plan to rewrite this template if we proceed after our proof-of-concept. Until then...

        Dell R740 by HTTP // PSU discovery
        1. Add a query parameter to the discovery rule:
          • Name: $expand
          • Value: *($levels=1)
          • Click image for larger version

Name:	image.png
Views:	1015
Size:	33.2 KB
ID:	485439
        2. Edit the 1st JavaScript function under the Preprocessing tab
          • Empty line 83 that is performing another request (we don't want to run more queries)
          • Since we aren't running anymore queries we need to point the sensor contents to the previous query to extract into.
            • Line 84: Change getField(response.body, 'ReadingType', '') to read sensor.ReadingType
            • Line 85: Change getField(response.body, 'Name', '')to read sensor.Name
            • Click image for larger version

Name:	image.png
Views:	831
Size:	23.0 KB
ID:	485440

        The full content of the preprocessor is:
        Code:
        function request(url) {
            api_request = new HttpRequest();
            api_request.addHeader('Authorization: Basic ' + btoa(params.user + ':' + params.password));
            Zabbix.log(4, '[ DELL ] Sending request: ' + url);
        
            try {
                response = api_request.get(url);
            } catch (error) {
                Zabbix.log(4, '[ DELL ] Get request returned error ' + error);
                throw 'Get request returned error ' + error + '. Check debug log for more information.';
            }
        
            Zabbix.log(4, '[ DELL ] Received response with status code ' +
                api_request.getStatus() + '\n' + response);
        
            if (api_request.getStatus() !== 200) {
                var message = 'Request failed with status code ' + api_request.getStatus();
                if (response !== null) {
                    if (typeof response.message === 'string') {
                        message += ': ' + response.message;
                    }
                }
        
                throw message + ' Check debug log for more information.';
            }
        
            if (response !== null) {
                try {
                    response = JSON.parse(response);
                }
                catch (error) {
                    Zabbix.log(4, '[ DELL ] Failed to parse response.');
                    response = null;
                }
            }
        
            return {
                status: api_request.getStatus(),
                body: response
            };
        }
        
        function getField(object, field, def) {
            var names = field.split('.');
            var name = names.shift();
        
            while (typeof name !== 'undefined') {
                if (typeof object === undefined || typeof object[name] === 'undefined') {
                    return def;
                }
        
                object = object[name];
                name = names.shift();
            }
        
            return object;
        }
        
        try {
            sensors = JSON.parse(value);
        }
        catch (error) {
            Zabbix.log(4, '[ DELL ] Failed to parse response.');
            sensors = null;
        }
        
        var params = {
            url: '{$API.URL}',
            user: '{$API.USER}',
            password: '{$API.PASSWORD}'
        };
        var result = [];
        
        var index = params.url.indexOf('://');
        index = params.url.indexOf('/', (index !== -1) ? (index + 3) : 0);
        if (index !== -1) {
            params.url = params.url.substring(0, index);
        }
        
        if (sensors.Members instanceof Array) {
            sensors.Members.forEach(function (sensor) {
                if (typeof sensor["@odata.id"] === 'string') {
                                                                
                    if (sensor.ReadingType === 'Voltage') {
                        result.push({ '{#SENSOR_NAME}': sensor.Name, '{#ODATA}': sensor["@odata.id"] });
                    }
                }
            });
        }
        
        return JSON.stringify(result);​

        Comment

        Working...