Ad Widget

Collapse

Make Use of SNMP Index Tables

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • mucknet
    Member
    • Dec 2004
    • 59

    #1

    Make Use of SNMP Index Tables

    Howdy --

    I'm a long time zabbix user, but I've been doing a lot of cacti work lately, and while I really don't like its use of rrdtool, I must say it does a pretty good job of taking advantage of SNMP indexes. This allows me to produce a series of graphs for a partition table with meaningful titles, without having to ever actually look and see what the names of the partitions are or what their Total Size are.
    Here is an example of a snmp partition table
    Code:
    snmptable -c public -v 1 localhost HOST-RESOURCES-MIB::hrStorageTable
    SNMP table: HOST-RESOURCES-MIB::hrStorageTable
    
     hrStorageIndex                                hrStorageType           hrStorageDescr hrStorageAllocationUnits hrStorageSize hrStorageUsed hrStorageAllocationFailures
                  1         HOST-RESOURCES-TYPES::hrStorageOther           Memory Buffers               1024 Bytes       1025472         41928                           ?
                  2           HOST-RESOURCES-TYPES::hrStorageRam              Real Memory               1024 Bytes       1025472        894756                           ?
                  3 HOST-RESOURCES-TYPES::hrStorageVirtualMemory               Swap Space               1024 Bytes       4192888           144                           ?
                  4     HOST-RESOURCES-TYPES::hrStorageFixedDisk                        /               4096 Bytes      59033266      32671452                           ?
                  5     HOST-RESOURCES-TYPES::hrStorageFixedDisk                     /sys               4096 Bytes             0             0                           ?
                  6     HOST-RESOURCES-TYPES::hrStorageFixedDisk            /proc/bus/usb               4096 Bytes             0             0                           ?
                  7     HOST-RESOURCES-TYPES::hrStorageFixedDisk                    /boot               1024 Bytes        101018         31441                           ?
                  8     HOST-RESOURCES-TYPES::hrStorageFixedDisk /proc/sys/fs/binfmt_misc               4096 Bytes             0             0                           ?
                  9     HOST-RESOURCES-TYPES::hrStorageFixedDisk  /var/lib/nfs/rpc_pipefs               4096 Bytes             0             0                           ?
                 10     HOST-RESOURCES-TYPES::hrStorageFixedDisk            /proc/fs/nfsd               4096 Bytes             0             0                           ?
                 11     HOST-RESOURCES-TYPES::hrStorageFixedDisk            /home/blah1               4096 Bytes      59033266      32671452                           ?
                 12     HOST-RESOURCES-TYPES::hrStorageFixedDisk               /home/blah2               4096 Bytes      59033266      32671452                           ?
                 13     HOST-RESOURCES-TYPES::hrStorageFixedDisk          /home/blah3               4096 Bytes      59033266      32671452                           ?
                 14     HOST-RESOURCES-TYPES::hrStorageFixedDisk             /home/blah4               4096 Bytes      59033266      32671452                           ?
    And the way it looks using snmpwalk, so the tables are not "assembled"
    Code:
    snmpwalk -c public -v 1 localhost HOST-RESOURCES-MIB::hrStorageTable
    HOST-RESOURCES-MIB::hrStorageIndex.1 = INTEGER: 1
    HOST-RESOURCES-MIB::hrStorageIndex.2 = INTEGER: 2
    HOST-RESOURCES-MIB::hrStorageIndex.3 = INTEGER: 3
    HOST-RESOURCES-MIB::hrStorageIndex.4 = INTEGER: 4
    HOST-RESOURCES-MIB::hrStorageIndex.5 = INTEGER: 5
    HOST-RESOURCES-MIB::hrStorageIndex.6 = INTEGER: 6
    HOST-RESOURCES-MIB::hrStorageIndex.7 = INTEGER: 7
    HOST-RESOURCES-MIB::hrStorageIndex.8 = INTEGER: 8
    HOST-RESOURCES-MIB::hrStorageIndex.9 = INTEGER: 9
    HOST-RESOURCES-MIB::hrStorageIndex.10 = INTEGER: 10
    HOST-RESOURCES-MIB::hrStorageIndex.11 = INTEGER: 11
    HOST-RESOURCES-MIB::hrStorageIndex.12 = INTEGER: 12
    HOST-RESOURCES-MIB::hrStorageIndex.13 = INTEGER: 13
    HOST-RESOURCES-MIB::hrStorageIndex.14 = INTEGER: 14
    HOST-RESOURCES-MIB::hrStorageType.1 = OID: HOST-RESOURCES-TYPES::hrStorageOther
    HOST-RESOURCES-MIB::hrStorageType.2 = OID: HOST-RESOURCES-TYPES::hrStorageRam
    HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
    HOST-RESOURCES-MIB::hrStorageType.4 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageType.5 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageType.7 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageType.8 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageType.9 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageType.10 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageType.11 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageType.12 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageType.13 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageType.14 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
    HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Memory Buffers
    HOST-RESOURCES-MIB::hrStorageDescr.2 = STRING: Real Memory
    HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Swap Space
    HOST-RESOURCES-MIB::hrStorageDescr.4 = STRING: /
    HOST-RESOURCES-MIB::hrStorageDescr.5 = STRING: /sys
    HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: /proc/bus/usb
    HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: /boot
    HOST-RESOURCES-MIB::hrStorageDescr.8 = STRING: /proc/sys/fs/binfmt_misc
    HOST-RESOURCES-MIB::hrStorageDescr.9 = STRING: /var/lib/nfs/rpc_pipefs
    HOST-RESOURCES-MIB::hrStorageDescr.10 = STRING: /proc/fs/nfsd
    HOST-RESOURCES-MIB::hrStorageDescr.11 = STRING: /home/blah1
    HOST-RESOURCES-MIB::hrStorageDescr.12 = STRING: /home/blah2
    HOST-RESOURCES-MIB::hrStorageDescr.13 = STRING: /home/blah3
    HOST-RESOURCES-MIB::hrStorageDescr.14 = STRING: /home/blah4
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.1 = INTEGER: 1024 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.2 = INTEGER: 1024 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.3 = INTEGER: 1024 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.4 = INTEGER: 4096 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.5 = INTEGER: 4096 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.6 = INTEGER: 4096 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.7 = INTEGER: 1024 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.8 = INTEGER: 4096 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.9 = INTEGER: 4096 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.10 = INTEGER: 4096 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.11 = INTEGER: 4096 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.12 = INTEGER: 4096 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.13 = INTEGER: 4096 Bytes
    HOST-RESOURCES-MIB::hrStorageAllocationUnits.14 = INTEGER: 4096 Bytes
    HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 1025472
    HOST-RESOURCES-MIB::hrStorageSize.2 = INTEGER: 1025472
    HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 4192888
    HOST-RESOURCES-MIB::hrStorageSize.4 = INTEGER: 59033266
    HOST-RESOURCES-MIB::hrStorageSize.5 = INTEGER: 0
    HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 0
    HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 101018
    HOST-RESOURCES-MIB::hrStorageSize.8 = INTEGER: 0
    HOST-RESOURCES-MIB::hrStorageSize.9 = INTEGER: 0
    HOST-RESOURCES-MIB::hrStorageSize.10 = INTEGER: 0
    HOST-RESOURCES-MIB::hrStorageSize.11 = INTEGER: 59033266
    HOST-RESOURCES-MIB::hrStorageSize.12 = INTEGER: 59033266
    HOST-RESOURCES-MIB::hrStorageSize.13 = INTEGER: 59033266
    HOST-RESOURCES-MIB::hrStorageSize.14 = INTEGER: 59033266
    HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 42092
    HOST-RESOURCES-MIB::hrStorageUsed.2 = INTEGER: 918772
    HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 144
    HOST-RESOURCES-MIB::hrStorageUsed.4 = INTEGER: 32671656
    HOST-RESOURCES-MIB::hrStorageUsed.5 = INTEGER: 0
    HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 0
    HOST-RESOURCES-MIB::hrStorageUsed.7 = INTEGER: 31441
    HOST-RESOURCES-MIB::hrStorageUsed.8 = INTEGER: 0
    HOST-RESOURCES-MIB::hrStorageUsed.9 = INTEGER: 0
    HOST-RESOURCES-MIB::hrStorageUsed.10 = INTEGER: 0
    HOST-RESOURCES-MIB::hrStorageUsed.11 = INTEGER: 32671656
    HOST-RESOURCES-MIB::hrStorageUsed.12 = INTEGER: 32671656
    HOST-RESOURCES-MIB::hrStorageUsed.13 = INTEGER: 32671656
    HOST-RESOURCES-MIB::hrStorageUsed.14 = INTEGER: 32671656
    This is a very, very commonly used design for SNMP MIB's, not just with disks, but with Interfaces, and many other items.

    For every Partition on the system, there is an index number associated with it. Every OID regarding that partition, ends in that index number. ie: if you look at the snmpwalk above, every item ending in .4 is a statistic for the / filesystem. This is shown very clearly in the snmptable output.

    This allows me to create a generic template and title it using something like:
    Partition $hrstoragedescr Using $hrstorageUsed Megabytes, out of $hrStorageSize.

    Does this make sense?

    I realize zabbix has better ways of getting this particular type of data, but this method applies for a lot of items (specifically I've been working with checkpoint firewalls, equallogic iscsi boxes, etc), and the partition table is an example everyone can query on their own box.

    I'm HOPING that this is the way you've decided to implemenet autodetection.
  • mucknet
    Member
    • Dec 2004
    • 59

    #2
    Here is an example of a cacti XML file for an equallogic iscsi box, where you define the items you want to get out of the Table, and how to find the index. In this case, there is no set index like the partition example above, but you can "make your own" index by just looking at the last 1 (or 2 in this case) octets of the OID (the oid_index_parse parameter). This is a more common problem/feature than one might initially imagine.
    Code:
    <equallogic_volumes>
            <name>Get Equallogic Volume statistics</name>
            <description>Queries an equallogic group for Volume Statistics</description>
            <oid_index>.1.3.6.1.4.1.12740.5.1.7.1.1.4</oid_index>
            <oid_index_parse>OID/REGEXP:.*\.([0-9]*\.[0-9]*)$</oid_index_parse>
            <index_order>eqVolName</index_order>
            <index_order_type>numeric</index_order_type>
            <index_title_format>|chosen_order_field|</index_title_format>
    
            <fields>
                    <eqVolName>
                            <name>Index</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>input</direction>
                            <oid>.1.3.6.1.4.1.12740.5.1.7.1.1.4</oid>
                    </eqVolName>
                    <eqVolPduCmd>
                            <name>Command Pdus</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>output</direction>
                            <oid>.1.3.6.1.4.1.12740.5.1.7.34.1.1</oid>
                    </eqVolPduCmd>
    
                    <eqVolPduRsp>
                            <name>Response Pdus</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>output</direction>
                            <oid>.1.3.6.1.4.1.12740.5.1.7.34.1.2</oid>
                    </eqVolPduRsp>
    
                    <eqVolTxDataBytes>
                            <name>Transmit Data Bytes</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>output</direction>
                            <oid>.1.3.6.1.4.1.12740.5.1.7.34.1.3</oid>
                    </eqVolTxDataBytes>
    
                    <eqVolRxDataBytes>
                            <name>Receive Data Bytes</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>output</direction>
                            <oid>.1.3.6.1.4.1.12740.5.1.7.34.1.4</oid>
                    </eqVolRxDataBytes>
    
                    <eqVolReadLatency>
                            <name>Read Latency</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>output</direction>
                            <oid>.1.3.6.1.4.1.12740.5.1.7.34.1.6</oid>
                    </eqVolReadLatency>
    
                    <eqVolWriteLatency>
                            <name>Write Latency</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>output</direction>
                            <oid>.1.3.6.1.4.1.12740.5.1.7.34.1.7</oid>
                    </eqVolWriteLatency>
            </fields>
    </equallogic_volumes>
    A slightly more straight forward one for the interface MIB, where you simply specify which OID contains the index.
    Code:
    <interface>
            <name>Get SNMP Interfaces</name>
            <description>Queries a host for a list of monitorable interfaces</description>
            <oid_index>.1.3.6.1.2.1.2.2.1.1</oid_index>
            <oid_num_indexes>.1.3.6.1.2.1.2.1.0</oid_num_indexes>
            <index_order>ifDescr:ifName:ifHwAddr:ifIndex</index_order>
            <index_order_type>numeric</index_order_type>
            <index_title_format>|chosen_order_field|</index_title_format>
    
            <fields>
                    <ifIndex>
                            <name>Index</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>input</direction>
                            <oid>.1.3.6.1.2.1.2.2.1.1</oid>
                    </ifIndex>
                    <ifOperStatus>
                            <name>Status</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>input</direction>
                            <oid>.1.3.6.1.2.1.2.2.1.8</oid>
                    </ifOperStatus>
                    <ifDescr>
                            <name>Description</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>input</direction>
                            <oid>.1.3.6.1.2.1.2.2.1.2</oid>
                    </ifDescr>
                    <ifName>
                            <name>Name (IF-MIB)</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>input</direction>
                            <oid>.1.3.6.1.2.1.31.1.1.1.1</oid>
                    </ifName>
                    <ifAlias>
                            <name>Alias (IF-MIB)</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>input</direction>
                            <oid>.1.3.6.1.2.1.31.1.1.1.18</oid>
                    </ifAlias>
                    ...
                    <ifOutErrors>
                            <name>Errors Out</name>
                            <method>walk</method>
                            <source>value</source>
                            <direction>output</direction>
                            <oid>.1.3.6.1.2.1.2.2.1.20</oid>
                    </ifOutErrors>
                    <ifIP>
                            <name>IP Address</name>
                            <method>walk</method>
                            <source>OID/REGEXP:.*\.([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$</source>
                            <direction>input</direction>
                            <oid>.1.3.6.1.2.1.4.20.1.2</oid>
                    </ifIP>
            </fields>
    </interface>
    I cut out a number of the actual OID's grabbed from the SNMP Interface xml file, because its quite long, but it should show you the idea. ifAlias and ifDescr are VERY useful, and allow you to quickly look at your graph and know what your'e looking at (if you have the ports on your switch labeled correctly )


    Hopefully I've explained this adequately, and I can certainly give it another shot if necessary. Aside from this one feature idea that cacti has, I can't stand cacti, and I'm waiting the moment I'm done with this contract and no longer use it
    Last edited by mucknet; 06-04-2007, 03:26.

    Comment

    • mucknet
      Member
      • Dec 2004
      • 59

      #3
      Alexei,

      Do you have any input on this? I'd love to hear your thoughts.

      Comment

      • joluinfante
        Junior Member
        • Jun 2007
        • 18

        #4
        Hi, Forum!
        I see this problem (we can use the value in the snmp tables) very important, and, I did work in the code of the zabbix. I have a first version (I testing now it). This version can use values of key like ".1.3.6.1.2.1.25.2.3.1.5.1,2" (Values of Storage). I did modify the program to convert this "oid" in two oids:
        .1.3.6.1.2.1.25.2.3.1.5.1
        .1.3.6.1.2.1.25.2.3.1.5.2
        ... and return the sum of two values.
        I suppouse is very import a syntax like ".1.3.6.1.2.1.25.2.3.1.5.*". On this case, the program need to walk for the table, and add each value, returning a unique value. But, I haven't this feature (I did work only 5 hours to change the program, and test it).
        For me, is very neccesary, to do the checks in any machine without install an agent.
        I'm attaching the source of zabbix modified (I did work in 1.4.2), and an test.c program, to test the method.
        If the owner of zabbix like this change, I can continue uploading the modified code.
        I'd like to receive your opinion, about the idea.

        TIA
        jorge infante
        Attached Files

        Comment

        • joluinfante
          Junior Member
          • Jun 2007
          • 18

          #5
          Hi, forum!
          I did continue on it.
          I prepare a test.c program to complete the format of input mib's.
          Samples:

          # ./test {host} .1.3.6.1.2.1.25.2.3.1.6.*
          0:HOST-RESOURCES-MIB::hrStorageUsed.1=9459293
          1:HOST-RESOURCES-MIB::hrStorageUsed.2=8126
          9467419 <--- Total size of Storage used

          # ./test {host} .1.3.6.1.2.1.25.2.3.1.6.1,2
          0:.1.3.6.1.2.1.25.2.3.1.6.1=9459293
          1:.1.3.6.1.2.1.25.2.3.1.6.2=8126
          9467419 <--- Total size of Storage used

          # ./test {host} .1.3.6.1.2.1.25.2.3.1.6.1
          0:.1.3.6.1.2.1.25.2.3.1.6.1=9459293
          9459293 <--- Storage used in drive "c"

          # ./test {host} .1.3.6.1.2.1.25.2.3.1.6.2
          0:.1.3.6.1.2.1.25.2.3.1.6.2=8126
          8126 <--- Storage used in drive "d"

          I'm implementing this changes in the source of zabbix.

          TIA
          jorge infante
          Attached Files

          Comment

          • Alexei
            Founder, CEO
            Zabbix Certified Trainer
            Zabbix Certified SpecialistZabbix Certified Professional
            • Sep 2004
            • 5654

            #6
            Thanks for your work. It has to be evaluated and integrated into ZABBIX in the best possible way. Added to our TODO list.
            Alexei Vladishev
            Creator of Zabbix, Product manager
            New York | Tokyo | Riga
            My Twitter

            Comment

            • joluinfante
              Junior Member
              • Jun 2007
              • 18

              #7
              Thanks.
              I'm sending my last source, with the last changes (to include oid.*). I'm thinking is a first idea. It need more work, to do eficient the process (like do a cache of oid's finded or similar).
              jorge
              Attached Files

              Comment

              • mcarbonneaux
                Member
                • Jul 2007
                • 31

                #8
                i'm very interesting on that beceause on alteon/nortel loadbalancer, i've list of virtual IP interface that load balance to target host (real server)...
                on this equipement add many virtual ip regulary...
                and on heach new virtual ip i'm oblige to add new monitoring manualy on zabbix...

                with cacti only one indexed table added and all new virtual ip are monitored when they are added...

                the idea to comulate plutiple value of index a beautyfull but multiple value of an index snmp table while be more more beautyfull!

                Comment

                • amish1975
                  Junior Member
                  • Jul 2013
                  • 14

                  #9
                  Is it a chance to have it implemented?

                  The great value of important SNMP data is available only as indexed tables.
                  There's need not just to summarize table data (as was shown above), but also to filter it out (based on certain criteria (RegExp?) and apply aggregate functions on the resultset (avg, count, etc).

                  The modifications, made my Joluinfante cannot be easily merged into Zabbix > 2.0 as the original source code was totally refactored.
                  Meanwhile the functionality is really HIGHLY DEMANDED, as now the only workaround is to call external scripts, do all the work (snmpwalk, parse the output) externally.

                  Please consider to extend the functionality of index search - something, like:

                  UCD-DISKIO-MIB::diskIOWrites["index","UCD-DISKIO-MIB::diskIODevice","dm-*","sum"]
                  - summarize the "diskIOWrites" values when the "diskIODevice" values match "dm-*" mask (SAN volumes, presented externally);

                  or:

                  HOST-RESOURCES-MIB::hrProcessorLoad["index","HOST-RESOURCES-MIB::hrProcessorFrwID","*","avg"]
                  - average load of all the server's processors

                  or:

                  HOST-RESOURCES-MIB::hrSWRunName["index","HOST-RESOURCES-MIB::hrSWRunName","*oracle*","cnt"]
                  - the number of spawned processes by oracle user

                  Hope this idea will be inflammatory!
                  Thanks!
                  Last edited by amish1975; 07-08-2013, 09:14.

                  Comment

                  • spidernik84
                    Junior Member
                    • Aug 2011
                    • 17

                    #10
                    Yeah I second that. We're trying to get data from a Cisco ACE load balancer SNMP table (rserver status) and the only solution I can currently only think about is the external script + parsing + dinamically create items via API.

                    How did you solve this amish1975?
                    Thanks

                    Comment

                    • amish1975
                      Junior Member
                      • Jul 2013
                      • 14

                      #11
                      Hi, spidernik84,

                      Glad there are two of us ) More chances to attract the developers attention...

                      Not solved yet. I was trying to inspect the source code of "checks_snmp.c" module v2.0.6 (most current for that moment) in order to try to merge changes, proposed by joluinfante for v.1.4.2

                      It was useless as this module was completely refactored (I think - mainly because of this https://www.zabbix.com/documentation...p/dynamicindex functionality).

                      So I duplicated it here:

                      as an enhancement request. Hope it'll be implemented one day...
                      Still no response... (((

                      Regards,
                      amish1975

                      Comment

                      • spidernik84
                        Junior Member
                        • Aug 2011
                        • 17

                        #12
                        I'll try to back you up as much as possible, this would be a nice to have feature!

                        Comment

                        Working...