This is a translation of the original English documentation page. Help us make it better.

8 Internal checks

Overview

Internal checks allow to monitor the internal processes of Zabbix. In other words, you can monitor what goes on with Zabbix server or Zabbix proxy.

Internal checks are calculated:

  • on Zabbix server - if the host is monitored by server
  • on Zabbix proxy - if the host is monitored by proxy

Internal checks are processed by server or proxy regardless of host maintenance status.

To use this item, choose the Zabbix internal item type.

Internal checks are processed by Zabbix pollers.

Performance

Using some internal items may negatively affect performance. These items are:

  • zabbix[host,,items]
  • zabbix[host,,items_unsupported]
  • zabbix[hosts]
  • zabbix[items]
  • zabbix[items_unsupported]
  • zabbix[queue]
  • zabbix[required_performance]
  • zabbix[stats,,,queue]
  • zabbix[triggers]

The System information and Queue frontend sections are also affected.

Supported checks

  • Parameters without angle brackets are constants - for example, 'host' and 'available' in zabbix[host,<type>,available]. Use them in the item key as is.
  • Values for items and item parameters that are "not supported on proxy" can only be gathered if the host is monitored by server. And vice versa, values "not supported on server" can only be gathered if the host is monitored by proxy.
Key
Description Return value Comments
zabbix[boottime]
Startup time of Zabbix server or Zabbix proxy process in seconds. Integer.
zabbix[cluster,discovery,nodes]
Discover high availability cluster nodes. JSON. This item can be used in low-level discovery.
zabbix[history]
Number of values stored in the HISTORY table. Integer. Do not use if MySQL InnoDB, Oracle or PostgreSQL is used!
(not supported on proxy)
zabbix[history_log]
Number of values stored in the HISTORY_LOG table. Integer. Do not use if MySQL InnoDB, Oracle or PostgreSQL is used!
(not supported on proxy)
zabbix[history_str]
Number of values stored in the HISTORY_STR table. Integer. Do not use if MySQL InnoDB, Oracle or PostgreSQL is used!
(not supported on proxy)
zabbix[history_text]
Number of values stored in the HISTORY_TEXT table. Integer. Do not use if MySQL InnoDB, Oracle or PostgreSQL is used!
(not supported on proxy)
zabbix[history_uint]
Number of values stored in the HISTORY_UINT table. Integer. Do not use if MySQL InnoDB, Oracle or PostgreSQL is used!
This item is supported since Zabbix 1.8.3.
(not supported on proxy)
zabbix[host,,items]
Number of enabled items (supported and not supported) on the host. Integer. This item is supported since Zabbix 3.0.0.
zabbix[host,,items_unsupported]
Number of enabled unsupported items on the host. Integer. This item is supported since Zabbix 3.0.0.*
zabbix[host,,maintenance]
Current maintenance status of a host. 0 - host in normal state,
1 - host in maintenance with data collection,
2 - host in maintenance without data collection.
This item is always processed by Zabbix server regardless of host location (on server or proxy). The proxy will not receive this item with configuration data.
The second parameter must be empty and is reserved for future use.
zabbix[host,discovery,interfaces]
Details of all configured interfaces of the host in Zabbix frontend. JSON object. This item can be used in low-level discovery.
This item is supported since Zabbix 3.4.0.
(not supported on proxy)
zabbix[host,<type>,available]
Availability of the main interface of a particular type of checks on the host. 0 - not available, 1 - available, 2 - unknown. Valid types are:
agent, snmp, ipmi, jmx

The item value is calculated according to configuration parameters regarding host unreachability/unavailability.

This item is supported since Zabbix 2.0.0.
zabbix[hosts]
Number of monitored hosts. Integer.
zabbix[items]
Number of enabled items (supported and not supported). Integer.
zabbix[items_unsupported]
Number of not supported items. Integer.
zabbix[java,,<param>]
Information about Zabbix Java gateway. If <param> is ping, "1" is returned. Can be used to check Java gateway availability using nodata() trigger function.

If <param> is version, version of Java gateway is returned. Example: "2.0.0".
Valid values for param are:
ping, version

Second parameter must be empty and is reserved for future use.
zabbix[lld_queue]
Count of values enqueued in the low-level discovery processing queue. Integer. This item can be used to monitor the low-level discovery processing queue length.

This item is supported since Zabbix 4.2.0.
zabbix[preprocessing_queue]
Count of values enqueued in the preprocessing queue. Integer. This item can be used to monitor the preprocessing queue length.

This item is supported since Zabbix 3.4.0.
zabbix[process,<type>,<mode>,<state>]
Time a particular Zabbix process or a group of processes (identified by <type> and <mode>) spent in <state> in percentage. It is calculated for the last minute only.

If <mode> is Zabbix process number that is not running (for example, with 5 pollers running <mode> is specified to be 6), such an item will turn into unsupported state.
Minimum and maximum refers to the usage percentage for a single process. So if in a group of 3 pollers usage percentages per process were 2, 18 and 66, min would return 2 and max would return 66.
Processes report what they are doing in shared memory and the self-monitoring process summarizes that data each second. State changes (busy/idle) are registered upon change - thus a process that becomes busy registers as such and doesn't change or update the state until it becomes idle. This ensures that even fully hung processes will be correctly registered as 100% busy.
Currently, "busy" means "not sleeping", but in the future additional states might be introduced - waiting for locks, performing database queries, etc.
On Linux and most other systems, resolution is 1/100 of a second.
Percentage of time.
Float.
Supported types of server processes:
alert manager, alert syncer, alerter, availability manager, configuration syncer, discoverer, escalator, history poller, history syncer, housekeeper, http poller, icmp pinger, ipmi manager, ipmi poller, java poller, lld manager, lld worker, odbc poller, poller, preprocessing manager, preprocessing worker, proxy poller, self-monitoring, snmp trapper, task manager, timer, trapper, unreachable poller, vmware collector

Supported types of proxy processes:
availability manager, configuration syncer, data sender, discoverer, heartbeat sender, history poller, history syncer, housekeeper, http poller, icmp pinger, ipmi manager, ipmi poller, java poller, odbs poller, poller, preprocessing manager, preprocessing worker, self-monitoring, snmp trapper, task manager, trapper, unreachable poller, vmware collector

Valid modes are:
avg - average value for all processes of a given type (default)
count - returns number of forks for a given process type, <state> should not be specified
max - maximum value
min - minimum value
<process number> - process number (between 1 and the number of pre-forked instances). For example, if 4 trappers are running, the value is between 1 and 4.

Valid states are:
busy - process is in busy state, for example, processing request (default).
idle - process is in idle state doing nothing.

Examples:
=> zabbix[process,poller,avg,busy] → average time of poller processes spent doing something during the last minute
=> zabbix[process,"icmp pinger",max,busy] → maximum time spent doing something by any ICMP pinger process during the last minute
=> zabbix[process,"history syncer",2,busy] → time spent doing something by history syncer number 2 during the last minute
=> zabbix[process,trapper,count] → amount of currently running trapper processes
zabbix[proxy,<name>,<param>]
Information about Zabbix proxy. Integer. name: proxy name

Valid values for param are:
lastaccess - timestamp of last heart beat message received from proxy
delay - how long collected values are unsent, calculated as "proxy delay" (difference between the current proxy time and the timestamp of the oldest unsent value on proxy) + ("current server time" - "proxy lastaccess")

Example:
=> zabbix[proxy,"Germany",lastaccess]

fuzzytime() function can be used to check availability of proxies.
This item is always processed by Zabbix server regardless of host location (on server or proxy).
zabbix[proxy_history]
Number of values in the proxy history table waiting to be sent to the server. Integer. (not supported on server)
zabbix[queue,<from>,<to>]
Number of monitored items in the queue which are delayed at least by <from> seconds but less than by <to> seconds. Integer. from - default: 6 seconds
to - default: infinity
Time-unit symbols (s,m,h,d,w) are supported for these parameters.
zabbix[rcache,<cache>,<mode>]
Availability statistics of Zabbix configuration cache. Integer (for size); float (for percentage). cache: buffer

Valid modes are:
total - total size of buffer
free - size of free buffer
pfree - percentage of free buffer
used - size of used buffer
pused - percentage of used buffer

pused mode is supported since Zabbix 4.0.0.
zabbix[requiredperformance]
Required performance of Zabbix server or Zabbix proxy, in new values per second expected. Float. Approximately correlates with "Required server performance, new values per second" in Reports → System information.
zabbix[stats,<ip>,<port>]
Remote Zabbix server or proxy internal metrics. JSON object. ip - IP/DNS/network mask list of servers/proxies to be remotely queried (default is 127.0.0.1)
port - port of server/proxy to be remotely queried (default is 10051)

Note that the stats request will only be accepted from the addresses listed in the 'StatsAllowedIP' server/proxy parameter on the target instance.

A selected set of internal metrics is returned by this item. For details, see Remote monitoring of Zabbix stats.

Supported since 4.2.0.
zabbix[stats,<ip>,<port>,queue,<from>,<to>]
Remote Zabbix server or proxy internal queue metrics (see zabbix[queue,<from>,<to>]). JSON object. ip - IP/DNS/network mask list of servers/proxies to be remotely queried (default is 127.0.0.1)
port - port of server/proxy to be remotely queried (default is 10051)
from - delayed by at least (default is 6 seconds)
to - delayed by at most (default is infinity)

Note that the stats request will only be accepted from the addresses listed in the 'StatsAllowedIP' server/proxy parameter on the target instance.

Supported since 4.2.0.
zabbix[tcache,cache,<parameter>]
Effectiveness statistics of the Zabbix trend function cache. Integer (for size); float (for percentage). Valid parameters are:
all - total cache requests (default)
hits - cache hits
phits - percentage of cache hits
misses - cache misses
pmisses - percentage of cache misses
items - the number of cached items
requests - the number of cached requests
pitems - percentage of cached items from cached items + requests. Low percentage most likely means that the cache size can be reduced.

Supported since 5.4.0.

(not supported on proxy)
zabbix[trends]
Number of values stored in the TRENDS table. Integer. Do not use if MySQL InnoDB, Oracle or PostgreSQL is used!
(not supported on proxy)
zabbix[trends_uint]
Number of values stored in the TRENDS_UINT table. Integer. Do not use if MySQL InnoDB, Oracle or PostgreSQL is used!
This item is supported since Zabbix 1.8.3.
(not supported on proxy)
zabbix[triggers]
Number of enabled triggers in Zabbix database, with all items enabled on enabled hosts. Integer. (not supported on proxy)
zabbix[uptime]
Uptime of Zabbix server or Zabbix proxy process in seconds. Integer.
zabbix[vcache,buffer,<mode>]
Availability statistics of Zabbix value cache. Integer (for size); float (for percentage). Valid modes are:
total - total size of buffer
free - size of free buffer
pfree - percentage of free buffer
used - size of used buffer
pused - percentage of used buffer

(not supported on proxy)
zabbix[vcache,cache,<parameter>]
Effectiveness statistics of Zabbix value cache. Integer.

With the mode parameter:
0 - normal mode,
1 - low memory mode
Valid parameter values are:
requests - total number of requests
hits - number of cache hits (history values taken from the cache)
misses - number of cache misses (history values taken from the database)
mode - value cache operating mode

This item is supported since Zabbix 2.2.0 and the mode parameter since Zabbix 3.0.0.
(not supported on proxy)

Once the low memory mode has been switched on, the value cache will remain in this state for 24 hours, even if the problem that triggered this mode is resolved sooner.

You may use this key with the Change per second preprocessing step in order to get values per second statistics.
zabbix[version]
Version of Zabbix server or proxy. String. This item is supported since Zabbix 5.0.0.

Example of return value: 5.0.0beta1
zabbix[vmware,buffer,<mode>]
Availability statistics of Zabbix vmware cache. Integer (for size); float (for percentage). Valid modes are:
total - total size of buffer
free - size of free buffer
pfree - percentage of free buffer
used - size of used buffer
pused - percentage of used buffer
zabbix[wcache,<cache>,<mode>]
Statistics and availability of Zabbix write cache. Specifying <cache> is mandatory.
Cache Mode
values all
(default)
Total number of values processed by Zabbix server or Zabbix proxy, except unsupported items. Integer. Counter.
You may use this key with the Change per second preprocessing step in order to get values per second statistics.
float Number of processed float values. Integer. Counter.
uint Number of processed unsigned integer values. Integer. Counter.
str Number of processed character/string values. Integer. Counter.
log Number of processed log values. Integer. Counter.
text Number of processed text values. Integer. Counter.
not supported Number of times item processing resulted in item becoming unsupported or keeping that state. Integer. Counter.
history pfree
(default)
Percentage of free history buffer. Float. History cache is used to store item values. A low number indicates performance problems on the database side.
free Size of free history buffer. Integer.
total Total size of history buffer. Integer.
used Size of used history buffer. Integer.
pused Percentage of used history buffer. Float. pused mode is supported since Zabbix 4.0.0.
index pfree
(default)
Percentage of free history index buffer. Float. History index cache is used to index values stored in history cache.
Index cache is supported since Zabbix 3.0.0.
free Size of free history index history buffer. Integer.
total Total size of history index history buffer. Integer.
used Size of used history index history buffer. Integer.
pused Percentage of used history index buffer. Float. pused mode is supported since Zabbix 4.0.0.
trend pfree
(default)
Percentage of free trend cache. Float. Trend cache stores aggregate for the current hour for all items that receive data.
(not supported on proxy)
free Size of free trend buffer. Integer. (not supported on proxy)
total Total size of trend buffer. Integer. (not supported on proxy)
used Size of used trend buffer. Integer. (not supported on proxy)
pused Percentage of used trend buffer. Float. (not supported on proxy)

pused mode is supported since Zabbix 4.0.0.

Item key details

  • Parameters without angle brackets are constants - for example, 'host' and 'available' in zabbix[host,<type>,available]. Use them in the item key as is.
  • Values for items and item parameters that are "not supported on proxy" can only be retrieved if the host is monitored by server. And vice versa, values "not supported on server" can only be retrieved if the host is monitored by proxy.
zabbix[boottime]


The startup time of Zabbix server or Zabbix proxy process in seconds.
Return value: Integer.

zabbix[cluster,discovery,nodes]


Discovers the high availability cluster nodes.
Return value: JSON object.

This item can be used in low-level discovery.

zabbix[connector_queue]


The count of values enqueued in the connector queue.
Return value: Integer.

This item is supported since Zabbix 6.4.0.

zabbix[host,,items]


The number of enabled items (supported and not supported) on the host.
Return value: Integer.

zabbix[host,,items_unsupported]


The number of enabled unsupported items on the host.
Return value: Integer.

zabbix[host,,maintenance]


The current maintenance status of the host.
Return values: 0 - normal state; 1 - maintenance with data collection; 2 - maintenance without data collection.

Comments:

  • This item is always processed by Zabbix server regardless of the host location (on server or proxy). The proxy will not receive this item with configuration data.
  • The second parameter must be empty and is reserved for future use.
zabbix[host,active_agent,available]


The availability of active agent checks on the host.
Return values: 0 - unknown; 1 - available; 2 - not available.

zabbix[host,discovery,interfaces]


The details of all configured interfaces of the host in Zabbix frontend.
Return value: JSON object.

Comments:

zabbix[host,<type>,available]


The availability of the main interface of a particular type of checks on the host.
Return values: 0 - not available; 1 - available; 2 - unknown.

Comments:

  • Valid types are: agent, snmp, ipmi, jmx;
  • The item value is calculated according to the configuration parameters regarding host unreachability/unavailability.
zabbix[hosts]


The number of monitored hosts.
Return value: Integer.

zabbix[items]


The number of enabled items (supported and not supported).
Return value: Integer.

zabbix[items_unsupported]


The number of unsupported items.
Return value: Integer.

zabbix[java,,<param>]


The information about Zabbix Java gateway.
Return values: 1 - if <param> is ping; Java gateway version - if <param> is version (for example: "2.0.0").

Comments:

  • Valid values for param are: ping, version;
  • This item can be used to check Java gateway availability using the nodata() trigger function;
  • The second parameter must be empty and is reserved for future use.
zabbix[lld_queue]


The count of values enqueued in the low-level discovery processing queue.
Return value: Integer.

This item can be used to monitor the low-level discovery processing queue length.

zabbix[preprocessing_queue]


The count of values enqueued in the preprocessing queue.
Return value: Integer.

This item can be used to monitor the preprocessing queue length.

zabbix[process,<type>,<mode>,<state>]


The percentage of time a particular Zabbix process or a group of processes (identified by <type> and <mode>) spent in <state>. It is calculated for the last minute only.
Return value: Float.

Parameters:

  • type - for server processes: alert manager, alert syncer, alerter, availability manager, configuration syncer, connector manager, connector worker, discoverer, escalator, history poller, history syncer, housekeeper, http poller, icmp pinger, ipmi manager, ipmi poller, java poller, lld manager, lld worker, odbc poller, poller, preprocessing manager, preprocessing worker, proxy poller, self-monitoring, snmp trapper, task manager, timer, trapper, unreachable poller, vmware collector;
    for proxy processes: availability manager, configuration syncer, data sender, discoverer, history syncer, housekeeper, http poller, icmp pinger, ipmi manager, ipmi poller, java poller, odbc poller, poller, preprocessing manager, preprocessing worker, self-monitoring, snmp trapper, task manager, trapper, unreachable poller, vmware collector
  • mode - avg - average value for all processes of a given type (default)
    count - returns number of forks for a given process type, <state> should not be specified
    max - maximum value
    min - minimum value
    <process number> - process number (between 1 and the number of pre-forked instances). For example, if 4 trappers are running, the value is between 1 and 4.
  • state - busy - process is in busy state, for example, the processing request (default); idle - process is in idle state doing nothing.

Comments:

  • If <mode> is a Zabbix process number that is not running (for example, with 5 pollers running the <mode> is specified to be 6), such an item will turn unsupported;
  • Minimum and maximum refers to the usage percentage for a single process. So if in a group of 3 pollers usage percentages per process were 2, 18 and 66, min would return 2 and max would return 66.
  • Processes report what they are doing in shared memory and the self-monitoring process summarizes that data each second. State changes (busy/idle) are registered upon change - thus a process that becomes busy registers as such and doesn't change or update the state until it becomes idle. This ensures that even fully hung processes will be correctly registered as 100% busy.
  • Currently, "busy" means "not sleeping", but in the future additional states might be introduced - waiting for locks, performing database queries, etc.
  • On Linux and most other systems, resolution is 1/100 of a second.

Examples:

zabbix[process,poller,avg,busy] #the average time of poller processes spent doing something during the last minute
       zabbix[process,"icmp pinger",max,busy] #the maximum time spent doing something by any ICMP pinger process during the last minute
       zabbix[process,"history syncer",2,busy] #the time spent doing something by history syncer number 2 during the last minute
       zabbix[process,trapper,count] #the amount of currently running trapper processes
zabbix[proxy,<name>,<param>]


The information about Zabbix proxy.
Return value: Integer.

Parameters:

  • name - the proxy name;
  • param - delay - how long the collected values are unsent, calculated as "proxy delay" (difference between the current proxy time and the timestamp of the oldest unsent value on proxy) + ("current server time" - "proxy lastaccess").
zabbix[proxy,discovery]


The list of Zabbix proxies with name, mode, encryption, compression, version, last seen, host count, item count, required values per second (vps) and version status (current/outdated/unsupported).
Return value: JSON object.

zabbix[proxy_history]


The number of values in the proxy history table waiting to be sent to the server.
Return values: Integer.

This item is not supported on Zabbix server.

zabbix[queue,<from>,<to>]


The number of monitored items in the queue which are delayed at least by <from> seconds, but less than <to> seconds.
Return value: Integer.

Parameters:

  • from - default: 6 seconds;
  • to - default: infinity.

Time-unit symbols (s,m,h,d,w) are supported in the parameters.

zabbix[rcache,<cache>,<mode>]


The availability statistics of the Zabbix configuration cache.
Return values: Integer (for size); Float (for percentage).

Parameters:

  • cache - buffer;
  • mode - total - the total size of buffer
    free - the size of free buffer
    pfree - the percentage of free buffer
    used - the size of used buffer
    pused - the percentage of used buffer
zabbix[requiredperformance]


The required performance of Zabbix server or Zabbix proxy, in new values per second expected.
Return value: Float.

Approximately correlates with "Required server performance, new values per second" in Reports → System information.

zabbix[stats,<ip>,<port>]


The internal metrics of a remote Zabbix server or proxy.
Return values: JSON object.

Parameters:

  • ip - the IP/DNS/network mask list of servers/proxies to be remotely queried (default is 127.0.0.1);
  • port - the port of server/proxy to be remotely queried (default is 10051).

Comments:

  • The stats request will only be accepted from the addresses listed in the 'StatsAllowedIP' server/proxy parameter on the target instance;
  • A selected set of internal metrics is returned by this item. For details, see Remote monitoring of Zabbix stats.
zabbix[stats,<ip>,<port>,queue,<from>,<to>]


The internal queue metrics (see zabbix[queue,<from>,<to>]) of a remote Zabbix server or proxy.
Return values: JSON object.

Parameters:

  • ip - the IP/DNS/network mask list of servers/proxies to be remotely queried (default is 127.0.0.1);
  • port - the port of server/proxy to be remotely queried (default is 10051);
  • from - delayed by at least (default is 6 seconds);
  • to - delayed by at most (default is infinity).

Comments:

  • The stats request will only be accepted from the addresses listed in the 'StatsAllowedIP' server/proxy parameter on the target instance;
  • A selected set of internal metrics is returned by this item. For details, see Remote monitoring of Zabbix stats.
zabbix[tcache,<cache>,<parameter>]


The effectiveness statistics of the Zabbix trend function cache.
Return values: Integer (for size); Float (for percentage).

Parameters:

  • cache - buffer;
  • mode - **all* - total cache requests (default)
    hits - cache hits
    phits - percentage of cache hits
    misses - cache misses
    pmisses - percentage of cache misses
    items - the number of cached items
    requests - the number of cached requests
    pitems - percentage of cached items from cached items + requests. Low percentage most likely means that the cache size can be reduced.

This item is not supported on Zabbix proxy.

zabbix[triggers]


The number of enabled triggers in Zabbix database, with all items enabled on enabled hosts.
Return value: Integer.

This item is not supported on Zabbix proxy.

zabbix[uptime]


The uptime of the Zabbix server or proxy process in seconds.
Return value: Integer.

zabbix[vcache,buffer,<mode>]


The availability statistics of the Zabbix value cache.
Return values: Integer (for size); Float (for percentage).

Parameter:

  • mode - total - the total size of buffer
    free - the size of free buffer
    pfree - the percentage of free buffer
    used - the size of used buffer
    pused - the percentage of used buffer

This item is not supported on Zabbix proxy.

zabbix[vcache,cache,<parameter>]


The effectiveness statistics of the Zabbix value cache.
Return values: Integer. With the mode parameter returns: 0 - normal mode; 1 - low memory mode.

Parameters:

  • parameter - requests - the total number of requests
    hits - the number of cache hits (history values taken from the cache)
    misses - the number of cache misses (history values taken from the database)
    mode - the value cache operating mode

Comments:

  • Once the low-memory mode has been switched on, the value cache will remain in this state for 24 hours, even if the problem that triggered this mode is resolved sooner;
  • You may use this key with the Change per second preprocessing step in order to get values-per-second statistics;
  • This item is not supported on Zabbix proxy.
zabbix[version]


The version of Zabbix server or proxy.
Return value: String. For example: 6.0.0beta1.

zabbix[vmware,buffer,<mode>]


The availability statistics of the Zabbix vmware cache.
Return values: Integer (for size); Float (for percentage).

Parameters:

  • mode - total - the total size of buffer
    free - the size of free buffer
    pfree - the percentage of free buffer
    used - the size of used buffer
    pused - the percentage of used buffer
zabbix[wcache,<cache>,<mode>]


The statistics and availability of the Zabbix write cache.
Return values: Integer (for number/size); Float (for percentage).

Parameters:

  • cache - values, history, index, or trend;
  • mode - (with values) all (default) - the total number of values processed by Zabbix server/proxy, except unsupported items (counter)
    float - the number of processed float values (counter)
    uint - the number of processed unsigned integer values (counter)
    str - the number of processed character/string values (counter)
    log - the number of processed log values (counter)
    text - the number of processed text values (counter)
    not supported - the number of times item processing resulted in item becoming unsupported or keeping that state (counter)
    (with history, index, trend cache) pfree (default) - the percentage of free buffer
    total - the total size of buffer
    free - the size of free buffer
    used - the size of used buffer
    pused - the percentage of used buffer

Comments:

  • Specifying <cache> is mandatory. The trend cache parameter is not supported with Zabbix proxy.
  • The history cache is used to store item values. A low number indicates performance problems on the database side.
  • The history index cache is used to index the values stored in the history cache;
  • The trend cache stores the aggregate for the current hour for all items that receive data;
  • You may use the zabbix[wcache,values] key with the Change per second preprocessing step in order to get values-per-second statistics.