-

-

-

-

-

Load per CPU considered sustainable. Tune if needed.

-

-

-

-

-

Filter out loopbacks, nulls, docker veth links and docker0 bridge by default

-

Ignore notPresent(7)

TCP Port node_exporter is listening on.

-

-

This macro is used in block devices discovery. Can be overridden on the host or linked template level

This macro is used in block devices discovery. Can be overridden on the host or linked template level

Disk read average response time (in ms) before the trigger would fire

Disk write average response time (in ms) before the trigger would fire

This macro is used in filesystems discovery. Can be overridden on the host or linked template level

This macro is used in filesystems discovery. Can be overridden on the host or linked template level

This macro is used in filesystems discovery. Can be overridden on the host or linked template level

This macro is used in filesystems discovery. Can be overridden on the host or linked template level

This macro is used in filesystems discovery. Can be overridden on the host or linked template level

This macro is used in filesystems discovery. Can be overridden on the host or linked template level

-

-

-

-

Discovery of network interfaces. Requires node_exporter v0.18 and up.

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_network_info$"}

Filter:

- A: { #IFNAME} MATCHES_REGEX {$NET.IF.IFNAME.MATCHES}

- B: { #IFNAME} NOT_MATCHES_REGEX {$NET.IF.IFNAME.NOT_MATCHES}

- C: { #IFALIAS} MATCHES_REGEX {$NET.IF.IFALIAS.MATCHES}

- D: { #IFALIAS} NOT_MATCHES_REGEX {$NET.IF.IFALIAS.NOT_MATCHES}

- E: { #IFOPERSTATUS} MATCHES_REGEX {$NET.IF.IFOPERSTATUS.MATCHES}

- F: { #IFOPERSTATUS} NOT_MATCHES_REGEX {$NET.IF.IFOPERSTATUS.NOT_MATCHES}

Discovery of file systems of different types.

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_filesystem_size(?:_bytes)?$", mountpoint=~".+"}

Filter:

- A: { #FSTYPE} MATCHES_REGEX {$VFS.FS.FSTYPE.MATCHES}

- B: { #FSTYPE} NOT_MATCHES_REGEX {$VFS.FS.FSTYPE.NOT_MATCHES}

- C: { #FSNAME} MATCHES_REGEX {$VFS.FS.FSNAME.MATCHES}

- D: { #FSNAME} NOT_MATCHES_REGEX {$VFS.FS.FSNAME.NOT_MATCHES}

- E: { #FSNAME} MATCHES_REGEX {$VFS.FS.FSDEVICE.MATCHES}

- F: { #FSDEVICE} NOT_MATCHES_REGEX {$VFS.FS.FSDEVICE.NOT_MATCHES}

-

Preprocessing:

- PROMETHEUS_TO_JSON: node_disk_io_now{device=~".+"}

Filter:

- A: { #DEVNAME} MATCHES_REGEX {$VFS.DEV.DEVNAME.MATCHES}

- B: { #DEVNAME} NOT_MATCHES_REGEX {$VFS.DEV.DEVNAME.NOT_MATCHES}

-

Preprocessing:

- PROMETHEUS_PATTERN: node_load1

-

Preprocessing:

- PROMETHEUS_PATTERN: node_load5

-

Preprocessing:

- PROMETHEUS_PATTERN: node_load15

-

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_cpu(?:_seconds_total)?$",cpu=~".+",mode="idle"}

- JAVASCRIPT: //count the number of cores return JSON.parse(value).length

CPU utilization in %

Preprocessing:

- JAVASCRIPT: //Calculate utilization return (100 - value)

The time the CPU has spent doing nothing.

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_cpu(?:_seconds_total)?$",cpu=~".+",mode="idle"}

- JAVASCRIPT: //calculates average, all cpu utilization var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0)/valueArr.length;

- CHANGE_PER_SECOND

- MULTIPLIER: 100

The time the CPU has spent running the kernel and its processes.

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_cpu(?:_seconds_total)?$",cpu=~".+",mode="system"}

- JAVASCRIPT: //calculates average, all cpu utilization var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0)/valueArr.length;

- CHANGE_PER_SECOND

- MULTIPLIER: 100

The time the CPU has spent running users' processes that are not niced.

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_cpu(?:_seconds_total)?$",cpu=~".+",mode="user"}

- JAVASCRIPT: //calculates average, all cpu utilization var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0)/valueArr.length;

- CHANGE_PER_SECOND

- MULTIPLIER: 100

The amount of CPU 'stolen' from this virtual machine by the hypervisor for other tasks (such as running another virtual machine).

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_cpu(?:_seconds_total)?$",cpu=~".+",mode="steal"}

- JAVASCRIPT: //calculates average, all cpu utilization var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0)/valueArr.length;

- CHANGE_PER_SECOND

- MULTIPLIER: 100

The amount of time the CPU has been servicing software interrupts.

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_cpu(?:_seconds_total)?$",cpu=~".+",mode="softirq"}

- JAVASCRIPT: //calculates average, all cpu utilization var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0)/valueArr.length;

- CHANGE_PER_SECOND

- MULTIPLIER: 100

The time the CPU has spent running users' processes that have been niced.

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_cpu(?:_seconds_total)?$",cpu=~".+",mode="nice"}

- JAVASCRIPT: //calculates average, all cpu utilization var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0)/valueArr.length;

- CHANGE_PER_SECOND

- MULTIPLIER: 100

Amount of time the CPU has been waiting for I/O to complete.

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_cpu(?:_seconds_total)?$",cpu=~".+",mode="iowait"}

- JAVASCRIPT: //calculates average, all cpu utilization var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0)/valueArr.length;

- CHANGE_PER_SECOND

- MULTIPLIER: 100

The amount of time the CPU has been servicing hardware interrupts.

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"^node_cpu(?:_seconds_total)?$",cpu=~".+",mode="irq"}

- JAVASCRIPT: //calculates average, all cpu utilization var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0)/valueArr.length;

- CHANGE_PER_SECOND

- MULTIPLIER: 100

Guest time (time spent running a virtual CPU for a guest operating system)

Preprocessing:

- PROMETHEUS_TO_JSON: `{name=~"^node_cpu(?:_guest_seconds_total)?$",cpu=~".+",mode=~"^(?:user

Time spent running a niced guest (virtual CPU for guest operating systems under the control of the Linux kernel)

Preprocessing:

- PROMETHEUS_TO_JSON: `{name=~"^node_cpu(?:_guest_seconds_total)?$",cpu=~".+",mode=~"^(?:nice

-

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"node_intr"}

- CHANGE_PER_SECOND

-

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"node_context_switches"}

- CHANGE_PER_SECOND

-

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"^node_boot_time(?:_seconds)?$"}

-

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"^node_time(?:_seconds)?$"}

System host name.

Preprocessing:

- PROMETHEUS_PATTERN: node_uname_info nodename

- DISCARD_UNCHANGED_HEARTBEAT: 1d

Labeled system information as provided by the uname system call.

Preprocessing:

- PROMETHEUS_TO_JSON: node_uname_info

- JAVASCRIPT: var info = JSON.parse(value)[0]; return info.labels.sysname+' version: '+info.labels.release+' '+info.labels.version

- DISCARD_UNCHANGED_HEARTBEAT: 1d

It could be increased by using sysctrl utility or modifying file /etc/sysctl.conf.

Preprocessing:

- PROMETHEUS_PATTERN: node_filefd_maximum

- DISCARD_UNCHANGED_HEARTBEAT: 1d

-

Preprocessing:

- PROMETHEUS_PATTERN: node_filefd_allocated

-

Preprocessing:

- DISCARD_UNCHANGED_HEARTBEAT: 1d

-

Preprocessing:

- PROMETHEUS_PATTERN: node_uname_info machine

- DISCARD_UNCHANGED_HEARTBEAT: 1d

Memory used percentage is calculated as (total-available)/total*100

Expression:

Total memory in Bytes

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"node_memory_MemTotal"}

Available memory, in Linux, available = free + buffers + cache. On other platforms calculation may vary. See also: https://www.zabbix.com/documentation/current/manual/appendix/items/vm.memory.size_params

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"node_memory_MemAvailable"}

-

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"node_memory_SwapTotal"}

-

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"node_memory_SwapFree"}

-

Expression:

-

Preprocessing:

- PROMETHEUS_PATTERN: node_exporter_build_info version

- DISCARD_UNCHANGED_HEARTBEAT: 1d

Preprocessing:

- PROMETHEUS_PATTERN: node_network_receive_bytes_total{device="{ #IFNAME}"}

- CHANGE_PER_SECOND

- MULTIPLIER: 8

Preprocessing:

- PROMETHEUS_PATTERN: node_network_transmit_bytes_total{device="{ #IFNAME}"}

- CHANGE_PER_SECOND

- MULTIPLIER: 8

Preprocessing:

- PROMETHEUS_PATTERN: node_network_transmit_errs_total{device="{ #IFNAME}"}

- CHANGE_PER_SECOND

Preprocessing:

- PROMETHEUS_PATTERN: node_network_receive_errs_total{device="{ #IFNAME}"}

- CHANGE_PER_SECOND

Preprocessing:

- PROMETHEUS_PATTERN: node_network_receive_drop_total{device="{ #IFNAME}"}

- CHANGE_PER_SECOND

Preprocessing:

- PROMETHEUS_PATTERN: node_network_transmit_drop_total{device="{ #IFNAME}"}

- CHANGE_PER_SECOND

Sets value to 0 if metric is missing in node_exporter output.

Preprocessing:

- PROMETHEUS_PATTERN: node_network_speed_bytes{device="{ #IFNAME}"}

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- MULTIPLIER: 8

node_network_protocol_type protocol_type value of /sys/class/net/.

Preprocessing:

- PROMETHEUS_PATTERN: node_network_protocol_type{device="{ #IFNAME}"}

Indicates the interface RFC2863 operational state as a string.

Possible values are:"unknown", "notpresent", "down", "lowerlayerdown", "testing","dormant", "up".

Reference: https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-class-net

Preprocessing:

- PROMETHEUS_PATTERN: node_network_info{device="{ #IFNAME}"} operstate

- JAVASCRIPT: var newvalue; switch(value) { case "up": newvalue = 1; break; case "down": newvalue = 2; break; case "testing": newvalue = 4; break; case "unknown": newvalue = 5; break; case "dormant": newvalue = 6; break; case "notPresent": newvalue = 7; break; default: newvalue = "Problem parsing interface operstate in JS"; } return newvalue;

-

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"^node_boot_time(?:_seconds)?$"}

- JAVASCRIPT: //use boottime to calculate uptime return (Math.floor(Date.now()/1000)-Number(value));

-

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"^node_filesystem_avail(?:_bytes)?$", mountpoint="{ #FSNAME}"}

Total space in Bytes

Preprocessing:

- PROMETHEUS_PATTERN: {__name__=~"^node_filesystem_size(?:_bytes)?$", mountpoint="{ #FSNAME}"}

Used storage in Bytes

Expression:

Space utilization in % for { #FSNAME}

Expression:

-

Preprocessing:

- PROMETHEUS_TO_JSON: {__name__=~"node_filesystem_files.*",mountpoint="{ #FSNAME}"}

- JAVASCRIPT: //count vfs.fs.inode.pfree var inode_free; var inode_total; JSON.parse(value).forEach(function(metric) { if (metric['name'] == 'node_filesystem_files'){ inode_total = metric['value']; } else if (metric['name'] == 'node_filesystem_files_free'){ inode_free = metric['value']; } }); return (inode_free/inode_total)*100;

r/s. The number (after merges) of read requests completed per second for the device.

Preprocessing:

- PROMETHEUS_PATTERN: node_disk_reads_completed_total{device="{ #DEVNAME}"}

- CHANGE_PER_SECOND

w/s. The number (after merges) of write requests completed per second for the device.

Preprocessing:

- PROMETHEUS_PATTERN: node_disk_writes_completed_total{device="{ #DEVNAME}"}

- CHANGE_PER_SECOND

This formula contains two boolean expressions that evaluates to 1 or 0 in order to set calculated metric to zero and to avoid division by zero exception.

Expression:

This formula contains two boolean expressions that evaluates to 1 or 0 in order to set calculated metric to zero and to avoid division by zero exception.

Expression:

-

Preprocessing:

- PROMETHEUS_PATTERN: node_disk_io_time_weighted_seconds_total{device="{ #DEVNAME}"}

- CHANGE_PER_SECOND

-

Preprocessing:

- PROMETHEUS_PATTERN: node_disk_io_time_seconds_total{device="{ #DEVNAME}"}

- CHANGE_PER_SECOND

- MULTIPLIER: 100

-

Rate of total read time counter. Used in r_await calculation

Preprocessing:

- PROMETHEUS_PATTERN: node_disk_read_time_seconds_total{device="{ #DEVNAME}"}

- CHANGE_PER_SECOND

Rate of total write time counter. Used in w_await calculation

Preprocessing:

- PROMETHEUS_PATTERN: node_disk_write_time_seconds_total{device="{ #DEVNAME}"}

- CHANGE_PER_SECOND

Per CPU load average is too high. Your system may be slow to respond.

-

Depends on:

- Load average is too high (per CPU load over {$LOAD_AVG_PER_CPU.MAX.WARN} for 5m)

-

Manual close: YES

System name has changed. Ack to close.

Manual close: YES

-

Depends on:

- Running out of file descriptors (less than < 20% free)

-

Operating system description has changed. Possible reasons that system has been updated or replaced. Ack to close.

Manual close: YES

Depends on:

- System name has changed (new name: {ITEM.VALUE})

-

Depends on:

- Lack of available memory ( < {$MEMORY.AVAILABLE.MIN} of {ITEM.VALUE2})

-

This trigger is ignored, if there is no swap configured

Depends on:

- High memory utilization ( >{$MEMORY.UTIL.MAX}% for 5m)

- Lack of available memory ( < {$MEMORY.AVAILABLE.MIN} of {ITEM.VALUE2})

-

Recovery expression:

Manual close: YES

Depends on:

- Interface { #IFNAME}({ #IFALIAS}): Link down

Recovers when below 80% of {$IF.ERRORS.WARN:"{ #IFNAME}"} threshold

Recovery expression:

Manual close: YES

Depends on:

- Interface { #IFNAME}({ #IFALIAS}): Link down

This Ethernet connection has transitioned down from its known maximum speed. This might be a sign of autonegotiation issues. Ack to close.

Recovery expression:

Manual close: YES

Depends on:

- Interface { #IFNAME}({ #IFALIAS}): Link down

This Ethernet connection has transitioned down from its known maximum speed. This might be a sign of autonegotiation issues. Ack to close.

Recovery expression:

Manual close: YES

Depends on:

- Interface { #IFNAME}({ #IFALIAS}): Link down

This trigger expression works as follows:

1. Can be triggered if operations status is down.

2. {$IFCONTROL:"{ #IFNAME}"}=1 - user can redefine Context macro to value - 0. That marks this interface as not important. No new trigger will be fired if this interface is down.

3. {TEMPLATE_NAME:METRIC.diff()}=1) - trigger fires only if operational status was up(1) sometime before. (So, do not fire 'ethernal off' interfaces.)

WARNING: if closed manually - won't fire again on next poll, because of .diff.

Recovery expression:

Manual close: YES

The device uptime is less than 10 minutes

Manual close: YES

Two conditions should match: First, space utilization should be above {$VFS.FS.PUSED.MAX.CRIT:"{ #FSNAME}"}.

Second condition should be one of the following:

- The disk free space is less than 5G.

- The disk will be full in less than 24 hours.

Manual close: YES

Two conditions should match: First, space utilization should be above {$VFS.FS.PUSED.MAX.CRIT:"{ #FSNAME}"}.

Second condition should be one of the following:

- The disk free space is less than 10G.

- The disk will be full in less than 24 hours.

Manual close: YES

Depends on:

- { #FSNAME}: Disk space is critically low (used > {$VFS.FS.PUSED.MAX.CRIT:"{ #FSNAME}"}%)

It may become impossible to write to disk if there are no index nodes left.

As symptoms, 'No space left on device' or 'Disk is full' errors may be seen even though free space is available.

It may become impossible to write to disk if there are no index nodes left.

As symptoms, 'No space left on device' or 'Disk is full' errors may be seen even though free space is available.

Depends on:

- { #FSNAME}: Running out of free inodes (free < {$VFS.FS.INODE.PFREE.MIN.CRIT:"{ #FSNAME}"}%)

This trigger might indicate disk { #DEVNAME} saturation.

Manual close: YES

Failed to fetch system metrics from node_exporter in time.

Manual close: YES