Filter of discoverable discovered services on local node.

Filter to exclude discovered services on local node.

Filter of discoverable discovered service by namespace on local node. Enterprise only, in case of Open Source version Namespace will be set to 'None'.

Filter to exclude discovered service by namespace on local node. Enterprise only, in case of Open Source version Namespace will be set to 'None'.

Consul instance URL.

Maximum acceptable value of node's health score for AVERAGE trigger expression.

Maximum acceptable value of node's health score for WARNING trigger expression.

Maximum percentage of used file descriptors.

Consul auth token.

Discovery HTTP API methods specific metrics.

Preprocessing:

- PROMETHEUS_TO_JSON: consul_api_http{method =~ ".*"}

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Discover metrics for services that are registered with the local agent.

Preprocessing:

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Filter:

- {#SERVICE_NAME} MATCHES_REGEX {$CONSUL.LLD.FILTER.LOCAL_SERVICE_NAME.MATCHES}

- {#SERVICE_NAME} NOT_MATCHES_REGEX {$CONSUL.LLD.FILTER.LOCAL_SERVICE_NAME.NOT_MATCHES}

- {#SERVICE_NAMESPACE} MATCHES_REGEX {$CONSUL.LLD.FILTER.SERVICE_NAMESPACE.MATCHES}

- {#SERVICE_NAMESPACE} NOT_MATCHES_REGEX {$CONSUL.LLD.FILTER.SERVICE_NAMESPACE.NOT_MATCHES}

Overrides:

aggregated status
- {#TYPE} MATCHES_REGEX aggregated_status
- ITEM_PROTOTYPE LIKE Aggregated status - DISCOVER

checks
- {#TYPE} MATCHES_REGEX service_check
- ITEM_PROTOTYPE LIKE Check - DISCOVER

Discover raft metrics for leader nodes.

Preprocessing:

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Discover raft metrics for server nodes.

Preprocessing:

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Role of current Consul agent.

Preprocessing:

- JSONPATH: $.Config.Server

- BOOL_TO_DECIMAL

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Version of Consul agent.

Preprocessing:

- JSONPATH: $.Config.Version

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Number of services on current node.

Preprocessing:

- JSONPATH: $.Stats.agent.services

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Number of checks on current node.

Preprocessing:

- JSONPATH: $.Stats.agent.checks

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Number of check monitors on current node.

Preprocessing:

- JSONPATH: $.Stats.agent.check_monitors

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Total user and system CPU time spent in seconds.

Preprocessing:

- PROMETHEUS_PATTERN: process_cpu_seconds_total

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

Virtual memory size in bytes.

Preprocessing:

- PROMETHEUS_PATTERN: process_virtual_memory_bytes

Resident memory size in bytes.

Preprocessing:

- PROMETHEUS_PATTERN: process_resident_memory_bytes

The number of Goroutines on Consul instance.

Preprocessing:

- PROMETHEUS_PATTERN: go_goroutines

Number of open file descriptors.

Preprocessing:

- PROMETHEUS_PATTERN: process_open_fds

Maximum number of open file descriptors.

Preprocessing:

- PROMETHEUS_PATTERN: process_max_fds

Number of times per second whenever a Consul agent in client mode makes an RPC request to a Consul server.

This gives a measure of how much a given agent is loading the Consul servers.

This is only generated by agents in client mode, not Consul servers.

Preprocessing:

- PROMETHEUS_PATTERN: consul_client_rpc

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

Number of times per second whenever a Consul agent in client mode makes an RPC request to a Consul server and fails.

Preprocessing:

- PROMETHEUS_PATTERN: consul_client_rpc_failed

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

This metric counts the number of times a Consul agent has accepted an incoming TCP stream connection per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_tcp_accept

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

This metric counts the number of times a Consul agent has initiated a push/pull sync with an other agent per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_tcp_connect

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

This metric measures the total number of bytes sent by a Consul agent through the TCP protocol per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_tcp_sent

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

This metric measures the total number of bytes received by a Consul agent through the UDP protocol per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_udp_received

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

This metric measures the total number of bytes sent by a Consul agent through the UDP protocol per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_udp_sent

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The 90 percentile for the number of nanoseconds consumed by stop-the-world garbage collection (GC) pauses since Consul started, in milliseconds.

Preprocessing:

- PROMETHEUS_PATTERN: consul_runtime_gc_pause_ns{quantile="0.9"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

- MULTIPLIER: 1.0E-9

The 50 percentile (median) for the number of nanoseconds consumed by stop-the-world garbage collection (GC) pauses since Consul started, in milliseconds.

Preprocessing:

- PROMETHEUS_PATTERN: consul_runtime_gc_pause_ns{quantile="0.5"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

- MULTIPLIER: 1.0E-9

This metric counts the number of times the Consul agent has performed failure detection on another agent at a slower probe rate.

The agent uses its own health metric as an indicator to perform this action.

If its health score is low, it means that the node is healthy, and vice versa.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_degraded

⛔️ON_FAIL: DISCARD_VALUE ->

This metric describes a node's perception of its own health based on how well it is meeting the soft real-time requirements of the protocol.

This metric ranges from 0 to 8, where 0 indicates "totally healthy".

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_health_score

⛔️ON_FAIL: DISCARD_VALUE ->

The 90 percentile for the number of gossips (messages) broadcasted to a set of randomly selected nodes.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_gossip{quantile="0.9"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 50 for the number of gossips (messages) broadcasted to a set of randomly selected nodes.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_gossip{quantile="0.5"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

This metric counts the number of alive Consul agents, that the agent has mapped out so far, based on the message information given by the network layer.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_msg_alive

⛔️ON_FAIL: DISCARD_VALUE ->

This metric counts the number of times a Consul agent has marked another agent to be a dead node.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_msg_dead

⛔️ON_FAIL: DISCARD_VALUE ->

The number of times a Consul agent suspects another as failed while probing during gossip protocol.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_msg_suspect

⛔️ON_FAIL: DISCARD_VALUE ->

The 90 percentile for the time taken to perform a single round of failure detection on a select Consul agent.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_probeNode{quantile="0.9"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 50 percentile (median) for the time taken to perform a single round of failure detection on a select Consul agent.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_probeNode{quantile="0.5"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 90 percentile for the number of Consul agents that have exchanged state with this agent.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_pushPullNode{quantile="0.9"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 50 percentile (median) for the number of Consul agents that have exchanged state with this agent.

Preprocessing:

- PROMETHEUS_PATTERN: consul_memberlist_pushPullNode{quantile="0.5"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 90 percentile for the time it takes to complete an update to the KV store.

Preprocessing:

- PROMETHEUS_PATTERN: consul_kvs_apply{quantile="0.9"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 50 percentile (median) for the time it takes to complete an update to the KV store.

Preprocessing:

- PROMETHEUS_PATTERN: consul_kvs_apply{quantile="0.5"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The number of updates to the KV store per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_kvs_apply_count

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

Increments when an agent is marked dead and then recovers within a short time period.

This can be an indicator of overloaded agents, network problems, or configuration errors where agents cannot connect to each other on the required ports.

Shown as events per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_member_flap

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

Increments when an agent is marked dead.

This can be an indicator of overloaded agents, network problems, or configuration errors where agents cannot connect to each other on the required ports.

Shown as events per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_member_failed

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

Increments when an agent joins the cluster. If an agent flapped or failed this counter also increments when it re-joins.

Shown as events per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_member_join

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

Increments when an agent leaves the cluster. Shown as events per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_member_left

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

Increments when a Consul agent updates. Shown as events per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_member_update

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of ACL resolves per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_acl_ResolveToken_count

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of catalog register operation per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_catalog_register_count

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of catalog deregister operation per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_catalog_deregister_count

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The 90 percentile for the time taken by the Consul agent to append an entry into the existing log.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_snapshot_appendLine{quantile="0.9"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 50 percentile (median) for the time taken by the Consul agent to append an entry into the existing log.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_snapshot_appendLine{quantile="0.5"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The number of snapshot appendLine operations per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_snapshot_appendLine_count

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The 90 percentile for the time taken by the Consul agent to compact a log.

This operation occurs only when the snapshot becomes large enough to justify the compaction.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_snapshot_compact{quantile="0.9"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 50 percentile (median) for the time taken by the Consul agent to compact a log.

This operation occurs only when the snapshot becomes large enough to justify the compaction.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_snapshot_compact{quantile="0.5"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The number of snapshot compact operations per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_serf_snapshot_compact_count

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

Data collection check.

Preprocessing:

- JSONPATH: $.error

⛔️ON_FAIL: CUSTOM_VALUE ->

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Aggregated values of all health checks for the service instance.

Preprocessing:

- JSONPATH: $[?(@.id == "{#SERVICE_ID}")].status.first()

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Current state of health check for the service.

Preprocessing:

- JSONPATH: $[?(@.id == "{#SERVICE_ID}")].checks[?(@.CheckID == "{#SERVICE_CHECK_ID}")].Status.first()

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Current output of health check for the service.

Preprocessing:

- JSONPATH: $[?(@.id == "{#SERVICE_ID}")].checks[?(@.CheckID == "{#SERVICE_CHECK_ID}")].Output.first()

- DISCARD_UNCHANGED_HEARTBEAT: 3h

The 90 percentile of how long it takes to service the given HTTP request for the given verb.

Preprocessing:

- PROMETHEUS_PATTERN: consul_api_http{method = "{#HTTP_METHOD}", quantile = "0.9"}: function: sum

⛔️ON_FAIL: DISCARD_VALUE ->

The 50 percentile (median) of how long it takes to service the given HTTP request for the given verb.

Preprocessing:

- PROMETHEUS_PATTERN: consul_api_http{method = "{#HTTP_METHOD}", quantile = "0.5"}: function: sum

⛔️ON_FAIL: DISCARD_VALUE ->

Thr number of HTTP request for the given verb per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_api_http_count{method = "{#HTTP_METHOD}"}: function: sum

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

Current state of Consul agent.

Preprocessing:

- JSONPATH: $.Stats.raft.state

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Increments when a server becomes a leader.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_state_leader

⛔️ON_FAIL: DISCARD_VALUE ->

The number of initiated leader elections.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_state_candidate

⛔️ON_FAIL: DISCARD_VALUE ->

Incremented whenever a leader first passes a message into the Raft commit process (called an Apply operation).

This metric describes the arrival rate of new logs into Raft per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_apply

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The 90 percentile of how long it takes a leader node to communicate with followers during a leader lease check, in milliseconds.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_leader_lastContact{quantile="0.9"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 50 percentile (median) of how long it takes a leader node to communicate with followers during a leader lease check, in milliseconds.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_leader_lastContact{quantile="0.5"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 90 percentile time it takes to commit a new entry to the raft log on the leader, in milliseconds.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_commitTime{quantile="0.9"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 50 percentile (median) time it takes to commit a new entry to the raft log on the leader, in milliseconds.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_commitTime{quantile="0.5"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 90 percentile time it takes for the leader to write log entries to disk, in milliseconds.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_leader_dispatchLog{quantile="0.9"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The 50 percentile (median) time it takes for the leader to write log entries to disk, in milliseconds.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_leader_dispatchLog{quantile="0.5"}

⛔️ON_FAIL: DISCARD_VALUE ->

- JAVASCRIPT: return (isNaN(value)) ? 0 : value;

The number of times a Raft leader writes a log to disk per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_leader_dispatchLog_count

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of commits a new entry to the Raft log on the leader per second.

Preprocessing:

- PROMETHEUS_PATTERN: consul_raft_commitTime_count

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

Tracks the overall health of the local server cluster. 1 if all servers are healthy, 0 if one or more are unhealthy.

Preprocessing:

- PROMETHEUS_PATTERN: consul_autopilot_healthy

⛔️ON_FAIL: DISCARD_VALUE ->

Get raw metrics from Consul instance /metrics endpoint.

Preprocessing:

- CHECK_NOT_SUPPORTED

⛔️ON_FAIL: DISCARD_VALUE ->

Get configuration and member information of the local agent.

Preprocessing:

- CHECK_NOT_SUPPORTED

⛔️ON_FAIL: DISCARD_VALUE ->

Get all the services that are registered with the local agent and their status.

Expression:

Consul version has changed. Ack to close.

Manual close: YES

"Heavy file descriptor usage (i.e., near the process’s file descriptor limit) indicates a potential file descriptor exhaustion issue."

This metric ranges from 0 to 8, where 0 indicates "totally healthy".

This health score is used to scale the time between outgoing probes, and higher scores translate into longer probing intervals.

For more details see section IV of the Lifeguard paper: https://arxiv.org/pdf/1707.00788.pdf

Depends on:

- Consul: Node's health score is critical

This metric ranges from 0 to 8, where 0 indicates "totally healthy".

This health score is used to scale the time between outgoing probes, and higher scores translate into longer probing intervals.

For more details see section IV of the Lifeguard paper: https://arxiv.org/pdf/1707.00788.pdf

Failed to get local services. Check debug log for more information.

Aggregated state of service on the local agent is 'warning'.

Aggregated state of service on the local agent is 'critical'.

Consul instance URL.

Consul auth token.

Maximum percentage of used file descriptors.

Filter of discoverable discovered services on local node.

Filter to exclude discovered services on local node.

Filter of discoverable discovered service by namespace on local node. Enterprise only, in case of Open Source version Namespace will be set to 'None'.

Filter to exclude discovered service by namespace on local node. Enterprise only, in case of Open Source version Namespace will be set to 'None'.

Maximum acceptable value of node's health score for WARNING trigger expression.

Maximum acceptable value of node's health score for AVERAGE trigger expression.

Get raw metrics from Consul instance /metrics endpoint.

Preprocessing

• Check for not supported value

  ⛔️Custom on fail: Discard value

Get configuration and member information of the local agent.

Preprocessing

• Check for not supported value

  ⛔️Custom on fail: Discard value

Role of current Consul agent.

Preprocessing

• JSON Path: $.Config.Server

• Boolean to decimal

• Discard unchanged with heartbeat: 3h

Version of Consul agent.

Preprocessing

• JSON Path: $.Config.Version

• Discard unchanged with heartbeat: 3h

Number of services on current node.

Preprocessing

• JSON Path: $.Stats.agent.services

• Discard unchanged with heartbeat: 3h

Number of checks on current node.

Preprocessing

• JSON Path: $.Stats.agent.checks

• Discard unchanged with heartbeat: 3h

Number of check monitors on current node.

Preprocessing

• JSON Path: $.Stats.agent.check_monitors

• Discard unchanged with heartbeat: 3h

Total user and system CPU time spent in seconds.

Preprocessing

• Prometheus pattern: VALUE(process_cpu_seconds_total)

  ⛔️Custom on fail: Discard value

• Change per second

Virtual memory size in bytes.

Preprocessing

• Prometheus pattern: VALUE(process_virtual_memory_bytes)

Resident memory size in bytes.

Preprocessing

• Prometheus pattern: VALUE(process_resident_memory_bytes)

The number of Goroutines on Consul instance.

Preprocessing

• Prometheus pattern: VALUE(go_goroutines)

Number of open file descriptors.

Preprocessing

• Prometheus pattern: VALUE(process_open_fds)

Maximum number of open file descriptors.

Preprocessing

• Prometheus pattern: VALUE(process_max_fds)

Number of times per second whenever a Consul agent in client mode makes an RPC request to a Consul server.

This gives a measure of how much a given agent is loading the Consul servers.

This is only generated by agents in client mode, not Consul servers.

Preprocessing

• Prometheus pattern: VALUE(consul_client_rpc)

  ⛔️Custom on fail: Discard value

• Change per second

Number of times per second whenever a Consul agent in client mode makes an RPC request to a Consul server and fails.

Preprocessing

• Prometheus pattern: VALUE(consul_client_rpc_failed)

  ⛔️Custom on fail: Discard value

• Change per second

This metric counts the number of times a Consul agent has accepted an incoming TCP stream connection per second.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_tcp_accept)

  ⛔️Custom on fail: Discard value

• Change per second

This metric counts the number of times a Consul agent has initiated a push/pull sync with an other agent per second.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_tcp_connect)

  ⛔️Custom on fail: Discard value

• Change per second

This metric measures the total number of bytes sent by a Consul agent through the TCP protocol per second.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_tcp_sent)

  ⛔️Custom on fail: Discard value

• Change per second

This metric measures the total number of bytes received by a Consul agent through the UDP protocol per second.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_udp_received)

  ⛔️Custom on fail: Discard value

• Change per second

This metric measures the total number of bytes sent by a Consul agent through the UDP protocol per second.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_udp_sent)

  ⛔️Custom on fail: Discard value

• Change per second

The 90 percentile for the number of nanoseconds consumed by stop-the-world garbage collection (GC) pauses since Consul started, in milliseconds.

Preprocessing

• Prometheus pattern: VALUE(consul_runtime_gc_pause_ns{quantile="0.9"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

• Custom multiplier: 1.0E-9

The 50 percentile (median) for the number of nanoseconds consumed by stop-the-world garbage collection (GC) pauses since Consul started, in milliseconds.

Preprocessing

• Prometheus pattern: VALUE(consul_runtime_gc_pause_ns{quantile="0.5"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

• Custom multiplier: 1.0E-9

This metric counts the number of times the Consul agent has performed failure detection on another agent at a slower probe rate.

The agent uses its own health metric as an indicator to perform this action.

If its health score is low, it means that the node is healthy, and vice versa.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_degraded)

  ⛔️Custom on fail: Discard value

This metric describes a node's perception of its own health based on how well it is meeting the soft real-time requirements of the protocol.

This metric ranges from 0 to 8, where 0 indicates "totally healthy".

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_health_score)

  ⛔️Custom on fail: Discard value

The 90 percentile for the number of gossips (messages) broadcasted to a set of randomly selected nodes.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_gossip{quantile="0.9"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 50 for the number of gossips (messages) broadcasted to a set of randomly selected nodes.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_gossip{quantile="0.5"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

This metric counts the number of alive Consul agents, that the agent has mapped out so far, based on the message information given by the network layer.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_msg_alive)

  ⛔️Custom on fail: Discard value

This metric counts the number of times a Consul agent has marked another agent to be a dead node.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_msg_dead)

  ⛔️Custom on fail: Discard value

The number of times a Consul agent suspects another as failed while probing during gossip protocol.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_msg_suspect)

  ⛔️Custom on fail: Discard value

The 90 percentile for the time taken to perform a single round of failure detection on a select Consul agent.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_probeNode{quantile="0.9"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 50 percentile (median) for the time taken to perform a single round of failure detection on a select Consul agent.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_probeNode{quantile="0.5"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 90 percentile for the number of Consul agents that have exchanged state with this agent.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_pushPullNode{quantile="0.9"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 50 percentile (median) for the number of Consul agents that have exchanged state with this agent.

Preprocessing

• Prometheus pattern: VALUE(consul_memberlist_pushPullNode{quantile="0.5"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 90 percentile for the time it takes to complete an update to the KV store.

Preprocessing

• Prometheus pattern: VALUE(consul_kvs_apply{quantile="0.9"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 50 percentile (median) for the time it takes to complete an update to the KV store.

Preprocessing

• Prometheus pattern: VALUE(consul_kvs_apply{quantile="0.5"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The number of updates to the KV store per second.

Preprocessing

• Prometheus pattern: VALUE(consul_kvs_apply_count)

  ⛔️Custom on fail: Discard value

• Change per second

Increments when an agent is marked dead and then recovers within a short time period.

This can be an indicator of overloaded agents, network problems, or configuration errors where agents cannot connect to each other on the required ports.

Shown as events per second.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_member_flap)

  ⛔️Custom on fail: Discard value

• Change per second

Increments when an agent is marked dead.

This can be an indicator of overloaded agents, network problems, or configuration errors where agents cannot connect to each other on the required ports.

Shown as events per second.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_member_failed)

  ⛔️Custom on fail: Discard value

• Change per second

Increments when an agent joins the cluster. If an agent flapped or failed this counter also increments when it re-joins.

Shown as events per second.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_member_join)

  ⛔️Custom on fail: Discard value

• Change per second

Increments when an agent leaves the cluster. Shown as events per second.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_member_left)

  ⛔️Custom on fail: Discard value

• Change per second

Increments when a Consul agent updates. Shown as events per second.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_member_update)

  ⛔️Custom on fail: Discard value

• Change per second

The number of ACL resolves per second.

Preprocessing

• Prometheus pattern: VALUE(consul_acl_ResolveToken_count)

  ⛔️Custom on fail: Discard value

• Change per second

The number of catalog register operation per second.

Preprocessing

• Prometheus pattern: VALUE(consul_catalog_register_count)

  ⛔️Custom on fail: Discard value

• Change per second

The number of catalog deregister operation per second.

Preprocessing

• Prometheus pattern: VALUE(consul_catalog_deregister_count)

  ⛔️Custom on fail: Discard value

• Change per second

The 90 percentile for the time taken by the Consul agent to append an entry into the existing log.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_snapshot_appendLine{quantile="0.9"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 50 percentile (median) for the time taken by the Consul agent to append an entry into the existing log.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_snapshot_appendLine{quantile="0.5"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The number of snapshot appendLine operations per second.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_snapshot_appendLine_count)

  ⛔️Custom on fail: Discard value

• Change per second

The 90 percentile for the time taken by the Consul agent to compact a log.

This operation occurs only when the snapshot becomes large enough to justify the compaction.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_snapshot_compact{quantile="0.9"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 50 percentile (median) for the time taken by the Consul agent to compact a log.

This operation occurs only when the snapshot becomes large enough to justify the compaction.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_snapshot_compact{quantile="0.5"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The number of snapshot compact operations per second.

Preprocessing

• Prometheus pattern: VALUE(consul_serf_snapshot_compact_count)

  ⛔️Custom on fail: Discard value

• Change per second

Get all the services that are registered with the local agent and their status.

Data collection check.

Preprocessing

• JSON Path: $.error

  ⛔️Custom on fail: Set value to

• Discard unchanged with heartbeat: 3h

Consul version has changed. Acknowledge to close the problem manually.

"Heavy file descriptor usage (i.e., near the process’s file descriptor limit) indicates a potential file descriptor exhaustion issue."

This metric ranges from 0 to 8, where 0 indicates "totally healthy".
This health score is used to scale the time between outgoing probes, and higher scores translate into longer probing intervals.
For more details see section IV of the Lifeguard paper: https://arxiv.org/pdf/1707.00788.pdf

• Consul: Node's health score is critical

This metric ranges from 0 to 8, where 0 indicates "totally healthy".
This health score is used to scale the time between outgoing probes, and higher scores translate into longer probing intervals.
For more details see section IV of the Lifeguard paper: https://arxiv.org/pdf/1707.00788.pdf

Failed to get local services. Check debug log for more information.

Discover metrics for services that are registered with the local agent.

Preprocessing

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 3h

Aggregated values of all health checks for the service instance.

Preprocessing

• JSON Path: $[?(@.id == "{#SERVICE_ID}")].status.first()

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 3h

Current state of health check for the service.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 3h

Current output of health check for the service.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 3h

Aggregated state of service on the local agent is 'warning'.

Aggregated state of service on the local agent is 'critical'.

Discovery HTTP API methods specific metrics.

Preprocessing

• Prometheus to JSON: consul_api_http{method =~ ".*"}

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 3h

The 90 percentile of how long it takes to service the given HTTP request for the given verb.

Preprocessing

• Prometheus pattern: The text is too long. Please see the template.

  ⛔️Custom on fail: Discard value

The 50 percentile (median) of how long it takes to service the given HTTP request for the given verb.

Preprocessing

• Prometheus pattern: The text is too long. Please see the template.

  ⛔️Custom on fail: Discard value

The number of HTTP request for the given verb per second.

Preprocessing

• Prometheus pattern: SUM(consul_api_http_count{method = "{#HTTP_METHOD}"})

  ⛔️Custom on fail: Discard value

• Change per second

Discover raft metrics for server nodes.

Preprocessing

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 3h

Current state of Consul agent.

Preprocessing

• JSON Path: $.Stats.raft.state

• Discard unchanged with heartbeat: 3h

Increments when a server becomes a leader.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_state_leader)

  ⛔️Custom on fail: Discard value

The number of initiated leader elections.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_state_candidate)

  ⛔️Custom on fail: Discard value

Incremented whenever a leader first passes a message into the Raft commit process (called an Apply operation).

This metric describes the arrival rate of new logs into Raft per second.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_apply)

  ⛔️Custom on fail: Discard value

• Change per second

Discover raft metrics for leader nodes.

Preprocessing

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 3h

The 90 percentile of how long it takes a leader node to communicate with followers during a leader lease check, in milliseconds.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_leader_lastContact{quantile="0.9"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 50 percentile (median) of how long it takes a leader node to communicate with followers during a leader lease check, in milliseconds.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_leader_lastContact{quantile="0.5"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 90 percentile time it takes to commit a new entry to the raft log on the leader, in milliseconds.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_commitTime{quantile="0.9"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 50 percentile (median) time it takes to commit a new entry to the raft log on the leader, in milliseconds.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_commitTime{quantile="0.5"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 90 percentile time it takes for the leader to write log entries to disk, in milliseconds.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_leader_dispatchLog{quantile="0.9"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The 50 percentile (median) time it takes for the leader to write log entries to disk, in milliseconds.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_leader_dispatchLog{quantile="0.5"})

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

The number of times a Raft leader writes a log to disk per second.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_leader_dispatchLog_count)

  ⛔️Custom on fail: Discard value

• Change per second

The number of commits a new entry to the Raft log on the leader per second.

Preprocessing

• Prometheus pattern: VALUE(consul_raft_commitTime_count)

  ⛔️Custom on fail: Discard value

• Change per second

Tracks the overall health of the local server cluster. 1 if all servers are healthy, 0 if one or more are unhealthy.

Preprocessing

• Prometheus pattern: VALUE(consul_autopilot_healthy)

  ⛔️Custom on fail: Discard value