The port of CockroachDB API and Prometheus endpoint.

Request scheme which may be http or https.

Number of days until the CA certificate expires.

Number of days until the node certificate expires.

Maximum clock offset of the node against the rest of the cluster in milliseconds for trigger expression.

Maximum percentage of used file descriptors.

Maximum number of SQL statements errors for trigger expression.

The critical threshold of the available disk space in percent.

The warning threshold of the available disk space in percent.

Discover per store metrics.

Preprocessing:

- PROMETHEUS_TO_JSON: capacity

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Check if HTTP/HTTPS service accepts TCP connections.

Preprocessing:

- DISCARD_UNCHANGED_HEARTBEAT: 10m

Mean clock offset of the node against the rest of the cluster.

Preprocessing:

- PROMETHEUS_PATTERN: clock_offset_meannanos: value: `</p><p>- MULTIPLIER:0.000000001`

Build information.

Preprocessing:

- PROMETHEUS_PATTERN: build_timestamp: label: tag

- DISCARD_UNCHANGED_HEARTBEAT: 3h

System CPU time.

Preprocessing:

- PROMETHEUS_PATTERN: sys_cpu_sys_ns: value: `</p><p>- CHANGE_PER_SECOND</p><p>- MULTIPLIER:0.000000001`

User CPU time.

Preprocessing:

- PROMETHEUS_PATTERN: sys_cpu_user_ns: value: `</p><p>- CHANGE_PER_SECOND</p><p>- MULTIPLIER:0.000000001`

CPU utilization in %.

Preprocessing:

- PROMETHEUS_PATTERN: sys_cpu_combined_percent_normalized: value: `</p><p>- MULTIPLIER:100`

Number of disk IO operations currently in progress on this host.

Preprocessing:

- PROMETHEUS_PATTERN: sys_host_disk_iopsinprogress: value: ``

- CHANGE_PER_SECOND

Bytes read from all disks per second since this process started

Preprocessing:

- PROMETHEUS_PATTERN: sys_host_disk_read_bytes: value: ``

- CHANGE_PER_SECOND

Number of disk read operations per second across all disks since this process started.

Preprocessing:

- PROMETHEUS_PATTERN: sys_host_disk_read_count: value: ``

- CHANGE_PER_SECOND

Bytes written to all disks per second since this process started.

Preprocessing:

- PROMETHEUS_PATTERN: sys_host_disk_write_bytes: value: ``

- CHANGE_PER_SECOND

Disk write operations per second across all disks since this process started.

Preprocessing:

- PROMETHEUS_PATTERN: sys_host_disk_write_count: value: ``

- CHANGE_PER_SECOND

Open file descriptors soft limit of the process.

Preprocessing:

- PROMETHEUS_PATTERN: sys_fd_softlimit: value: `</p><p>- DISCARD_UNCHANGED_HEARTBEAT:3h`

The number of open file descriptors.

Preprocessing:

- PROMETHEUS_PATTERN: sys_fd_open: value: ``

The amount of processor time used by Go's garbage collector across all nodes. During garbage collection, application code execution is paused.

Preprocessing:

- PROMETHEUS_PATTERN: sys_gc_pause_ns: value: `</p><p>- CHANGE_PER_SECOND</p><p>- MULTIPLIER:0.000000001`

The number of times that Go's garbage collector was invoked per second across all nodes.

Preprocessing:

- PROMETHEUS_PATTERN: sys_gc_count: value: ``

- CHANGE_PER_SECOND

Current number of Goroutines. This count should rise and fall based on load.

Preprocessing:

- PROMETHEUS_PATTERN: sys_goroutines: value: ``

Number of aborted KV transactions per second.

Preprocessing:

- PROMETHEUS_PATTERN: txn_aborts: value: ``

- CHANGE_PER_SECOND

Number of KV transactions (including 1PC) committed per second.

Preprocessing:

- PROMETHEUS_PATTERN: txn_commits: value: ``

- CHANGE_PER_SECOND

The number of live nodes in the cluster (will be 0 if this node is not itself live).

Preprocessing:

- PROMETHEUS_PATTERN: liveness_livenodes: value: `</p><p>- DISCARD_UNCHANGED_HEARTBEAT:3h`

Number of successful node liveness heartbeats per second from this node.

Preprocessing:

- PROMETHEUS_PATTERN: liveness_heartbeatsuccesses: value: ``

- CHANGE_PER_SECOND

Current bytes of memory allocated by the C layer.

Preprocessing:

- PROMETHEUS_PATTERN: sys_cgo_allocbytes: value: ``

Current bytes of memory allocated by the Go layer.

Preprocessing:

- PROMETHEUS_PATTERN: sys_go_allocbytes: value: ``

Total bytes of memory managed by the C layer.

Preprocessing:

- PROMETHEUS_PATTERN: sys_cgo_totalbytes: value: ``

Total bytes of memory managed by the Go layer.

Preprocessing:

- PROMETHEUS_PATTERN: sys_go_totalbytes: value: ``

Resident set size (RSS) of memory in use by the node.

Preprocessing:

- PROMETHEUS_PATTERN: sys_rss: value: ``

Bytes received per second on all network interfaces since this process started.

Preprocessing:

- PROMETHEUS_PATTERN: sys_host_net_recv_bytes: value: ``

- CHANGE_PER_SECOND

Bytes sent per second on all network interfaces since this process started.

Preprocessing:

- PROMETHEUS_PATTERN: sys_host_net_send_bytes: value: ``

- CHANGE_PER_SECOND

The number of errors encountered while attempting to write metrics to disk, per second.

Preprocessing:

- PROMETHEUS_PATTERN: timeseries_write_errors: value: ``

- CHANGE_PER_SECOND

The number of successfully written metric samples per second.

Preprocessing:

- PROMETHEUS_PATTERN: timeseries_write_samples: value: ``

- CHANGE_PER_SECOND

Number of RPCs stuck or retrying for a long time.

Preprocessing:

- PROMETHEUS_PATTERN: requests_slow_distsender: value: ``

Total amount of incoming SQL client network traffic in bytes per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_bytesin: value: ``

- CHANGE_PER_SECOND

Total amount of outgoing SQL client network traffic in bytes per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_bytesout: value: ``

- CHANGE_PER_SECOND

Current SQL statement memory usage for root.

Preprocessing:

- PROMETHEUS_PATTERN: sql_mem_root_current: value: ``

Total number of SQL DDL statements successfully executed per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_ddl_count: value: ``

- CHANGE_PER_SECOND

Total number of open SQL sessions.

Preprocessing:

- PROMETHEUS_PATTERN: sql_conns: value: ``

Total number of SQL statements currently active.

Preprocessing:

- PROMETHEUS_PATTERN: sql_distsql_queries_active: value: ``

A moving average of the number of DELETE statements successfully executed per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_delete_count: value: ``

- CHANGE_PER_SECOND

Number of SQL queries executed per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_query_count: value: ``

- CHANGE_PER_SECOND

The number of statements denied per second by a feature flag.

Preprocessing:

- PROMETHEUS_PATTERN: sql_feature_flag_denial: value: ``

- CHANGE_PER_SECOND

The number of distributed SQL flows currently active per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_distsql_flows_active: value: ``

- CHANGE_PER_SECOND

A moving average of the number of INSERT statements successfully executed per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_insert_count: value: ``

- CHANGE_PER_SECOND

A moving average of the number of SELECT statements successfully executed per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_select_count: value: ``

- CHANGE_PER_SECOND

A moving average of the number of UPDATE statements successfully executed per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_update_count: value: ``

- CHANGE_PER_SECOND

Total number of SQL statements that experienced contention per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_distsql_contended_queries_count: value: ``

- CHANGE_PER_SECOND

Total number of statements which returned a planning or runtime error per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_failure_count: value: ``

- CHANGE_PER_SECOND

Total number of currently open SQL transactions.

Preprocessing:

- PROMETHEUS_PATTERN: sql_txns_open: value: ``

Total number of SQL transaction abort errors per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_txn_abort_count: value: ``

- CHANGE_PER_SECOND

Total number of SQL transaction COMMIT statements successfully executed per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_txn_commit_count: value: ``

- CHANGE_PER_SECOND

Total number of SQL transaction BEGIN statements successfully executed per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_txn_begin_count: value: ``

- CHANGE_PER_SECOND

Total number of SQL transaction ROLLBACK statements successfully executed per second.

Preprocessing:

- PROMETHEUS_PATTERN: sql_txn_rollback_count: value: ``

- CHANGE_PER_SECOND

Process uptime.

Preprocessing:

- PROMETHEUS_PATTERN: sys_uptime: value: ``

Node certificate expires at that date.

Preprocessing:

- PROMETHEUS_PATTERN: security_certificate_expiration_node: value: `</p><p>⛔️ON_FAIL:DISCARD_VALUE -> </p><p>- DISCARD_UNCHANGED_HEARTBEAT:6h`

CA certificate expires at that date.

Preprocessing:

- PROMETHEUS_PATTERN: security_certificate_expiration_ca: value: `</p><p>⛔️ON_FAIL:DISCARD_VALUE -> </p><p>- DISCARD_UNCHANGED_HEARTBEAT:6h`

Number of logical bytes stored in live key-value pairs on this node. Live data excludes historical and deleted data.

Preprocessing:

- PROMETHEUS_PATTERN: livebytes{store="{#STORE}"}: value: ``

Number of physical bytes stored in system key-value pairs.

Preprocessing:

- PROMETHEUS_PATTERN: sysbytes{store="{#STORE}"}: value: ``

Available storage capacity.

Preprocessing:

- PROMETHEUS_PATTERN: capacity_available{store="{#STORE}"}: value: ``

Total storage capacity. This value may be explicitly set using --store. If a store size has not been set, this metric displays the actual disk capacity.

Preprocessing:

- PROMETHEUS_PATTERN: capacity{store="{#STORE}"}: value: `</p><p>- DISCARD_UNCHANGED_HEARTBEAT:3h`

Disk space in use by CockroachDB data on this node. This excludes the Cockroach binary, operating system, and other system files.

Preprocessing:

- PROMETHEUS_PATTERN: capacity_used{store="{#STORE}"}: value: ``

Available storage capacity in %.

Expression:

Number of lease holders.

Preprocessing:

- PROMETHEUS_PATTERN: replicas_leaseholders{store="{#STORE}"}: value: ``

Number of logical bytes stored in key-value pairs on this node. This includes historical and deleted data.

Preprocessing:

- PROMETHEUS_PATTERN: totalbytes{store="{#STORE}"}: value: ``

Number of kv-level requests received per second by the store, averaged over a large time period as used in rebalancing decisions.

Preprocessing:

- PROMETHEUS_PATTERN: rebalancing_queriespersecond{store="{#STORE}"}: value: ``

Number of keys written (i.e. applied by raft) per second to the store, averaged over a large time period as used in rebalancing decisions.

Preprocessing:

- PROMETHEUS_PATTERN: rebalancing_writespersecond{store="{#STORE}"}: value: ``

Number of replicas which failed processing in the consistency checker queue per second.

Preprocessing:

- PROMETHEUS_PATTERN: queue_consistency_process_failure{store="{#STORE}"}: value: ``

- CHANGE_PER_SECOND

Number of replicas which failed processing in the GC queue per second.

Preprocessing:

- PROMETHEUS_PATTERN: queue_gc_process_failure{store="{#STORE}"}: value: ``

- CHANGE_PER_SECOND

Number of replicas which failed processing in the Raft log queue per second.

Preprocessing:

- PROMETHEUS_PATTERN: queue_raftlog_process_failure{store="{#STORE}"}: value: ``

- CHANGE_PER_SECOND

Number of replicas which failed processing in the Raft repair queue per second.

Preprocessing:

- PROMETHEUS_PATTERN: queue_raftsnapshot_process_failure{store="{#STORE}"}: value: ``

- CHANGE_PER_SECOND

Number of replicas which failed processing in the replica GC queue per second.

Preprocessing:

- PROMETHEUS_PATTERN: queue_replicagc_process_failure{store="{#STORE}"}: value: ``

- CHANGE_PER_SECOND

Number of replicas which failed processing in the replicate queue per second.

Preprocessing:

- PROMETHEUS_PATTERN: queue_replicate_process_failure{store="{#STORE}"}: value: ``

- CHANGE_PER_SECOND

Number of replicas which failed processing in the split queue per second.

Preprocessing:

- PROMETHEUS_PATTERN: queue_split_process_failure{store="{#STORE}"}: value: ``

- CHANGE_PER_SECOND

Number of replicas which failed processing in the time series maintenance queue per second.

Preprocessing:

- PROMETHEUS_PATTERN: queue_tsmaintenance_process_failure{store="{#STORE}"}: value: ``

- CHANGE_PER_SECOND

Number of ranges.

Preprocessing:

- PROMETHEUS_PATTERN: ranges{store="{#STORE}"}: value: ``

Number of ranges with fewer live replicas than needed for quorum.

Preprocessing:

- PROMETHEUS_PATTERN: ranges_unavailable{store="{#STORE}"}: value: ``

Number of ranges with fewer live replicas than the replication target.

Preprocessing:

- PROMETHEUS_PATTERN: ranges_underreplicated{store="{#STORE}"}: value: ``

The average number of real read operations executed per logical read operation.

Preprocessing:

- PROMETHEUS_PATTERN: rocksdb_read_amplification{store="{#STORE}"}: value: ``

Count of block cache hits per second.

Preprocessing:

- PROMETHEUS_PATTERN: rocksdb_block_cache_hits{store="{#STORE}"}: value: ``

- CHANGE_PER_SECOND

Count of block cache misses per second.

Preprocessing:

- PROMETHEUS_PATTERN: rocksdb_block_cache_misses{store="{#STORE}"}: value: ``

- CHANGE_PER_SECOND

Block cache hit ratio in %.

Expression:

Number of replicas.

Preprocessing:

- PROMETHEUS_PATTERN: replicas{store="{#STORE}"}: value: ``

Number of quiesced replicas.

Preprocessing:

- PROMETHEUS_PATTERN: replicas_quiescent{store="{#STORE}"}: value: ``

Number of requests that have been stuck for a long time acquiring latches.

Preprocessing:

- PROMETHEUS_PATTERN: requests_slow_latch{store="{#STORE}"}: value: ``

Number of requests that have been stuck for a long time acquiring a lease.

Preprocessing:

- PROMETHEUS_PATTERN: requests_slow_lease{store="{#STORE}"}: value: ``

Number of requests that have been stuck for a long time in raft.

Preprocessing:

- PROMETHEUS_PATTERN: requests_slow_raft{store="{#STORE}"}: value: ``

The number of SSTables in use.

Preprocessing:

- PROMETHEUS_PATTERN: rocksdb_num_sstables{store="{#STORE}"}: value: ``

Get raw metrics from the Prometheus endpoint.

Preprocessing:

- CHECK_NOT_SUPPORTED

⛔️ON_FAIL: DISCARD_VALUE ->

Get node /health endpoint

Preprocessing:

- CHECK_NOT_SUPPORTED

⛔️ON_FAIL: DISCARD_VALUE ->

- REGEX: HTTP.*\s(\d+): \1

- DISCARD_UNCHANGED_HEARTBEAT: 3h

Get node /health?ready=1 endpoint

Preprocessing:

- CHECK_NOT_SUPPORTED

⛔️ON_FAIL: DISCARD_VALUE ->

- REGEX: HTTP.*\s(\d+): \1

- DISCARD_UNCHANGED_HEARTBEAT: 3h

-

Cockroach-measured clock offset is nearing limit (by default, servers kill themselves at 400ms from the mean).

-

Getting close to open file descriptor limit.

Node is not executing SQL despite having connections.

-

Uptime is less than 10 minutes.

Zabbix has not received data for items for the last 5 minutes.

Depends on:

- CockroachDB: Service is down

Node certificate expires soon.

CA certificate expires soon.

Storage is running low on free space (less than {$COCKROACHDB.STORE.USED.MIN.WARN}% available).

Recovery expression:

Depends on:

- CockroachDB: Storage [{#STORE}]: Available storage capacity is critically low

Storage is running critically low on free space (less than {$COCKROACHDB.STORE.USED.MIN.CRIT}% available).

Recovery expression:

Node's /health endpoint has returned HTTP 500 Internal Server Error which indicates unhealthy mode.

Depends on:

- CockroachDB: Service is down

Node's /health?ready=1 endpoint has returned HTTP 503 Service Unavailable. Possible reasons:

- node is in the wait phase of the node shutdown sequence;

- node is unable to communicate with a majority of the other nodes in the cluster, likely because the cluster is unavailable due to too many nodes being down.

Depends on:

- CockroachDB: Service is down