Maximum size of distributed files queue to insert for trigger expression.

Maximum number of delayed inserts for trigger expression.

Filter of discoverable databases

Filter to exclude discovered databases

Filter of discoverable dictionaries

Filter to exclude discovered dictionaries

Maximum diff between log_pointer and log_max_index.

Maximum number of smth for trigger expression

Maximum number of parts per partition for trigger expression.

-

The port of ClickHouse HTTP endpoint

Maximum ClickHouse query time in seconds for trigger expression

Maximum size of the queue for operations waiting to be performed for trigger expression.

Replication lag across all tables for trigger expression.

Request scheme which may be http or https

-

Info about dictionaries

Filter:

- {#NAME} MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DICT.MATCHES}

- {#NAME} NOT_MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DICT.NOT_MATCHES}

Info about replicas

Filter:

- {#DB} MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.MATCHES}

- {#DB} NOT_MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.NOT_MATCHES}

Info about tables

Filter:

- {#DB} MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.MATCHES}

- {#DB} NOT_MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.NOT_MATCHES}

Get longest running query.

-

Preprocessing:

- DISCARD_UNCHANGED_HEARTBEAT: 10m

Preprocessing:

- REGEX: Ok\. 1

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- DISCARD_UNCHANGED_HEARTBEAT: 10m

Version of the server

Preprocessing:

- DISCARD_UNCHANGED_HEARTBEAT: 1d

Revision of the server.

Preprocessing:

- JSONPATH: $[?(@.metric == "Revision")].value.first()

Number of seconds since ClickHouse server start

Preprocessing:

- JSONPATH: $[?(@.metric == "Uptime")].value.first()

Number of queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing:

- JSONPATH: $[?(@.data.event == "Query")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Number of SELECT queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing:

- JSONPATH: $[?(@.event == "SelectQuery")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Number of INSERT queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing:

- JSONPATH: $[?(@.event == "InsertQuery")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

"Number of INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree table."

Preprocessing:

- JSONPATH: $[?(@.metric == "DelayedInserts")].value.first()

Number of executing queries

Preprocessing:

- JSONPATH: $[?(@.metric == "Query")].value.first()

Number of executing background merges

Preprocessing:

- JSONPATH: $[?(@.metric == "Merge")].value.first()

The number of uncompressed bytes inserted in all tables.

Preprocessing:

- JSONPATH: $[?(@.event == "InsertedBytes")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

"Number of bytes (the number of bytes before decompression) read from compressed sources (files, network)."

Preprocessing:

- JSONPATH: $[?(@.event == "ReadCompressedBytes")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

The number of rows inserted in all tables.

Preprocessing:

- JSONPATH: $[?(@.event == "InsertedRows")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Rows read for background merges.

Preprocessing:

- JSONPATH: $[?(@.event == "MergedRows")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Uncompressed bytes that were read for background merges

Preprocessing:

- JSONPATH: $[?(@.event == "MergedUncompressedBytes")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Clickhouse MergeTree table engine split each INSERT query to partitions (PARTITION BY expression) and add one or more PARTS per INSERT inside each partition,

after that background merge process run.

Preprocessing:

- JSONPATH: $[?(@.metric == "MaxPartCountForPartition")].value.first()

Number of connections to TCP server (clients with native interface).

Preprocessing:

- JSONPATH: $[?(@.metric == "TCPConnection")].value.first()

Number of connections to HTTP server.

Preprocessing:

- JSONPATH: $[?(@.metric == "HTTPConnection")].value.first()

Number of connections to remote servers sending data that was INSERTed into Distributed tables.

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedSend")].value.first()

Number of connections to MySQL server.

Preprocessing:

- JSONPATH: $[?(@.metric == "MySQLConnection")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

Number of connections from other replicas to fetch parts.

Preprocessing:

- JSONPATH: $[?(@.metric == "InterserverConnection")].value.first()

Network errors (timeouts and connection failures) during query execution, background pool tasks and DNS cache update.

Preprocessing:

- JSONPATH: $[?(@.event == "NetworkErrors")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Number of read (read, pread, io_getevents, etc.) syscalls in fly

Preprocessing:

- JSONPATH: $[?(@.metric == "Read")].value.first()

Number of write (write, pwrite, io_getevents, etc.) syscalls in fly

Preprocessing:

- JSONPATH: $[?(@.metric == "Write")].value.first()

"Total number of bytes allocated by the application."

Preprocessing:

- JSONPATH: $[?(@.metric == "jemalloc.allocated")].value.first()

Maximum number of bytes in physically resident data pages mapped by the allocator,

comprising all pages dedicated to allocator metadata, pages backing active allocations,

and unused dirty pages.

Preprocessing:

- JSONPATH: $[?(@.metric == "jemalloc.resident")].value.first()

"Total number of bytes in active extents mapped by the allocator."

Preprocessing:

- JSONPATH: $[?(@.metric == "jemalloc.mapped")].value.first()

"Total amount of memory (bytes) allocated in currently executing queries."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTracking")].value.first()

"Total amount of memory (bytes) allocated in background processing pool (that is dedicated for background merges, mutations and fetches).

Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa. This happens naturally due to caches for tables indexes and doesn't indicate memory leaks."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingInBackgroundProcessingPool")].value.first()

"Total amount of memory (bytes) allocated in background processing pool (that is dedicated for background moves). Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa.

This happens naturally due to caches for tables indexes and doesn't indicate memory leaks."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingInBackgroundMoveProcessingPool")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

"Total amount of memory (bytes) allocated in background schedule pool (that is dedicated for bookkeeping tasks of Replicated tables)."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingInBackgroundSchedulePool")].value.first()

Total amount of memory (bytes) allocated for background merges. Included in MemoryTrackingInBackgroundProcessingPool. Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa.

This happens naturally due to caches for tables indexes and doesn't indicate memory leaks.

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingForMerges")].value.first()

Number of pending files to process for asynchronous insertion into Distributed tables. Number of files for every shard is summed.

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedFilesToInsert")].value.first()

Connection retries in replicated DB connection pool

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedConnectionFailTry")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

"Connection failures after all retries in replicated DB connection pool"

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedConnectionFailAtAll")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Maximum replica queue delay relative to current time

Preprocessing:

- JSONPATH: $[?(@.metric == "ReplicasMaxAbsoluteDelay")].value.first()

Preprocessing:

- JSONPATH: $[?(@.metric == "ReplicasSumQueueSize")].value.first()

Number of Replicated tables that are currently in readonly state

due to re-initialization after ZooKeeper session loss

or due to startup without ZooKeeper configured.

Preprocessing:

- JSONPATH: $[?(@.metric == "ReadonlyReplica")].value.first()

Table size in bytes. Database: {#DB}, table: {#TABLE}

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].bytes.first()

Number of parts of the table. Database: {#DB}, table: {#TABLE}

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].parts.first()

Number of rows in the table. Database: {#DB}, table: {#TABLE}

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].rows.first()

Database size in bytes.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}")].bytes.sum()

Whether the replica is in read-only mode.

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when re-initializing sessions in ZooKeeper, and during session re-initialization in ZooKeeper.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].is_readonly.first()

True if the ZooKeeper session expired

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].is_session_expired.first()

Number of data parts that will appear as the result of INSERTs or merges that haven't been done yet.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].future_parts.first()

Number of data parts in the queue for verification. A part is put in the verification queue if there is suspicion that it might be damaged.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].parts_to_check.first()

Size of the queue for operations waiting to be performed.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].queue_size.first()

Number of inserts of blocks of data that need to be made.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].inserts_in_queue.first()

Number of merges waiting to be made.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].merges_in_queue.first()

Maximum entry number in the log of general activity. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].log_max_index.first()

Maximum entry number in the log of general activity that the replica copied to its execution queue, plus one. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].log_pointer.first()

Total number of known replicas of this table. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].total_replicas.first()

Number of replicas of this table that have a session in ZooKeeper (i.e., the number of functioning replicas). (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].active_replicas.first()

Difference between log_max_index and log_pointer

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].replica_lag.first()

The amount of RAM the dictionary uses.

Preprocessing:

- JSONPATH: $[?(@.name == "{#NAME}")].bytes_allocated.first()

Number of items stored in the dictionary.

Preprocessing:

- JSONPATH: $[?(@.name == "{#NAME}")].element_count.first()

The percentage filled in the dictionary (for a hashed dictionary, the percentage filled in the hash table).

Preprocessing:

- JSONPATH: $[?(@.name == "{#NAME}")].bytes_allocated.first()

- MULTIPLIER: 100

Number of sessions (connections) to ZooKeeper. Should be no more than one.

Preprocessing:

- JSONPATH: $[?(@.metric == "ZooKeeperSession")].value.first()

Number of watches (e.g., event subscriptions) in ZooKeeper.

Preprocessing:

- JSONPATH: $[?(@.metric == "ZooKeeperWatch")].value.first()

Number of requests to ZooKeeper in progress.

Preprocessing:

- JSONPATH: $[?(@.metric == "ZooKeeperRequest")].value.first()

Time spent in waiting for ZooKeeper operations.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperWaitMicroseconds")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- MULTIPLIER: 0.000001

- CHANGE_PER_SECOND

Count of ZooKeeper exceptions that does not belong to user/hardware exceptions.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperOtherExceptions")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Count of ZooKeeper exceptions caused by session moved/expired, connection loss, marshalling error, operation timed out and invalid zhandle state.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperHardwareExceptions")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Count of ZooKeeper exceptions caused by no znodes, bad version, node exists, node empty and no children for ephemeral.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperUserExceptions")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Get information about the number of events that have occurred in the system.

Preprocessing:

- JSONPATH: $.data

Get metrics which can be calculated instantly, or have a current value format JSONEachRow

Preprocessing:

- JSONPATH: $.data

Get metrics that are calculated periodically in the background

Preprocessing:

- JSONPATH: $.data

Get information about settings that are currently in use.

Preprocessing:

- JSONPATH: $.data

- DISCARD_UNCHANGED_HEARTBEAT: 1h

-

Preprocessing:

- JSONPATH: $.data

-

Preprocessing:

- JSONPATH: $.data

-

Preprocessing:

- JSONPATH: $.data

-

Manual close: YES

-

Manual close: YES

-

Manual close: YES

Depends on:

- ClickHouse: Port {$CLICKHOUSE.PORT} is unavailable

ClickHouse version has changed. Ack to close.

Manual close: YES

Uptime is less than 10 minutes.

Manual close: YES

Zabbix has not received data for items for the last 30 minutes

Manual close: YES

Depends on:

- ClickHouse: Service is down

Clickhouse have INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree, please decrease INSERT frequency

Manual close: YES

Descease INSERT queries frequency.

Clickhouse MergeTree table engine split each INSERT query to partitions (PARTITION BY expression)

and add one or more PARTS per INSERT inside each partition,

after that background merge process run, and when you have too much unmerged parts inside partition,

SELECT queries performance can significate degrade, so clickhouse try delay insert, or abort it.

Manual close: YES

Number of errors (timeouts and connection failures) during query execution, background pool tasks and DNS cache update is too high.

"Clickhouse servers and <remote_servers> in config.xml

https://clickhouse.tech/docs/en/operations/table_engines/distributed/"

Manual close: YES

When replica have too much lag, it can be skipped from Distributed SELECT Queries without errors

and you will have wrong query results.

Manual close: YES

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when re-initializing sessions in ZooKeeper, and during session re-initialization in ZooKeeper.

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when re-initializing sessions in ZooKeeper, and during session re-initialization in ZooKeeper.

-

-

-

Number of sessions (connections) to ZooKeeper.

Should be no more than one, because using more than one connection to ZooKeeper may lead to bugs due to lack of linearizability (stale reads) that ZooKeeper consistency model allows.

ClickHouse configuration has been changed. Ack to close.

Manual close: YES

Maximum number of network errors for trigger expression

The port of ClickHouse HTTP endpoint

Request scheme which may be http or https

Filter of discoverable databases

Filter to exclude discovered databases

Filter of discoverable dictionaries

Filter to exclude discovered dictionaries

Filter of discoverable tables

Filter to exclude discovered tables

Maximum ClickHouse query time in seconds for trigger expression

Maximum size of the queue for operations waiting to be performed for trigger expression.

Maximum diff between log_pointer and log_max_index.

Replication lag across all tables for trigger expression.

Maximum size of distributed files queue to insert for trigger expression.

Maximum number of parts per partition for trigger expression.

Maximum number of delayed inserts for trigger expression.

Get information about the number of events that have occurred in the system.

Preprocessing

• JSON Path: $.data

Get metrics which can be calculated instantly, or have a current value format JSONEachRow

Preprocessing

• JSON Path: $.data

Get metrics that are calculated periodically in the background

Preprocessing

• JSON Path: $.data

Get information about settings that are currently in use.

Preprocessing

• JSON Path: $.data

• Discard unchanged with heartbeat: 1h

Get longest running query.

Preprocessing

• Discard unchanged with heartbeat: 10m

Preprocessing

• Regular expression: Ok\. 1

  ⛔️Custom on fail: Set value to: 0

• Discard unchanged with heartbeat: 10m

Version of the server

Preprocessing

• Discard unchanged with heartbeat: 1d

Revision of the server.

Preprocessing

• JSON Path: $[?(@.metric == "Revision")].value.first()

Number of seconds since ClickHouse server start

Preprocessing

• JSON Path: $[?(@.metric == "Uptime")].value.first()

Number of queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing

• JSON Path: $[?(@.data.event == "Query")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Number of SELECT queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing

• JSON Path: $[?(@.event == "SelectQuery")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Number of INSERT queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing

• JSON Path: $[?(@.event == "InsertQuery")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Number of INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree table.

Preprocessing

• JSON Path: $[?(@.metric == "DelayedInserts")].value.first()

Number of executing queries

Preprocessing

• JSON Path: $[?(@.metric == "Query")].value.first()

Number of executing background merges

Preprocessing

• JSON Path: $[?(@.metric == "Merge")].value.first()

The number of uncompressed bytes inserted in all tables.

Preprocessing

• JSON Path: $[?(@.event == "InsertedBytes")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Number of bytes (the number of bytes before decompression) read from compressed sources (files, network).

Preprocessing

• JSON Path: $[?(@.event == "ReadCompressedBytes")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

The number of rows inserted in all tables.

Preprocessing

• JSON Path: $[?(@.event == "InsertedRows")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Rows read for background merges.

Preprocessing

• JSON Path: $[?(@.event == "MergedRows")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Uncompressed bytes that were read for background merges

Preprocessing

• JSON Path: $[?(@.event == "MergedUncompressedBytes")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Clickhouse MergeTree table engine split each INSERT query to partitions (PARTITION BY expression) and add one or more PARTS per INSERT inside each partition, after that background merge process run.

Preprocessing

• JSON Path: $[?(@.metric == "MaxPartCountForPartition")].value.first()

Number of connections to TCP server (clients with native interface).

Preprocessing

• JSON Path: $[?(@.metric == "TCPConnection")].value.first()

Number of connections to HTTP server.

Preprocessing

• JSON Path: $[?(@.metric == "HTTPConnection")].value.first()

Number of connections to remote servers sending data that was INSERTed into Distributed tables.

Preprocessing

• JSON Path: $[?(@.metric == "DistributedSend")].value.first()

Number of connections to MySQL server.

Preprocessing

• JSON Path: $[?(@.metric == "MySQLConnection")].value.first()

  ⛔️Custom on fail: Set value to: 0

Number of connections from other replicas to fetch parts.

Preprocessing

• JSON Path: $[?(@.metric == "InterserverConnection")].value.first()

Network errors (timeouts and connection failures) during query execution, background pool tasks and DNS cache update.

Preprocessing

• JSON Path: $[?(@.event == "NetworkErrors")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Number of sessions (connections) to ZooKeeper. Should be no more than one.

Preprocessing

• JSON Path: $[?(@.metric == "ZooKeeperSession")].value.first()

Number of watches (e.g., event subscriptions) in ZooKeeper.

Preprocessing

• JSON Path: $[?(@.metric == "ZooKeeperWatch")].value.first()

Number of requests to ZooKeeper in progress.

Preprocessing

• JSON Path: $[?(@.metric == "ZooKeeperRequest")].value.first()

Time spent in waiting for ZooKeeper operations.

Preprocessing

• JSON Path: $[?(@.event == "ZooKeeperWaitMicroseconds")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Custom multiplier: 1.0E-6

• Change per second

Count of ZooKeeper exceptions that does not belong to user/hardware exceptions.

Preprocessing

• JSON Path: $[?(@.event == "ZooKeeperOtherExceptions")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Count of ZooKeeper exceptions caused by session moved/expired, connection loss, marshalling error, operation timed out and invalid zhandle state.

Preprocessing

• JSON Path: $[?(@.event == "ZooKeeperHardwareExceptions")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Count of ZooKeeper exceptions caused by no znodes, bad version, node exists, node empty and no children for ephemeral.

Preprocessing

• JSON Path: $[?(@.event == "ZooKeeperUserExceptions")].value.first()

  ⛔️Custom on fail: Set value to: 0

• Change per second

Number of read (read, pread, io_getevents, etc.) syscalls in fly

Preprocessing

• JSON Path: $[?(@.metric == "Read")].value.first()

Number of write (write, pwrite, io_getevents, etc.) syscalls in fly

Preprocessing

• JSON Path: $[?(@.metric == "Write")].value.first()

Total number of bytes allocated by the application.

Preprocessing

• JSON Path: $[?(@.metric == "jemalloc.allocated")].value.first()

Maximum number of bytes in physically resident data pages mapped by the allocator,

comprising all pages dedicated to allocator metadata, pages backing active allocations,

and unused dirty pages.

Preprocessing

• JSON Path: $[?(@.metric == "jemalloc.resident")].value.first()

Total number of bytes in active extents mapped by the allocator.

Preprocessing

• JSON Path: $[?(@.metric == "jemalloc.mapped")].value.first()

Total amount of memory (bytes) allocated in currently executing queries.

Preprocessing

• JSON Path: $[?(@.metric == "MemoryTracking")].value.first()

Total amount of memory (bytes) allocated in background processing pool (that is dedicated for background merges, mutations and fetches).

Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa. This happens naturally due to caches for tables indexes and doesn't indicate memory leaks.

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Set value to: 0

Total amount of memory (bytes) allocated in background processing pool (that is dedicated for background moves). Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa.

This happens naturally due to caches for tables indexes and doesn't indicate memory leaks.

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Set value to: 0

Total amount of memory (bytes) allocated in background schedule pool (that is dedicated for bookkeeping tasks of Replicated tables).

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Set value to: 0

Total amount of memory (bytes) allocated for background merges. Included in MemoryTrackingInBackgroundProcessingPool. Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa.

This happens naturally due to caches for tables indexes and doesn't indicate memory leaks.

Preprocessing

• JSON Path: $[?(@.metric == "MemoryTrackingForMerges")].value.first()

  ⛔️Custom on fail: Set value to: 0

Number of pending files to process for asynchronous insertion into Distributed tables. Number of files for every shard is summed.

Preprocessing

• JSON Path: $[?(@.metric == "DistributedFilesToInsert")].value.first()

Connection retries in replicated DB connection pool

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Set value to: 0

• Change per second

Connection failures after all retries in replicated DB connection pool

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Set value to: 0

• Change per second

Maximum replica queue delay relative to current time

Preprocessing

• JSON Path: $[?(@.metric == "ReplicasMaxAbsoluteDelay")].value.first()

Number of replication tasks in queue

Preprocessing

• JSON Path: $[?(@.metric == "ReplicasSumQueueSize")].value.first()

Number of Replicated tables that are currently in readonly state due to re-initialization after ZooKeeper session loss or due to startup without ZooKeeper configured.

Preprocessing

• JSON Path: $[?(@.metric == "ReadonlyReplica")].value.first()

Get information about replicas.

Preprocessing

• JSON Path: $.data

Get information about databases.

Preprocessing

• JSON Path: $.data

Get information about tables.

Preprocessing

• JSON Path: $.data

Get information about dictionaries.

Preprocessing

• JSON Path: $.data

ClickHouse configuration has been changed. Acknowledge to close the problem manually.

• ClickHouse: Port {$CLICKHOUSE.PORT} is unavailable

The ClickHouse version has changed. Acknowledge to close the problem manually.

The host uptime is less than 10 minutes.

Zabbix has not received any data for items for the last 30 minutes.

• ClickHouse: Service is down

Clickhouse have INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree, please decrease INSERT frequency

Descease INSERT queries frequency.
Clickhouse MergeTree table engine split each INSERT query to partitions (PARTITION BY expression)
and add one or more PARTS per INSERT inside each partition,
after that background merge process run, and when you have too much unmerged parts inside partition,
SELECT queries performance can significate degrade, so clickhouse try delay insert, or abort it.

Number of errors (timeouts and connection failures) during query execution, background pool tasks and DNS cache update is too high.

Number of sessions (connections) to ZooKeeper.
Should be no more than one, because using more than one connection to ZooKeeper may lead to bugs due to lack of linearizability (stale reads) that ZooKeeper consistency model allows.

Clickhouse servers and <remote_servers> in config.xml (https://clickhouse.tech/docs/en/operations/table_engines/distributed/)

When replica have too much lag, it can be skipped from Distributed SELECT Queries without errors
and you will have wrong query results.

Info about tables

The item gets information about {#TABLE} table of {#DB} database.

Preprocessing

• JSON Path: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].first()

  ⛔️Custom on fail: Discard value

Table size in bytes. Database: {#DB}, table: {#TABLE}

Preprocessing

• JSON Path: $.bytes

Number of parts of the table. Database: {#DB}, table: {#TABLE}

Preprocessing

• JSON Path: $.parts

Number of rows in the table. Database: {#DB}, table: {#TABLE}

Preprocessing

• JSON Path: $.rows

Info about replicas

The item gets information about replicas of {#TABLE} table of {#DB} database.

Preprocessing

• JSON Path: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].first()

  ⛔️Custom on fail: Discard value

Whether the replica is in read-only mode.

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when re-initializing sessions in ZooKeeper, and during session re-initialization in ZooKeeper.

Preprocessing

• JSON Path: $.is_readonly

True if the ZooKeeper session expired

Preprocessing

• JSON Path: $.is_session_expired

Number of data parts that will appear as the result of INSERTs or merges that haven't been done yet.

Preprocessing

• JSON Path: $.future_parts

Number of data parts in the queue for verification. A part is put in the verification queue if there is suspicion that it might be damaged.

Preprocessing

• JSON Path: $.parts_to_check

Size of the queue for operations waiting to be performed.

Preprocessing

• JSON Path: $.queue_size

Number of inserts of blocks of data that need to be made.

Preprocessing

• JSON Path: $.inserts_in_queue

Number of merges waiting to be made.

Preprocessing

• JSON Path: $.merges_in_queue

Maximum entry number in the log of general activity. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing

• JSON Path: $.log_max_index

Maximum entry number in the log of general activity that the replica copied to its execution queue, plus one. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing

• JSON Path: $.log_pointer

Total number of known replicas of this table. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing

• JSON Path: $.total_replicas

Number of replicas of this table that have a session in ZooKeeper (i.e., the number of functioning replicas). (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing

• JSON Path: $.active_replicas

Difference between log_max_index and log_pointer

Preprocessing

• JSON Path: $.replica_lag

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when re-initializing sessions in ZooKeeper, and during session re-initialization in ZooKeeper.

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when re-initializing sessions in ZooKeeper, and during session re-initialization in ZooKeeper.

Info about dictionaries

The item gets information about {#NAME} dictionary.

Preprocessing

• JSON Path: $[?(@.name == "{#NAME}")].first()

  ⛔️Custom on fail: Discard value

The amount of RAM the dictionary uses.

Preprocessing

• JSON Path: $.bytes_allocated

Number of items stored in the dictionary.

Preprocessing

• JSON Path: $.element_count

The percentage filled in the dictionary (for a hashed dictionary, the percentage filled in the hash table).

Preprocessing

• JSON Path: $.bytes_allocated

• Custom multiplier: 100

Info about databases

The item gets information about {#DB} database.

Preprocessing

• JSON Path: $[?(@.database == "{#DB}")].first()

  ⛔️Custom on fail: Discard value

Database size in bytes.

Preprocessing

• JSON Path: $.bytes

Number of tables in {#DB} database.

Preprocessing

• JSON Path: $.tables

Maximum size of distributed files queue to insert for trigger expression.

Maximum number of delayed inserts for trigger expression.

Filter of discoverable databases

Filter to exclude discovered databases

Filter of discoverable dictionaries

Filter to exclude discovered dictionaries

Maximum diff between log_pointer and log_max_index.

Maximum number of smth for trigger expression

Maximum number of parts per partition for trigger expression.

-

The port of ClickHouse HTTP endpoint

Maximum ClickHouse query time in seconds for trigger expression

Maximum size of the queue for operations waiting to be performed for trigger expression.

Replication lag across all tables for trigger expression.

Request scheme which may be http or https

-

Info about tables

Filter:

- {#DB} MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.MATCHES}

- {#DB} NOT_MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.NOT_MATCHES}

Info about replicas

Filter:

- {#DB} MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.MATCHES}

- {#DB} NOT_MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.NOT_MATCHES}

Info about dictionaries

Filter:

- {#NAME} MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DICT.MATCHES}

- {#NAME} NOT_MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DICT.NOT_MATCHES}

Get longest running query.

-

Preprocessing:

- DISCARD_UNCHANGED_HEARTBEAT: 10m

Preprocessing:

- REGEX: Ok\. 1

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- DISCARD_UNCHANGED_HEARTBEAT: 10m

Version of the server

Preprocessing:

- DISCARD_UNCHANGED_HEARTBEAT: 1d

Revision of the server.

Preprocessing:

- JSONPATH: $[?(@.metric == "Revision")].value.first()

Number of seconds since ClickHouse server start

Preprocessing:

- JSONPATH: $[?(@.metric == "Uptime")].value.first()

Number of queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing:

- JSONPATH: $[?(@.data.event == "Query")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Number of SELECT queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing:

- JSONPATH: $[?(@.event == "SelectQuery")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Number of INSERT queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing:

- JSONPATH: $[?(@.event == "InsertQuery")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

"Number of INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree table."

Preprocessing:

- JSONPATH: $[?(@.metric == "DelayedInserts")].value.first()

Number of executing queries

Preprocessing:

- JSONPATH: $[?(@.metric == "Query")].value.first()

Number of executing background merges

Preprocessing:

- JSONPATH: $[?(@.metric == "Merge")].value.first()

The number of uncompressed bytes inserted in all tables.

Preprocessing:

- JSONPATH: $[?(@.event == "InsertedBytes")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

"Number of bytes (the number of bytes before decompression) read from compressed sources (files, network)."

Preprocessing:

- JSONPATH: $[?(@.event == "ReadCompressedBytes")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

The number of rows inserted in all tables.

Preprocessing:

- JSONPATH: $[?(@.event == "InsertedRows")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Rows read for background merges.

Preprocessing:

- JSONPATH: $[?(@.event == "MergedRows")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Uncompressed bytes that were read for background merges

Preprocessing:

- JSONPATH: $[?(@.event == "MergedUncompressedBytes")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Clickhouse MergeTree table engine split each INSERT query to partitions (PARTITION BY expression) and add one or more PARTS per INSERT inside each partition,

after that background merge process run.

Preprocessing:

- JSONPATH: $[?(@.metric == "MaxPartCountForPartition")].value.first()

Number of connections to TCP server (clients with native interface).

Preprocessing:

- JSONPATH: $[?(@.metric == "TCPConnection")].value.first()

Number of connections to HTTP server.

Preprocessing:

- JSONPATH: $[?(@.metric == "HTTPConnection")].value.first()

Number of connections to remote servers sending data that was INSERTed into Distributed tables.

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedSend")].value.first()

Number of connections to MySQL server.

Preprocessing:

- JSONPATH: $[?(@.metric == "MySQLConnection")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

Number of connections from other replicas to fetch parts.

Preprocessing:

- JSONPATH: $[?(@.metric == "InterserverConnection")].value.first()

Network errors (timeouts and connection failures) during query execution, background pool tasks and DNS cache update.

Preprocessing:

- JSONPATH: $[?(@.event == "NetworkErrors")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Number of read (read, pread, io_getevents, etc.) syscalls in fly

Preprocessing:

- JSONPATH: $[?(@.metric == "Read")].value.first()

Number of write (write, pwrite, io_getevents, etc.) syscalls in fly

Preprocessing:

- JSONPATH: $[?(@.metric == "Write")].value.first()

"Total number of bytes allocated by the application."

Preprocessing:

- JSONPATH: $[?(@.metric == "jemalloc.allocated")].value.first()

Maximum number of bytes in physically resident data pages mapped by the allocator,

comprising all pages dedicated to allocator metadata, pages backing active allocations,

and unused dirty pages.

Preprocessing:

- JSONPATH: $[?(@.metric == "jemalloc.resident")].value.first()

"Total number of bytes in active extents mapped by the allocator."

Preprocessing:

- JSONPATH: $[?(@.metric == "jemalloc.mapped")].value.first()

"Total amount of memory (bytes) allocated in currently executing queries."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTracking")].value.first()

"Total amount of memory (bytes) allocated in background processing pool (that is dedicated for background merges, mutations and fetches).

Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa. This happens naturally due to caches for tables indexes and doesn't indicate memory leaks."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingInBackgroundProcessingPool")].value.first()

"Total amount of memory (bytes) allocated in background processing pool (that is dedicated for background moves). Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa.

This happens naturally due to caches for tables indexes and doesn't indicate memory leaks."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingInBackgroundMoveProcessingPool")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

"Total amount of memory (bytes) allocated in background schedule pool (that is dedicated for bookkeeping tasks of Replicated tables)."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingInBackgroundSchedulePool")].value.first()

Total amount of memory (bytes) allocated for background merges. Included in MemoryTrackingInBackgroundProcessingPool. Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa.

This happens naturally due to caches for tables indexes and doesn't indicate memory leaks.

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingForMerges")].value.first()

Number of pending files to process for asynchronous insertion into Distributed tables. Number of files for every shard is summed.

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedFilesToInsert")].value.first()

Connection retries in replicated DB connection pool

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedConnectionFailTry")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

"Connection failures after all retries in replicated DB connection pool"

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedConnectionFailAtAll")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Maximum replica queue delay relative to current time

Preprocessing:

- JSONPATH: $[?(@.metric == "ReplicasMaxAbsoluteDelay")].value.first()

Preprocessing:

- JSONPATH: $[?(@.metric == "ReplicasSumQueueSize")].value.first()

Number of Replicated tables that are currently in readonly state

due to re-initialization after ZooKeeper session loss

or due to startup without ZooKeeper configured.

Preprocessing:

- JSONPATH: $[?(@.metric == "ReadonlyReplica")].value.first()

Table size in bytes. Database: {#DB}, table: {#TABLE}

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].bytes.first()

Number of parts of the table. Database: {#DB}, table: {#TABLE}

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].parts.first()

Number of rows in the table. Database: {#DB}, table: {#TABLE}

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].rows.first()

Database size in bytes.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}")].bytes.sum()

Whether the replica is in read-only mode.

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when reinitializing sessions in ZooKeeper, and during session reinitialization in ZooKeeper.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].is_readonly.first()

True if the ZooKeeper session expired

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].is_session_expired.first()

Number of data parts that will appear as the result of INSERTs or merges that haven't been done yet.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].future_parts.first()

Number of data parts in the queue for verification. A part is put in the verification queue if there is suspicion that it might be damaged.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].parts_to_check.first()

Size of the queue for operations waiting to be performed.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].queue_size.first()

Number of inserts of blocks of data that need to be made.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].inserts_in_queue.first()

Number of merges waiting to be made.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].merges_in_queue.first()

Maximum entry number in the log of general activity. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].log_max_index.first()

Maximum entry number in the log of general activity that the replica copied to its execution queue, plus one. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].log_pointer.first()

Total number of known replicas of this table. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].total_replicas.first()

Number of replicas of this table that have a session in ZooKeeper (i.e., the number of functioning replicas). (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].active_replicas.first()

Difference between log_max_index and log_pointer

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].replica_lag.first()

The amount of RAM the dictionary uses.

Preprocessing:

- JSONPATH: $[?(@.name == "{#NAME}")].bytes_allocated.first()

Number of items stored in the dictionary.

Preprocessing:

- JSONPATH: $[?(@.name == "{#NAME}")].element_count.first()

The percentage filled in the dictionary (for a hashed dictionary, the percentage filled in the hash table).

Preprocessing:

- JSONPATH: $[?(@.name == "{#NAME}")].bytes_allocated.first()

- MULTIPLIER: 100

Number of sessions (connections) to ZooKeeper. Should be no more than one.

Preprocessing:

- JSONPATH: $[?(@.metric == "ZooKeeperSession")].value.first()

Number of watches (e.g., event subscriptions) in ZooKeeperr.

Preprocessing:

- JSONPATH: $[?(@.metric == "ZooKeeperWatch")].value.first()

Number of requests to ZooKeeper in progress.

Preprocessing:

- JSONPATH: $[?(@.metric == "ZooKeeperRequest")].value.first()

Time spent in waiting for ZooKeeper operations.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperWaitMicroseconds")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- MULTIPLIER: 0.000001

- CHANGE_PER_SECOND

Count of ZooKeeper exceptions that does not belong to user/hardware exceptions.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperOtherExceptions")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Count of ZooKeeper exceptions caused by session moved/expired, connection loss, marshalling error, operation timed out and invalid zhandle state.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperHardwareExceptions")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Count of ZooKeeper exceptions caused by no znodes, bad version, node exists, node empty and no children for ephemeral.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperUserExceptions")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Get information about the number of events that have occurred in the system.

Preprocessing:

- JSONPATH: $.data

Get metrics which can be calculated instantly, or have a current value format JSONEachRow

Preprocessing:

- JSONPATH: $.data

Get metrics that are calculated periodically in the background

Preprocessing:

- JSONPATH: $.data

Get information about settings that are currently in use.

Preprocessing:

- JSONPATH: $.data

- DISCARD_UNCHANGED_HEARTBEAT: 1h

-

Preprocessing:

- JSONPATH: $.data

-

Preprocessing:

- JSONPATH: $.data

-

Preprocessing:

- JSONPATH: $.data

-

Manual close: YES

-

Manual close: YES

-

Manual close: YES

Depends on:

- ClickHouse: Port {$CLICKHOUSE.PORT} is unavailable

ClickHouse version has changed. Ack to close.

Manual close: YES

Uptime is less than 10 minutes

Manual close: YES

Zabbix has not received data for items for the last 30 minutes

Manual close: YES

Depends on:

- ClickHouse: Service is down

Clickhouse have INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree, please decrease INSERT frequency

Manual close: YES

Descease INSERT queries frequency.

Clickhouse MergeTree table engine split each INSERT query to partitions (PARTITION BY expression)

and add one or more PARTS per INSERT inside each partition,

after that background merge process run, and when you have too much unmerged parts inside partition,

SELECT queries performance can significate degrade, so clickhouse try delay insert, or abort it.

Manual close: YES

Number of errors (timeouts and connection failures) during query execution, background pool tasks and DNS cache update is too high.

"Clickhouse servers and <remote_servers> in config.xml

https://clickhouse.tech/docs/en/operations/table_engines/distributed/"

Manual close: YES

When replica have too much lag, it can be skipped from Distributed SELECT Queries without errors

and you will have wrong query results.

Manual close: YES

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when reinitializing sessions in ZooKeeper, and during session reinitialization in ZooKeeper.

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when reinitializing sessions in ZooKeeper, and during session reinitialization in ZooKeeper.

-

-

-

Number of sessions (connections) to ZooKeeper.

Should be no more than one, because using more than one connection to ZooKeeper may lead to bugs due to lack of linearizability (stale reads) that ZooKeeper consistency model allows.

ClickHouse configuration has been changed. Ack to close.

Manual close: YES

Maximum size of distributed files queue to insert for trigger expression.

Maximum number of delayed inserts for trigger expression.

Filter of discoverable databases

Filter to exclude discovered databases

Filter of discoverable dictionaries

Filter to exclude discovered dictionaries

Maximum diff between log_pointer and log_max_index.

Maximum number of smth for trigger expression

Maximum number of parts per partition for trigger expression.

-

The port of ClickHouse HTTP endpoint

Maximum ClickHouse query time in seconds for trigger expression

Maximum size of the queue for operations waiting to be performed for trigger expression.

Replication lag across all tables for trigger expression.

Request scheme which may be http or https

-

Info about tables

Filter:

- A: {#DB} MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.MATCHES}

- B: {#DB} NOT_MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.NOT_MATCHES}

Info about replicas

Filter:

- A: {#DB} MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.MATCHES}

- B: {#DB} NOT_MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DB.NOT_MATCHES}

Info about dictionaries

Filter:

- A: {#NAME} MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DICT.MATCHES}

- B: {#NAME} NOT_MATCHES_REGEX {$CLICKHOUSE.LLD.FILTER.DICT.NOT_MATCHES}

Get longest running query.

-

Preprocessing:

- DISCARD_UNCHANGED_HEARTBEAT: 10m

Preprocessing:

- REGEX: Ok\. 1

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- DISCARD_UNCHANGED_HEARTBEAT: 10m

Version of the server

Preprocessing:

- DISCARD_UNCHANGED_HEARTBEAT: 1d

Revision of the server.

Preprocessing:

- JSONPATH: $[?(@.metric == "Revision")].value.first()

Number of seconds since ClickHouse server start

Preprocessing:

- JSONPATH: $[?(@.metric == "Uptime")].value.first()

Number of queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing:

- JSONPATH: $[?(@.data.event == "Query")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Number of SELECT queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing:

- JSONPATH: $[?(@.event == "SelectQuery")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Number of INSERT queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.

Preprocessing:

- JSONPATH: $[?(@.event == "InsertQuery")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

"Number of INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree table."

Preprocessing:

- JSONPATH: $[?(@.metric == "DelayedInserts")].value.first()

Number of executing queries

Preprocessing:

- JSONPATH: $[?(@.metric == "Query")].value.first()

Number of executing background merges

Preprocessing:

- JSONPATH: $[?(@.metric == "Merge")].value.first()

The number of uncompressed bytes inserted in all tables.

Preprocessing:

- JSONPATH: $[?(@.event == "InsertedBytes")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

"Number of bytes (the number of bytes before decompression) read from compressed sources (files, network)."

Preprocessing:

- JSONPATH: $[?(@.event == "ReadCompressedBytes")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

The number of rows inserted in all tables.

Preprocessing:

- JSONPATH: $[?(@.event == "InsertedRows")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Rows read for background merges.

Preprocessing:

- JSONPATH: $[?(@.event == "MergedRows")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Uncompressed bytes that were read for background merges

Preprocessing:

- JSONPATH: $[?(@.event == "MergedUncompressedBytes")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

"Clickhouse MergeTree table engine split each INSERT query to partitions (PARTITION BY expression) and add one or more PARTS per INSERT inside each partition,

after that background merge process run."

Preprocessing:

- JSONPATH: $[?(@.metric == "MaxPartCountForPartition")].value.first()

Number of connections to TCP server (clients with native interface).

Preprocessing:

- JSONPATH: $[?(@.metric == "TCPConnection")].value.first()

Number of connections to HTTP server.

Preprocessing:

- JSONPATH: $[?(@.metric == "HTTPConnection")].value.first()

Number of connections to remote servers sending data that was INSERTed into Distributed tables.

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedSend")].value.first()

Number of connections to MySQL server.

Preprocessing:

- JSONPATH: $[?(@.metric == "MySQLConnection")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

Number of connections from other replicas to fetch parts.

Preprocessing:

- JSONPATH: $[?(@.metric == "InterserverConnection")].value.first()

Network errors (timeouts and connection failures) during query execution, background pool tasks and DNS cache update.

Preprocessing:

- JSONPATH: $[?(@.event == "NetworkErrors")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Number of read (read, pread, io_getevents, etc.) syscalls in fly

Preprocessing:

- JSONPATH: $[?(@.metric == "Read")].value.first()

Number of write (write, pwrite, io_getevents, etc.) syscalls in fly

Preprocessing:

- JSONPATH: $[?(@.metric == "Write")].value.first()

"Total number of bytes allocated by the application."

Preprocessing:

- JSONPATH: $[?(@.metric == "jemalloc.allocated")].value.first()

"Maximum number of bytes in physically resident data pages mapped by the allocator,

comprising all pages dedicated to allocator metadata, pages backing active allocations,

and unused dirty pages."

Preprocessing:

- JSONPATH: $[?(@.metric == "jemalloc.resident")].value.first()

"Total number of bytes in active extents mapped by the allocator."

Preprocessing:

- JSONPATH: $[?(@.metric == "jemalloc.mapped")].value.first()

"Total amount of memory (bytes) allocated in currently executing queries."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTracking")].value.first()

"Total amount of memory (bytes) allocated in background processing pool (that is dedicated for background merges, mutations and fetches).

Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa. This happens naturally due to caches for tables indexes and doesn't indicate memory leaks."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingInBackgroundProcessingPool")].value.first()

"Total amount of memory (bytes) allocated in background processing pool (that is dedicated for background moves). Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa.

This happens naturally due to caches for tables indexes and doesn't indicate memory leaks."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingInBackgroundMoveProcessingPool")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

"Total amount of memory (bytes) allocated in background schedule pool (that is dedicated for bookkeeping tasks of Replicated tables)."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingInBackgroundSchedulePool")].value.first()

"Total amount of memory (bytes) allocated for background merges. Included in MemoryTrackingInBackgroundProcessingPool. Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa.

This happens naturally due to caches for tables indexes and doesn't indicate memory leaks."

Preprocessing:

- JSONPATH: $[?(@.metric == "MemoryTrackingForMerges")].value.first()

Number of pending files to process for asynchronous insertion into Distributed tables. Number of files for every shard is summed.

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedFilesToInsert")].value.first()

Connection retries in replicated DB connection pool

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedConnectionFailTry")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

"Connection failures after all retries in replicated DB connection pool"

Preprocessing:

- JSONPATH: $[?(@.metric == "DistributedConnectionFailAtAll")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Maximum replica queue delay relative to current time

Preprocessing:

- JSONPATH: $[?(@.metric == "ReplicasMaxAbsoluteDelay")].value.first()

Preprocessing:

- JSONPATH: $[?(@.metric == "ReplicasSumQueueSize")].value.first()

"Number of Replicated tables that are currently in readonly state

due to re-initialization after ZooKeeper session loss

or due to startup without ZooKeeper configured."

Preprocessing:

- JSONPATH: $[?(@.metric == "ReadonlyReplica")].value.first()

Table size in bytes. Database: {#DB}, table: {#TABLE}

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].bytes.first()

Number of parts of the table. Database: {#DB}, table: {#TABLE}

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].parts.first()

Number of rows in the table. Database: {#DB}, table: {#TABLE}

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].rows.first()

Database size in bytes.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}")].bytes.sum()

Whether the replica is in read-only mode.

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when reinitializing sessions in ZooKeeper, and during session reinitialization in ZooKeeper.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].is_readonly.first()

True if the ZooKeeper session expired

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].is_session_expired.first()

Number of data parts that will appear as the result of INSERTs or merges that haven't been done yet.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].future_parts.first()

Number of data parts in the queue for verification. A part is put in the verification queue if there is suspicion that it might be damaged.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].parts_to_check.first()

Size of the queue for operations waiting to be performed.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].queue_size.first()

Number of inserts of blocks of data that need to be made.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].inserts_in_queue.first()

Number of merges waiting to be made.

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].merges_in_queue.first()

Maximum entry number in the log of general activity. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].log_max_index.first()

Maximum entry number in the log of general activity that the replica copied to its execution queue, plus one. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].log_pointer.first()

Total number of known replicas of this table. (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].total_replicas.first()

Number of replicas of this table that have a session in ZooKeeper (i.e., the number of functioning replicas). (Have a non-zero value only where there is an active session with ZooKeeper).

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].active_replicas.first()

Difference between log_max_index and log_pointer

Preprocessing:

- JSONPATH: $[?(@.database == "{#DB}" && @.table == "{#TABLE}")].replica_lag.first()

The amount of RAM the dictionary uses.

Preprocessing:

- JSONPATH: $[?(@.name == "{#NAME}")].bytes_allocated.first()

Number of items stored in the dictionary.

Preprocessing:

- JSONPATH: $[?(@.name == "{#NAME}")].element_count.first()

The percentage filled in the dictionary (for a hashed dictionary, the percentage filled in the hash table).

Preprocessing:

- JSONPATH: $[?(@.name == "{#NAME}")].bytes_allocated.first()

- MULTIPLIER: 100

Number of sessions (connections) to ZooKeeper. Should be no more than one.

Preprocessing:

- JSONPATH: $[?(@.metric == "ZooKeeperSession")].value.first()

Number of watches (e.g., event subscriptions) in ZooKeeperr.

Preprocessing:

- JSONPATH: $[?(@.metric == "ZooKeeperWatch")].value.first()

Number of requests to ZooKeeper in progress.

Preprocessing:

- JSONPATH: $[?(@.metric == "ZooKeeperRequest")].value.first()

Time spent in waiting for ZooKeeper operations.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperWaitMicroseconds")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- MULTIPLIER: 0.000001

- CHANGE_PER_SECOND

Count of ZooKeeper exceptions that does not belong to user/hardware exceptions.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperOtherExceptions")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Count of ZooKeeper exceptions caused by session moved/expired, connection loss, marshalling error, operation timed out and invalid zhandle state.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperHardwareExceptions")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Count of ZooKeeper exceptions caused by no znodes, bad version, node exists, node empty and no children for ephemeral.

Preprocessing:

- JSONPATH: $[?(@.event == "ZooKeeperUserExceptions")].value.first()

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

Get information about the number of events that have occurred in the system.

Preprocessing:

- JSONPATH: $.data

Get metrics which can be calculated instantly, or have a current value format JSONEachRow

Preprocessing:

- JSONPATH: $.data

Get metrics that are calculated periodically in the background

Preprocessing:

- JSONPATH: $.data

Get information about settings that are currently in use.

Preprocessing:

- JSONPATH: $.data

- DISCARD_UNCHANGED_HEARTBEAT: 1h

-

Preprocessing:

- JSONPATH: $.data

-

Preprocessing:

- JSONPATH: $.data

-

Preprocessing:

- JSONPATH: $.data

-

Manual close: YES

-

Manual close: YES

-

Manual close: YES

Depends on:

- ClickHouse: Port {$CLICKHOUSE.PORT} is unavailable

ClickHouse version has changed. Ack to close.

Manual close: YES

Uptime is less than 10 minutes.

Manual close: YES

Zabbix has not received data for items for the last 30 minutes

Manual close: YES

Depends on:

- ClickHouse: Service is down

Clickhouse have INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree, please decrease INSERT frequency

Manual close: YES

"Descease INSERT queries frequency.

Clickhouse MergeTree table engine split each INSERT query to partitions (PARTITION BY expression)

and add one or more PARTS per INSERT inside each partition,

after that background merge process run, and when you have too much unmerged parts inside partition,

SELECT queries performance can significate degrade, so clickhouse try delay insert, or abort it"

Manual close: YES

Number of errors (timeouts and connection failures) during query execution, background pool tasks and DNS cache update is too high.

"Clickhouse servers and <remote_servers> in config.xml

https://clickhouse.tech/docs/en/operations/table_engines/distributed/"

Manual close: YES

"When replica have too much lag, it can be skipped from Distributed SELECT Queries without errors

and you will have wrong query results."

Manual close: YES

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when reinitializing sessions in ZooKeeper, and during session reinitialization in ZooKeeper.

This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when reinitializing sessions in ZooKeeper, and during session reinitialization in ZooKeeper.

-

-

-

"Number of sessions (connections) to ZooKeeper.

Should be no more than one, because using more than one connection to ZooKeeper may lead to bugs due to lack of linearizability (stale reads) that ZooKeeper consistency model allows."

ClickHouse configuration has been changed. Ack to close.

Manual close: YES