The port of TiDB server metrics web endpoint

TiDB server URL

Maximum percentage of used file descriptors

Maximum heap memory used

Maximum number of DDL tasks that are waiting

Maximum number of times that the operating system rewinds every second

Maximum number of schema lease errors

Maximum number of load schema errors

Maximum number of GC-related operations failures

Maximum number of region related errors

Minimum number of keep alive operations

Get TiDB instance metrics.

Preprocessing

• Check for not supported value

  ⛔️Custom on fail: Discard value

• Prometheus to JSON

Get TiDB instance status info.

Preprocessing

• Check for not supported value

  ⛔️Custom on fail: Set value to: {"status": "0"}

Status of PD instance.

Preprocessing

• JSON Path: $.status

  ⛔️Custom on fail: Set value to: 1

• Discard unchanged with heartbeat: 1h

Get information about server queries.

Preprocessing

• JSON Path: $[?(@.name == "tidb_server_query_total")]

  ⛔️Custom on fail: Discard value

The number of queries on TiDB instance per second with failure of command execution results.

Preprocessing

• JSON Path: $[?(@.labels.result == "Error")].value.sum()

• Change per second

The number of queries on TiDB instance per second with success of command execution results.

Preprocessing

• JSON Path: $[?(@.labels.result == "OK")].value.sum()

• Change per second

The number of queries per second on TiDB instance.

Preprocessing

• JSON Path: $..value.sum()

• Change per second

Get SQL statements metrics.

Preprocessing

• JSON Path: $[?(@.name=="tidb_executor_statement_total")]

  ⛔️Custom on fail: Discard value

The total number of SQL statements executed per second.

Preprocessing

• JSON Path: $..value.sum()

• Change per second

The number of error occurred when executing SQL statements per second (such as syntax errors and primary key conflicts).

Preprocessing

• JSON Path: $[?(@.name=="tidb_server_execute_error_total")].value.sum()

  ⛔️Custom on fail: Discard value

• Change per second

Get TiKV client metrics.

Preprocessing

• JSON Path: $[?(@.name=~"tidb_tikvclient_*")]

  ⛔️Custom on fail: Discard value

The number of executed KV commands per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of TSO commands that TiDB obtains from PD per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of TSO requests that TiDB obtains from PD per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of region related errors returned by TiKV per second.

Preprocessing

• JSON Path: $[?(@.name=="tidb_tikvclient_region_err_total")].value.sum()

• Change per second

The number of DDL tasks that are waiting.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing

• JSON Path: $[?(@.name=="tidb_ddl_waiting_jobs")].value.sum()

  ⛔️Custom on fail: Set value to: 0

The statistics of the schemas that TiDB obtains from TiKV per second.

Preprocessing

• JSON Path: $[?(@.name=="tidb_domain_load_schema_total")].value.sum()

• Change per second

The total number of failures to reload the latest schema information in TiDB per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Discard value

• Change per second

The number of schema lease errors per second.

"outdate" errors means that the schema cannot be updated, which is a more serious error and triggers an alert.

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Discard value

• Change per second

The number of schema lease errors per second.

"change" means that the schema has changed

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Discard value

• Change per second

The number of errors returned by TiKV.

Preprocessing

• JSON Path: $[?(@.name=="tidb_tikvclient_backoff_total")].value.sum()

  ⛔️Custom on fail: Discard value

• Change per second

The number of times that the metrics are refreshed on TiDB instance per minute.

Preprocessing

• JSON Path: $[?(@.name=="tidb_monitor_keep_alive_total")].value.first()

  ⛔️Custom on fail: Discard value

• Simple change

The connection number of current TiDB instance.

Preprocessing

• JSON Path: $[?(@.name=="tidb_server_connections")].value.first()

Number of heap bytes that are in use.

Preprocessing

• JSON Path: $[?(@.name=="go_memstats_heap_inuse_bytes")].value.first()

Resident memory size in bytes.

Preprocessing

• JSON Path: $[?(@.name=="process_resident_memory_bytes")].value.first()

The number of Goroutines on TiDB instance.

Preprocessing

• JSON Path: $[?(@.name=="go_goroutines")].value.first()

Number of open file descriptors.

Preprocessing

• JSON Path: $[?(@.name=="process_open_fds")].value.first()

Maximum number of open file descriptors.

Preprocessing

• JSON Path: $[?(@.name=="process_max_fds")].value.first()

Total user and system CPU usage ratio.

Preprocessing

• JSON Path: $[?(@.name=="process_cpu_seconds_total")].value.first()

• Change per second

• Custom multiplier: 100

The runtime of each TiDB instance.

Preprocessing

• JSON Path: $[?(@.name=="process_start_time_seconds")].value.first()

• JavaScript: The text is too long. Please see the template.

Version of the TiDB instance.

Preprocessing

• JSON Path: $.version

• Discard unchanged with heartbeat: 3h

The number of times that the operating system rewinds every second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of critical errors occurred in TiDB per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of panics occurred in TiDB per second.

Preprocessing

• JSON Path: $[?(@.name=="tidb_server_panic_total")].value.first()

  ⛔️Custom on fail: Discard value

• Change per second

The latest schema information is not reloaded in TiDB within one lease.

Indicates whether the TiDB process still exists. If the number of times for tidb_monitor_keep_alive_total increases less than 10 per minute, the TiDB process might already exit and an alert is triggered.

Heavy file descriptor usage (i.e., near the process's file descriptor limit) indicates a potential file descriptor exhaustion issue.

Uptime is less than 10 minutes.

TiDB version has changed. Acknowledge to close the problem manually.

When a panic occurs, an alert is triggered. The thread is often recovered, otherwise, TiDB will frequently restart.

Discovery QPS specific metrics.

Preprocessing

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

Get QPS metrics of {#TYPE}.

Preprocessing

• JSON Path: $[?(@.labels.type == "{#TYPE}")]

  ⛔️Custom on fail: Discard value

The number of queries on TiDB instance per second with success of command execution results.

Preprocessing

• JSON Path: $[?(@.labels.result == "OK")].value.first()

• Change per second

The number of queries on TiDB instance per second with failure of command execution results.

Preprocessing

• JSON Path: $[?(@.labels.result == "Error")].value.first()

• Change per second

Discovery statement specific metrics.

Preprocessing

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

The number of SQL statements executed per second.

Preprocessing

• JSON Path: $[?(@.labels.type == "{#TYPE}")].value.first()

• Change per second

Discovery KV specific metrics.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

The number of executed KV commands per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

Discovery lock resolves specific metrics.

Preprocessing

• JSON Path: $[?(@.name=="tidb_tikvclient_lock_resolver_actions_total")]

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

Discovery KV backoff specific metrics.

Preprocessing

• JSON Path: $[?(@.name=="tidb_tikvclient_backoff_total")]

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

Discovery GC action results metrics.

Preprocessing

• JSON Path: $[?(@.name=="tidb_tikvclient_gc_action_result")]

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

The number of results of GC-related operations per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

Maximum number of DDL tasks that are waiting

Maximum number of GC-related operations failures

Maximum heap memory used

Minimum number of keep alive operations

Maximum percentage of used file descriptors

The port of TiDB server metrics web endpoint

Maximum number of region related errors

Maximum number of schema lease errors

Maximum number of load schema errors

Maximum number of times that the operating system rewinds every second

TiDB server URL

Discovery GC action results metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_gc_action_result")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Overrides:

Failed GC-related operations trigger
- {#TYPE} MATCHES_REGEX failed
- TRIGGER_PROTOTYPE LIKE Too many failed GC-related operations - DISCOVER

Discovery KV backoff specific metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_backoff_total")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Discovery KV specific metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_txn_cmd_duration_seconds_count")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Discovery lock resolves specific metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_lock_resolver_actions_total")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Discovery QPS specific metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_server_query_total")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Discovery statement specific metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_executor_statement_total")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Status of PD instance.

Preprocessing:

- JSONPATH: $.status

⛔️ON_FAIL: CUSTOM_VALUE -> 1

- DISCARD_UNCHANGED_HEARTBEAT: 1h

The number of queries on TiDB instance per second with failure of command execution results.

Preprocessing:

- JSONPATH: $[?(@.name == "tidb_server_query_total" && @.labels.result == "Error")].value.sum()

- CHANGE_PER_SECOND

The number of queries on TiDB instance per second with success of command execution results.

Preprocessing:

- JSONPATH: $[?(@.name == "tidb_server_query_total" && @.labels.result == "OK")].value.sum()

- CHANGE_PER_SECOND

The number of queries per second on TiDB instance.

Preprocessing:

- JSONPATH: $[?(@.name == "tidb_server_query_total")].value.sum()

- CHANGE_PER_SECOND

The total number of SQL statements executed per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_executor_statement_total")].value.sum()

- CHANGE_PER_SECOND

The number of error occurred when executing SQL statements per second (such as syntax errors and primary key conflicts).

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_server_execute_error_total")].value.sum()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of executed KV commands per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_txn_cmd_duration_seconds_count")].value.sum()

- CHANGE_PER_SECOND

The number of TSO commands that TiDB obtains from PD per second.

Preprocessing:

- JSONPATH: $[?(@.name=="pd_client_cmd_handle_cmds_duration_seconds_count" && @.labels.type == "tso")].value.first()

- CHANGE_PER_SECOND

The number of TSO requests that TiDB obtains from PD per second.

Preprocessing:

- JSONPATH: $[?(@.name=="pd_client_request_handle_requests_duration_seconds_count" && @.labels.type == "tso")].value.first()

- CHANGE_PER_SECOND

The number of region related errors returned by TiKV per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_region_err_total")].value.sum()

- CHANGE_PER_SECOND

The number of DDL tasks that are waiting.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_lock_resolver_actions_total")].value.sum()

- CHANGE_PER_SECOND

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_ddl_waiting_jobs")].value.sum()

The statistics of the schemas that TiDB obtains from TiKV per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_domain_load_schema_total")].value.sum()

- CHANGE_PER_SECOND

The total number of failures to reload the latest schema information in TiDB per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_domain_load_schema_total && @.labels.type == "failed"")].value.first()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of schema lease errors per second.

"outdate" errors means that the schema cannot be updated, which is a more serious error and triggers an alert.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_session_schema_lease_error_total && @.labels.type == "outdate"")].value.first()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of schema lease errors per second.

"change" means that the schema has changed

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_session_schema_lease_error_total && @.labels.type == "change"")].value.first()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of errors returned by TiKV.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_backoff_total")].value.sum()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of times that the metrics are refreshed on TiDB instance per minute.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_monitor_keep_alive_total")].value.first()

⛔️ON_FAIL: DISCARD_VALUE ->

- SIMPLE_CHANGE

The connection number of current TiDB instance.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_server_connections")].value.first()

Number of heap bytes that are in use.

Preprocessing:

- JSONPATH: $[?(@.name=="go_memstats_heap_inuse_bytes")].value.first()

Resident memory size in bytes.

Preprocessing:

- JSONPATH: $[?(@.name=="process_resident_memory_bytes")].value.first()

The number of Goroutines on TiDB instance.

Preprocessing:

- JSONPATH: $[?(@.name=="go_goroutines")].value.first()

Number of open file descriptors.

Preprocessing:

- JSONPATH: $[?(@.name=="process_open_fds")].value.first()

Maximum number of open file descriptors.

Preprocessing:

- JSONPATH: $[?(@.name=="process_max_fds")].value.first()

Total user and system CPU usage ratio.

Preprocessing:

- JSONPATH: $[?(@.name=="process_cpu_seconds_total")].value.first()

- CHANGE_PER_SECOND

- MULTIPLIER: 100

The runtime of each TiDB instance.

Preprocessing:

- JSONPATH: $[?(@.name=="process_start_time_seconds")].value.first()

- JAVASCRIPT: //use boottime to calculate uptime return (Math.floor(Date.now()/1000)-Number(value));

Version of the TiDB instance.

Preprocessing:

- JSONPATH: $.version

- DISCARD_UNCHANGED_HEARTBEAT: 3h

The number of times that the operating system rewinds every second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_monitor_time_jump_back_total")].value.first()

- CHANGE_PER_SECOND

The number of critical errors occurred in TiDB per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_server_critical_error_total")].value.first()

- CHANGE_PER_SECOND

The number of panics occurred in TiDB per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_server_panic_total")].value.first()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of queries on TiDB instance per second with success of command execution results.

Preprocessing:

- JSONPATH: $[?(@.name == "tidb_server_query_total" && @.labels.result == "OK" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of queries on TiDB instance per second with failure of command execution results.

Preprocessing:

- JSONPATH: $[?(@.name == "tidb_server_query_total" && @.labels.result == "Error" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of SQL statements executed per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_executor_statement_total" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of executed KV commands per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_txn_cmd_duration_seconds_count" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_lock_resolver_actions_total" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_backoff_total" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of results of GC-related operations per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_gc_action_result" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

Get TiDB instance metrics.

Preprocessing:

- CHECK_NOT_SUPPORTED

⛔️ON_FAIL: DISCARD_VALUE ->

- PROMETHEUS_TO_JSON

Get TiDB instance status info.

Preprocessing:

- CHECK_NOT_SUPPORTED

⛔️ON_FAIL: CUSTOM_VALUE -> {"status": "0"}

-

-

-

-

The latest schema information is not reloaded in TiDB within one lease.

Indicates whether the TiDB process still exists. If the number of times for tidb_monitor_keep_alive_total increases less than 10 per minute, the TiDB process might already exit and an alert is triggered.

-

Heavy file descriptor usage (i.e., near the process's file descriptor limit) indicates a potential file descriptor exhaustion issue.

Uptime is less than 10 minutes.

Manual close: YES

TiDB version has changed. Ack to close.

Manual close: YES

-

When a panic occurs, an alert is triggered. The thread is often recovered, otherwise, TiDB will frequently restart.

-

The port of TiDB server metrics web endpoint

TiDB server URL

Maximum percentage of used file descriptors

Maximum heap memory used

Maximum number of DDL tasks that are waiting

Maximum number of times that the operating system rewinds every second

Maximum number of schema lease errors

Maximum number of load schema errors

Maximum number of GC-related operations failures

Maximum number of region related errors

Minimum number of keep alive operations

Get TiDB instance metrics.

Preprocessing

• Check for not supported value

  ⛔️Custom on fail: Discard value

• Prometheus to JSON

Get TiDB instance status info.

Preprocessing

• Check for not supported value

  ⛔️Custom on fail: Set value to: {"status": "0"}

Status of PD instance.

Preprocessing

• JSON Path: $.status

  ⛔️Custom on fail: Set value to: 1

• Discard unchanged with heartbeat: 1h

Get information about server queries.

Preprocessing

• JSON Path: $[?(@.name == "tidb_server_query_total")]

  ⛔️Custom on fail: Discard value

The number of queries on TiDB instance per second with failure of command execution results.

Preprocessing

• JSON Path: $[?(@.labels.result == "Error")].value.sum()

• Change per second

The number of queries on TiDB instance per second with success of command execution results.

Preprocessing

• JSON Path: $[?(@.labels.result == "OK")].value.sum()

• Change per second

The number of queries per second on TiDB instance.

Preprocessing

• JSON Path: $..value.sum()

• Change per second

Get SQL statements metrics.

Preprocessing

• JSON Path: $[?(@.name=="tidb_executor_statement_total")]

  ⛔️Custom on fail: Discard value

The total number of SQL statements executed per second.

Preprocessing

• JSON Path: $..value.sum()

• Change per second

The number of error occurred when executing SQL statements per second (such as syntax errors and primary key conflicts).

Preprocessing

• JSON Path: $[?(@.name=="tidb_server_execute_error_total")].value.sum()

  ⛔️Custom on fail: Discard value

• Change per second

Get TiKV client metrics.

Preprocessing

• JSON Path: $[?(@.name=~"tidb_tikvclient_*")]

  ⛔️Custom on fail: Discard value

The number of executed KV commands per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of TSO commands that TiDB obtains from PD per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of TSO requests that TiDB obtains from PD per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of region related errors returned by TiKV per second.

Preprocessing

• JSON Path: $[?(@.name=="tidb_tikvclient_region_err_total")].value.sum()

• Change per second

The number of DDL tasks that are waiting.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing

• JSON Path: $[?(@.name=="tidb_ddl_waiting_jobs")].value.sum()

  ⛔️Custom on fail: Set value to: 0

The statistics of the schemas that TiDB obtains from TiKV per second.

Preprocessing

• JSON Path: $[?(@.name=="tidb_domain_load_schema_total")].value.sum()

• Change per second

The total number of failures to reload the latest schema information in TiDB per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Discard value

• Change per second

The number of schema lease errors per second.

"outdate" errors means that the schema cannot be updated, which is a more serious error and triggers an alert.

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Discard value

• Change per second

The number of schema lease errors per second.

"change" means that the schema has changed

Preprocessing

• JSON Path: The text is too long. Please see the template.

  ⛔️Custom on fail: Discard value

• Change per second

The number of errors returned by TiKV.

Preprocessing

• JSON Path: $[?(@.name=="tidb_tikvclient_backoff_total")].value.sum()

  ⛔️Custom on fail: Discard value

• Change per second

The number of times that the metrics are refreshed on TiDB instance per minute.

Preprocessing

• JSON Path: $[?(@.name=="tidb_monitor_keep_alive_total")].value.first()

  ⛔️Custom on fail: Discard value

• Simple change

The connection number of current TiDB instance.

Preprocessing

• JSON Path: $[?(@.name=="tidb_server_connections")].value.first()

Number of heap bytes that are in use.

Preprocessing

• JSON Path: $[?(@.name=="go_memstats_heap_inuse_bytes")].value.first()

Resident memory size in bytes.

Preprocessing

• JSON Path: $[?(@.name=="process_resident_memory_bytes")].value.first()

The number of Goroutines on TiDB instance.

Preprocessing

• JSON Path: $[?(@.name=="go_goroutines")].value.first()

Number of open file descriptors.

Preprocessing

• JSON Path: $[?(@.name=="process_open_fds")].value.first()

Maximum number of open file descriptors.

Preprocessing

• JSON Path: $[?(@.name=="process_max_fds")].value.first()

Total user and system CPU usage ratio.

Preprocessing

• JSON Path: $[?(@.name=="process_cpu_seconds_total")].value.first()

• Change per second

• Custom multiplier: 100

The runtime of each TiDB instance.

Preprocessing

• JSON Path: $[?(@.name=="process_start_time_seconds")].value.first()

• JavaScript: The text is too long. Please see the template.

Version of the TiDB instance.

Preprocessing

• JSON Path: $.version

• Discard unchanged with heartbeat: 3h

The number of times that the operating system rewinds every second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of critical errors occurred in TiDB per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

The number of panics occurred in TiDB per second.

Preprocessing

• JSON Path: $[?(@.name=="tidb_server_panic_total")].value.first()

  ⛔️Custom on fail: Discard value

• Change per second

The latest schema information is not reloaded in TiDB within one lease.

Indicates whether the TiDB process still exists. If the number of times for tidb_monitor_keep_alive_total increases less than 10 per minute, the TiDB process might already exit and an alert is triggered.

Heavy file descriptor usage (i.e., near the process's file descriptor limit) indicates a potential file descriptor exhaustion issue.

Uptime is less than 10 minutes.

TiDB version has changed. Acknowledge to close the problem manually.

When a panic occurs, an alert is triggered. The thread is often recovered, otherwise, TiDB will frequently restart.

Discovery QPS specific metrics.

Preprocessing

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

Get QPS metrics of {#TYPE}.

Preprocessing

• JSON Path: $[?(@.labels.type == "{#TYPE}")]

  ⛔️Custom on fail: Discard value

The number of queries on TiDB instance per second with success of command execution results.

Preprocessing

• JSON Path: $[?(@.labels.result == "OK")].value.first()

• Change per second

The number of queries on TiDB instance per second with failure of command execution results.

Preprocessing

• JSON Path: $[?(@.labels.result == "Error")].value.first()

• Change per second

Discovery statement specific metrics.

Preprocessing

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

The number of SQL statements executed per second.

Preprocessing

• JSON Path: $[?(@.labels.type == "{#TYPE}")].value.first()

• Change per second

Discovery KV specific metrics.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

The number of executed KV commands per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

Discovery lock resolves specific metrics.

Preprocessing

• JSON Path: $[?(@.name=="tidb_tikvclient_lock_resolver_actions_total")]

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

Discovery KV backoff specific metrics.

Preprocessing

• JSON Path: $[?(@.name=="tidb_tikvclient_backoff_total")]

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

Discovery GC action results metrics.

Preprocessing

• JSON Path: $[?(@.name=="tidb_tikvclient_gc_action_result")]

  ⛔️Custom on fail: Discard value

• JavaScript: The text is too long. Please see the template.

• Discard unchanged with heartbeat: 1h

The number of results of GC-related operations per second.

Preprocessing

• JSON Path: The text is too long. Please see the template.

• Change per second

Maximum number of DDL tasks that are waiting

Maximum number of GC-related operations failures

Maximum heap memory used

Minimum number of keep alive operations

Maximum percentage of used file descriptors

The port of TiDB server metrics web endpoint

Maximum number of region related errors

Maximum number of schema lease errors

Maximum number of load schema errors

Maximum number of times that the operating system rewinds every second

TiDB server URL

Discovery QPS specific metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_server_query_total")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Discovery statement specific metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_executor_statement_total")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Discovery KV specific metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_txn_cmd_duration_seconds_count")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Discovery lock resolves specific metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_lock_resolver_actions_total")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Discovery KV backoff specific metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_backoff_total")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Discovery GC action results metrics.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_gc_action_result")]

- JAVASCRIPT: The text is too long. Please see the template.

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Overrides:

Failed GC-related operations trigger
- {#TYPE} MATCHES_REGEX failed
- TRIGGER_PROTOTYPE LIKE Too many failed GC-related operations - DISCOVER

Status of PD instance.

Preprocessing:

- JSONPATH: $.status

⛔️ON_FAIL: CUSTOM_VALUE -> 1

- DISCARD_UNCHANGED_HEARTBEAT: 1h

The number of queries on TiDB instance per second with failure of command execution results.

Preprocessing:

- JSONPATH: $[?(@.name == "tidb_server_query_total" && @.labels.result == "Error")].value.sum()

- CHANGE_PER_SECOND

The number of queries on TiDB instance per second with success of command execution results.

Preprocessing:

- JSONPATH: $[?(@.name == "tidb_server_query_total" && @.labels.result == "OK")].value.sum()

- CHANGE_PER_SECOND

The number of queries per second on TiDB instance.

Preprocessing:

- JSONPATH: $[?(@.name == "tidb_server_query_total")].value.sum()

- CHANGE_PER_SECOND

The total number of SQL statements executed per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_executor_statement_total")].value.sum()

- CHANGE_PER_SECOND

The number of error occurred when executing SQL statements per second (such as syntax errors and primary key conflicts).

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_server_execute_error_total")].value.sum()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of executed KV commands per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_txn_cmd_duration_seconds_count")].value.sum()

- CHANGE_PER_SECOND

The number of TSO commands that TiDB obtains from PD per second.

Preprocessing:

- JSONPATH: $[?(@.name=="pd_client_cmd_handle_cmds_duration_seconds_count" && @.labels.type == "tso")].value.first()

- CHANGE_PER_SECOND

The number of TSO requests that TiDB obtains from PD per second.

Preprocessing:

- JSONPATH: $[?(@.name=="pd_client_request_handle_requests_duration_seconds_count" && @.labels.type == "tso")].value.first()

- CHANGE_PER_SECOND

The number of region related errors returned by TiKV per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_region_err_total")].value.sum()

- CHANGE_PER_SECOND

The number of DDL tasks that are waiting.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_lock_resolver_actions_total")].value.sum()

- CHANGE_PER_SECOND

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_ddl_waiting_jobs")].value.sum()

The statistics of the schemas that TiDB obtains from TiKV per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_domain_load_schema_total")].value.sum()

- CHANGE_PER_SECOND

The total number of failures to reload the latest schema information in TiDB per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_domain_load_schema_total && @.labels.type == "failed"")].value.first()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of schema lease errors per second.

"outdate" errors means that the schema cannot be updated, which is a more serious error and triggers an alert.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_session_schema_lease_error_total && @.labels.type == "outdate"")].value.first()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of schema lease errors per second.

"change" means that the schema has changed

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_session_schema_lease_error_total && @.labels.type == "change"")].value.first()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of errors returned by TiKV.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_backoff_total")].value.sum()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of times that the metrics are refreshed on TiDB instance per minute.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_monitor_keep_alive_total")].value.first()

⛔️ON_FAIL: DISCARD_VALUE ->

- SIMPLE_CHANGE

The connection number of current TiDB instance.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_server_connections")].value.first()

Number of heap bytes that are in use.

Preprocessing:

- JSONPATH: $[?(@.name=="go_memstats_heap_inuse_bytes")].value.first()

Resident memory size in bytes.

Preprocessing:

- JSONPATH: $[?(@.name=="process_resident_memory_bytes")].value.first()

The number of Goroutines on TiDB instance.

Preprocessing:

- JSONPATH: $[?(@.name=="go_goroutines")].value.first()

Number of open file descriptors.

Preprocessing:

- JSONPATH: $[?(@.name=="process_open_fds")].value.first()

Maximum number of open file descriptors.

Preprocessing:

- JSONPATH: $[?(@.name=="process_max_fds")].value.first()

Total user and system CPU usage ratio.

Preprocessing:

- JSONPATH: $[?(@.name=="process_cpu_seconds_total")].value.first()

- CHANGE_PER_SECOND

- MULTIPLIER: 100

The runtime of each TiDB instance.

Preprocessing:

- JSONPATH: $[?(@.name=="process_start_time_seconds")].value.first()

- JAVASCRIPT: //use boottime to calculate uptime return (Math.floor(Date.now()/1000)-Number(value));

Version of the TiDB instance.

Preprocessing:

- JSONPATH: $.version

- DISCARD_UNCHANGED_HEARTBEAT: 3h

The number of times that the operating system rewinds every second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_monitor_time_jump_back_total")].value.first()

- CHANGE_PER_SECOND

The number of critical errors occurred in TiDB per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_server_critical_error_total")].value.first()

- CHANGE_PER_SECOND

The number of panics occurred in TiDB per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_server_panic_total")].value.first()

⛔️ON_FAIL: DISCARD_VALUE ->

- CHANGE_PER_SECOND

The number of queries on TiDB instance per second with success of command execution results.

Preprocessing:

- JSONPATH: $[?(@.name == "tidb_server_query_total" && @.labels.result == "OK" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of queries on TiDB instance per second with failure of command execution results.

Preprocessing:

- JSONPATH: $[?(@.name == "tidb_server_query_total" && @.labels.result == "Error" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of SQL statements executed per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_executor_statement_total" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of executed KV commands per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_txn_cmd_duration_seconds_count" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_lock_resolver_actions_total" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of TiDB operations that resolve locks per second. When TiDB's read or write request encounters a lock, it tries to resolve the lock.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_backoff_total" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

The number of results of GC-related operations per second.

Preprocessing:

- JSONPATH: $[?(@.name=="tidb_tikvclient_gc_action_result" && @.labels.type == "{#TYPE}")].value.first()

- CHANGE_PER_SECOND

Get TiDB instance metrics.

Preprocessing:

- CHECK_NOT_SUPPORTED

⛔️ON_FAIL: DISCARD_VALUE ->

- PROMETHEUS_TO_JSON

Get TiDB instance status info.

Preprocessing:

- CHECK_NOT_SUPPORTED

⛔️ON_FAIL: CUSTOM_VALUE -> {"status": "0"}

-

-

-

-

The latest schema information is not reloaded in TiDB within one lease.

Indicates whether the TiDB process still exists. If the number of times for tidb_monitor_keep_alive_total increases less than 10 per minute, the TiDB process might already exit and an alert is triggered.

-

Heavy file descriptor usage (i.e., near the process's file descriptor limit) indicates a potential file descriptor exhaustion issue.

Uptime is less than 10 minutes

Manual close: YES

TiDB version has changed. Ack to close.

Manual close: YES

-

When a panic occurs, an alert is triggered. The thread is often recovered, otherwise, TiDB will frequently restart.

-