Maximum number of gRPC requests failures

Filter of discoverable gRPC codes https://github.com/grpc/grpc/blob/master/doc/statuscodes.md

Filter to exclude discovered gRPC codes https://github.com/grpc/grpc/blob/master/doc/statuscodes.md

Filter of discoverable gRPC codes which will be create triggers

Maximum number of HTTP requests failures

Maximum number of leader changes

Maximum percentage of used file descriptors

-

The port of Etcd API endpoint

Maximum number of proposal failures

Maximum number of proposals in queue

Request scheme which may be http or https

-

Preprocessing:

- PROMETHEUS_TO_JSON: grpc_server_handled_total

- JAVASCRIPT: var data = JSON.parse(value), lookup = {}, result =[]; for (var item, i = 0; item = data[i++];) { var code = item.labels.grpc_code; if (!(code in lookup)) { lookup[code] = 1; result.push({ "{#GRPC.CODE}": code}); } } return JSON.stringify(result);

- DISCARD_UNCHANGED_HEARTBEAT: 1h

Filter:

- A: {#GRPC.CODE} NOT_MATCHES_REGEX {$ETCD.GRPC_CODE.NOT_MATCHES}

- B: {#GRPC.CODE} MATCHES_REGEX {$ETCD.GRPC_CODE.MATCHES}

Preprocessing:

- PROMETHEUS_TO_JSON: etcd_network_peer_sent_bytes_total

-

Preprocessing:

- DISCARD_UNCHANGED_HEARTBEAT: 10m

-

Preprocessing:

- JSONPATH: $.health

- BOOL_TO_DECIMAL

- DISCARD_UNCHANGED_HEARTBEAT: 10m

Whether or not this member is a leader. 1 if is, 0 otherwise.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_server_is_leader

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- DISCARD_UNCHANGED_HEARTBEAT: 10m

Whether or not a leader exists. 1 is existence, 0 is not.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_server_has_leader

- DISCARD_UNCHANGED_HEARTBEAT: 10m

The the number of leader changes the member has seen since its start.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_server_leader_changes_seen_total

The number of consensus proposals committed.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_server_proposals_committed_total

- CHANGE_PER_SECOND

The number of consensus proposals applied.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_server_proposals_applied_total

- CHANGE_PER_SECOND

The number of failed proposals seen.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_server_proposals_failed_total

- CHANGE_PER_SECOND

The current number of pending proposals to commit.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_server_proposals_pending

Number of reads action by (get/getRecursive), local to this member.

Preprocessing:

- PROMETHEUS_TO_JSON: etcd_debugging_store_reads_total

- JAVASCRIPT: //calculates total reads var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0);

- CHANGE_PER_SECOND

Number of writes (e.g. set/compareAndDelete) seen by this member.

Preprocessing:

- PROMETHEUS_TO_JSON: etcd_debugging_store_writes_total

- JAVASCRIPT: var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0);

- CHANGE_PER_SECOND

The number of bytes received from grpc clients per second

Preprocessing:

- PROMETHEUS_PATTERN: etcd_network_client_grpc_received_bytes_total

- CHANGE_PER_SECOND

The number of bytes sent from grpc clients per second

Preprocessing:

- PROMETHEUS_PATTERN: etcd_network_client_grpc_sent_bytes_total

- CHANGE_PER_SECOND

Number of requests received into the system (successfully parsed and authd).

Preprocessing:

- PROMETHEUS_TO_JSON: etcd_http_received_total

- JAVASCRIPT: var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0);

- CHANGE_PER_SECOND

Number of handle failures of requests (non-watches), by method (GET/PUT etc.), and code 5XX.

Preprocessing:

- PROMETHEUS_TO_JSON: etcd_http_failed_total{code=~"5.+"}

- JAVASCRIPT: var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0);

- CHANGE_PER_SECOND

Number of handle failures of requests (non-watches), by method (GET/PUT etc.), and code 4XX.

Preprocessing:

- PROMETHEUS_TO_JSON: etcd_http_failed_total{code=~"4.+"}

- JAVASCRIPT: var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0);

- CHANGE_PER_SECOND

The number of RPC stream messages received on the server.

Preprocessing:

- PROMETHEUS_TO_JSON: grpc_server_msg_received_total

- JAVASCRIPT: var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0);

- CHANGE_PER_SECOND

The number of gRPC stream messages sent by the server.

Preprocessing:

- PROMETHEUS_TO_JSON: grpc_server_msg_sent_total

- JAVASCRIPT: var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0);

- CHANGE_PER_SECOND

The number of RPCs started on the server.

Preprocessing:

- PROMETHEUS_TO_JSON: grpc_server_started_total

- JAVASCRIPT: var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0);

- CHANGE_PER_SECOND

Version of the Etcd server.

Preprocessing:

- JSONPATH: $.etcdserver

- DISCARD_UNCHANGED_HEARTBEAT: 1d

Version of the Etcd cluster.

Preprocessing:

- JSONPATH: $.etcdcluster

- DISCARD_UNCHANGED_HEARTBEAT: 1d

Total size of the underlying database.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_debugging_mvcc_db_total_size_in_bytes

The number of DB keys compacted per second.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_debugging_mvcc_db_compaction_keys_total

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

The number of expired keys per second.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_debugging_store_expires_total

- CHANGE_PER_SECOND

Total number of keys.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_debugging_mvcc_keys_total

Etcd server uptime.

Preprocessing:

- PROMETHEUS_PATTERN: process_start_time_seconds

- JAVASCRIPT: //use boottime to calculate uptime return (Math.floor(Date.now()/1000)-Number(value));

Virtual memory size in bytes.

Preprocessing:

- PROMETHEUS_PATTERN: process_virtual_memory_bytes

Resident memory size in bytes.

Preprocessing:

- PROMETHEUS_PATTERN: process_resident_memory_bytes

Total user and system CPU time spent in seconds.

Preprocessing:

- PROMETHEUS_PATTERN: process_cpu_seconds_total

- CHANGE_PER_SECOND

Number of open file descriptors.

Preprocessing:

- PROMETHEUS_PATTERN: process_open_fds

The Maximum number of open file descriptors.

Preprocessing:

- PROMETHEUS_PATTERN: process_max_fds

The number of deletes seen by this member per second.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_debugging_mvcc_delete_total

- CHANGE_PER_SECOND

The number of puts seen by this member per second.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_debugging_mvcc_put_total

- CHANGE_PER_SECOND

The number of ranges seen by this member per second.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_debugging_mvcc_range_total

- CHANGE_PER_SECOND

The number of transactions seen by this member per second.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_debugging_mvcc_range_total

- CHANGE_PER_SECOND

The number of events sent by this member per second

Preprocessing:

- PROMETHEUS_PATTERN: etcd_debugging_mvcc_events_total

- CHANGE_PER_SECOND

Total number of pending events to be sent.

Preprocessing:

- PROMETHEUS_PATTERN: etcd_debugging_mvcc_pending_events_total

The number of RPCs completed on the server with grpc_code {#GRPC.CODE}

Preprocessing:

- PROMETHEUS_TO_JSON: grpc_server_handled_total{grpc_method="{#GRPC.CODE}"}

- JAVASCRIPT: var valueArr = JSON.parse(value); return valueArr.reduce(function(acc,obj){ return acc + parseFloat(obj['value']) },0);

- CHANGE_PER_SECOND

The number of bytes sent to peer with ID {#ETCD.PEER}

Preprocessing:

- PROMETHEUS_PATTERN: etcd_network_peer_sent_bytes_total{To="{#ETCD.PEER}"}

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

The number of bytes received from peer with ID {#ETCD.PEER}

Preprocessing:

- PROMETHEUS_PATTERN: etcd_network_peer_received_bytes_total{From="{#ETCD.PEER}"}

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

The number of send failures from peer with ID {#ETCD.PEER}

Preprocessing:

- PROMETHEUS_PATTERN: etcd_network_peer_sent_failures_total{To="{#ETCD.PEER}"}

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

The number of receive failures from the peer with ID {#ETCD.PEER}

Preprocessing:

- PROMETHEUS_PATTERN: etcd_network_peer_received_failures_total{To="{#ETCD.PEER}"}

⛔️ON_FAIL: CUSTOM_VALUE -> 0

- CHANGE_PER_SECOND

-

-

-

Manual close: YES

https://etcd.io/docs/v3.4.0/op-guide/monitoring/#health-check

Depends on:

- Etcd: Service is unavailable

Zabbix has not received data for items for the last 30 minutes

Manual close: YES

Depends on:

- Etcd: Service is unavailable

"If a member does not have a leader, it is totally unavailable."

Rapid leadership changes impact the performance of etcd significantly. It also signals that the leader is unstable, perhaps due to network connectivity issues or excessive load hitting the etcd cluster.

"Normally related to two issues: temporary failures related to a leader election or

longer downtime caused by a loss of quorum in the cluster."

"Rising pending proposals suggests there is a high client load or the member cannot commit proposals."

"Too many reqvests failed on etcd instance with 5xx HTTP code"

Etcd version has changed. Ack to close.

Manual close: YES

Etcd version has changed. Ack to close.

Manual close: YES

Uptime is less than 10 minutes

Manual close: YES

"Heavy file descriptor usage (i.e., near the process’s file descriptor limit) indicates a potential file descriptor exhaustion issue.

If the file descriptors are exhausted, etcd may panic because it cannot create new WAL files."

-