ZooKeeper

ZooKeeper

Apache ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Available solutions




This template is for Zabbix version: 7.4

Source: https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/zookeeper_http?at=release/7.4

Zookeeper by HTTP

Overview

This template is designed for the effortless deployment of Apache Zookeeper monitoring by Zabbix via HTTP and doesn't require any external scripts.

This template works with standalone and cluster instances. Metrics are collected from each Zookeeper node by requests to AdminServer.

Requirements

Zabbix version: 7.4 and higher.

Tested versions

This template has been tested on:

  • Apache Zookeeper, version 3.6+, 3.8+

Configuration

Zabbix should be configured according to the instructions in the Templates out of the box section.

Setup

  1. Enable the AdminServer and configure the parameters according to the official documentation.

  2. Set the hostname or IP address of the Apache Zookeeper host in the {$ZOOKEEPER.HOST} macro. You can also change the {$ZOOKEEPER.COMMAND_URL}, {$ZOOKEEPER.PORT} and {$ZOOKEEPER.SCHEME} macros if necessary.

Macros used

Name Description Default
{$ZOOKEEPER.HOST}

The hostname or IP address of the Apache Zookeeper host.

 <SET ZOOKEEPER HOST>
{$ZOOKEEPER.PORT}

The port the embedded Jetty server listens on (admin.serverPort).

 8080
{$ZOOKEEPER.COMMAND_URL}

The URL for listing and issuing commands relative to the root URL (admin.commandURL).

 commands
{$ZOOKEEPER.SCHEME}

Request scheme which may be http or https

 http
{$ZOOKEEPER.FILE_DESCRIPTORS.MAX.WARN}

Maximum percentage of file descriptors usage alert threshold (for trigger expression).

 85
{$ZOOKEEPER.OUTSTANDING_REQ.MAX.WARN}

Maximum number of outstanding requests (for trigger expression).

 10
{$ZOOKEEPER.PENDING_SYNCS.MAX.WARN}

Maximum number of pending syncs from the followers (for trigger expression).

 10

Items

Name Description Type Key and additional info
Get server metrics HTTP agent zookeeper.get_metrics
Get connections stats

Get information on client connections to server. Note, depending on the number of client connections this operation may be expensive (i.e. impact server performance).

 HTTP agent zookeeper.get_connections_stats
Server mode

Mode of the server. In an ensemble, this may either be leader or follower. Otherwise, it is standalone

 Dependent item zookeeper.server_state

Preprocessing

  • JSON Path: $.server_state

  • Discard unchanged with heartbeat: 1h
Uptime

Uptime that a peer has been in a table leading/following/observing state.

 Dependent item zookeeper.uptime

Preprocessing

  • JSON Path: $.uptime

  • Custom multiplier: 0.001
Version

Version of Zookeeper server.

 Dependent item zookeeper.version

Preprocessing

  • JSON Path: $.version

  • Regular expression: ^([0-9\.]+) \1

  • Discard unchanged with heartbeat: 3h
Approximate data size

Data tree size in bytes.The size includes the znode path and its value.

 Dependent item zookeeper.approximate_data_size

Preprocessing

  • JSON Path: $.approximate_data_size
File descriptors, max

Maximum number of file descriptors that a zookeeper server can open.

 Dependent item zookeeper.max_file_descriptor_count

Preprocessing

  • JSON Path: $.max_file_descriptor_count

  • Discard unchanged with heartbeat: 1h
File descriptors, open

Number of file descriptors that a zookeeper server has open.

 Dependent item zookeeper.open_file_descriptor_count

Preprocessing

  • JSON Path: $.open_file_descriptor_count
Outstanding requests

The number of queued requests when the server is under load and is receiving more sustained requests than it can process.

 Dependent item zookeeper.outstanding_requests

Preprocessing

  • JSON Path: $.outstanding_requests
Commit per sec

The number of commits performed per second

 Dependent item zookeeper.commit_count.rate

Preprocessing

  • JSON Path: $.commit_count

  • Change per second
Diff syncs per sec

Number of diff syncs performed per second

 Dependent item zookeeper.diff_count.rate

Preprocessing

  • JSON Path: $.diff_count

  • Change per second
Snap syncs per sec

Number of snap syncs performed per second

 Dependent item zookeeper.snap_count.rate

Preprocessing

  • JSON Path: $.snap_count

  • Change per second
Looking per sec

Rate of transitions into looking state.

 Dependent item zookeeper.looking_count.rate

Preprocessing

  • JSON Path: $.looking_count

  • Change per second
Alive connections

Number of active clients connected to a zookeeper server.

 Dependent item zookeeper.num_alive_connections

Preprocessing

  • JSON Path: $.num_alive_connections
Global sessions

Number of global sessions.

 Dependent item zookeeper.global_sessions

Preprocessing

  • JSON Path: $.global_sessions
Local sessions

Number of local sessions.

 Dependent item zookeeper.local_sessions

Preprocessing

  • JSON Path: $.local_sessions
Drop connections per sec

Rate of connection drops.

 Dependent item zookeeper.connection_drop_count.rate

Preprocessing

  • JSON Path: $.connection_drop_count

  • Change per second
Rejected connections per sec

Rate of connection rejected.

 Dependent item zookeeper.connection_rejected.rate

Preprocessing

  • JSON Path: $.connection_rejected

  • Change per second
Revalidate connections per sec

Rate of connection revalidations.

 Dependent item zookeeper.connection_revalidate_count.rate

Preprocessing

  • JSON Path: $.connection_revalidate_count

  • Change per second
Revalidate per sec

Rate of revalidations.

 Dependent item zookeeper.revalidate_count.rate

Preprocessing

  • JSON Path: $.revalidate_count

  • Change per second
Latency, max

The maximum amount of time it takes for the server to respond to a client request.

 Dependent item zookeeper.max_latency

Preprocessing

  • JSON Path: $.max_latency
Latency, min

The minimum amount of time it takes for the server to respond to a client request.

 Dependent item zookeeper.min_latency

Preprocessing

  • JSON Path: $.min_latency
Latency, avg

The average amount of time it takes for the server to respond to a client request.

 Dependent item zookeeper.avg_latency

Preprocessing

  • JSON Path: $.avg_latency
Znode count

The number of znodes in the ZooKeeper namespace (the data)

 Dependent item zookeeper.znode_count

Preprocessing

  • JSON Path: $.znode_count

  • Discard unchanged with heartbeat: 1h
Ephemeral nodes count

Number of ephemeral nodes that a zookeeper server has in its data tree.

 Dependent item zookeeper.ephemerals_count

Preprocessing

  • JSON Path: $.ephemerals_count
Watch count

Number of watches currently set on the local ZooKeeper process.

 Dependent item zookeeper.watch_count

Preprocessing

  • JSON Path: $.watch_count
Packets sent per sec

The number of zookeeper packets sent from a server per second.

 Dependent item zookeeper.packets_sent

Preprocessing

  • JSON Path: $.packets_sent

  • Change per second
Packets received per sec

The number of zookeeper packets received by a server per second.

 Dependent item zookeeper.packets_received.rate

Preprocessing

  • JSON Path: $.packets_received

  • Change per second
Bytes received per sec

Number of bytes received per second.

 Dependent item zookeeper.bytes_received_count.rate

Preprocessing

  • JSON Path: $.bytes_received_count

  • Change per second
Election time, avg

Time between entering and leaving election.

 Dependent item zookeeper.avg_election_time

Preprocessing

  • JavaScript: The text is too long. Please see the template.
Elections

Number of elections happened.

 Dependent item zookeeper.cnt_election_time

Preprocessing

  • JavaScript: The text is too long. Please see the template.
Fsync time, avg

Time to fsync transaction log.

 Dependent item zookeeper.avg_fsynctime

Preprocessing

  • JavaScript: The text is too long. Please see the template.
Fsync

Count of performed fsyncs.

 Dependent item zookeeper.cnt_fsynctime

Preprocessing

  • JavaScript: The text is too long. Please see the template.
Snapshot write time, avg

Average time to write a snapshot.

 Dependent item zookeeper.avg_snapshottime

Preprocessing

  • JavaScript: The text is too long. Please see the template.
Snapshot writes

Count of performed snapshot writes.

 Dependent item zookeeper.cnt_snapshottime

Preprocessing

  • JavaScript: The text is too long. Please see the template.

Triggers

Name Description Expression Severity Dependencies and additional info
Zookeeper: Server mode has changed

Zookeeper node state has changed. Acknowledge to close the problem manually.

 last(/Zookeeper by HTTP/zookeeper.server_state,#1)<>last(/Zookeeper by HTTP/zookeeper.server_state,#2) and length(last(/Zookeeper by HTTP/zookeeper.server_state))>0 Info Manual close: Yes
Zookeeper: Failed to fetch info data

Zabbix has not received data for items for the last 10 minutes

 nodata(/Zookeeper by HTTP/zookeeper.uptime,10m)=1 Warning Manual close: Yes
Zookeeper: Version has changed

Zookeeper version has changed. Acknowledge to close the problem manually.

 last(/Zookeeper by HTTP/zookeeper.version,#1)<>last(/Zookeeper by HTTP/zookeeper.version,#2) and length(last(/Zookeeper by HTTP/zookeeper.version))>0 Info Manual close: Yes
Zookeeper: Too many file descriptors used

Number of file descriptors used more than {$ZOOKEEPER.FILE_DESCRIPTORS.MAX.WARN}% of the available number of file descriptors.

 min(/Zookeeper by HTTP/zookeeper.open_file_descriptor_count,5m) * 100 / last(/Zookeeper by HTTP/zookeeper.max_file_descriptor_count) > {$ZOOKEEPER.FILE_DESCRIPTORS.MAX.WARN} Warning
Zookeeper: Too many queued requests

Number of queued requests in the server. This goes up when the server receives more requests than it can process.

 min(/Zookeeper by HTTP/zookeeper.outstanding_requests,5m)>{$ZOOKEEPER.OUTSTANDING_REQ.MAX.WARN} Average Manual close: Yes

LLD rule Leader metrics discovery

Name Description Type Key and additional info
Leader metrics discovery

Additional metrics for leader node

 Dependent item zookeeper.metrics.leader

Preprocessing

  • JSON Path: $.server_state

  • JavaScript: The text is too long. Please see the template.

Item prototypes for Leader metrics discovery

Name Description Type Key and additional info
Pending syncs{#SINGLETON}

Number of pending syncs to carry out to ZooKeeper ensemble followers.

 Dependent item zookeeper.pending_syncs[{#SINGLETON}]

Preprocessing

  • JSON Path: $.pending_syncs
Quorum size{#SINGLETON} Dependent item zookeeper.quorum_size[{#SINGLETON}]

Preprocessing

  • JSON Path: $.quorum_size
Synced followers{#SINGLETON}

Number of synced followers reported when a node server_state is leader.

 Dependent item zookeeper.synced_followers[{#SINGLETON}]

Preprocessing

  • JSON Path: $.synced_followers
Synced non-voting follower{#SINGLETON}

Number of synced voting followers reported when a node server_state is leader.

 Dependent item zookeeper.synced_non_voting_followers[{#SINGLETON}]

Preprocessing

  • JSON Path: $.synced_non_voting_followers
Synced observers{#SINGLETON}

Number of synced observers.

 Dependent item zookeeper.synced_observers[{#SINGLETON}]

Preprocessing

  • JSON Path: $.synced_observers
Learners{#SINGLETON}

Number of learners.

 Dependent item zookeeper.learners[{#SINGLETON}]

Preprocessing

  • JSON Path: $.learners

Trigger prototypes for Leader metrics discovery

Name Description Expression Severity Dependencies and additional info
Zookeeper: Too many pending syncs min(/Zookeeper by HTTP/zookeeper.pending_syncs[{#SINGLETON}],5m)>{$ZOOKEEPER.PENDING_SYNCS.MAX.WARN} Average Manual close: Yes
Zookeeper: Too few active followers

The number of followers should equal the total size of your ZooKeeper ensemble, minus 1 (the leader is not included in the follower count). If the ensemble fails to maintain quorum, all automatic failover features are suspended.

 last(/Zookeeper by HTTP/zookeeper.synced_followers[{#SINGLETON}]) < last(/Zookeeper by HTTP/zookeeper.quorum_size[{#SINGLETON}])-1 Average

LLD rule Clients discovery

Name Description Type Key and additional info
Clients discovery

Get list of client connections.

Note, depending on the number of client connections this operation may be expensive (i.e. impact server performance).

 HTTP agent zookeeper.clients

Preprocessing

  • JavaScript: The text is too long. Please see the template.

Item prototypes for Clients discovery

Name Description Type Key and additional info
Zookeeper client {#TYPE} [{#CLIENT}]: Get client info

The item gets information about "{#CLIENT}" client of "{#TYPE}" type.

 Dependent item zookeeper.client_info[{#TYPE},{#CLIENT}]

Preprocessing

  • JSON Path: The text is too long. Please see the template.

    ⛔️Custom on fail: Discard value
Zookeeper client {#TYPE} [{#CLIENT}]: Latency, max

The maximum amount of time it takes for the server to respond to a client request.

 Dependent item zookeeper.max_latency[{#TYPE},{#CLIENT}]

Preprocessing

  • JSON Path: $.max_latency
Zookeeper client {#TYPE} [{#CLIENT}]: Latency, min

The minimum amount of time it takes for the server to respond to a client request.

 Dependent item zookeeper.min_latency[{#TYPE},{#CLIENT}]

Preprocessing

  • JSON Path: $.min_latency
Zookeeper client {#TYPE} [{#CLIENT}]: Latency, avg

The average amount of time it takes for the server to respond to a client request.

 Dependent item zookeeper.avg_latency[{#TYPE},{#CLIENT}]

Preprocessing

  • JSON Path: $.avg_latency
Zookeeper client {#TYPE} [{#CLIENT}]: Packets sent per sec

The number of packets sent.

 Dependent item zookeeper.packets_sent[{#TYPE},{#CLIENT}]

Preprocessing

  • JSON Path: $.packets_sent

  • Change per second
Zookeeper client {#TYPE} [{#CLIENT}]: Packets received per sec

The number of packets received.

 Dependent item zookeeper.packets_received[{#TYPE},{#CLIENT}]

Preprocessing

  • JSON Path: $.packets_received

  • Change per second
Zookeeper client {#TYPE} [{#CLIENT}]: Outstanding requests

The number of queued requests when the server is under load and is receiving more sustained requests than it can process.

 Dependent item zookeeper.outstanding_requests[{#TYPE},{#CLIENT}]

Preprocessing

  • JSON Path: $.outstanding_requests

Feedback

Please report any issues with the template at https://support.zabbix.com

You can also provide feedback, discuss the template, or ask for help at ZABBIX forums

Articles and documentation

+ Propose new article

Didn't find integration you need?

Request custom integration

Zabbix integration team will develop custom integration based on your requirements and Zabbix best practices.

Request

Propose your integration

Have you already developed high quality integration and want to submit to Zabbix integration repository?

Propose