Source: https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/gitlab_http?at=release/6.4
GitLab by HTTP
Overview
For Zabbix version: 6.4 and higher.
This template is designed to monitor GitLab by Zabbix that works without any external scripts.
Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
The template GitLab by HTTP
— collects metrics by an HTTP agent from the GitLab /-/metrics
endpoint.
See https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_metrics.html.
This template was tested on:
- GitLab, version 13.5.3 EE
Setup
See Zabbix template operation for basic instructions.
This template works with self-hosted GitLab instances. Internal service metrics are collected from the GitLab /-/metrics
endpoint.
To access metrics following two methods are available:
- Explicitly allow monitoring instance IP address in gitlab whitelist configuration.
- Get token from Gitlab
Admin -> Monitoring -> Health check
page: http://your.gitlab.address/admin/health_check; Use this token in macro{$GITLAB.HEALTH.TOKEN}
as variable path, like:?token=your_token
. Remember to change the macros{$GITLAB.URL}
. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. Some metrics may not be collected depending on your Gitlab instance version and configuration. See Gitlab's documentation for further information about its metric collection.
Zabbix configuration
No specific Zabbix configuration is required.
Macros used
Name | Description | Default |
---|---|---|
{$GITLAB.HEALTH.TOKEN} | The token path for Gitlab health check. Example |
`` |
{$GITLAB.HTTP.FAIL.MAX.WARN} | The maximum number of HTTP request failures for a trigger expression. |
2 |
{$GITLAB.OPEN.FDS.MAX.WARN} | The maximum percentage of used file descriptors for a trigger expression. |
90 |
{$GITLAB.PUMA.QUEUE.MAX.WARN} | The maximum number of Puma queued requests for a trigger expression. |
1 |
{$GITLAB.PUMA.UTILIZATION.MAX.WARN} | The maximum percentage of Puma thread utilization for a trigger expression. |
90 |
{$GITLAB.REDIS.FAIL.MAX.WARN} | The maximum number of Redis client exceptions for a trigger expression. |
2 |
{$GITLAB.UNICORN.QUEUE.MAX.WARN} | The maximum number of Unicorn queued requests for a trigger expression. |
1 |
{$GITLAB.UNICORN.UTILIZATION.MAX.WARN} | The maximum percentage of Unicorn workers utilization for a trigger expression. |
90 |
{$GITLAB.URL} | URL of a GitLab instance. |
http://localhost |
Template links
There are no template links in this template.
Discovery rules
Name | Description | Type | Key and additional info |
---|---|---|---|
Puma metrics discovery | Discovery of Puma specific metrics when Puma is used. |
HTTP_AGENT | gitlab.puma.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: |
Unicorn metrics discovery | DiscoveryUnicorn specific metrics, when Unicorn is used. |
HTTP_AGENT | gitlab.unicorn.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: |
Items collected
Group | Name | Description | Type | Key and additional info |
---|---|---|---|---|
GitLab | GitLab: Instance readiness check | The readiness probe checks whether the GitLab instance is ready to accept traffic via Rails Controllers. |
HTTP_AGENT | gitlab.readiness Preprocessing: - CHECK_NOT_SUPPORTED ⛔️ON_FAIL: - JSONPATH: - BOOL_TO_DECIMAL ⛔️ON_FAIL: - DISCARD_UNCHANGED_HEARTBEAT: |
GitLab | GitLab: Application server status | Checks whether the application server is running. This probe is used to know if Rails Controllers are not deadlocked due to a multi-threading. |
HTTP_AGENT | gitlab.liveness Preprocessing: - CHECK_NOT_SUPPORTED ⛔️ON_FAIL: - JSONPATH: - BOOL_TO_DECIMAL ⛔️ON_FAIL: - DISCARD_UNCHANGED_HEARTBEAT: |
GitLab | GitLab: Version | Version of the GitLab instance. |
DEPENDENT | gitlab.deployments.version Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
GitLab | GitLab: Ruby: First process start time | Minimum UNIX timestamp of ruby processes start time. |
DEPENDENT | gitlab.ruby.process_start_time_seconds.first Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
GitLab | GitLab: Ruby: Last process start time | Maximum UNIX timestamp ruby processes start time. |
DEPENDENT | gitlab.ruby.process_start_time_seconds.last Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
GitLab | GitLab: User logins, total | Counter of how many users have logged in since GitLab was started or restarted. |
DEPENDENT | gitlab.user_session_logins_total Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab | GitLab: User CAPTCHA logins failed, total | Counter of failed CAPTCHA attempts during login. |
DEPENDENT | gitlab.failed_login_captcha_total Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab | GitLab: User CAPTCHA logins, total | Counter of successful CAPTCHA attempts during login. |
DEPENDENT | gitlab.successful_login_captcha_total Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab | GitLab: Upload file does not exist | Number of times an upload record could not find its file. |
DEPENDENT | gitlab.upload_file_does_not_exist Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab | GitLab: Pipelines: Processing events, total | Total amount of pipeline processing events. |
DEPENDENT | gitlab.pipeline.processing_events_total Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab | GitLab: Pipelines: Created, total | Counter of pipelines created. |
DEPENDENT | gitlab.pipeline.created_total Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab | GitLab: Pipelines: Auto DevOps pipelines, total | Counter of completed Auto DevOps pipelines. |
DEPENDENT | gitlab.pipeline.auto_devops_completed.total Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab | GitLab: Pipelines: Auto DevOps pipelines, failed | Counter of completed Auto DevOps pipelines with status "failed". |
DEPENDENT | gitlab.pipeline.auto_devops_completed_total.failed Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab | GitLab: Pipelines: CI/CD creation duration | The sum of the time in seconds it takes to create a CI/CD pipeline. |
DEPENDENT | gitlab.pipeline.pipeline_creation Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab | GitLab: Pipelines: Pipelines: CI/CD creation count | The count of the time it takes to create a CI/CD pipeline. |
DEPENDENT | gitlab.pipeline.pipeline_creation.count Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab | GitLab: Database: Connection pool, busy | Connections to the main database in use where the owner is still alive. |
DEPENDENT | gitlab.database.connection_pool_busy Preprocessing: - JSONPATH: |
GitLab | GitLab: Database: Connection pool, current | Current connections to the main database in the pool. |
DEPENDENT | gitlab.database.connection_pool_connections Preprocessing: - JSONPATH: |
GitLab | GitLab: Database: Connection pool, dead | Connections to the main database in use where the owner is not alive. |
DEPENDENT | gitlab.database.connection_pool_dead Preprocessing: - JSONPATH: |
GitLab | GitLab: Database: Connection pool, idle | Connections to the main database not in use. |
DEPENDENT | gitlab.database.connection_pool_idle Preprocessing: - JSONPATH: |
GitLab | GitLab: Database: Connection pool, size | Total connection to the main database pool capacity. |
DEPENDENT | gitlab.database.connection_pool_size Preprocessing: - JSONPATH: |
GitLab | GitLab: Database: Connection pool, waiting | Threads currently waiting on this queue. |
DEPENDENT | gitlab.database.connection_pool_waiting Preprocessing: - JSONPATH: |
GitLab | GitLab: Redis: Client requests rate, queues | Number of Redis client requests per second. (Instance: queues) |
DEPENDENT | gitlab.redis.client_requests.queues.rate Preprocessing: - JSONPATH: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
GitLab | GitLab: Redis: Client requests rate, cache | Number of Redis client requests per second. (Instance: cache) |
DEPENDENT | gitlab.redis.client_requests.cache.rate Preprocessing: - JSONPATH: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
GitLab | GitLab: Redis: Client requests rate, shared_state | Number of Redis client requests per second. (Instance: shared_state) |
DEPENDENT | gitlab.redis.client_requests.shared_state.rate Preprocessing: - JSONPATH: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
GitLab | GitLab: Redis: Client exceptions rate, queues | Number of Redis client exceptions per second. (Instance: queues) |
DEPENDENT | gitlab.redis.client_exceptions.queues.rate Preprocessing: - JSONPATH: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
GitLab | GitLab: Redis: Client exceptions rate, cache | Number of Redis client exceptions per second. (Instance: cache) |
DEPENDENT | gitlab.redis.client_exceptions.cache.rate Preprocessing: - JSONPATH: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
GitLab | GitLab: Redis: client exceptions rate, shared_state | Number of Redis client exceptions per second. (Instance: shared_state) |
DEPENDENT | gitlab.redis.client_exceptions.shared_state.rate Preprocessing: - JSONPATH: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
GitLab | GitLab: Cache: Misses rate, total | The cache read miss count. |
DEPENDENT | gitlab.cache.misses_total.rate Preprocessing: - JSONPATH: - CHANGE_PER_SECOND |
GitLab | GitLab: Cache: Operations rate, total | The count of cache operations. |
DEPENDENT | gitlab.cache.operations_total.rate Preprocessing: - JSONPATH: - CHANGE_PER_SECOND |
GitLab | GitLab: Ruby: CPU usage per second | Average CPU time util in seconds. |
DEPENDENT | gitlab.ruby.process_cpu_seconds.rate Preprocessing: - JSONPATH: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
GitLab | GitLab: Ruby: Running_threads | Number of running Ruby threads. |
DEPENDENT | gitlab.ruby.threads_running Preprocessing: - JSONPATH: |
GitLab | GitLab: Ruby: File descriptors opened, avg | Average number of opened file descriptors. |
DEPENDENT | gitlab.ruby.file_descriptors.avg Preprocessing: - JSONPATH: |
GitLab | GitLab: Ruby: File descriptors opened, max | Maximum number of opened file descriptors. |
DEPENDENT | gitlab.ruby.file_descriptors.max Preprocessing: - JSONPATH: |
GitLab | GitLab: Ruby: File descriptors opened, min | Minimum number of opened file descriptors. |
DEPENDENT | gitlab.ruby.file_descriptors.min Preprocessing: - JSONPATH: |
GitLab | GitLab: Ruby: File descriptors, max | Maximum number of open file descriptors per process. |
DEPENDENT | gitlab.ruby.process_max_fds Preprocessing: - JSONPATH: |
GitLab | GitLab: Ruby: RSS memory, avg | Average RSS Memory usage in bytes. |
DEPENDENT | gitlab.ruby.process_resident_memory_bytes.avg Preprocessing: - JSONPATH: |
GitLab | GitLab: Ruby: RSS memory, min | Minimum RSS Memory usage in bytes. |
DEPENDENT | gitlab.ruby.process_resident_memory_bytes.min Preprocessing: - JSONPATH: |
GitLab | GitLab: Ruby: RSS memory, max | Maximum RSS Memory usage in bytes. |
DEPENDENT | gitlab.ruby.process_resident_memory_bytes.max Preprocessing: - JSONPATH: |
GitLab | GitLab: HTTP requests rate, total | Number of requests received into the system. |
DEPENDENT | gitlab.http.requests.rate Preprocessing: - JSONPATH: - CHANGE_PER_SECOND |
GitLab | GitLab: HTTP requests rate, 5xx | Number of handle failures of requests with HTTP-code 5xx. |
DEPENDENT | gitlab.http.requests.5xx.rate Preprocessing: - JSONPATH: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
GitLab | GitLab: HTTP requests rate, 4xx | Number of handle failures of requests with code 4XX. |
DEPENDENT | gitlab.http.requests.4xx.rate Preprocessing: - JSONPATH: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
GitLab | GitLab: Transactions per second | Transactions per second (gitlabtransaction* metrics). |
DEPENDENT | gitlab.transactions.rate Preprocessing: - JSONPATH: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
GitLab: Puma stats | GitLab: Active connections | Number of puma threads processing a request. |
DEPENDENT | gitlab.puma.active_connections[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Puma stats | GitLab: Workers | Total number of puma workers. |
DEPENDENT | gitlab.puma.workers[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Puma stats | GitLab: Running workers | The number of booted puma workers. |
DEPENDENT | gitlab.puma.running_workers[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Puma stats | GitLab: Stale workers | The number of old puma workers. |
DEPENDENT | gitlab.puma.stale_workers[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Puma stats | GitLab: Running threads | The number of running puma threads. |
DEPENDENT | gitlab.puma.running[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Puma stats | GitLab: Queued connections | The number of connections in that puma worker's "todo" set waiting for a worker thread. |
DEPENDENT | gitlab.puma.queued_connections[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Puma stats | GitLab: Pool capacity | The number of requests the puma worker is capable of taking right now. |
DEPENDENT | gitlab.puma.pool_capacity[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Puma stats | GitLab: Max threads | The maximum number of puma worker threads. |
DEPENDENT | gitlab.puma.max_threads[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Puma stats | GitLab: Idle threads | The number of spawned puma threads which are not processing a request. |
DEPENDENT | gitlab.puma.idle_threads[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Puma stats | GitLab: Killer terminations, total | The number of workers terminated by PumaWorkerKiller. |
DEPENDENT | gitlab.puma.killer_terminations_total[{#SINGLETON}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
GitLab: Unicorn stats | GitLab: Unicorn: Workers | The number of Unicorn workers |
DEPENDENT | gitlab.unicorn.unicorn_workers[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Unicorn stats | GitLab: Unicorn: Active connections | The number of active Unicorn connections. |
DEPENDENT | gitlab.unicorn.active_connections[{#SINGLETON}] Preprocessing: - JSONPATH: |
GitLab: Unicorn stats | GitLab: Unicorn: Queued connections | The number of queued Unicorn connections. |
DEPENDENT | gitlab.unicorn.queued_connections[{#SINGLETON}] Preprocessing: - JSONPATH: |
Zabbix raw items | GitLab: Get instance metrics | - |
HTTP_AGENT | gitlab.get_metrics Preprocessing: - CHECK_NOT_SUPPORTED ⛔️ON_FAIL: - PROMETHEUS_TO_JSON |
Triggers
Name | Description | Expression | Severity | Dependencies and additional info |
---|---|---|---|---|
GitLab: Gitlab instance is not able to accept traffic | - |
last(/GitLab by HTTP/gitlab.readiness)=0 |
HIGH | Depends on: - GitLab: Liveness check was failed |
GitLab: Liveness check was failed | The application server is not running or Rails Controllers are deadlocked. |
last(/GitLab by HTTP/gitlab.liveness)=0 |
HIGH | |
GitLab: Version has changed | The GitLab version has changed. Perform Ack to close. |
last(/GitLab by HTTP/gitlab.deployments.version,#1)<>last(/GitLab by HTTP/gitlab.deployments.version,#2) and length(last(/GitLab by HTTP/gitlab.deployments.version))>0 |
INFO | Manual close: YES |
GitLab: Too many Redis queues client exceptions | "Too many Redis client exceptions during the requests to Redis instance queues." |
min(/GitLab by HTTP/gitlab.redis.client_exceptions.queues.rate,5m)>{$GITLAB.REDIS.FAIL.MAX.WARN} |
WARNING | |
GitLab: Too many Redis cache client exceptions | "Too many Redis client exceptions during the requests to Redis instance cache." |
min(/GitLab by HTTP/gitlab.redis.client_exceptions.cache.rate,5m)>{$GITLAB.REDIS.FAIL.MAX.WARN} |
WARNING | |
GitLab: Too many Redis shared_state client exceptions | "Too many Redis client exceptions during the requests to Redis instance shared_state." |
min(/GitLab by HTTP/gitlab.redis.client_exceptions.shared_state.rate,5m)>{$GITLAB.REDIS.FAIL.MAX.WARN} |
WARNING | |
GitLab: Failed to fetch info data | Zabbix has not received a metrics data for the last 30 minutes |
nodata(/GitLab by HTTP/gitlab.ruby.threads_running,30m)=1 |
WARNING | Manual close: YES Depends on: - GitLab: Liveness check was failed |
GitLab: Current number of open files is too high | - |
min(/GitLab by HTTP/gitlab.ruby.file_descriptors.max,5m)/last(/GitLab by HTTP/gitlab.ruby.process_max_fds)*100>{$GITLAB.OPEN.FDS.MAX.WARN} |
WARNING | |
GitLab: Too many HTTP requests failures | "Too many requests failed on GitLab instance with 5xx HTTP code" |
min(/GitLab by HTTP/gitlab.http.requests.5xx.rate,5m)>{$GITLAB.HTTP.FAIL.MAX.WARN} |
WARNING | |
GitLab: Puma instance thread utilization is too high | - |
min(/GitLab by HTTP/gitlab.puma.active_connections[{#SINGLETON}],5m)/last(/GitLab by HTTP/gitlab.puma.max_threads[{#SINGLETON}])*100>{$GITLAB.PUMA.UTILIZATION.MAX.WARN} |
WARNING | |
GitLab: Puma is queueing requests | - |
min(/GitLab by HTTP/gitlab.puma.queued_connections[{#SINGLETON}],15m)>{$GITLAB.PUMA.QUEUE.MAX.WARN} |
WARNING | |
GitLab: Unicorn worker utilization is too high | - |
min(/GitLab by HTTP/gitlab.unicorn.active_connections[{#SINGLETON}],5m)/last(/GitLab by HTTP/gitlab.unicorn.unicorn_workers[{#SINGLETON}])*100>{$GITLAB.UNICORN.UTILIZATION.MAX.WARN} |
WARNING | |
GitLab: Unicorn is queueing requests | - |
min(/GitLab by HTTP/gitlab.unicorn.queued_connections[{#SINGLETON}],5m)>{$GITLAB.UNICORN.QUEUE.MAX.WARN} |
WARNING |
Feedback
Please report any issues with the template at https://support.zabbix.com.
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums.