OpenTelemetry

Bring application telemetry into Zabbix to gain observability into service behavior, identify errors, and investigate performance issues in one place.

Available solutions

Zabbix version 8.0
Download

Generic OpenTelemetry by OTLP

Overview

This template collects OpenTelemetry (OTLP) telemetry — traces, metrics and logs — via Zabbix's native APM query items (Telemetry query), without external scripts or a Zabbix proxy script layer. It works with any OTLP-instrumented application or with an OpenTelemetry Collector in front of it.

Requirements

Zabbix version: 8.0 and higher.

Tested versions

This template has been tested on:

  • OpenTelemetry Collector

Configuration

Zabbix should be configured according to the instructions in the Templates out of the box section.

Setup

  1. Point your application's OpenTelemetry SDK (or an OpenTelemetry Collector in front of it) at your Zabbix server/proxy's OTLP endpoint.
  2. Apply this template to a host representing the instrumented application or service.
  3. Adjust the interval macros if the defaults don't match how often your source actually emits data.
  4. Adjust thresholds macros.

Macros used

Name Description Default
{$OTLP.FREQUENCY}

The update interval for OpenTelemetry items. Could be used with context.

 1m
{$OTLP.LOOPBACK}

The loopback for OpenTelemetry items. Could be used with context.

 10m
{$OTLP.TIMESHIFT}

The TimeShift for OpenTelemetry items. Could be used with context.

 10s
{$OTLP.GRANULARITY}

The granularity for OpenTelemetry items. Could be used with context.

 1m
{$SPAN.ERROR.WARN}

Span error rate threshold, in %.

 5
{$LOG.SEVERITY.FATAL}

Threshold for Fatal severity entries in logs.

 0
{$LOG.SEVERITY.ERROR}

Threshold for Error severity entries in logs.

 5
{$LOG.SEVERITY.WARN}

Threshold for Warning severity entries in logs.

 10
{$SPAN.OK.DURATION.WARN}

Threshold for Ok span duration, in seconds.

 0.5
{$SPAN.UNSET.DURATION.WARN}

Threshold for Unset span duration, in seconds.

 1

Items

Name Description Type Key and additional info
Get logs

Collect OpenTelemetry log counts grouped by severity.

 Telemetry query otlp.logs.get
Span: Error count

Number of spans with status code "Error".

 Telemetry query otlp.span.status.error

Preprocessing

  • Check for not supported value: error matches "^No value$"

    ⛔️Custom on fail: Set value to: {"columns":{"count":0}}

  • JSON Path: $.columns.count

  • Discard unchanged with heartbeat: 1h
Span: Error duration

Average duration of spans with status code "Error".

 Telemetry query otlp.span.status.error.duration

Preprocessing

  • JSON Path: $.columns.avg

  • Custom multiplier: 1.0E-9
Span: Ok count

Number of spans with status code "Ok".

 Telemetry query otlp.span.status.ok

Preprocessing

  • Check for not supported value: error matches "^No value$"

    ⛔️Custom on fail: Set value to: {"columns":{"count":0}}

  • JSON Path: $.columns.count

  • Discard unchanged with heartbeat: 1h
Span: Ok duration

Average duration of spans with status code "Ok".

 Telemetry query otlp.span.status.ok.duration

Preprocessing

  • JSON Path: $.columns.avg

  • Custom multiplier: 1.0E-9
Span: Unset count

Number of spans with status code "Unset".

 Telemetry query otlp.span.status.unset

Preprocessing

  • Check for not supported value: error matches "^No value$"

    ⛔️Custom on fail: Set value to: {"columns":{"count":0}}

  • JSON Path: $.columns.count

  • Discard unchanged with heartbeat: 1h
Span: Unset duration

Average duration of spans with status code "Unset".

 Telemetry query otlp.span.status.unset.duration

Preprocessing

  • JSON Path: $.columns.avg

  • Custom multiplier: 1.0E-9
System: Uptime

System uptime.

 Telemetry query otlp.metrics.system.uptime

Preprocessing

  • JSON Path: $.columns.avg
Metrics: Count

Amount of incoming metrics over the {$OTLP.FREQUENCY:metrics}.

 Telemetry query otlp.metrics.count

Preprocessing

  • Check for not supported value: error matches "^No value$"

    ⛔️Custom on fail: Set value to: {"columns":{"count":0}}

  • JSON Path: $.columns.count

  • Discard unchanged with heartbeat: 1h
Metrics: Gauge count

Amount of incoming gauge metrics over the {$OTLP.FREQUENCY:gauge}.

 Telemetry query otlp.metrics_gauge.count

Preprocessing

  • Check for not supported value: error matches "^No value$"

    ⛔️Custom on fail: Set value to: {"columns":{"count":0}}

  • JSON Path: $.columns.count

  • Discard unchanged with heartbeat: 1h
Metrics: Histogram count

Amount of incoming histogram metrics over the {$OTLP.FREQUENCY:histogram}.

 Telemetry query otlp.metrics_histogram.count

Preprocessing

  • Check for not supported value: error matches "^No value$"

    ⛔️Custom on fail: Set value to: {"columns":{"count":0}}

  • JSON Path: $.columns.count

  • Discard unchanged with heartbeat: 1h
Logs: Info entries count

Number of Info severity entries in logs.

 Dependent item otlp.logs.info

Preprocessing

  • JSON Path: The text is too long. Please see the template.

    ⛔️Custom on fail: Discard value

  • Discard unchanged with heartbeat: 1h
Logs: Warning entries count

Number of Warning severity entries in logs.

 Dependent item otlp.logs.warn

Preprocessing

  • JSON Path: The text is too long. Please see the template.

    ⛔️Custom on fail: Discard value

  • Discard unchanged with heartbeat: 1h
Logs: Error entries count

Number of Error severity entries in logs.

 Dependent item otlp.logs.average

Preprocessing

  • JSON Path: The text is too long. Please see the template.

    ⛔️Custom on fail: Discard value

  • Discard unchanged with heartbeat: 1h
Logs: Fatal entries count

Number of Fatal severity entries in logs.

 Dependent item otlp.logs.high

Preprocessing

  • JSON Path: The text is too long. Please see the template.

    ⛔️Custom on fail: Discard value

  • Discard unchanged with heartbeat: 1h
Span: Error percentage

Percentage of error spans.

 Calculated otlp.span.error.percentage

Preprocessing

  • Check for not supported value: any error

    ⛔️Custom on fail: Discard value

Triggers

Name Description Expression Severity Dependencies and additional info
OpenTelemetry: Span duration is too big min(/Generic OpenTelemetry by OTLP/otlp.span.status.ok.duration,5m)>{$SPAN.OK.DURATION.WARN} Warning
OpenTelemetry: Unset span duration is too big min(/Generic OpenTelemetry by OTLP/otlp.span.status.unset.duration,5m)>{$SPAN.UNSET.DURATION.WARN} Warning
OpenTelemetry: Warning severity level entries in logs last(/Generic OpenTelemetry by OTLP/otlp.logs.warn)>{$LOG.SEVERITY.WARN} Warning
OpenTelemetry: Error severity level entries in logs last(/Generic OpenTelemetry by OTLP/otlp.logs.average)>{$LOG.SEVERITY.ERROR} Average
OpenTelemetry: Fatal severity level entries in logs last(/Generic OpenTelemetry by OTLP/otlp.logs.high)>{$LOG.SEVERITY.FATAL} High
OpenTelemetry: High span error rate last(/Generic OpenTelemetry by OTLP/otlp.span.error.percentage)>{$SPAN.ERROR.WARN} Warning

Feedback

Please report any issues with the template at https://support.zabbix.com

You can also provide feedback, discuss the template, or ask for help at ZABBIX forums

Articles and documentation

+ Propose new article

Sie können die Integration nicht finden, die Sie benötigen?

Request custom integration

Zabbix integration team will develop custom integration based on your requirements and Zabbix best practices.

Request

Propose integration

Have you already developed high quality integration and want to submit to Zabbix integration repository?

Propose