Traces
Overview
The APM > Traces section displays traces collected from your applications and services.
A trace represents the path that a request takes through your applications and services. For example, when a user clicks the Place order button in an online shop, the request can pass from the frontend service, through the checkout and payment services, to the database.
A trace consists of spans. Each span represents a single operation, such as an HTTP request that the frontend service sends to the checkout service, a database query that saves the order, or a function call that calculates the shipping cost.
Spans form a hierarchy. The root span represents the first operation of the request. Every other span is a child span of the span during which its operation was started. For example, the frontend service receiving the Place order request is the root span, and the checkout service processing the order is its child span.
Each span is produced by a service (e.g., zabbix-frontend; named by the OpenTelemetry service.name resource attribute), and by an instrumentation scope, the part of the code that created it (e.g., a library that traces HTTP requests).
To provide context about each operation, spans also have key-value attributes: span attributes describe the operation itself (e.g., http.method: POST), and resource attributes describe the service and the environment it runs in (e.g., host.name: eabf8c2a5ddc).
For more details on traces, see the OpenTelemetry documentation.
The APM > Traces section contains the following elements:
- Kiosk mode button - display only the page content, without the menu and header.
- Time period selector - select the time period for which to display traces.
- Filter - display only the traces you are interested in.
- Trace list - view the traces, described by their root spans.
- Trace panel - view all spans of the selected trace on a timeline.
Trace list
Each row in the trace list represents a trace; the columns show the data of its root span.
Traces are displayed in a customizable table.
The 
Customize table button in the table header lets you show, hide, or reorder columns, and reset the layout to default.
You can also resize each column by dragging its right border; double-click the border to reset.
All customizations are applied immediately and remembered for the current user.
The trace list can have the following columns:
Each column corresponds to a column in the otel_traces table of the telemetry data source; for the list of columns, see Supported telemetry data columns.
Most columns have the same name as in the data source (e.g., Trace ID corresponds to the TraceId column).
Where the names differ, the data source column is specified in the table below.
| Column | Description |
|---|---|
| Trace ID | ID of the trace, shared by all spans that belong to the trace. |
| Span ID | ID of the root span. |
| Trace state | Additional data, if any, that tracing tools attach to the trace (see W3C Trace Context). |
| Service name | Service that produced the root span, followed by the number of spans in the trace. If any spans have the Error status code (see below), a red icon shows the number of them. |
| Operation name | Operation that the root span represents (e.g., POST /zabbix.php).Data source column: SpanName. |
| Span kind | Role of the root span: Unspecified - not specified. Internal - operation performed within an application. Server - handling of an incoming request. Client - outgoing request. Producer - creation of an asynchronous operation (e.g., sending a message). Consumer - processing of an asynchronous operation. |
| Scope name | Instrumentation scope that produced the root span. |
| Scope version | Version of the instrumentation scope. |
| Status code | Status of the operation that the root span represents: Unset - no explicit status has been assigned to the root span. Ok - the operation was completed successfully. Error - the operation was completed with an error. |
| Status message | Additional information, if any, about the status. |
| Start time | Date and time when the root span started. Data source column: Timestamp. |
| Span attributes | Span attributes of the root span. Click the  three-dot button to display all attributes. |
| Resource attributes | Resource attributes of the root span. Click the three-dot button to display all attributes. |
| Duration | Duration of the trace; the gray bar shows it relative to the other traces. |
Trace panel
Click a trace in the trace list to open the trace panel, which shows all spans of the trace on a timeline: which operations were performed, in what order, and how long each took.
The trace panel can have the following elements:
| Element | Description |
|---|---|
| Trace ID | ID of the selected trace. |
| Trace timeline overview | The whole trace, with spans in up to 8 rows (several spans can share a row). Drag the borders of the selected range, or the range itself, to zoom the span timeline to that part of the trace. |
| Span tree | Spans in their hierarchy; each shows its service (in bold), operation name, and number of direct child spans. |
| Span timeline | Spans as bars: the position and length show when each span started and how long it took. Markers show events (e.g., exceptions); hover over or click a marker for details. |
| Span details | Click a span in the span tree to see its details and attributes. |
Using filter
You can use the filter to display only the traces you are interested in.
To filter by attributes, enter the attribute key, select the operator, and enter the attribute value.
The following operators are supported:
- Contains - (default) the attribute value contains the entered string.
- Equals - the attribute value equals the entered string.
- Does not contain - the attribute value does not contain the entered string.
- Does not equal - the attribute value does not equal the entered string.
- Exists - the attribute with the entered key exists.
- Does not exist - the attribute with the entered key does not exist.
If you add several conditions, select how to evaluate them:
- And/Or - (default) all conditions must be met; conditions with the same attribute key are combined with Or.
- Or - at least one condition must be met.