[#registry-observability-otel]
= Configuring {registry} observability with OpenTelemetry

[role="_abstract"]
You can configure {registry} to export telemetry data using OpenTelemetry (OTel) for comprehensive observability.
This includes distributed tracing, metrics export via OTLP protocol, and log correlation with trace context.

OpenTelemetry support is disabled by default.
When enabled, {registry} exports telemetry data to an OpenTelemetry-compatible collector such as Jaeger, Grafana Tempo, or the OpenTelemetry Collector.

.Prerequisites
* You have already installed {registry} using the {operator}.
* You have an OpenTelemetry-compatible backend available (for example, Jaeger, Grafana Tempo, or OpenTelemetry Collector).

.Procedure
. Configure the `ApicurioRegistry3` CR with the `spec.app.otel` section:
+
[source,yaml]
----
include::ROOT:example$apicurioregistry3_otel_cr.yaml[]
----

. Apply the CR to your cluster:
+
[source,shell]
----
kubectl apply -f apicurioregistry3_otel_cr.yaml
----

.OpenTelemetry configuration options
[%header,cols="3,2,2,4"]
|===
| Configuration option | Type | Default | Description

| `spec.app.otel.enabled`
| boolean
| `false`
| Enable OpenTelemetry for distributed tracing, metrics export via OTLP, and log correlation.

| `spec.app.otel.endpoint`
| string
| _none_
| The endpoint URL of the OpenTelemetry collector. Supports both gRPC (typically port 4317) and HTTP (typically port 4318) protocols.

| `spec.app.otel.protocol`
| string
| `grpc`
| The protocol to use for exporting telemetry data. Valid values are `grpc` or `http/protobuf`.

| `spec.app.otel.traceSamplingRatio`
| number
| _none_ (samples all)
| The sampling ratio for traces (0.0 to 1.0). A value of `0.1` means 10% of traces are sampled. When set, uses ratio-based sampling. Recommended for production deployments.
|===

.OpenTelemetry features
When OpenTelemetry is enabled, {registry} provides the following observability features:

* *Distributed tracing*: All REST API requests are automatically traced with spans containing request details, path parameters, and Apicurio-specific attributes such as `groupId`, `artifactId`, and `version`.

* *Storage layer tracing*: All storage operations create child spans, enabling you to trace the complete request flow from REST API to database.

* *Kafka tracing*: When using KafkaSQL storage, Kafka operations are automatically traced with context propagation.

* *Custom metrics*: OpenTelemetry metrics for artifact operations, schema validations, and search requests are exported alongside existing Prometheus metrics.

* *Log correlation*: When JSON logging is enabled, trace context is automatically injected into log entries for easy correlation.

.Performance considerations
OpenTelemetry instrumentation adds a small performance overhead. With all telemetry signals enabled at 100% sampling, the measured impact is:

* *Latency increase*: 1-9% depending on operation type (approximately 0.05-0.35ms per request)
* *Throughput decrease*: 1-7% depending on operation type
* *Average overhead*: approximately 4-6% with 100% sampling

With the recommended 10% sampling ratio (`spec.app.otel.traceSamplingRatio: 0.1`), the overhead is reduced to less than 1%. OpenTelemetry is disabled by default to avoid any overhead for users who do not require observability features.

To reproduce these benchmarks, see the `OpenTelemetryPerformanceTest` and `OpenTelemetryPerformanceEnabledTest` classes in the `app` module, or run:

[source,shell]
----
./mvnw test -pl app -Dtest=OpenTelemetryPerformanceTest -DOpenTelemetryPerformanceTest=enabled
./mvnw test -pl app -Dtest=OpenTelemetryPerformanceEnabledTest -DOpenTelemetryPerformanceTest=enabled
----

.Backwards compatibility
OpenTelemetry support is fully backwards compatible:

* The existing Prometheus metrics endpoint (`/metrics`) remains available and unchanged.
* Health check endpoints (`/health/*`) continue to work as before.
* All existing Micrometer-based metrics continue to function.

[role="_additional-resources"]
.Additional resources
* link:https://opentelemetry.io/docs/[OpenTelemetry Documentation]
* link:https://quarkus.io/guides/opentelemetry[Quarkus OpenTelemetry Guide]
* link:https://www.jaegertracing.io/docs/[Jaeger Documentation]
