# Debezium CDC with OpenTelemetry Distributed Tracing

This example demonstrates end-to-end distributed tracing with Debezium Change Data Capture (CDC), showing how traces flow from an application writing to a database, through Debezium capturing changes, to schema registration in Apicurio Registry, and finally to a consumer processing CDC events.

## Why This Matters

In microservices architectures using CDC, debugging issues across services is challenging because:
- Database writes and CDC events are typically disconnected in traces
- Schema registry interactions appear as separate, unrelated requests
- There's no single view showing the complete data flow

This example provides **full observability** of the CDC pipeline, making it easy to:
- Debug latency issues across the entire flow
- Identify which component is causing delays
- Correlate application requests with their downstream CDC effects

## Key Feature: Complete Trace Propagation

This example solves a common challenge with Debezium and OpenTelemetry: ensuring that **all HTTP calls** made during CDC processing (including schema registration to Apicurio Registry) appear in the same distributed trace as the original database write.

This is achieved through a custom OpenTelemetry SMT (`OtelActivateTracingSpan`) that keeps the trace context active during format conversion, allowing the OTEL Java Agent to automatically instrument HTTP calls to the schema registry.

See the related Debezium issue: https://github.com/debezium/dbz/issues/1557

## Architecture

```
                                                        ┌─────────────────┐
                                                        │    Apicurio     │
                                                   ┌───▶│    Registry     │◀───┐
                                                   │    └─────────────────┘    │
                                                   │                           │
┌─────────────────┐     ┌─────────────────┐     ┌──┴──────────────┐            │
│  Order Service  │────▶│   PostgreSQL    │────▶│    Debezium     │            │
│  (Quarkus)      │     │   (CDC source)  │     │    Server       │            │
└────────┬────────┘     └─────────────────┘     └────────┬────────┘            │
         │                                               │                     │
         │              ┌─────────────────┐              │                     │
         │              │     Kafka       │◀─────────────┘                     │
         │              │   (Strimzi)     │                                    │
         │              └────────┬────────┘                                    │
         │                       │                                             │
         │              ┌────────▼────────┐                                    │
         │              │  CDC Consumer   │────────────────────────────────────┘
         │              │   (Quarkus)     │
         │              └────────┬────────┘
         │                       │
         └───────────────────────┘
                 │
                 ▼ (all services send traces)
         ┌───────────────┐     ┌─────────────────┐
         │  OpenTelemetry │────▶│     Jaeger      │
         │   Collector    │     │  (Visualization)│
         └───────────────┘     └─────────────────┘
```

All services (Order Service, Debezium Server, Apicurio Registry, CDC Consumer) are configured to send traces to the OpenTelemetry Collector, which exports them to Jaeger for visualization.

## Components

- **Order Service**: Quarkus application that creates/updates orders in PostgreSQL with OpenTelemetry tracing
- **PostgreSQL (orders)**: Source database with CDC enabled (Debezium's postgres image with logical replication)
- **Debezium Server**: Standalone Debezium deployment with OpenTelemetry Java Agent and custom OTEL SMT
- **Kafka**: Message broker for CDC events (Strimzi image)
- **Apicurio Registry**: Schema registry for Avro schemas with OpenTelemetry enabled
- **CDC Consumer**: Quarkus application that consumes CDC events from Kafka
- **OpenTelemetry Collector**: Collects and exports traces to Jaeger
- **Jaeger**: Distributed tracing visualization

## Trace Flow

1. **Order Service** creates an order in PostgreSQL, storing the W3C trace context in a `tracingspancontext` column
2. **Debezium Server** captures the database change via the custom `OtelActivateTracingSpan` SMT which:
   - Extracts the trace context from the record
   - Activates an OpenTelemetry span that remains active during format conversion
3. **Avro format converter** serializes the event and registers the schema with Apicurio Registry (HTTP call is traced)
4. **CDC Consumer** receives the event, deserializes using schema from Registry, and processes it
5. All spans are collected and visible in **Jaeger** under the same trace ID

## Prerequisites

- Docker 20.10+ and Docker Compose v2.0+
- Java 21+ and Maven 3.8+
- ~4GB available RAM for all containers

## Quick Start

### 1. Build the applications

```bash
cd examples/debezium-otel-tracing

# Build all modules (order-service, cdc-consumer, debezium-converter)
mvn clean package -DskipTests
```

### 2. Start the infrastructure

```bash
docker compose up -d --build
```

Wait for all services to be healthy (typically 2-3 minutes on first run):

```bash
docker compose ps
```

All services should show "healthy" or "running" status before proceeding.

### 3. Verify setup

```bash
./scripts/setup.sh
```

### 4. Access the services

- **Jaeger UI**: http://localhost:16686
- **Order Service**: http://localhost:8084
- **CDC Consumer**: http://localhost:8085
- **Debezium Server**: http://localhost:8083
- **Apicurio Registry**: http://localhost:8086

### 5. Create an order and observe the trace

```bash
# Create an order
curl -X POST http://localhost:8084/orders \
  -H "Content-Type: application/json" \
  -d '{
    "customerName": "Alice Smith",
    "product": "Widget Pro",
    "quantity": 2,
    "price": 49.99
  }'
```

Example response:
```json
{
  "id": 1,
  "status": "created",
  "traceId": "a1b2c3d4e5f67890a1b2c3d4e5f67890",
  "message": "Order created and will be captured by Debezium CDC"
}
```

```bash
# Wait for CDC propagation
sleep 3

# Check CDC events received
curl http://localhost:8085/cdc/events
```

### 6. View traces in Jaeger

Open http://localhost:16686 and:
1. Select service: `order-service`, `debezium-server`, `apicurio-registry`, or `cdc-consumer`
2. Click "Find Traces"
3. Observe the trace spanning from order creation through CDC to consumer, **including schema registration**

### 7. Stop the environment

```bash
docker compose down -v
```

> **Note**: The `-v` flag removes volumes, which cleans up the PostgreSQL replication slot. This is recommended when restarting the example to avoid slot conflicts.

## Expected Trace Structure

A successful order creation should produce a trace with approximately 20+ spans across all services:

```
order-service: POST /orders
├── create-order-entity (database insert)
│
└── debezium-server: debezium-cdc-orders (custom OTEL SMT)
    ├── POST apicurio-registry/apis/registry/v2/... (schema registration)
    │   └── apicurio-registry: createArtifact
    ├── Kafka producer send
    │
    └── cdc-consumer: process-cdc-event
        └── GET apicurio-registry/apis/registry/v3/... (schema lookup)
            └── apicurio-registry: getContentByGlobalId
```

The key indicator of success is seeing the `POST` to Apicurio Registry (schema registration) appearing as a **child span** under `debezium-server`, not as a separate disconnected trace.

## Testing Scenarios

### Basic Order Flow

```bash
# Create order
curl -X POST http://localhost:8084/orders \
  -H "Content-Type: application/json" \
  -d '{"customerName":"Test","product":"Widget","quantity":1,"price":9.99}'

# Check CDC consumer stats
curl http://localhost:8085/cdc/stats
```

Example stats response:
```json
{
  "totalReceived": 1,
  "totalProcessed": 0,
  "queueSize": 1,
  "consumerRunning": true
}
```

```bash
# Get CDC events
curl http://localhost:8085/cdc/events
```

### Order Status Update (UPDATE CDC event)

```bash
# Update order status (assuming order ID 1)
curl -X PUT "http://localhost:8084/orders/1/status?status=SHIPPED"
```

### Batch Orders

```bash
# Create 5 orders
curl -X POST "http://localhost:8084/orders/batch?count=5"

# Get batch of CDC events
curl "http://localhost:8085/cdc/events/batch?count=5"
```

## Verification

Run the verification script:

```bash
./scripts/verify-tracing.sh
```

This checks:
- Service health (Order Service, CDC Consumer, Debezium Server)
- Order creation with trace ID
- CDC event propagation
- Trace correlation in events
- Jaeger accessibility

## What to Look for in Jaeger

### Services
- `order-service`: HTTP requests and database operations
- `debezium-server`: CDC capture, custom OTEL SMT spans, and schema registration HTTP calls
- `apicurio-registry`: Schema storage operations
- `cdc-consumer`: Kafka consumer and event processing spans

### Key Observation

In the trace, you should see HTTP POST/GET calls to `apicurio-registry` appearing as child spans under the `debezium-server` service. This confirms that schema registration is properly traced as part of the CDC flow.

### Span Attributes
- `order.id`, `order.product`, `order.quantity`
- `cdc.operation` (CREATE, UPDATE, DELETE)
- `kafka.topic`, `kafka.partition`, `kafka.offset`
- `http.method`, `http.url` (for registry calls)

## Custom OTEL SMT

The `debezium-converter` module contains `OtelActivateTracingSpan`, a custom Single Message Transform that:

1. Extracts W3C trace context from the `tracingspancontext` field in CDC records
2. Creates an OpenTelemetry span as a child of the extracted trace
3. **Keeps the span scope active** (unlike Debezium's built-in SMT) so that downstream HTTP calls inherit the context

Configuration in `debezium-server/application.properties`:
```properties
debezium.transforms=oteltracing,unwrap
debezium.transforms.oteltracing.type=io.apicurio.registry.examples.debezium.converter.OtelActivateTracingSpan
debezium.transforms.oteltracing.tracing.operation.name=debezium-cdc-orders
debezium.transforms.oteltracing.tracing.span.context.field=tracingspancontext
```

## API Reference

### Order Service Endpoints

| Method | Path | Description |
|--------|------|-------------|
| POST | `/orders` | Create a new order |
| GET | `/orders` | List all orders |
| GET | `/orders/{id}` | Get order by ID |
| PUT | `/orders/{id}/status?status=X` | Update order status |
| POST | `/orders/batch?count=N` | Create batch of orders |
| GET | `/orders/health` | Health check |
| GET | `/orders/trace-info` | Current trace context |

### CDC Consumer Endpoints

| Method | Path | Description |
|--------|------|-------------|
| GET | `/cdc/events` | Get next CDC event |
| GET | `/cdc/events/batch?count=N` | Get batch of CDC events |
| GET | `/cdc/stats` | Consumer statistics |
| GET | `/cdc/health` | Health check |
| GET | `/cdc/trace-info` | Current trace context |

## Troubleshooting

### No CDC events appearing

1. Check Debezium Server health:
   ```bash
   curl http://localhost:8083/q/health
   ```

2. Check Debezium Server logs:
   ```bash
   docker compose logs debezium-server
   ```

3. Verify PostgreSQL replication:
   ```bash
   docker compose exec postgres-orders psql -U orderuser -d orders -c "SELECT * FROM pg_replication_slots;"
   ```

### Traces not appearing in Jaeger

1. Check OTel Collector logs:
   ```bash
   docker compose logs otel-collector
   ```

2. Verify services are sending traces:
   ```bash
   docker compose logs order-service | grep -i trace
   ```

### Schema registration not in trace

1. Verify OTEL Java Agent is loaded:
   ```bash
   docker compose logs debezium-server | grep "opentelemetry-javaagent"
   ```

2. Check that the custom SMT is being used:
   ```bash
   docker compose logs debezium-server | grep "OtelActivateTracingSpan"
   ```

### Restarting the example

If restarting after a previous run, clean up volumes to avoid replication slot conflicts:

```bash
docker compose down -v  # -v removes volumes including replication slot state
```

If you see errors about the replication slot already existing, manually drop it:

```bash
docker compose exec postgres-orders psql -U orderuser -d orders -c "SELECT pg_drop_replication_slot('orders_slot');"
```

## Related Documentation

- [Debezium Distributed Tracing](https://debezium.io/documentation/reference/stable/integrations/tracing.html)
- [OpenTelemetry Documentation](https://opentelemetry.io/docs/)
- [Apicurio Registry Documentation](https://www.apicur.io/registry/docs/)
- [Debezium Issue: OTEL span context not active during format conversion](https://github.com/debezium/dbz/issues/1557)
