What happens when your telemetry backend goes down?

Your application is healthy. Requests are completing, logs are being emitted, and the Collector is running. Then your observability backend becomes unavailable.

For a while, nothing obvious changes. The Collector keeps receiving telemetry. Your dashboards stop moving. When the backend recovers, you discover a gap covering exactly the period you wanted to investigate.

Where did the data go?

The resilience lessons in our OTel Track work through three layers: in-memory queues, persistent queues, and external messaging. To choose between them, start with the failure you need to survive. A backend outage and a Collector restart ask different things of your pipeline.

Follow the failed export backward

Consider this path:

1. Application Emits telemetry
2. Receiver Collector ingress
3. Processors Collector processing
4. Exporter queue Buffers pending exports
5. Backend Receives exports

When the backend stops accepting data, exporter workers encounter failures. For retryable errors, they wait and try again. Meanwhile, new telemetry fills the sending queue. A short outage can end before either capacity or retries run out. A longer one exposes those limits. The Collector resiliency guide describes both failure boundaries.

There are two separate questions here: how long should a worker keep trying to export a request, and how much additional data can wait behind it?

Retry time cannot answer the capacity question. A queue cannot make a permanent export error retryable.

Retries buy opportunities; queues buy capacity

For the OTLP/gRPC exporter in Collector Contrib 0.163.1, you can make the policy explicit:

exporters:
  otlp_grpc/backend:
    endpoint: backend:4317
    tls:
      insecure: true
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 5s
      max_elapsed_time: 10m
    sending_queue:
      enabled: true
      sizer: requests
      queue_size: 5000
      num_consumers: 4

otlp_grpc is the current gRPC exporter name; otlp remains a deprecated exporter alias. The OTLP receiver is still named otlp, including when it accepts HTTP. The tagged exporter metadata identifies the current name and alias.

The plaintext connection is for a local lab. Use your destination’s TLS and authentication configuration in production.

Here, capacity is measured in requests, not spans or bytes. The queue limit covers queued requests and requests still being processed, including retrying exports; workers do not provide extra capacity outside it. max_elapsed_time limits a request’s retry sequence; it does not set a retention period for the whole queue. Setting it to 0 removes that deadline, but permanent errors still terminate retries. See the version-pinned exporter helper configuration and retry implementation.

Suppose you measure 20 requests per second arriving at an initially empty exporter queue. During a complete outage, 5,000 request slots give roughly 250 seconds of buffering. This is a planning estimate: request sizes, occupied slots, in-flight work, and traffic bursts affect the result.

Now ask about recovery. If the backend accepts 30 requests per second while 20 new requests per second keep arriving, only 10 requests per second go toward the backlog. A backlog of 5,000 requests would take about 500 seconds to drain under those assumptions.

An outage budget needs a recovery budget too. If export throughput merely matches ingestion, the backlog stays with you.

Backpressure has to reach something that can retry

With the queue’s default nonblocking overflow behavior, a full queue rejects new enqueue attempts. Those records never reach the exporter’s retry loop. Whether they are recovered depends on the components upstream. Trace the error path all the way back to the source instead of assuming that every rejected record will be retried.

Memory pressure introduces another boundary. The memory_limiter processor can refuse incoming data with a non-permanent error, allowing compatible upstream components to retry. Put it first in the processor list, and size the process with headroom. It helps protect the Collector from exhausting memory; it does not provide storage. The memory limiter documentation explains these responsibilities.

A Collector restart is a different failure

An in-memory queue can bridge a backend outage while the Collector remains alive. Kill the Collector, and the queued data has no durable home.

An exporter queue backed by file_storage addresses that restart boundary. The extension must be enabled under service.extensions and referenced by sending_queue.storage. Its directory needs writable storage that survives replacement of the Collector process or container. See the file storage extension.

That protection starts when data reaches the persistent queue. A separate batch processor can still hold data earlier in the pipeline. Its size threshold and timeout control flushing, rather than disk persistence. See the batch processor documentation.

This distinction makes a useful test: send an identifiable record while the backend is down, confirm it reaches the exporter queue, kill the Collector, and then look for that same record after recovery.

Try the outage yourself

This lab uses two Collectors. One is the gateway under test; the other stands in for the backend and prints received logs. It uses Contrib 0.163.1, the latest release checked on 18 October 2026, pinned by its multi-platform image digest for reproducibility. Both Collectors use the same artifact. The 0.163.1 distribution manifest bundles 0.163.0 component modules, so the source citations use those module tags. This is a fresh-storage experiment, not a migration test for an existing queue.

You need Docker with Compose v2 and curl. Save these three files in an empty directory. In gateway.yaml, deliberately omit a separate batch processor so this experiment isolates the exporter queue:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    create_directory: true
    fsync: true

exporters:
  otlp_grpc/backend:
    endpoint: backend:4317
    tls:
      insecure: true
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 5s
      max_elapsed_time: 0
    sending_queue:
      enabled: true
      sizer: requests
      queue_size: 100
      num_consumers: 1
      storage: file_storage

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp_grpc/backend]

Use backend.yaml to make delivery visible:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug]

Finally, save compose.yaml:

name: collector-outage-lab
services:
  gateway:
    image: otel/opentelemetry-collector-contrib:0.163.1@sha256:1175b03003053962012f5a231f5567f66a3e8fdb960c7835c0e48f7aa1555806
    user: "0:0"
    mem_limit: 512m
    environment:
      GOMEMLIMIT: 200MiB
    ports:
      - "127.0.0.1:24318:4318"
    volumes:
      - ./gateway.yaml:/etc/otelcol-contrib/config.yaml:ro,z
      - queue-data:/var/lib/otelcol
  backend:
    image: otel/opentelemetry-collector-contrib:0.163.1@sha256:1175b03003053962012f5a231f5567f66a3e8fdb960c7835c0e48f7aa1555806
    mem_limit: 512m
    volumes:
      - ./backend.yaml:/etc/otelcol-contrib/config.yaml:ro,z
volumes:
  queue-data:

Exporter queue batching is opt-in in this release; this lab leaves sending_queue.batch unset and enables no batching feature gate. That keeps the request-sized queue experiment focused.

The gateway runs as root only to simplify named-volume permissions in this local lab. In production, give the Collector’s service user ownership of its storage directory. The named volume is what preserves the queue when we recreate the container.

Validate and start the configuration:

docker compose run --rm --no-deps gateway validate --config /etc/otelcol-contrib/config.yaml
docker compose run --rm --no-deps backend validate --config /etc/otelcol-contrib/config.yaml
docker compose up -d
docker compose stop backend

Send one record with a distinctive body. Wait for the gateway’s startup message if the request initially cannot connect:

curl --fail-with-body --retry 15 --retry-connrefused --retry-delay 1 --retry-max-time 30 \
  http://localhost:24318/v1/logs \
  -H 'Content-Type: application/json' \
  --data '{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"outage-lab"}}]},"scopeLogs":[{"scope":{"name":"outage-lab"},"logRecords":[{"severityNumber":9,"severityText":"INFO","body":{"stringValue":"survive-restart-001"}}]}]}]}'

Inspect the gateway logs for export retry messages. A successful response from the gateway alone does not prove backend delivery.

docker compose logs gateway

Once the gateway is retrying, kill it and recreate its container while the backend remains stopped:

docker compose kill -s SIGKILL gateway
docker compose up -d --no-deps --force-recreate gateway
docker compose start backend
docker compose logs -f backend

Look for survive-restart-001 in the backend’s output. Its arrival demonstrates recovery of that record across both the backend outage and the gateway process failure. It does not establish an exactly-once guarantee or prove survival of a disk failure.

For a comparison, remove storage: file_storage from sending_queue, clean up the lab, and repeat with a different marker. Killing the gateway then removes the in-memory buffer before the backend can receive it.

Running these experiments on Linux/arm64 with the pinned 0.163.1 image produced the following results:

Queue	Record sent while backend was stopped	Result after gateway crash and backend recovery
Persistent	survive-restart-001	Appeared in the backend’s debug output
In memory	memory-only-001	Did not appear; a fresh recovery-control-001 record did arrive

The control record checks that the second experiment’s missing marker was not simply caused by a backend that remained unavailable. Separate checks with the same binary confirmed that a one-request in-memory queue rejects a second request with HTTP 503 while the first is retrying, and that an expired two-second retry budget drops its record before recovery. Persistence does not remove either boundary.

After each experiment, stop the log follower with Ctrl+C and clean up:

docker compose down -v

The -v deletes this lab’s queue volume. Wait until you have finished inspecting the results.

Watch recovery as closely as the outage

A running process is only one piece of evidence. Track exporter queue occupancy and capacity, enqueue failures, receiver refusals, and successful exports. For logs, useful metric names include otelcol_exporter_queue_size, otelcol_exporter_queue_capacity, otelcol_exporter_enqueue_failed_log_records, and otelcol_exporter_sent_log_records. Export format and version can affect their rendered names; check the internal telemetry documentation.

Pair those measurements with an identifiable record at the destination. A shrinking queue is encouraging, but you still need to establish whether data was delivered or discarded. Keep a monitoring path available during the backend outage you are testing.

When local persistence is not enough

If the failure budget includes losing a gateway’s storage, or retaining a substantial backlog independently of Collector instances, an external broker such as Kafka may fit the requirements. That introduces its own retention, replication, capacity, and operational decisions.

Also inspect when the consuming Collector marks messages as processed. Acceptance into a downstream asynchronous buffer is a different milestone from successful backend delivery. The Kafka receiver configuration exposes message-marking and commit controls; test the complete path rather than treating Kafka as a blanket delivery guarantee.

Before sizing a production pipeline, write down the outage it must tolerate: expected arrival rate, maximum downtime, whether the Collector or its disk may disappear, acceptable loss, and required catch-up time. Then run that failure against the design.

These are the questions we explore in the queues, WAL, messaging, and monitoring lessons of the OTel Track. The most useful result is a measured answer to: when this backend fails, which telemetry survives, and how do we know?