Source: https://telemetrydrops.com/blog/collector-backend-outage/

How Collector retries, queues, and persistent storage handle backend outages—and what survives a Collector restart.

 

1.  [Home](https://telemetrydrops.com/) 
2.  / [Blog](https://telemetrydrops.com/blog/) 
3.  / What happens when your telemetry backend goes down? 

[Collector](https://telemetrydrops.com/blog/tags/collector/)[Architecture](https://telemetrydrops.com/blog/tags/architecture/) 9 min read

# What happens when your telemetry backend goes down?

![Juraci Paixão Kröhling](https://telemetrydrops.com/authors/juraci.webp)

[Juraci Paixão Kröhling](https://telemetrydrops.com/instructors/juraci-paixao-krohling/)

October 19, 2026

![A finite reservoir of telemetry blocks beside an interrupted connection to a backend.](https://telemetrydrops.com/images/blog/collector-backend-outage.webp) 

TL;DR

A sending queue buys time during a backend outage. Persistent storage lets queued telemetry survive a Collector restart. Neither removes capacity limits or guarantees delivery: test the failure boundaries before you need them.

Your application is healthy. Requests are completing, logs are being emitted, and the Collector is running. Then your observability backend becomes unavailable.

For a while, nothing obvious changes. The Collector keeps receiving telemetry. Your dashboards stop moving. When the backend recovers, you discover a gap covering exactly the period you wanted to investigate.

Where did the data go?

The resilience lessons in our OTel Track work through three layers: in-memory queues, persistent queues, and external messaging. To choose between them, start with the failure you need to survive. A backend outage and a Collector restart ask different things of your pipeline.

## Follow the failed export backward

Consider this path:

1.  **Application** Emits telemetry
2.  **Receiver** Collector ingress
3.  **Processors** Collector processing
4.  **Exporter queue** Buffers pending exports
5.  **Backend** Receives exports

The exporter queue holds pending telemetry. Retrying exports consume capacity while the backend is unavailable.

When the backend stops accepting data, exporter workers encounter failures. For retryable errors, they wait and try again. Meanwhile, new telemetry fills the sending queue. A short outage can end before either capacity or retries run out. A longer one exposes those limits. The [Collector resiliency guide](https://opentelemetry.io/docs/collector/resiliency/) describes both failure boundaries.

There are two separate questions here: how long should a worker keep trying to export a request, and how much additional data can wait behind it?

Retry time cannot answer the capacity question. A queue cannot make a permanent export error retryable.

## Retries buy opportunities; queues buy capacity

For the OTLP/gRPC exporter in Collector Contrib 0.163.1, you can make the policy explicit:

```yaml
exporters:
  otlp_grpc/backend:
    endpoint: backend:4317
    tls:
      insecure: true
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 5s
      max_elapsed_time: 10m
    sending_queue:
      enabled: true
      sizer: requests
      queue_size: 5000
      num_consumers: 4
```

`otlp_grpc` is the current gRPC exporter name; `otlp` remains a deprecated exporter alias. The OTLP receiver is still named `otlp`, including when it accepts HTTP. The [tagged exporter metadata](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.163.0/exporter/otlpexporter/metadata.yaml) identifies the current name and alias.

The plaintext connection is for a local lab. Use your destination’s TLS and authentication configuration in production.

Here, capacity is measured in requests, not spans or bytes. The [queue limit](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.163.0/exporter/exporterhelper/internal/queuebatch/config.go) covers queued requests and requests still being processed, including retrying exports; workers do not provide extra capacity outside it. `max_elapsed_time` limits a request’s retry sequence; it does not set a retention period for the whole queue. Setting it to `0` removes that deadline, but permanent errors still terminate retries. See the version-pinned [exporter helper configuration](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.163.0/exporter/exporterhelper/README.md) and [retry implementation](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.163.0/exporter/exporterhelper/internal/retry_sender.go).

Suppose you measure 20 requests per second arriving at an initially empty exporter queue. During a complete outage, 5,000 request slots give roughly 250 seconds of buffering. This is a planning estimate: request sizes, occupied slots, in-flight work, and traffic bursts affect the result.

Now ask about recovery. If the backend accepts 30 requests per second while 20 new requests per second keep arriving, only 10 requests per second go toward the backlog. A backlog of 5,000 requests would take about 500 seconds to drain under those assumptions.

An outage budget needs a recovery budget too. If export throughput merely matches ingestion, the backlog stays with you.

## Backpressure has to reach something that can retry

With the queue’s default nonblocking overflow behavior, a full queue rejects new enqueue attempts. Those records never reach the exporter’s retry loop. Whether they are recovered depends on the components upstream. Trace the error path all the way back to the source instead of assuming that every rejected record will be retried.

Memory pressure introduces another boundary. The `memory_limiter` processor can refuse incoming data with a non-permanent error, allowing compatible upstream components to retry. Put it first in the processor list, and size the process with headroom. It helps protect the Collector from exhausting memory; it does not provide storage. The [memory limiter documentation](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.163.0/processor/memorylimiterprocessor/README.md) explains these responsibilities.

## A Collector restart is a different failure

An in-memory queue can bridge a backend outage while the Collector remains alive. Kill the Collector, and the queued data has no durable home.

An exporter queue backed by `file_storage` addresses that restart boundary. The extension must be enabled under `service.extensions` and referenced by `sending_queue.storage`. Its directory needs writable storage that survives replacement of the Collector process or container. See the [file storage extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.163.0/extension/storage/filestorage/README.md).

That protection starts when data reaches the persistent queue. A separate batch processor can still hold data earlier in the pipeline. Its size threshold and timeout control flushing, rather than disk persistence. See the [batch processor documentation](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.163.0/processor/batchprocessor/README.md).

This distinction makes a useful test: send an identifiable record while the backend is down, confirm it reaches the exporter queue, kill the Collector, and then look for that same record after recovery.

## Try the outage yourself

This lab uses two Collectors. One is the gateway under test; the other stands in for the backend and prints received logs. It uses Contrib **0.163.1**, the latest release checked on 18 October 2026, pinned by its multi-platform image digest for reproducibility. Both Collectors use the same artifact. The [0.163.1 distribution manifest](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.1/distributions/otelcol-contrib/manifest.yaml) bundles 0.163.0 component modules, so the source citations use those module tags. This is a fresh-storage experiment, not a migration test for an existing queue.

You need Docker with Compose v2 and curl. Save these three files in an empty directory. In `gateway.yaml`, deliberately omit a separate batch processor so this experiment isolates the exporter queue:

```yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    create_directory: true
    fsync: true

exporters:
  otlp_grpc/backend:
    endpoint: backend:4317
    tls:
      insecure: true
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 5s
      max_elapsed_time: 0
    sending_queue:
      enabled: true
      sizer: requests
      queue_size: 100
      num_consumers: 1
      storage: file_storage

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp_grpc/backend]
```

Use `backend.yaml` to make delivery visible:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug]
```

Finally, save `compose.yaml`:

```yaml
name: collector-outage-lab
services:
  gateway:
    image: otel/opentelemetry-collector-contrib:0.163.1@sha256:1175b03003053962012f5a231f5567f66a3e8fdb960c7835c0e48f7aa1555806
    user: "0:0"
    mem_limit: 512m
    environment:
      GOMEMLIMIT: 200MiB
    ports:
      - "127.0.0.1:24318:4318"
    volumes:
      - ./gateway.yaml:/etc/otelcol-contrib/config.yaml:ro,z
      - queue-data:/var/lib/otelcol
  backend:
    image: otel/opentelemetry-collector-contrib:0.163.1@sha256:1175b03003053962012f5a231f5567f66a3e8fdb960c7835c0e48f7aa1555806
    mem_limit: 512m
    volumes:
      - ./backend.yaml:/etc/otelcol-contrib/config.yaml:ro,z
volumes:
  queue-data:
```

Exporter queue batching is [opt-in in this release](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.163.0/exporter/exporterhelper/internal/queuebatch/config.go); this lab leaves `sending_queue.batch` unset and enables no batching feature gate. That keeps the request-sized queue experiment focused.

The gateway runs as root only to simplify named-volume permissions in this local lab. In production, give the Collector’s service user ownership of its storage directory. The named volume is what preserves the queue when we recreate the container.

Validate and start the configuration:

```bash
docker compose run --rm --no-deps gateway validate --config /etc/otelcol-contrib/config.yaml
docker compose run --rm --no-deps backend validate --config /etc/otelcol-contrib/config.yaml
docker compose up -d
docker compose stop backend
```

Send one record with a distinctive body. Wait for the gateway’s startup message if the request initially cannot connect:

```bash
curl --fail-with-body --retry 15 --retry-connrefused --retry-delay 1 --retry-max-time 30 \
  http://localhost:24318/v1/logs \
  -H 'Content-Type: application/json' \
  --data '{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"outage-lab"}}]},"scopeLogs":[{"scope":{"name":"outage-lab"},"logRecords":[{"severityNumber":9,"severityText":"INFO","body":{"stringValue":"survive-restart-001"}}]}]}]}'
```

Inspect the gateway logs for export retry messages. A successful response from the gateway alone does not prove backend delivery.

```bash
docker compose logs gateway
```

Once the gateway is retrying, kill it and recreate its container while the backend remains stopped:

```bash
docker compose kill -s SIGKILL gateway
docker compose up -d --no-deps --force-recreate gateway
docker compose start backend
docker compose logs -f backend
```

Look for `survive-restart-001` in the backend’s output. Its arrival demonstrates recovery of that record across both the backend outage and the gateway process failure. It does not establish an exactly-once guarantee or prove survival of a disk failure.

For a comparison, remove `storage: file_storage` from `sending_queue`, clean up the lab, and repeat with a different marker. Killing the gateway then removes the in-memory buffer before the backend can receive it.

Running these experiments on Linux/arm64 with the pinned 0.163.1 image produced the following results:

| Queue | Record sent while backend was stopped | Result after gateway crash and backend recovery |
| --- | --- | --- |
| Persistent | `survive-restart-001` | Appeared in the backend’s debug output |
| In memory | `memory-only-001` | Did not appear; a fresh `recovery-control-001` record did arrive |

The control record checks that the second experiment’s missing marker was not simply caused by a backend that remained unavailable. Separate checks with the same binary confirmed that a one-request in-memory queue rejects a second request with HTTP 503 while the first is retrying, and that an expired two-second retry budget drops its record before recovery. Persistence does not remove either boundary.

After each experiment, stop the log follower with Ctrl+C and clean up:

```bash
docker compose down -v
```

The `-v` deletes this lab’s queue volume. Wait until you have finished inspecting the results.

## Watch recovery as closely as the outage

A running process is only one piece of evidence. Track exporter queue occupancy and capacity, enqueue failures, receiver refusals, and successful exports. For logs, useful metric names include `otelcol_exporter_queue_size`, `otelcol_exporter_queue_capacity`, `otelcol_exporter_enqueue_failed_log_records`, and `otelcol_exporter_sent_log_records`. Export format and version can affect their rendered names; check the [internal telemetry documentation](https://opentelemetry.io/docs/collector/internal-telemetry/).

Pair those measurements with an identifiable record at the destination. A shrinking queue is encouraging, but you still need to establish whether data was delivered or discarded. Keep a monitoring path available during the backend outage you are testing.

## When local persistence is not enough

If the failure budget includes losing a gateway’s storage, or retaining a substantial backlog independently of Collector instances, an external broker such as Kafka may fit the requirements. That introduces its own retention, replication, capacity, and operational decisions.

Also inspect when the consuming Collector marks messages as processed. Acceptance into a downstream asynchronous buffer is a different milestone from successful backend delivery. The [Kafka receiver configuration](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.163.0/receiver/kafkareceiver/README.md) exposes message-marking and commit controls; test the complete path rather than treating Kafka as a blanket delivery guarantee.

Before sizing a production pipeline, write down the outage it must tolerate: expected arrival rate, maximum downtime, whether the Collector or its disk may disappear, acceptable loss, and required catch-up time. Then run that failure against the design.

These are the questions we explore in the queues, WAL, messaging, and monitoring lessons of the [OTel Track](https://telemetrydrops.com/products/otel-track/). The most useful result is a measured answer to: **when this backend fails, which telemetry survives, and how do we know?**

## Key takeaways

*   • Retries handle temporary export failures; queue capacity determines how much incoming telemetry can wait.
*   • A persistent exporter queue protects data that reaches it, provided the storage survives the restart.
*   • Recovery requires export capacity above the ongoing ingestion rate, or the backlog cannot drain.
*   • Test backend outages and Collector restarts separately, and verify individual records at the destination.

## Frequently asked questions

### Does the OpenTelemetry Collector lose data when a backend is unavailable?

It can buffer data and retry temporary failures, but data can still be lost through queue overflow, exhausted retries, permanent export errors, or failures elsewhere in the pipeline. The outcome depends on the configured exporter and upstream retry behavior.

### Does file\_storage make the entire Collector pipeline persistent?

No. When referenced by sending\_queue.storage, it persists that exporter's queue. Data still buffered in an SDK or an earlier processor is outside that protection, and the storage volume must survive the Collector restart.

### Does max\_elapsed\_time set the maximum age of queued telemetry?

No. It limits a request's export retry sequence, rather than acting as a queue-wide retention policy. Setting it to zero removes that retry deadline, but does not remove queue capacity limits or make permanent errors retryable.

![Juraci Paixão Kröhling](https://telemetrydrops.com/authors/juraci.webp)

[Juraci Paixão Kröhling](https://telemetrydrops.com/instructors/juraci-paixao-krohling/)

OpenTelemetry Maintainer & Instructor

OpenTelemetry maintainer and founder of Telemetry Drops, where he teaches observability engineering through hands-on courses.

[LinkedIn](https://www.linkedin.com/in/jpkroehling) [GitHub](https://github.com/jpkrohling)

Telemetry Drops offers hands-on OpenTelemetry courses for engineering teams. [Learn more](https://telemetrydrops.com/products/).

[Collector](https://telemetrydrops.com/blog/tags/collector/)[Architecture](https://telemetrydrops.com/blog/tags/architecture/)