# Managing a Collector Fleet with OpAMP

> Source: https://www.ymotongpoo.com/books/observability-platform-with-otel/40-collector_fleet/


To change the tail sampling conditions in the gateway layer, you merge a PR into the configuration repository. **GitOps** then rolls the change out to a few gateways. GitOps is an operating method that treats the contents of a Git repository as the source of truth and applies that state to the real environment automatically.

Agents that run as a DaemonSet on hundreds of nodes are a different case. To change them, you update the ConfigMap and do a rolling restart of every Pod. This works while changes are few. When you add more filters, or turn on debug logs on a subset of nodes more often, a restart of every Pod for each configuration change becomes a heavy burden. Distributing configuration to many Collectors and managing their state is called **fleet management**.

## The challenges of fleet management

Fleet management needs the following capabilities.

- Distributing configuration: deliver a new configuration to hundreds of Collectors, and confirm that each one applied it
- Knowing the state: list which Collector runs with which configuration, and whether it is healthy
- Preparing for failure: when you distribute a broken configuration, keep the range where telemetry stops as small as possible, and roll back automatically

If you change configuration only by redistributing images and doing rolling restarts, every change restarts every Collector. The only way to know which Collectors received the new configuration is the log of the deployment tool. You cannot ask the Collectors themselves. If configuration changes become more frequent, separate the distribution of configuration from deployment.

## An overview of OpAMP

[**OpAMP** (Open Agent Management Protocol)](https://opentelemetry.io/docs/specs/opamp/) is a protocol for managing many agents remotely from a server. In this book, the managed agents are Collectors. The specification covers configuration distribution, health reporting, connection credential management, and package updates.

Each agent connects to the management server over WebSocket or HTTP, and the two sides exchange AgentToServer and ServerToAgent messages. The agent reports its identity, its effective configuration, and its health. The server returns configuration and instructions. The two sides negotiate the features that they use as capabilities when the agent connects, so you can use configuration distribution alone.

As of September 2026, the [OpAMP specification](https://github.com/open-telemetry/opamp-spec/blob/main/specification.md) is in Beta. Releases have reached v0.20.0, but they include breaking changes, and the specification has not reached 1.0. The reference implementation, [opamp-go](https://github.com/open-telemetry/opamp-go), is also at v0.24.0. On the other hand, the Supervisor ships as an official distributed artifact, and several vendor products implement OpAMP. Production use has started, but you adopt OpAMP on the assumption that you will keep up with changes to the specification.

![Fleet management with OpAMP](20260926-opamp-topology.png)
*Figure 1: Solid lines represent configuration distribution, and dotted lines represent reports from the agents. The source of truth for configuration lives in Git, and OpAMP handles the distribution from the management server to the agents.*

## OpAMP Supervisor

On the Collector side, there are two approaches. The opampextension approach builds the OpAMP client into the Collector itself. The other approach places a supervisor process outside the Collector. That process is the **OpAMP Supervisor**. The Supervisor approach also puts opampextension in the Collector. Its role changes, though: the Collector uses it to talk to the Supervisor, not to the OpAMP server.

The Supervisor keeps the connection to the OpAMP server and starts the Collector as a child process. It merges the configuration that it receives from the server with the local configuration, applies the result, and restarts the Collector when needed. Because the Supervisor runs outside the Collector, it can report the state and try to recover even when the Collector is unhealthy.

The configuration looks like this.

```yaml
server:
  endpoint: wss://opamp.internal.example.com/v1/opamp

capabilities:
  accepts_remote_config: true
  reports_effective_config: true
  reports_health: true

agent:
  executable: /usr/local/bin/otelcol-internal
  config_files:
    - /etc/otelcol/base.yaml

storage:
  directory: /var/lib/otelcol/supervisor
```

For `agent.executable`, specify the internal Collector that you built in Chapter 4. The OCB manifest must include opampextension, plus nopreceiver and nopexporter for the startup check. If you start a Collector without them under the Supervisor, the bootstrap fails. I confirmed this behavior with the reference implementation (Chapter 9).

The Supervisor handles the connection to the OpAMP server and configuration management. The Supervisor uses opampextension for local communication to get the state of the Collector. Even with a supervisor process outside, the Collector still needs the extension inside it.

In `config_files`, you can list local configuration and the remote configuration (`$REMOTE_CONFIG`). The configuration that the Supervisor reads later takes precedence. If you place the platform-managed configuration after the remote configuration, the required settings go on top last. The order is not a permission boundary, though. Before distribution, CI checks two things: the effective configuration includes the required processors, and the endpoints are allowed. After distribution, compare the configuration against the effective configuration that the Supervisor reports.

The Supervisor is developed in [cmd/opampsupervisor in contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor), and the project distributes official binaries and container images. As of September 2026, its stability is alpha. It implements these features: receiving and applying remote configuration, reporting the effective configuration and health, restart instructions, and connection credential updates.

On the other hand, package management, which updates the Collector binary, is not implemented. The OpAMP specification defines it, but you cannot distribute binaries through the Supervisor. Redistribute the binary as an image, and use OpAMP to distribute configuration.

## Operating remote configuration

When you introduce OpAMP, configuration lives in two places: the definition in Git, and the configuration that the OpAMP server distributes. If each place holds its own settings, the configuration in Git and the distributed configuration drift apart. So make Git the source of truth for configuration. A merge into the configuration repository triggers the OpAMP server to load the configuration and distribute it to the Collectors. OpAMP does not replace GitOps. It handles the distribution from the management server to the Collectors.

The OpAMP server can return a different configuration to each agent. You can therefore split agents into groups by attributes and apply a change in stages. For example, widen the target from one machine, to one cluster, to the whole fleet.

The Supervisor has `automatic_config_rollback`, which returns to the previous configuration when the Collector cannot start after a change. It also has `startup_fallback_configs`, which it uses when it cannot connect to the server at startup. Automatic rollback is disabled by default, so enable it explicitly.

Automatic rollback detects only one kind of failure: the Collector cannot start. Some configurations are syntactically correct, and the Collector starts with them, so automatic rollback does not detect them. Examples are a configuration that drops all data with filter, a wrong destination, a configuration that fails authentication, and a configuration that exceeds processing capacity. In the reference implementation (Chapter 9), I distributed a configuration that drops every span. The status stayed APPLIED and healthy, and only the telemetry stopped.

Even for startup failures, you cannot treat the alpha implementation as an operational guarantee. I tested the Supervisor at 0.159.0. Even with `automatic_config_rollback` enabled, it could not recover by itself. It persisted the configuration that had failed to start as the "last working configuration." Recovery depended on redistributing a fixed configuration. The server side also needed a control that does not resend a configuration for which an agent reported FAILED. Chapter 9 records the course of the experiment and the numbers.

Contain the damage from configuration distribution with these mechanisms: a staged rollout that starts with a canary, monitoring of the Collector's sent and received counts and the exporter failure rate, and checks that synthetic telemetry arrives. Also prepare the conditions for stopping a rollout, and the procedure for redistributing a fixed version. Even when automatic rollback works, it does not replace these checks and recovery procedures.

![State transitions in a staged rollout](20260926-staged-rollout.png)
*Figure 2: Solid lines represent state transitions when a check passes, and dotted lines represent transitions when an anomaly is detected. At every stage, you recover by redistributing a fixed version.*

## Choices for the OpAMP server

As of September 2026, there is no official standalone OpAMP server product. What opamp-go provides is a library for implementing a server and a sample server for demos. You implement the features that a production management plane needs yourself.

The reference implementation (Chapter 9) includes a minimal server that uses the server library of opamp-go. It is an implementation for learning, to observe the protocol. It is not a reference implementation for production.

A production management plane needs these features: authentication and identification of agents, tenant isolation, management of target groups, persistence of state, and audit of changes. It also needs redundancy and upgrades for the server itself, staged rollouts, and handling of conflicting updates. If you estimate the work from the configuration-sending part alone, you leave out these operational features.

In production, choose one of the following approaches based on how often the fleet changes and which features you need.

| Choice | When it fits |
|---|---|
| Keep using GitOps | Changes are rare, and applying them to the whole fleet at once is enough |
| Use a management product that supports OpAMP | You want per-agent delivery, staged rollouts, and fleet visibility quickly |
| Implement OpAMP in your existing internal management plane | You already have a configuration management platform, and you do not want a separate system only for Collectors |
| Build a dedicated management plane | The fleet is large, and control over distribution is an area where you compete |

## When GitOps alone is enough

While the fleet is small, GitOps alone, with ConfigMap updates and rolling restarts, can manage it. Base the decision on how often configuration changes and how wide each change is, not on the number of Collectors.

- If configuration changes a few times a month and applying changes to the whole fleet at once causes no problems, GitOps alone is probably enough
- If changes happen several times a week and you need to change only some nodes, consider OpAMP
- If you only want a list of health and effective configurations, you can start by installing the Supervisor in report-only mode (accepts_remote_config set to false)

In a setup that manages agents on hundreds of nodes, make Git the source of truth for configuration. Have CI check the effective configuration, then distribute it to each Collector with OpAMP. The canary and the monitoring stop the rollout of an abnormal configuration, and when a change fails, you redistribute a fixed version. OpAMP separates the path that applies reviewed and checked configuration to the fleet from the deployment of the Collectors.