# Designing the Collector Layer and Building a Custom Collector

> Source: https://www.ymotongpoo.com/books/observability-platform-with-otel/30-collector_build/


The SDK distribution and zero-code instrumentation both send telemetry to the Collector in each environment. If applications sent telemetry directly to the backend, you would not need to run a Collector. I still put a relay layer in between. The reason is to separate decisions about the delivery path from the application.

When applications send directly, decisions about the backend enter the application side. To change the backend, you must change the configuration of every service and redeploy it, and you must distribute the authentication token to every service. If you want to remove an attribute before storage, you have to ask each team for a change. The **Collector layer** gathers these decisions about the delivery path in one place.

## The role of the Collector layer

The [**OpenTelemetry Collector**](https://opentelemetry.io/docs/collector/) is a pipeline that receives, processes, and sends telemetry. You declare the pipeline in YAML: **receivers** take telemetry in, **processors** process it, and exporters send it out. The platform gives the Collector the following roles.

- Abstracting the destination: the application only sends OTLP to the Collector, and does not know where the backend is. You can change the backend, or use several backends at once, with a change to the Collector configuration alone
- Processing telemetry: the Collector applies additions and removals of attributes, transformations, filters, and sampling uniformly, without touching application code
- Isolating credentials: only the Collector holds the API keys for the backend, and the platform does not distribute them to applications
- Smoothing delivery: the Collector takes over batching, retries, and backpressure from the application

Smoothing delivery does not guarantee that no data is lost. By default, the Collector's queue lives in memory. If the process or the node stops, the contents of the queue are lost. A persistent queue lets the Collector resend data after a restart, but it needs a disk. Whether the Collector drops data or applies backpressure when it exceeds capacity also depends on the configuration. With a Collector in the path, you can design the conditions and the extent of data loss outside the application.

## Deployment patterns

The Collector has three main deployment patterns.

In the **agent pattern**, the Collector runs on the same host or node as the application. On Kubernetes, you deploy it as a DaemonSet, and it gives the application a nearby destination. It also attaches host metadata, such as the node name and Pod information, as resource attributes.

In the **gateway pattern**, you place a cluster of Collectors as an aggregation layer for each organization or cluster. You deploy it as a Deployment, scale it horizontally, and make it the single exit to the backend.

In the **sidecar pattern**, a Collector runs alongside each Pod. You can use it to isolate tenants from each other. Because the number of Collectors grows with the number of Pods, adopt it only when agents or gateways cannot give you the isolation that you need.

For an environment where several teams run services on Kubernetes and the organization manages the destinations, this book recommends a **two-tier configuration of agents and gateways**. The OpenTelemetry specification does not define this as a standard configuration. This is a decision based on this book's assumptions, as described in "The environment this book assumes" in Chapter 1. In a small environment with a single team, gateways alone are enough, and if you do not need node metadata, you can omit the agents. In the two-tier configuration, the application sends to the agent on its node, the agent forwards to the gateway, and the gateway sends to the backend.

Put organizational policy in the gateway. The rules for removing personal information and sensitive information change, and so do the destinations. A change to the agent configuration, on the other hand, affects every node. Limit the agent to attaching host metadata and receiving from applications, and use the same configuration on every node. The gateway handles sampling, attribute governance, routing, and authentication to the backend.

The agent runs on the same node as the application, so a node failure affects both. The two-tier configuration does not remove the conditions for data loss. Design the queue and the retries for each tier, and decide how much data each kind of failure can lose.

![Two-tier topology of agents and gateways](20260926-agent-gateway.png)
*Figure 1: The arrows represent the flow of telemetry. The agents on the two nodes have the same role, and the platform distributes the same configuration to every node. Organizational policy is concentrated in the gateway. Add the dashed load-balancing layer only when you scale tail sampling horizontally.*

## Designing the processing in the gateway layer

The gateway layer holds the processing that the organization decides on as a whole.

**Tail sampling** decides whether to keep a trace after the trace is complete, based on whether it has errors and on its latency. For example, you can implement a policy that keeps every trace with an error and 1% of normal traces. The decision needs all spans that make up a trace, so tail sampling goes in the gateway.

If you scale the gateway horizontally, you need the Load Balancing exporter in front of it. This exporter sends spans with the same trace ID to the same gateway instance. In practice, a two-tier configuration with tail sampling therefore includes a load-balancing layer. Also, tail sampling reduces the volume that you store, but not the volume that flows to the gateway. Control that flow together with head sampling on the SDK side.

For **attribute governance**, use the Transform processor. While the telemetry is in transit, it applies redaction and renames deprecated attributes. Redaction removes personally identifiable information (PII), such as email addresses, and sensitive information, such as tokens.

Renaming an attribute is not a standalone string replacement, though. The SDK attaches the version of the conventions that it follows to the telemetry as a **schema URL**. Backends and conversion tools use this declaration to convert between old and new attribute names. If you rename only the attribute and leave the old schema URL, a conversion can run twice, or a required conversion can fail to apply. Design the input schema URL, the conversion to apply, and the output schema URL as one set. Chapter 6 explains this relationship from the side of the conventions.

**Routing** reads resource attributes and changes the destination for each tenant or environment. For example, you can send telemetry from the development environment to low-cost storage, and send data from a specific team to a dedicated tenant.

For **load control**, use the Memory Limiter processor and the Batch processor. Place them at the start and the end of the pipeline to keep a sudden surge in volume from stopping the Collector. Place these two processors in the agent as well.

## Why a custom build

The Collector has official distributions such as core and contrib[^otel-distro], and contrib bundles the components from the community. This book uses a custom build that contains only the components that you choose.

[^otel-distro]: As of September 2026, there are [five distributions](https://opentelemetry.io/docs/collector/distributions/).

contrib contains more than 100 components, and a single organization does not use most of them. Unused components are still in scope when you check for vulnerabilities, and a configuration mistake can open a receiver that you did not intend to open. A binary that contains only the parts you need has a smaller attack surface. It also serves as the list of components that the organization approves for use.

You also need a custom build to add an extension that works with your internal authentication system, or a receiver for a proprietary protocol.

## Building with OCB

The official tool for custom builds is **OCB** (OpenTelemetry Collector Builder)[^ocb]. You list the parts that you use in `manifest.yaml`, and OCB generates Go code and builds it.

[^ocb]: The official documentation is at [Building a custom Collector](https://opentelemetry.io/docs/collector/extend/ocb/). It moved from the earlier "custom-collector" URL, so watch out for old links.

```yaml
dist:
  module: github.com/example/otelcol-internal
  name: otelcol-internal
  description: OpenTelemetry Collector, internal standard build
  output_path: ./build

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.161.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.161.0
  - gomod: go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.161.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.161.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor v0.161.0

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.161.0

extensions:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.161.0
```

As of September 2026, a manifest needs care with versions and configuration fields. This example specifies v0.161.0, so build it with the matching OCB v0.161.0. The measurements in Chapter 9 use v0.159.0. To reproduce the conditions of those measurements, use the versions that the reference implementation pins.

The Collector core repository releases two versions at the same time, v1.67.0 and v0.161.0. The modules that have reached v1 are in the API layer: pdata, which handles the internal representation of telemetry, confmap, which loads configuration, and others. Components such as receivers and processors are v0 modules, so the manifest uses versions such as v0.161.0.

Note that the module version and the maturity of a component are separate measures. Each component declares its stability for each signal in its own `metadata.yaml`. For example, the OTLP receiver is part of a v0 module, but it declares stable for traces, metrics, and logs. When you decide whether to adopt a component, do not look at whether its module is v0. Look at the stability that the component declares for the signal that you use.

OCB removed the `otelcol_version` field in November 2024. Old articles still show examples that set it in the dist section, but you do not use it now. OCB checks version compatibility between components at build time.

The mechanism that expands configuration values from sources such as environment variables is called a **confmap provider**. You can omit the `providers` section of the manifest. If you omit it, OCB includes env, file, http, https, and yaml. The section "Distributing the configuration file" below covers how to write the expansions.

The official OCB Docker image lets you reproduce the same build environment in CI. With `--skip-compilation`, OCB runs only the code generation. With `--skip-generate --skip-get-modules`, it runs only the compilation. You can also commit the generated code to the repository and make it part of code review. For a setup that includes containerization, you can refer to the [opentelemetry-collector-releases](https://github.com/open-telemetry/opentelemetry-collector-releases) repository, which builds the official distributions.

![Build pipeline with OCB](20260926-ocb-pipeline.png)
*Figure 2: The arrows represent artifacts passing to the next step. Starting from the manifest, CI runs the build, the configuration validation, and the containerization.*

The Collector releases every two weeks, and contrib components also receive breaking changes. Use a tool such as Renovate to open PRs that update the versions in the manifest, and have CI build the Collector and validate the configuration. When the platform side takes over the updates, each team no longer has to keep up on its own.

## Distributing the configuration file

Along with the custom binary, the platform distributes a predefined config.yaml. Development teams do not write arbitrary Collector configuration. They specify only the fields that the platform allows.

To inject the differences between environments, use expansion through confmap providers.

```yaml
exporters:
  otlp:
    endpoint: ${env:GATEWAY_ENDPOINT}
    headers:
      authorization: ${env:GATEWAY_TOKEN}
```

`${env:VAR}` expands an environment variable, and `${file:PATH}` expands the contents of a file. Keep one configuration template for all environments, and move the differences between environments into environment variables and secret management. You then do not need a copy of the configuration file for each environment.

You can pass the `--config` flag to the Collector more than once, and the Collector merges each later configuration into the earlier ones. You can build a setup that layers a team's file over base.yaml, but this merge is not a permission boundary. The values that the Collector reads later take precedence. If the team's file redefines `service.pipelines.traces.processors`, it replaces the whole list, including the shared Memory Limiter and PII removal. Even if the processor definitions remain, the Collector does not run a processor that no pipeline refers to.

You can confirm this replacement by running it. Take a base.yaml that includes the required PII removal processor. Layer a team file over it that contains only `processors: [memory_limiter]`, and run `print-config`. The PII removal disappears from the pipeline in the effective configuration. On top of that, `otelcol validate` reports this configuration as valid (exit code 0). The configuration is valid as syntax, and the validation command knows nothing about organizational policy.

Therefore, check governance against the effective configuration. A CI job that looks only at whether `validate` passes lets through a configuration that has dropped a required processor.

To protect what you govern, restrict the fields that teams can write. In this book, a team can add only a named pipeline dedicated to that team (for example, `traces/team-checkout`), and the platform's CI merges the configuration. The CI uses `otelcol validate` and the output of the effective configuration. It checks two things: every pipeline has the required processors, and the exporter endpoints are on the allowlist. The target of the check is the configuration that the Collector actually uses, not the shape of the files.

If you let teams define their own pipelines, split Collectors by trust boundary. The team-managed Collector sends to a gateway that the platform manages, and PII removal and authentication run in that gateway, which teams cannot change. You run more Collectors, but you create the boundary with process isolation instead of configuration rules.

The platform distributes the Collector binary, the pipelines, and the update path, and exposes the allowed configuration fields to development teams. For a team that needs arbitrary configuration, provide an isolated pipeline or a separate Collector.