# Distributing Zero-Code Instrumentation

> Source: https://www.ymotongpoo.com/books/observability-platform-with-otel/20-zero_code_instrumentation/


Even with an SDK distribution in place, some services cannot call `Setup`. Examples are legacy services that are hard to maintain, and third-party binaries whose source code you cannot change. Others belong to teams that cannot start work on instrumentation. The platform must be able to produce minimum telemetry from these services too, through work on the platform side.

## The guaranteed minimum of instrumentation

OpenTelemetry calls the methods that instrument code without changing it [**zero-code instrumentation**](https://opentelemetry.io/docs/zero-code/). On a self-service platform, zero-code instrumentation does not replace manual instrumentation. It serves as the **guaranteed minimum of instrumentation**. It records the inputs and outputs of HTTP and gRPC as traces and metrics, with no work from development teams. Services that need business-specific information add manual instrumentation on top.

When minimum traces already exist, development teams can judge concretely which business information is missing. When nothing is recorded, they cannot even judge which instrumentation to add first.

![Comparison of instrumentation injection points](20260926-injection-points.png)
*Figure 1: The vertical axis shows the places where you can insert instrumentation, from the source code down to the kernel. Solid lines show the relationships on the path to execution, and dotted lines show where each instrumentation method intervenes. The places that you can change determine which methods you can choose.*

## Options by language

The runtime characteristics of each language determine the method of zero-code instrumentation. The following table summarizes the options for the major languages.

| Language | Method | Notes |
|---|---|---|
| Java | Bytecode rewriting by the javaagent | Supports many libraries |
| Python | Run-time patching with the opentelemetry-instrument command | A mechanism to replace the distro also exists |
| Node.js | Run-time patching through require hooks | Consolidated in auto-instrumentations-node |
| .NET | Injection through the CLR profiler API | Official zero-code instrumentation is available |
| Go | eBPF (at run time) or code rewriting at build time | The injection method differs from the one in languages with a dynamic runtime |

In languages with a dynamic runtime, such as Java and Python, you can replace code at run time. A statically compiled single binary in Go has no mechanism to load an agent at run time. So Go uses one of two methods. The eBPF method observes the program from the kernel side. The compile-time method adds instrumentation code at compile time.

## eBPF-based zero-code instrumentation for Go

The eBPF instrumentation for Go lives in the [**OBI** (OpenTelemetry eBPF Instrumentation)](https://opentelemetry.io/docs/zero-code/obi/) project. The project started when Grafana Labs donated Beyla, and the latest version as of September 2026 is v0.13.0. It is still v0, so breaking changes can happen[^obi]. OBI uses eBPF to observe system calls and function calls from outside the target process, so it does not change the binary. It can capture HTTP, gRPC, the protocols of major databases and messaging systems, and calls to GenAI APIs such as OpenAI and Anthropic.

[^obi]: v0.11.0, released on August 17, 2026, added zero-code instrumentation of Go's trace API. OBI is taking over the area that opentelemetry-go-instrumentation used to cover. The project's goal for 2026 is a stable 1.0 ([official blog](https://opentelemetry.io/blog/2026/obi-goals/)).

eBPF instrumentation of Go binaries at the function level has constraints, though. Goroutines and OS threads do not map one to one, which makes it hard to track the context of a request. Compiler optimizations change the structure of functions. Context propagation also requires writes to the memory of a running process, and those writes carry risk. My book *[Behind the Scenes of OpenTelemetry eBPF Instrumentation](https://zenn.dev/ymotongpoo/books/go-ebpf-primer)* (in Japanese) explains the details, starting from how the CPU and memory work. This book assumes that eBPF instrumentation for Go has limits in the granularity and the stability of the information that it collects. The distribution methods in this chapter build on that assumption.

## Compile-time instrumentation

In Go, you can also build instrumentation code in at compile time. [**otelc**](https://opentelemetry.io/docs/zero-code/go/compile-time/) is the official tool for this, developed by merging Datadog's Orchestrion and Alibaba's instrumentation tool. It reached stable with v1.0 in July 2026. The latest version as of September 2026 is v1.1.0[^otelc].

[^otelc]: The [v1 announcement on the official blog](https://opentelemetry.io/blog/2026/go-compile-time-instrumentation-v1/) describes the background. The maintainers retracted v1.0.0 because of a bug, so use v1.0.1 or later.

You only replace the build command, and the instrumentation goes in.

```console
$ go get -tool go.opentelemetry.io/otelc/tool/cmd/otelc
$ otelc go build ./...
```

The `otelc pin` command fixes the instrumentation configuration in the repository. As of September 2026, though, it does not support committing the generated artifacts[^otelcpin]. For CI and team development, use the commands above, which generate the configuration on every build.

[^otelcpin]: The instrumentation packages use pseudo-versions, and they resolve only inside the otelc executable. So when the generated artifacts are committed, the build fails with `package ... is not part of a module`. The work to separate them is in progress in [issue #585](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/issues/585).

`otelc` uses Go's `-toolexec` mechanism. During compilation, it inserts instrumentation code into the calls to target libraries. As of September 2026, it supports major libraries such as `net/http`, `database/sql`, gRPC, Redis, and Kafka, and it produces traces and metrics. It also supports injecting the trace context into logs.

In Go, you can choose eBPF, build-time instrumentation, or manual instrumentation. The following table compares what each one changes and how they differ in operation.

| Aspect | eBPF (OBI) | Build time (otelc) | Manual instrumentation (distribution) |
|---|---|---|---|
| Code changes | Not needed | Not needed | Needed |
| Build changes | Not needed | Replace the build command | Add a dependency |
| Privileges at run time | Needed | Not needed | Not needed |
| Coverage | Inputs and outputs of supported protocols | Calls to supported libraries | Anything that you can write in code |
| Available attributes | Information that the protocol exposes | Up to the arguments of library calls | Anything, including business context |
| Avoiding duplication | Exclusion settings for target processes | Remove the service from the build targets | Manual instrumentation is the baseline |
| Unit of instrumentation updates | Agent update on the host | Rebuild and redeploy of the service | Distribution update and redeploy |

The default method depends on where the platform can make changes. For an organization that has a shared CI template and runs mainly on Kubernetes, this book recommends otelc for zero-code instrumentation of Go. It rolls out through a change to the build command, it needs no privileges, and it puts instrumentation updates into normal deploys. OpenTelemetry does not define this order of preference. This is a decision based on this book's assumptions.

If you cannot rebuild the binary and cannot change the CI either, choose the eBPF method. When you instrument several languages together per host, OBI is also a candidate, because it works regardless of language.

## Distributing on Kubernetes

On Kubernetes, the [OpenTelemetry Operator](https://opentelemetry.io/docs/platforms/kubernetes/operator/automatic/) provides the `Instrumentation` custom resource. Once the platform team sets one up in the cluster, a development team can enable zero-code instrumentation by adding one annotation to a Pod.

```yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default-instrumentation
spec:
  exporter:
    endpoint: http://otel-agent.observability:4317
  propagators:
    - tracecontext
    - baggage
```

The development team adds the following annotation to the Pod.

```yaml
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
```

The Operator's Admission Webhook detects the creation of the Pod and injects the agent and the environment variables for the language. The configuration uses `OTEL_*` environment variables, so the SDK distribution and zero-code instrumentation share the same configuration method.

![How the Operator injects zero-code instrumentation](20260926-operator-injection.png)
*Figure 2: The arrows show the time order of processing. The development team specifies the annotation, and the Operator's Admission Webhook injects instrumentation at Pod creation.*

Injection for Go is off by default. Pass the `--enable-go-instrumentation=true` flag to the Operator. Also add an annotation that points to the path of the target executable.

```yaml
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-go: "true"
    instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/app/server"
```

The injected agent uses eBPF, so it runs as a privileged container, and it does not support multi-container Pods. As of September 2026, the Operator injects the older opentelemetry-go-instrumentation, not OBI. Its latest release is v0.24.0 from April 27, 2026, and only dependency updates have landed since then. Because of these constraints, consider otelc first in environments where you can change the shared CI.

## Distributing outside Kubernetes

Where you cannot use the Operator, choose the injection method by which part of the build and deploy you can change.

- Bundle the agent in the base image. This fits methods such as Java's javaagent, where you place a file and enable it with an environment variable
- Build it into the CI template. Go's otelc uses this method. It replaces the build command, so it rolls out easily in organizations with a shared CI
- Run an eBPF agent per host. OBI runs as a daemon on the host and observes every process on that host

Whichever method you use, set the destination to the agent Collector in each environment (Chapter 4). If zero-code instrumentation alone sends directly to the backend, the transformations and controls in the Collector do not apply to its telemetry.

## Designing stages toward manual instrumentation

Zero-code instrumentation records the inputs and outputs of HTTP. It does not tell you which internal step took time, or which customer the request concerned. This business-specific information needs manual instrumentation. Here, the platform team does not provide the manual instrumentation itself. It provides a state in which teams can choose the stage. First, zero-code instrumentation records minimum information. Then a team that needs more can add manual instrumentation at its own discretion. The central team does not decide how much to record. Development teams choose that, and this choice is the core of self-service.

When you combine zero-code and manual instrumentation, the instrumentation can overlap. For example, zero-code instrumentation and the distribution's otelhttp can both create a span for the same HTTP request. The same operation then appears twice. In this book, manual instrumentation is the baseline, and services that have SDK instrumentation leave the scope of zero-code instrumentation. With the Operator, remove the annotation. With per-host OBI, exclude the process in the selection criteria for targets.

The SDK distribution distributes the organization's defaults to code that teams can change. Zero-code instrumentation distributes minimum instrumentation to services whose code teams cannot change. Which one to use depends on where you can make changes, rather than on which one to adopt first.