Introduction

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/00-introduction.

This book accompanies my Go Conference 2026 talk “Behind the Scenes of OpenTelemetry eBPF Instrumentation” (in Japanese). The 40-minute talk assumes an advanced audience. The book starts with the computer fundamentals needed to follow the same material.

eBPF (a mechanism for inserting small monitoring programs into the Linux kernel without modifying the application; the details come later in the book) can capture distributed traces of HTTP and gRPC without changing application source code. This is known as zero-code instrumentation. Product descriptions promise that it requires neither a rebuild nor a redeployment and works regardless of language.

Apply a standard return hook to a Go binary, and the observed program can crash with this error.

fatal error: unknown caller pc

The observer has crashed its target. A technique used with other languages fails on Go before collecting a trace.

This book groups these failures into four hurdles. These hurdles come from the very design that makes Go fast and easy to write. Standard eBPF instrumentation assumes that the OS manages a fixed stack and that functions use the platform’s calling convention, as C and Java threads do. Go’s movable goroutine stacks, custom calling convention, and runtime-managed scheduling and memory violate those assumptions.

This book uses OpenTelemetry eBPF Instrumentation (OBI) as its case study. OBI grew out of Grafana Beyla after Grafana donated Beyla to the OpenTelemetry project. Go is the only language for which OBI instruments individual functions; other languages use a generic path that interprets network bytes as protocols. Chapter 8 explains the difference. Each hurdle includes code you can run and the corresponding OBI implementation. All execution results were measured with go1.26.5 linux/amd64, and the OBI code comes from release v0.11.0.

Distributed traces and traceparent

Instrumentation means adding observation points to a program to record processing times and call relationships. In this book, binary means the executable file produced by compilation, not an arbitrary sequence of 0s and 1s.

A distributed trace is a record that lets you see one operation spanning multiple services as a single flow. When service A calls service B, giving the work in both services the same trace ID lets the backend (the server that collects and displays the records) join the two records after receiving them separately. Each segment is called a span, and B’s span records A’s span ID as its parent.

The HTTP header that carries this identification is traceparent. Its format is defined by W3C Trace Context and looks like 00-<trace ID>-<span ID>-01. With SDK-based instrumentation, the application attaches this header itself. Attaching it from the outside, without changing the application’s code, is what this book calls zero-code instrumentation.

Figure 1: Solid arrows show the flow of requests; dashed arrows show records being sent to the backend. Because the trace IDs match, the two spans can be displayed as one trace, and B’s span holds A’s span ID as its parent.

There are two paths to achieving this with eBPF. Instrumentation that watches communication sees byte sequences like GET /items flowing through sockets, recognizes them as HTTP, and measures elapsed time by matching requests to responses. Instrumentation that watches functions reads the arguments of specific net/http functions at the moment they are called. The former is language-agnostic; the latter needs an implementation per language. All four hurdles in this book arise in the latter.

How to read this book

Chapters 2 through 6 explain how computers and operating systems run programs and what the Go runtime manages. They establish where the stack lives, where arguments are placed, and how a tool finds a function’s address. Once you understand this, you’ll see why each of the four hurdles is genuinely hard.

The background chapters connect to the hurdles as follows.

Chapters 7 and 8 introduce eBPF and OBI. Chapters 9 through 12 cover the four hurdles. Chapter 13 examines the consequences for Go developers, Chapter 14 covers developments in upstream Go, and the appendix contains a glossary.

The order of the background chapters does not match the timeline along which your source code becomes a running program. You can always check where in that flow the current discussion sits with the following figure.

Figure 2: Arrows show the flow of time and transformation. Below each stage is the chapter that covers it.

The only prerequisite is that you can read Go code. No knowledge of operating systems, assembly, eBPF, or distributed tracing is required. I explain the necessary concepts as they come up.

The sample code links to the Go Playground. Its isolated environment makes some results differ from a local run: process IDs stay constant, and repeated runs can return a cached result. To watch addresses change, edit at least one character between runs or run the code locally. The disassembly and build-setting sections require a local build.

Readers comfortable with Go internals and Linux’s execution model can skip Chapters 2 through 6 and start from Chapter 7, “What Is eBPF?” The glossary points back to the chapter that introduces each term.

The hurdle chapters quote OBI’s implementation code, but the surrounding explanation stands on its own. Long excerpts are collapsed, and each hurdle ends with a summary.