Yoshi Yamaguchi

What Go Developers Should Know About Being Observed

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/60-for_go_developers.

Go developers encounter OBI as the observer attached to their application. The Go version, build flags, and dependency versions determine how much it can instrument.

Built with Go 1.17 or later

OBI’s support matrix states Go 1.17+ for library-level instrumentation. The dividing line is the version that switched to ABIInternal, the internal convention by which Go functions pass arguments. Every standard-library entry in offsets.json also lists 1.17.0 as its oldest.

The register ABI from Hurdle 2 was introduced in Go 1.17. Supporting older stack-based binaries would require a separate argument-reading path. Go 1.17 was released in August 2021, so few current deployments need that path.

What happens with a stripped binary

Adding -ldflags="-s -w" shrinks a distributed binary by removing some sections described in Chapter 4. The following commands show what survives in a small program with one net/http handler.

$ go build -o app main.go
$ go build -ldflags="-s -w" -o app_stripped main.go

$ ls -l app app_stripped | awk '{print $5, $9}'
5444719 app
3748002 app_stripped

$ readelf -S app         | grep -c debug_    # number of DWARF sections
8
$ readelf -S app_stripped | grep -c debug_
0

$ go tool nm app_stripped
reading app_stripped: no symbol section

$ readelf -S app_stripped | grep -o gopclntab
gopclntab

DWARF (the debug information containing types and field positions) and the symbol table are gone. The .gopclntab table mapping instruction addresses to Go function names survives because the Go runtime needs it to assemble stack traces on panic. The build info blob also remains, so go version -m keeps working.

Function addresses can still be located. go tool nm and go tool objdump refuse to run without a symbol table, but OBI doesn’t depend on them. pkg/internal/goexec/instructions.go parses .gopclntab on its own. Since runtime.moduledata has neither a symbol nor a fixed address, a comment describes the search as follows: scan the binary for an 8-byte-aligned value that points to .gopclntab, then check whether the surrounding data matches the layout of moduledata. This design lets OBI determine function positions even without symbols.

Stripping DWARF changes offset resolution. OBI first tries to read field offsets from the binary’s own DWARF; stripping DWARF with -w leaves nothing to read, so it must resolve every entry through offsets.json instead. It moves from reading the ground truth written in the binary to hoping a pre-built table covers its version. A library version absent from that table leaves the corresponding instrumentation entry silently unresolved.

Keeping DWARF makes instrumentation more reliable. Whether to add -w is a trade-off between distribution size and instrumentation reliability.

How build settings relate to instrumentation Figure 1: The arrows show which part of instrumentation is affected by what remains in the binary (from cause to effect). Adding -s -w drops DWARF and the symbol table. .gopclntab survives, so function addresses can still be located, but offset resolution falls back entirely on offsets.json.

Instrumentation points that vanish through inlining

As we saw in Chapter 5, an inlined function loses its call instruction entirely, so there’s nowhere to place a uprobe. The smaller the accessor or wrapper, the more invisible it becomes to instrumentation.

OBI chooses large functions such as net/http.serverHandler.ServeHTTP and google.golang.org/grpc.(*Server).handleStream, which are large enough that the compiler does not inline them. When writing an eBPF tool for another function, check the compiler’s decision with -gcflags=-m.

The versions of your libraries

As the flip side of Hurdle 3 (field offsets), the versions of your dependencies affect whether instrumentation works. If method names stopped showing up right after you upgraded gRPC, the problem may not be in your application: offsets.json may not have caught up yet.

When traces do not appear, investigate in this order.

  • Is the process being detected?
  • If it is, are the function addresses being resolved?
  • If they are, are the field values being read plausible?

Wrong values at the last step indicate a likely gap in offset tracking.