Hurdle 3: Version-Dependent Field Offsets

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/50-hurdle3_offsets.

OBI identifies a g without reading its contents, but HTTP methods and gRPC targets live inside structs. Extracting them requires field reads.

The protocol-level instrumentation from Chapter 8 can read the method and path of plaintext HTTP/1.1 from leading bytes such as GET /items HTTP/1.1. Go instrumentation still needs structs for values that the captured byte stream does not expose.

One is TLS. Go includes its own statically linked crypto/tls, so OBI cannot use the Chapter 6 technique of placing uprobes on functions in the OpenSSL shared library. The socket only exposes bytes after encryption. For Go processes, OBI places uprobes on crypto/tls.(*Conn).Read and Write and reads the plaintext there, before encryption and after decryption. The other case is HTTP/2 and gRPC. HTTP/2 uses HPACK to compress header names and values, replacing them with indexes into a table built for each connection. From the second request onward, :path may be only an index, so an observer that did not see the connection from its first byte cannot recover the original string. OBI then falls back to * for the span name. In addition, OBI captures only the first 256 bytes of a request from the socket, and values such as routing templates never appear in the byte stream at all.

Go instrumentation reads structs before serialization, including Method and URL in http.Request and method in gRPC’s transport.Stream. An external tool must locate each field by byte position. This is the most troublesome of the four hurdles: like Hurdle 2, a mistake here produces incorrect trace contents instead of a crash.

What is a field offset?

As Chapter 2 showed, an external observer sees byte positions. The number of bytes between the start of a struct and a field is the field offset, the value returned by unsafe.Offsetof.

Go source accesses fields by names such as req.Method, but eBPF sees only bytes and addresses. The external reader needs a byte-level rule such as “Header starts 56 bytes after the start of http.Request.”

Checking your Go against OBI’s table

You can extract this offset from Go itself with unsafe.Offsetof.

• Run in the Go Playground

package main

import (
	"fmt"
	"net/http"
	"runtime"
	"unsafe"
)

func main() {
	var r http.Request
	fmt.Println(runtime.Version())
	fmt.Printf("Method        = %d\n", unsafe.Offsetof(r.Method))
	fmt.Printf("URL           = %d\n", unsafe.Offsetof(r.URL))
	fmt.Printf("Header        = %d\n", unsafe.Offsetof(r.Header))
	fmt.Printf("ContentLength = %d\n", unsafe.Offsetof(r.ContentLength))
}

go1.26.5
Method        = 0
URL           = 16
Header        = 56
ContentLength = 88

These four numbers match the contents of the offset table OBI carries, pkg/internal/goexec/offsets.json.

"net/http.Request": {
  "Method":        { "versions": {"oldest": "1.17.0", "newest": "1.26.4"},
                     "offsets": [{"offset": 0,  "since": "1.17.0"}] },
  "URL":           { "versions": {"oldest": "1.17.0", "newest": "1.26.4"},
                     "offsets": [{"offset": 16, "since": "1.17.0"}] },
  "Header":        { "versions": {"oldest": "1.17.0", "newest": "1.26.4"},
                     "offsets": [{"offset": 56, "since": "1.17.0"}] },
  "ContentLength": { "versions": {"oldest": "1.17.0", "newest": "1.26.4"},
                     "offsets": [{"offset": 88, "since": "1.17.0"}] }
}

versions says across which range of versions this entry has been verified; offsets says since when the field has been at that position.

The local Go output matches OBI’s precomputed table. Reading a struct from outside boils down to trusting this table and reading byte +56 for Header.

net/http.Request is stable in this range: each offsets array has one element because these fields retained their positions from Go 1.17 through 1.26.

The shift when one field is added

Adding one field in the middle of a struct shows how later offsets change.

• Run in the Go Playground

package main

import (
	"fmt"
	"unsafe"
)

type streamV1 struct {
	ctx    any
	id     uint32
	method string
}

type streamV2 struct { // one internal field added
	ctx    any
	cancel func()
	id     uint32
	method string
}

func main() {
	var v1 streamV1
	var v2 streamV2
	fmt.Printf("v1: id=%2d  method=%2d\n", unsafe.Offsetof(v1.id), unsafe.Offsetof(v1.method))
	fmt.Printf("v2: id=%2d  method=%2d\n", unsafe.Offsetof(v2.id), unsafe.Offsetof(v2.method))
}

v1: id=16  method=24
v2: id=24  method=32

Both fields after cancel shift by 8 bytes. The byte layout explains the values. An any occupies 16 bytes for its type and data pointers; a func() occupies one 8-byte pointer; and a string occupies 16 bytes for a pointer and length. The 4-byte uint32 leaves a gap before the following string, whose pointer requires an 8-byte boundary, so padding fills the gap.

Figure 1: The arrow represents the change of adding one field. The position and size of the padding change too, so the fields behind shift by exactly the same 8 bytes that were added.

Code that hardcoded 24 as the position of method reads, against v2, a meaningless position straddling id and the padding.

Offsets that change without notice

The streamV1/streamV2 example is not hypothetical. gRPC’s internal/transport.Stream, which OBI reads to extract the method name, shows the same shift in the wild. The offset table records four positions for method, reflecting three changes.

It is a type in the internal package internal/transport, so no compatibility promise applies. In Go, a package whose path contains internal cannot be imported from outside, and this type is not part of the library’s public API. Its developers are therefore free to rearrange the fields in any update. OBI bears the cost of that freedom because it reads memory positions from outside the process. In the same table, golang.org/x/net/http2.ClientConn.fr has moved even more: eight recorded ranges and seven position changes, including one move back to a previous position. Across all of offsets.json, OBI tracks 70 structs and 133 fields; 42 of those fields have moved at least once.

Figure 2: Arrows show which position the instrumentation code reads. A hardcoded +88 points at method in grpc 1.66 but at a different field in 1.69. The read can still produce a plausible value instead of crashing.

A hardcoded offset can work for one version and read the wrong location after a dependency upgrade. The tool still produces traces, masking the error behind plausible but incorrect contents. As an observability tool, this behavior does the most damage to trust in the system it observes.

Read DWARF first, fill the gaps from the table

OBI combines two sources of information. The selection begins in pkg/internal/goexec/structmembers.go.

func structMemberOffsets(elfFile *elf.File) (FieldOffsets, error) {
	// first, try to read offsets from DWARF debug info
	var offs FieldOffsets
	var expected map[GoOffset]struct{}
	dwarfData, err := elfFile.DWARF()
	if err == nil {
		offs, expected = structMemberOffsetsFromDwarf(dwarfData)
		if len(expected) > 0 {
			log().Debug("Fields not found in the DWARF file", "fields", expected)
		} else {
			libVersions, err := findLibraryVersions(elfFile)
			if err != nil {
				return nil, fmt.Errorf("searching for library versions: %w", err)
			}
			offs = offsetsForLibVersions(offs, libVersions.versions, log())
			setGoAutoSDKActivationSupport(offs, libVersions, elfFile)
			return offs, nil
		}
	} else {
		// initialize empty offsets
		offs = FieldOffsets{}
	}

	log().Debug("Can't read all offsets from DWARF info. Checking in prefetched database")

	// if it is not possible, query from prefetched offsets
	return structMemberPreFetchedOffsets(elfFile, offs)
}

The first thing it tries is DWARF, the debug information containing types and field positions, embedded in the target binary itself. That is the .debug_* sections listed in Chapter 4.

DWARF holds the right answer because the compiler writes down, for the debugger’s sake, “this field of this type is this many bytes from the start.” It is the same information that lets a debugger display req.Method by name. So as long as OBI can read DWARF, it gets values that match the very binary in front of it. There is no version-tracking problem.

The expected value is the set of fields that could not be read from DWARF. If it is empty, DWARF alone was sufficient and the function returns. If any field remains, the function calls structMemberPreFetchedOffsets and passes the offsets it already found. It does not discard values read from DWARF. The receiving function checks if _, found := fieldOffsets[constantName]; found { continue }, skipping fields that already have values and filling only the missing ones from the table. The test TestPrefetchedOffsetsPreserveResolvedOffsets verifies this behavior.

What fills the gaps is a table prepared in advance. That is offsets.json, embedded in the binary with //go:embed.

//go:embed offsets.json
var prefetchedOffsets string

A tool called go-offsets-tracker extracts and updates the table for releases of Go and major libraries. Maintaining zero-code instrumentation therefore includes tracking changes to their internal structures.

The mapping between the C and Go sides remains manual. The constant sequence in structmembers.go is preceded by this comment.

// this const table must match what's in go_offsets.h
type GoOffset uint32

const (
	// go common
	ConnFdPos GoOffset = iota + 1
	FdLaddrPos
	FdRaddrPos
	TCPAddrPortPtrPos
	TCPAddrIPPtrPos
	// http
	URLPtrPos
	PathPtrPos
	RawQueryPtrPos
	HostPtrPos
	SchemePtrPos
	MethodPtrPos
	StatusCodePtrPos
	ResponseLengthPtrPos
	ContentLengthPtrPos
	ReqHeaderPtrPos
	IoWriterBufPtrPos
	IoWriterNPos
	IoWriterWrPos
	// ...
)

If the iota ordering on the Go side differs from the eBPF C header, OBI reads another field. No compiler check or test verifies this correspondence. Comments on both sides tell maintainers to keep the sequences aligned; the C header says // start at 1, must match what's in structmembers.go.

The three constants at the tail of this excerpt, IoWriterBufPtrPos, IoWriterNPos, and IoWriterWrPos, correspond to buf, n, and wr of bufio.Writer. Offset tracking covers these unexported standard-library fields because Hurdle 4 uses them.

Working out the version from the binary

Before consulting the table, OBI parses the linker’s build info blob to determine the Go and dependency versions used for the binary.

// The build info blob left by the linker is identified by
// a 16-byte header, consisting of buildInfoMagic (14 bytes),
// the binary's pointer size (1 byte),
// and whether the binary is big endian (1 byte).
var buildInfoMagic = []byte("\xff Go buildinf:")

From there it extracts runtime.buildVersion and runtime.modinfo. This is the same data go version -m reads: the Go version together with the paths, versions, and hashes of the dependency modules.

$ go version -m ./myapp
./myapp: go1.26.5
	path	example.com/myapp
	mod	example.com/myapp	(devel)
	dep	golang.org/x/arch	v0.30.0	h1:sB9h+1gRGa2+LauFSV0tm8bK1J2yo1bx6/Uyi/P6DTU=
	build	-buildmode=exe
	build	-compiler=gc
	...

Once the dependency module versions are known, the correct row can be picked from the Stream.method table above. In some places OBI goes a step further and holds the module hashes themselves.

var goAutoSDKActivationModules = [...]activationModule{
	{
		path: "go.opentelemetry.io/auto/sdk",
		sums: map[string]string{
			"v1.1.0": "h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=",
			"v1.2.0": "h1:YpRtUFjvhSymycLS2T81lT6IGhcUP+LUPtv0iv1N8bM=",
			"v1.2.1": "h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=",
		},
	},
	// ...
}

These are the same hashes recorded in go.sum. OBI enables release-specific behavior only after confirming both the version string and the identity of the module contents.

Figure 3: Arrows show the processing flow, top to bottom. Fields readable from DWARF are used as-is; only the ones that were not readable are filled in from offsets.json. The two merge into a single offset table. Only the part filled from the table depends on the binary’s version.

Hurdle 3 has no fix that removes the fragility. OBI’s answer is a mechanism that keeps absorbing it: reading available field positions from DWARF and filling the gaps from an automatically updated table. This division of labor lets OBI keep up with changes in Go and its libraries.