# Hurdle 3: Version-Dependent Field Offsets > Source: https://www.ymotongpoo.com/books/go-ebpf-primer/50-hurdle3_offsets/ OBI identifies a `g` without reading its contents. Other values are not so easy. To get the method name of an HTTP request, or the target of a gRPC call, OBI reads inside a struct. The protocol instrumentation from Chapter 8 looks like it could get the same values from the byte sequence that passes through the socket. For plaintext HTTP/1.1 it does. OBI reads the method and the path from a leading byte sequence such as `GET /items HTTP/1.1`. Go instrumentation still reads structs, because some values never reach the byte sequence. One reason is TLS. Go carries its own `crypto/tls` and links it statically, so the Chapter 6 technique of placing uprobes on OpenSSL functions does not apply. Only the encrypted byte sequence reaches the socket. For a Go process, OBI places uprobes on `crypto/tls.(*Conn).Read` and `Write`, and reads the plaintext there: before encryption, and after decryption. The other reason is HTTP/2 and gRPC. HTTP/2 compresses header names and values with HPACK, which replaces them with indexes into a table built for each connection. From the second request onward, `:path` can be an index alone. An observer that did not watch the connection from its first byte cannot recover the original string. OBI then falls back to `*` for the span name. What the socket path reads by default is also just the first 256 bytes of a request; enable body capture in the configuration and HTTP/1.1 can go further. Some values, such as a routing template, never appear in the byte sequence at all. So Go instrumentation reads the struct before serialization. It reads `Method` and `URL` in `http.Request`, and `method` in gRPC's `transport.Stream`, by byte position. An external tool has to know where each field sits. This is the hardest of the four hurdles to deal with. A mistake here behaves like a mistake in Hurdle 2: no crash, and a trace whose contents are wrong. ## What is a field offset? As Chapter 2 showed, an external observer sees byte positions only. The number of bytes from the start of a struct to a field is the **field offset**. `unsafe.Offsetof` returns that same number. Go source accesses a field by name, as in `req.Method`, but eBPF looks at the process from outside, where the name does not exist. Running memory holds byte sequences and addresses. So the external reader needs a rule stated in bytes: `Header` sits 56 bytes after the start of `http.Request`. ## Checking your Go against OBI's table You can extract this offset from Go itself with `unsafe.Offsetof`. * [Run in the Go Playground](https://go.dev/play/p/WkccXeBRibC) ```go package main import ( "fmt" "net/http" "runtime" "unsafe" ) func main() { var r http.Request fmt.Println(runtime.Version()) fmt.Printf("Method = %d\n", unsafe.Offsetof(r.Method)) fmt.Printf("URL = %d\n", unsafe.Offsetof(r.URL)) fmt.Printf("Header = %d\n", unsafe.Offsetof(r.Header)) fmt.Printf("ContentLength = %d\n", unsafe.Offsetof(r.ContentLength)) } ``` ``` go1.26.5 Method = 0 URL = 16 Header = 56 ContentLength = 88 ``` These four numbers match the offset table that OBI carries, `pkg/internal/goexec/offsets.json`. ```json "net/http.Request": { "Method": { "versions": {"oldest": "1.17.0", "newest": "1.27.1"}, "offsets": [{"offset": 0, "since": "1.17.0"}] }, "URL": { "versions": {"oldest": "1.17.0", "newest": "1.27.1"}, "offsets": [{"offset": 16, "since": "1.17.0"}] }, "Header": { "versions": {"oldest": "1.17.0", "newest": "1.27.1"}, "offsets": [{"offset": 56, "since": "1.17.0"}] }, "ContentLength": { "versions": {"oldest": "1.17.0", "newest": "1.27.1"}, "offsets": [{"offset": 88, "since": "1.17.0"}] } } ``` `versions` gives the range of versions across which OBI verified this entry. `offsets` gives the version from which the field sits at that position. The value that the local Go printed matches the value in the table that OBI prepared in advance. To read a struct from outside is to trust this table and read byte `+56`. `net/http.Request` is a lucky case, and each `offsets` array holds one element. These fields did not move once between Go 1.17 and 1.27. ## The shift when one field is added Add one field in the middle of a struct, then compare the two layouts. * [Run in the Go Playground](https://go.dev/play/p/HDiUH0QUv3Q) ```go package main import ( "fmt" "unsafe" ) type streamV1 struct { ctx any id uint32 method string } type streamV2 struct { // one internal field added ctx any cancel func() id uint32 method string } func main() { var v1 streamV1 var v2 streamV2 fmt.Printf("v1: id=%2d method=%2d\n", unsafe.Offsetof(v1.id), unsafe.Offsetof(v1.method)) fmt.Printf("v2: id=%2d method=%2d\n", unsafe.Offsetof(v2.id), unsafe.Offsetof(v2.method)) } ``` ``` v1: id=16 method=24 v2: id=24 method=32 ``` Everything behind the added field moved together. The byte layout explains the numbers. An `any` takes 16 bytes: one pointer to the type, one to the data. A `func()` takes a single pointer, so 8 bytes, and a `string` takes 16 for a pointer and a length. A `uint32` takes 4 bytes. The `string` behind it starts with a pointer, which must begin on an 8-byte boundary, so padding fills the gap. ![Adding one field shifts everything after it](20260911-struct-byte-band.png) *Figure 1: The arrow represents the change of adding one field. The padding changes position and size too. The fields behind shift by the same 8 bytes that the new field added.* Code that hardcoded 24 as the position of `method` reads a meaningless position in `v2`, one that straddles `id` and the padding. ## Offsets that change without notice The `streamV1` and `streamV2` pair is not an invented example. OBI reads gRPC's `internal/transport.Stream` to get the method name, and that struct shows the same shift. The table records four positions for `method`, so the field moved three times. | gRPC version | Offset of `Stream.method` | |---|---| | 1.40.0 and later | 80 | | 1.66.0 and later | 88 | | 1.69.0 and later | 24 | | 1.77.0 and later | 16 | The type lives in the internal package `internal/transport`, so no compatibility promise covers it. In Go, nothing outside can import a package whose path contains `internal`, and the library never shows this type to its users. Its developers can therefore rearrange the fields in any update, and OBI pays for that freedom, because it reads memory positions from outside. In the same table, `golang.org/x/net/http2.ClientConn.fr` moved more: eight recorded ranges, seven changes of position, one of them back to a place the field held before. Across `offsets.json`, OBI tracks 89 structs and 154 fields, and 43 of them record at least one move. These counts are as of v0.13.0, and they exclude the entries for struct sizes and constants. ![Offsets shifting between versions](20260911-offset-shift.png) *Figure 2: Arrows show which position the instrumentation code reads. A hardcoded `+88` points at `method` in grpc 1.66, and at a different field in 1.69. The worst outcome is a read that produces a plausible value without crashing.* Hardcode an offset, and the code works perfectly against one version. Point it at an application that the user rebuilt on a newer dependency, and it reads the wrong place. The worst case is that **it does not crash, and keeps working plausibly**. The traces still arrive, with the wrong contents inside. For an observability tool, nothing damages trust in the observed system more. ## Read DWARF first, fill the gaps from the table OBI solves this by combining two sources. The entry point in `pkg/internal/goexec/structmembers.go` is where the two paths divide. ```go func structMemberOffsets(elfFile *elf.File) (FieldOffsets, error) { // first, try to read offsets from DWARF debug info var offs FieldOffsets var expected map[GoOffset]struct{} dwarfData, err := elfFile.DWARF() if err == nil { offs, expected = structMemberOffsetsFromDwarf(dwarfData) if len(expected) > 0 { log().Debug("Fields not found in the DWARF file", "fields", expected) } else { libVersions, err := findLibraryVersions(elfFile) if err != nil { return nil, fmt.Errorf("searching for library versions: %w", err) } offs = offsetsForLibVersions(offs, libVersions.versions, log()) setGoAutoSDKActivationSupport(offs, libVersions, elfFile) return offs, nil } } else { // initialize empty offsets offs = FieldOffsets{} } log().Debug("Can't read all offsets from DWARF info. Checking in prefetched database") // if it is not possible, query from prefetched offsets return structMemberPreFetchedOffsets(elfFile, offs) } ``` OBI tries DWARF first. DWARF is the debug information that holds types and field positions, and the compiler embeds it in the target binary itself. Chapter 4 listed it as the `.debug_*` sections. DWARF holds the right answer because the compiler writes it down for the debugger: this field of this type sits this many bytes from the start. The same information lets a debugger show `req.Method` by name. So while OBI can read DWARF, it gets values that match the binary in front of it. Nothing here depends on the version. A tool generates the offset table below automatically, so you may ask why OBI does not use the table alone. DWARF comes first because it holds the answer written into the binary in front of you. The table can carry only the versions that were known and built ahead of time. A freshly released version, a fork, or a private module is absent from it. So the order is to read the real answer first, then fill only the gaps from the table. `expected` in the code is the set of fields that DWARF did not supply. If it is empty, DWARF was enough, and the function returns. If even one field remains, the function calls `structMemberPreFetchedOffsets`, and it passes the offsets that it already has. It never discards a value that it read from DWARF. The receiving function checks `if _, found := fieldOffsets[constantName]; found { continue }`, skips a field that already has a value, and fills only the missing ones from the table. The test `TestPrefetchedOffsetsPreserveResolvedOffsets` pins this behavior. The table that fills the gaps is `offsets.json`, which `//go:embed` puts into the OBI binary. ```go //go:embed offsets.json var prefetchedOffsets string ``` Nobody writes this table by hand. A tool called `go-offsets-tracker` extracts the offsets again at every release of Go and of the major libraries, and updates the table. Part of the work of developing OBI therefore goes into tracking the internal structures of Go and those libraries. This is the hidden cost of zero-code instrumentation. The mapping between the C side and the Go side stays manual. This comment sits above the constant sequence in `structmembers.go`. ```go // this const table must match what's in go_offsets.h type GoOffset uint32 const ( // go common ConnFdPos GoOffset = iota + 1 FdLaddrPos FdRaddrPos TCPAddrPortPtrPos TCPAddrIPPtrPos // http URLPtrPos PathPtrPos RawQueryPtrPos HostPtrPos SchemePtrPos MethodPtrPos StatusCodePtrPos ResponseLengthPtrPos ContentLengthPtrPos ReqHeaderPtrPos IoWriterBufPtrPos IoWriterNPos IoWriterWrPos // ... ) ``` If the `iota` order on the Go side and the order in the eBPF C header disagree, OBI reads an entirely different field. Nothing checks the correspondence. The C header carries its own note, `// start at 1, must match what's in structmembers.go`, and the maintainers hold the two sequences together through those comments alone. The compiler says nothing, and no test covers it. The three constants at the tail of this excerpt, `IoWriterBufPtrPos`, `IoWriterNPos`, and `IoWriterWrPos`, correspond to `buf`, `n`, and `wr` of `bufio.Writer`. Offset tracking therefore covers unexported fields of the standard library. Hurdle 4 shows what they are for. ## Working out the version from the binary To use the table, OBI first has to know which version built this binary. It parses the build info blob that the Go linker leaves in the binary. ```go // The build info blob left by the linker is identified by // a 16-byte header, consisting of buildInfoMagic (14 bytes), // the binary's pointer size (1 byte), // and whether the binary is big endian (1 byte). var buildInfoMagic = []byte("\xff Go buildinf:") ``` From there OBI extracts `runtime.buildVersion` and `runtime.modinfo`. `go version -m` reads the same data: the Go version, and the path, version, and hash of every dependency module. ``` $ go version -m ./myapp ./myapp: go1.26.5 path example.com/myapp mod example.com/myapp (devel) dep golang.org/x/arch v0.30.0 h1:sB9h+1gRGa2+LauFSV0tm8bK1J2yo1bx6/Uyi/P6DTU= build -buildmode=exe build -compiler=gc ... ``` Once OBI knows the versions of the dependency modules, it can pick the right row from the `Stream.method` table above. In some places OBI goes further and holds the module hashes themselves. ```go var goAutoSDKActivationModules = [...]activationModule{ { path: "go.opentelemetry.io/auto/sdk", sums: map[string]string{ "v1.1.0": "h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=", "v1.2.0": "h1:YpRtUFjvhSymycLS2T81lT6IGhcUP+LUPtv0iv1N8bM=", "v1.2.1": "h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=", }, }, // ... } ``` These are the same hashes that `go.sum` records. OBI confirms the version string and the identity of the contents, and only then turns on the behavior specific to that version. ![Assembling offsets from two sources](20260911-offset-resolution.png) *Figure 3: Arrows show the processing flow, top to bottom. OBI keeps every field that it read from DWARF, and fills only the unreadable ones from `offsets.json`. Looking the table up first requires working out the version from the build info, and the table returns the newest record at or below that version. The two join into a single offset table. Only the part that comes from the table depends on the version of the binary.* The answer to Hurdle 3 does not remove the fragility. It absorbs the fragility through a mechanism, release after release. OBI reads a field position from DWARF when the binary carries it, and fills the rest from a table that updates itself. That division of labor is how OBI keeps up with changes in Go and in its libraries.