Hurdle 1: uretprobe Doesn't Work
Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/40-hurdle1_uretprobe.
Symptoms
Placing a uretprobe on a Go binary can crash the instrumented program with fatal error: unknown caller pc, the error shown in Chapter 1.
An observer cannot be allowed to terminate its target. The crash comes from how uretprobe implements a return hook.
How uretprobe works
A uretprobe fires a uprobe at function entry and replaces the stack’s return address with the address of a kernel-provided trampoline. As Chapter 5 showed, RET pops the return address from one stack slot and jumps there. Replacing that number redirects RET.
Uretprobe assumes that the return address remains in the same place until RET executes.
Figure 1: Solid arrows represent control flow; the dashed one shows where the rewritten return address leads. The moment the function is entered, uretprobe rewrites the return address on the stack to the trampoline’s address. This scheme assumes that the stack location holding the return address never moves afterward.
The collision with Go’s movable stacks
In Go, this assumption does not hold. We saw in Chapter 6 that a goroutine’s stack relocates to a different region every time it grows.
The runtime has to walk the stack before it can adjust pointers, and the walk exposes the rewritten return address.
The runtime first allocates a new region and copies the stack’s active range. It then walks the copied frames and fixes their pointers. Starting from the currently executing function, the runtime reads each frame’s return address and looks it up in .gopclntab, the table from Chapter 4 that maps instruction addresses to Go functions. The function record gives the frame size and pointer locations. The runtime rewrites those pointers, advances by the frame size, and repeats the process with the caller until it reaches the goroutine’s origin.
Processing a frame depends on mapping its return address to a function. If the lookup fails, the runtime does not know the frame’s size and cannot find the start of the next frame. Reading the value is not enough; the value must identify a function.
Figure 2: Solid arrows represent the order of operations; the dashed one represents moving to the caller’s frame and repeating the same procedure. The frame’s size and the positions of its pointers come from the function information looked up from the return address.
The return address rewritten by a uretprobe points to a kernel-provided trampoline rather than a function in the Go binary. This region lies outside the range that contains Go code, and /proc/<PID>/maps lists it as [uprobes]. The function table therefore has no matching entry. Stack copying and GC scans cannot stop halfway, so the runtime terminates with fatal error: unknown caller pc. In go1.26, unwinder.next in runtime/traceback.go throws this error when findfunc cannot resolve the return address.
The output from an actual crash along this path is preserved in Go issue #27077. Arguments and intermediate lines are omitted.
runtime: unexpected return pc for crypto/x509.(*Certificate).Verify called from 0x7fffffffe000
...
fatal error: unknown caller pc
runtime stack:
runtime.throw(0x5538ab, 0x11)
/usr/local/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, ...)
/usr/local/go/src/runtime/traceback.go:257 +0x1bdb
runtime.copystack(0xc420000180, 0x4000, 0x7fff74e5e901)
/usr/local/go/src/runtime/stack.go:891 +0x270
runtime.newstack()
/usr/local/go/src/runtime/stack.go:1063 +0x30f
runtime.morestack()
/usr/local/go/src/runtime/asm_amd64.s:480 +0x89
Reading from the bottom up: the stack runs short and execution enters morestack; newstack calls copystack, which performs the relocation; and the frame walk fails inside it. The 0x7fffffffe000 in the first line is the trampoline’s address: the report says that the return target of crypto/x509.(*Certificate).Verify had become that address. The report dates from go1.10, when the frame-walking implementation was called gentraceback. This is why Go and uretprobe are incompatible (golang/go#22008).
Figure 3: Solid arrows represent the flow of processing; dashed ones represent values read from the stack. The runtime uses the return address as its clue to look up which function’s frame it is. The trampoline’s address is not in the table, so the walk gets stuck there.
Watching a stack move
Most Go code hides stack movement. The following program exposes it by comparing a local variable’s address before and after stack growth.
//go:noinline preserves the function call. The pad inside grow and the deep recursion exhaust the stack, while unsafe.Pointer and uintptr expose a variable’s address for comparison. Ordinary application code does not need these measures. The code runs on any OS; macOS produces different address values but the same behavior as the linux/amd64 measurements used here.
package main
import (
"fmt"
"unsafe"
)
//go:noinline
func grow(n int) int {
var pad [256]byte
if n == 0 {
return int(pad[0])
}
return grow(n-1) + int(pad[1])
}
func main() {
var anchor [16]byte
before := uintptr(unsafe.Pointer(&anchor[0]))
grow(3000) // deep recursion to grow the goroutine's stack
after := uintptr(unsafe.Pointer(&anchor[0]))
fmt.Printf("before = %#x\n", before)
fmt.Printf("after = %#x\n", after)
fmt.Printf("moved = %v\n", before != after)
}
anchor is a local variable of main, and nothing touches it before or after the recursion in grow. Yet when you run this, its address changes.
before = 0x2522317acee8
after = 0x2522319dfee8
moved = true
The specific addresses change from run to run. With go1.26.5 linux/amd64, the version used for this book, moved came out true every time. The recursion exhausted the stack while main’s frame was still active, so the runtime allocated a larger region and relocated every frame, including main’s. If any pointer had been pointing at anchor at that moment, the runtime would have rewritten it to the new address as well.
If a uretprobe had written a trampoline address in place of a real return address, this relocation would not complete. While walking the frames for the copy, the runtime would fail to find a function corresponding to that value and stop.
uprobes on every RET instruction
Because OBI cannot use uretprobes, it places ordinary uprobes directly on the instructions that return from the function.
A uprobe can target any instruction address, including each RET at a function’s exits. OBI follows three steps.
- Disassemble the target function.
- Find the addresses of every
RETinstruction in the machine code. - Set an ordinary uprobe on every one of them.
This makes the eBPF program fire just before the CPU executes RET and returns to the caller.
This approach leaves the stack untouched. A uprobe swaps one byte in the code area for a breakpoint instruction, so the stack retains a return address inside a Go function. The runtime can resolve that address and complete its scan when it moves the stack.
The RET instruction also stays at its build-time address in the machine code. Stack relocation therefore does not affect whether its uprobe fires.
There is exactly one exit this scheme can’t catch: a function that ends in a panic that is never recovered. In that case the function terminates along with its goroutine without executing RET, so the exit hook doesn’t fire. But the request itself has failed by then, so the practical harm of a missing latency record is limited.
Figure 4: The arrows represent the order of operations. The exit hook is an ordinary uprobe placed on each RET found by disassembly. Even with a single return in the source, one defer makes it two RETs.
Two RETs from one return
There is a reason I wrote “every.” Even when the Go source has only one return, the machine code can contain multiple RETs. Writing a single defer is enough to create this situation.
defer is the construct that registers work to run when the function exits. Even if the function appears to have one exit, a separate path is needed that runs the registered work before leaving. The //go:noinline in the following code is, as before, the directive that keeps the function from being inlined.
package main
import (
"fmt"
"sync"
)
var (
mu sync.Mutex
cache = map[string]int{}
)
//go:noinline
func Lookup(key string) int {
mu.Lock()
defer mu.Unlock()
return cache[key]
}
func main() {
fmt.Println(Lookup("go"))
}
Run go build and then go tool objdump, and two RETs show up. The output below is from a binary built with GOOS=linux GOARCH=amd64. Even on a Mac or Windows machine, building with these two environment variables lets you observe the same thing (go tool objdump can read cross-compiled binaries). The left column is the source line number, next comes the address, and to its right the instruction. The only lines you need to look at are the two that say RET; feel free to skip the rest.
$ go tool objdump -s 'main\.Lookup$' s2_ret
s2_ret.go:14 0x49e1c0 CMPQ SP, 0x10(R14)
s2_ret.go:14 0x49e1c4 JBE 0x49e2a1
...
s2_ret.go:17 0x49e28f POPQ BP
s2_ret.go:17 0x49e290 RET ← the normal path
s2_ret.go:17 0x49e291 CALL runtime.deferreturn(SB)
s2_ret.go:17 0x49e296 MOVQ 0x28(SP), AX
s2_ret.go:17 0x49e29b ADDQ $0x50, SP
s2_ret.go:17 0x49e29f POPQ BP
s2_ret.go:17 0x49e2a0 RET ← the path through defer
s2_ret.go:14 0x49e2ab CALL runtime.morestack_noctxt.abi0(SB)
0x49e290 leaves the function normally, while 0x49e2a0 leaves after runtime.deferreturn. Both correspond to the same source-level return at s2_ret.go:17.
The compiler normally expands deferred work at the end of the function and runs it inline. A function that recovers from a panic goes through runtime.deferreturn to finish the remaining work before leaving. The same path handles cases the compiler cannot expand, such as a defer inside a loop. A uprobe placed only at 0x49e290 therefore misses some error cases, which normal-path testing may not reveal.
This output also contains two elements covered later. The first line’s CMPQ SP, 0x10(R14) is the stack-growth check just described: it compares the stack pointer with the value 16 bytes into the structure R14 points at, and if the stack falls short, jumps to the last line’s runtime.morestack_noctxt. Hurdle 2 explains what R14 contains.
OBI’s implementation
The following function in pkg/internal/goexec/instructions_amd64.go finds the RET instructions for OBI. It is ordinary Go, not eBPF C code.
func FindReturnOffsets(baseOffset uint64, data []byte) ([]uint64, error) {
var returnOffsets []uint64
index := 0
for index < len(data) {
// FIXME remove this once x86asm is able to recognize and decode
// ENDBR64
if isENDBRXX(data[index:]) {
index += endbrSize
continue
}
instruction, err := x86asm.Decode(data[index:], 64)
if err != nil {
return nil, fmt.Errorf("failed to decode x64 instruction at offset %d: %w", index, err)
}
if instruction.Op == x86asm.RET {
returnOffsets = append(returnOffsets, baseOffset+uint64(index))
}
index += instruction.Len
}
return returnOffsets, nil
}
Using golang.org/x/arch/x86/x86asm, the function decodes machine code from the beginning and records the position of each x86asm.RET. It sweeps the byte sequence without following branches or jumps.
The isENDBRXX branch skips 4 bytes because x86asm cannot yet decode ENDBR64, the CPU’s indirect-branch protection instruction. The attached // FIXME remove this once ... comment records the compatibility workaround. These lines show that zero-code instrumentation adds up from unglamorous, incremental work.
The code in pkg/ebpf/instrumenter.go binds a uprobe to each collected offset.
if probe.End != nil {
if len(probe.ReturnOffsets) == 0 {
// ...
return closers, errors.New("setting uretprobe (attaching to offset): missing return offsets")
}
for _, offset := range probe.ReturnOffsets {
up, err := exe.Uprobe("", probe.End, &link.UprobeOptions{
Address: offset,
})
// ...
closers = append(closers, up)
}
}
The first argument to exe.Uprobe is the empty string, and the offset goes in Address, using the “direct address” form from Chapter 7. Although the error message says uretprobe, the code attaches an ordinary uprobe at the address of a RET instruction. It serves as an exit hook without using the uretprobe mechanism.
Go’s movable stack makes return-address rewriting unsafe. OBI leaves the return address alone, finds every RET in the machine code, and places an ordinary uprobe on each one.