# Hurdle 1: uretprobe Doesn't Work

> Source: https://www.ymotongpoo.com/books/go-ebpf-primer/40-hurdle1_uretprobe/


## Symptoms

Placing a probe at a function's entry and its exit looks like something Go should take as readily as any other language. Yet a uretprobe on a Go binary can crash the instrumented program with `fatal error: unknown caller pc`. That is the error I put in Chapter 1.

Instrumentation exists to observe, and it must never kill the program that it observes. The cause of the crash lies in the way uretprobe implements a return hook.

## How uretprobe works

As Chapter 7 showed, a uretprobe acts the moment the uprobe at the function's entry fires. The kernel then replaces the return address on the stack with the address of a trampoline that it provides. Chapter 5 showed what that return address is. It is a plain number in one stack slot, and the `RET` instruction at the end of the function reads it before it jumps.

At the entry, `SP` points at that slot. The stack only ever works last in, first out, and the return address is the last thing pushed onto it. The kernel therefore reads the eight bytes at `SP`, saves them, and writes a different value into the same place. Change the number and `RET` goes somewhere else.

So a uretprobe assumes that the address that it writes stays in the same place until `RET` runs.

![Return-address rewriting by uretprobe](20260911-uretprobe-rewrite.png)
*Figure 1: Solid arrows represent control flow; the dashed one shows where the rewritten return address leads. The vertical position inside each table shows the height of the address. The moment execution enters the function, SP points at the slot holding the return address, and uretprobe rewrites that slot to the address of a trampoline that the kernel maps into the target process's address space (the `[uprobes]` region). This scheme assumes that the stack location holding the return address never moves afterward.*

## The collision with Go's movable stacks

In Go, that assumption does not hold. Chapter 6 showed that a goroutine's stack moves to a different region every time it grows.

The pointer adjustment itself is not the problem. To adjust the pointers, the runtime **has to walk** the stack.

The runtime first allocates a new region and copies the whole range of the stack that is in use. It then walks the frames one at a time and fixes the pointers in the copy, beginning with the frame of the function that runs now. It reads the return address stored in that frame, and that address points at an instruction inside the calling function. So `.gopclntab`, the table from Chapter 4 that maps an instruction address to a Go function, tells the runtime which function the frame belongs to. Once the runtime knows the function, it knows the size of the frame and the position of every pointer in it. It rewrites those pointers to the new addresses, and the frame size tells it where the next frame starts. It then moves to the caller's frame and repeats the same steps until it reaches the goroutine's origin.

Everything that the runtime needs to process one frame sits behind a single step: mapping the return address to a function. If that lookup fails, the runtime does not know the size of the frame and cannot find the start of the next one. Reading the value is not enough; the value must identify a function.

![The frame-by-frame scan when moving a stack](20260911-stack-walk.png)
*Figure 2: Solid arrows represent the order of operations; the dashed one represents moving to the caller's frame and repeating the same procedure. The size of the frame and the positions of its pointers come from the function that the runtime looks up from the return address.*

The return address that a uretprobe writes points at a trampoline that the kernel provides, not at a function in the Go binary. That region lies outside the range that holds Go code, and `/proc/<PID>/maps` lists it as `[uprobes]`. The function table therefore has no matching entry. The runtime cannot abandon a stack copy or a GC scan halfway, so it dies with `fatal error: unknown caller pc`. In go1.26, `unwinder.next` in `runtime/traceback.go` throws this error when `findfunc` cannot resolve the return address.

Go issue [#27077](https://github.com/golang/go/issues/27077) preserves the output from a real crash along this path. I have cut the arguments and the intermediate lines.

```
runtime: unexpected return pc for crypto/x509.(*Certificate).Verify called from 0x7fffffffe000
...
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x5538ab, 0x11)
	/usr/local/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, ...)
	/usr/local/go/src/runtime/traceback.go:257 +0x1bdb
runtime.copystack(0xc420000180, 0x4000, 0x7fff74e5e901)
	/usr/local/go/src/runtime/stack.go:891 +0x270
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:1063 +0x30f
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:480 +0x89
```

Read it from the bottom up. The stack runs short, so execution enters `morestack`. `newstack` then calls `copystack`, which does the relocation, and the frame walk fails inside it. The `0x7fffffffe000` on the first line is the trampoline's address; the report says that `crypto/x509.(*Certificate).Verify` was returning to it. The report dates from go1.10, when the function that walked the frames was still called `gentraceback`. This is why Go and uretprobe cannot work together ([golang/go#22008](https://github.com/golang/go/issues/22008)).

![When a return address maps to no function, the runtime cannot walk the stack](20260911-unwind-failure.png)
*Figure 3: Solid arrows represent the flow of processing; dashed ones represent values read from the stack. The runtime uses the return address as its clue to look up which function's frame it is in `.gopclntab`. The trampoline lives in a region that the Go linker knows nothing about, so it is not registered in that table, and the walk gets stuck there.*

## Watching a stack move

A moving stack is a behavior you never notice in day-to-day Go development. Watching one move makes the rest of this chapter concrete. The program below compares the address of a local variable before and after the stack grows.

The code carries three devices that make the result easy to read. `//go:noinline` tells the compiler to leave the call in place instead of expanding it. The `pad` inside `grow` and the deep recursion make sure the stack runs out. `unsafe.Pointer` and `uintptr` pull the address of a variable out as a number so that you can compare it before and after. None of the three belongs in ordinary application code. The code runs on any OS: the measurements here come from linux/amd64, and macOS gives different address values with the same result.

* [Run it on the Go Playground](https://go.dev/play/p/k2fKCPbMYCv)

```go
package main

import (
	"fmt"
	"unsafe"
)

//go:noinline
func grow(n int) int {
	var pad [256]byte
	if n == 0 {
		return int(pad[0])
	}
	return grow(n-1) + int(pad[1])
}

func main() {
	var anchor [16]byte
	before := uintptr(unsafe.Pointer(&anchor[0]))
	grow(3000) // deep recursion to grow the goroutine's stack
	after := uintptr(unsafe.Pointer(&anchor[0]))

	fmt.Printf("before = %#x\n", before)
	fmt.Printf("after  = %#x\n", after)
	fmt.Printf("moved  = %v\n", before != after)
}
```

`anchor` is a local variable of `main`, and nothing touches it before or after the recursion in `grow`. Yet when you run this, its address changes.

```
before = 0x2522317acee8
after  = 0x2522319dfee8
moved  = true
```

The specific addresses change from run to run. On go1.26.5 linux/amd64, the version that this book uses, `moved` came out `true` every time. The recursion consumed the stack past the depth of `main`'s frame, so the runtime took a larger region and moved every frame, `main`'s included. If a pointer to `anchor` exists at that moment, the runtime rewrites it to the new address as well.

Suppose a uretprobe had written a false return address. This relocation would never finish. While the runtime walks the frames for the copy, it looks that value up, finds no entry in the table, and can only stop there.

## uprobes on every `RET` instruction

Since the uretprobe built for exits is unavailable, OBI places ordinary uprobes directly on the instructions that correspond to the exits.

As Chapter 7 showed, you can place a uprobe at any instruction address. It can go on a `RET` at the end of a function rather than on the function's first address. OBI takes three steps.

1. Disassemble the target function.
2. Find the addresses of every `RET` instruction in the machine code.
3. Place an ordinary uprobe on every one of them.

The eBPF program then fires just before the CPU executes `RET` and returns to the caller.

The difference in effort against a uretprobe shows up here. With a uretprobe, the only address you have to know is the function's entry. The kernel takes on the exit itself, by rewriting the return address that it saved at the entry into the address of the trampoline. The instrumenting side never learns where `RET` sits.

OBI's form does not hand that work to the kernel, so it has to find every exit address on its own. The symbol table gives the start address of a function and its size, and it records nothing about where inside that range a `RET` sits. That is why step 1 disassembles.

The mechanism from Chapter 7 explains why this form does not crash. To place a uprobe is to swap one byte at that instruction address for a breakpoint instruction. That byte lives in the code area, and nothing goes onto the stack. The return address on the stack therefore stays what it was, the address of a real Go function. When the runtime moves the stack, it can still look that address up, so the walk runs to the end.

The address of the `RET` instruction does not move either. Chapter 4 showed that the position of a function is fixed at build time and does not change while the program runs. What moves is the stack, not the place where the machine code sits. Whether the probe fires has nothing to do with a stack relocation.

One probe per `RET` has a price, and it is worth seeing where the price falls. It does not fall on each call of the function. A single call executes one `RET`, so the number of firings matches what a uretprobe would give you.

The price falls on setup and teardown. Registering with the kernel takes one call per `RET`. Later in this chapter, `instrumenter.go` walks the offsets it collected one by one, calls `exe.Uprobe` for each, and keeps the link that comes back. Removing the instrumentation closes every link it kept. As the number of instrumented functions grows, this count becomes the number of functions times the number of `RET`s. The analysis side pays too, because it scans the machine code of each function one instruction at a time.

Some exits escape this scheme: the functions that a panic skips as it unwinds. When nothing `recover`s the panic the whole program ends, but even when a `defer` in a caller does `recover`, the inner functions that the unwinding skipped never execute `RET`. The only function that comes back through the `RET` path is the one that registered the `defer` that ran `recover`. `runtime.Goexit` and `os.Exit` do the same thing. The `net/http` server `recover`s a handler panic in `conn.serve`, so the exits of instrumented functions inside a handler go missing in this shape. The request has already failed at that point, so losing the record of the elapsed time does limited harm.

![uprobes on every RET instruction](20260911-ret-uprobes.png)
*Figure 4: The arrows represent the order of operations. For the exit hook, OBI places an ordinary uprobe on each `RET` that the disassembly finds. Even with a single `return` in the source, one `defer` makes two `RET`s.*

## Two `RET`s from one `return`

I wrote "every" for a reason. Even when the Go source has a single `return`, the compiler can emit more than one `RET`. Writing one `defer` is enough to produce that.

`defer` is the construct that registers work to run when the function exits. Even a function that appears to have one exit needs a separate path that runs the registered work before it leaves. The `//go:noinline` in the code below is, as before, the directive that tells the compiler not to expand the function.

* [Open in the Go Playground](https://go.dev/play/p/QNKx9E4rpJM)

```go
package main

import (
	"fmt"
	"sync"
)

var (
	mu    sync.Mutex
	cache = map[string]int{}
)

//go:noinline
func Lookup(key string) int {
	mu.Lock()
	defer mu.Unlock()
	return cache[key]
}

func main() {
	fmt.Println(Lookup("go"))
}
```

Build it with `go build` and run `go tool objdump` on the result, and two `RET`s appear. The output below comes from a binary built with `GOOS=linux GOARCH=amd64`. On a Mac or a Windows machine, set those two environment variables and you see the same thing, because `go tool objdump` reads cross-compiled binaries too. The left column is the source line number, the next one is the address, and the one on its right is the instruction. Only the two lines that say `RET` matter here, so skip the rest.

```
$ go tool objdump -s 'main\.Lookup$' s2_ret
  s2_ret.go:14  0x49e1c0   CMPQ SP, 0x10(R14)
  s2_ret.go:14  0x49e1c4   JBE 0x49e2a1
  ...
  s2_ret.go:17  0x49e28f   POPQ BP
  s2_ret.go:17  0x49e290   RET                                   ← the normal path
  s2_ret.go:17  0x49e291   CALL runtime.deferreturn(SB)
  s2_ret.go:17  0x49e296   MOVQ 0x28(SP), AX
  s2_ret.go:17  0x49e29b   ADDQ $0x50, SP
  s2_ret.go:17  0x49e29f   POPQ BP
  s2_ret.go:17  0x49e2a0   RET                                   ← the path through defer
  s2_ret.go:14  0x49e2ab   CALL runtime.morestack_noctxt.abi0(SB)
```

`0x49e290` leaves the function normally, while `0x49e2a0` leaves after `runtime.deferreturn`. Both correspond to the same source-level `return` at `s2_ret.go:17`.

The compiler normally expands the work registered with `defer` at the end of the function and runs it there. When a panic occurs and `recover` brings the program back, the function goes through `runtime.deferreturn` to finish the remaining work before it leaves. The same path serves the cases that the compiler cannot expand, such as a `defer` inside a loop. A uprobe at `0x49e290` alone therefore misses the exit of every call in which a panic occurred. The record of the elapsed time goes missing only on errors, which makes it a gap that testing the normal path never shows.

Two things in this output belong to later sections. The `CMPQ SP, 0x10(R14)` on the first line is the stack-growth check I described above. It compares the stack pointer with the value 16 bytes into the struct that `R14` points at, and it jumps to `runtime.morestack_noctxt` on the last line when the stack falls short. Hurdle 2 covers what `R14` is.

## OBI's implementation

The following function in `pkg/internal/goexec/instructions_amd64.go` finds the `RET` instructions for OBI. It is ordinary Go, not eBPF C code.

```go
func FindReturnOffsets(baseOffset uint64, data []byte) ([]uint64, error) {
	var returnOffsets []uint64
	index := 0
	for index < len(data) {
		// FIXME remove this once x86asm is able to recognize and decode
		// ENDBR64
		if isENDBRXX(data[index:]) {
			index += endbrSize
			continue
		}

		instruction, err := x86asm.Decode(data[index:], 64)
		if err != nil {
			return nil, fmt.Errorf("failed to decode x64 instruction at offset %d: %w", index, err)
		}

		if instruction.Op == x86asm.RET {
			returnOffsets = append(returnOffsets, baseOffset+uint64(index))
		}

		index += instruction.Len
	}

	return returnOffsets, nil
}
```

The function uses `golang.org/x/arch/x86/x86asm` to decode the machine code one instruction at a time from the start, and it records the position of every `x86asm.RET`. That is all it does. It follows no branch and no jump; it reads the byte sequence straight through.

x86asm cannot yet decode `ENDBR64`, the CPU instruction that guards indirect branches, so the `isENDBRXX` branch skips 4 bytes to work around it. The `// FIXME remove this once ...` comment still sits above it. These few lines show how much of zero-code instrumentation is plain, incremental work.

`pkg/ebpf/instrumenter.go` then binds a uprobe to each of the offsets that OBI collected.

```go
	if probe.End != nil {
		if len(probe.ReturnOffsets) == 0 {
			// ...
			return closers, errors.New("setting uretprobe (attaching to offset): missing return offsets")
		}

		for _, offset := range probe.ReturnOffsets {
			up, err := exe.Uprobe("", probe.End, &link.UprobeOptions{
				Address: offset,
			})
			// ...
			closers = append(closers, up)
		}
	}
```

The first argument to `exe.Uprobe` is an empty string and the offset goes into `Address`, which is the direct-address form from Chapter 7. Note that this `offset` is an offset within the file. As Chapter 7 said, what goes into `Address` is not a virtual address. In `pkg/internal/goexec/instructions.go`, OBI converts the virtual address it used for disassembly into a file offset from the ELF segment information. The error message says `uretprobe`, but the call attaches an ordinary uprobe. Only the name of the role survives; what sits at the address of a `RET` is the same kind of hook as an entry hook.

By now one probe sits at the entry, and as many probes as there are `RET`s sit at the exits. That leaves the question of how an exit probe knows it belongs to the same function as the entry probe. Nothing works out at run time which function execution is inside. The answer comes in two stages.

Which function an entry belongs to, and which function an exit belongs to, are both fixed at attach time. OBI holds a pair of eBPF programs for each instrumented function, one for the entry and one for the exit. `probe.End` in the excerpt above is the exit program. The entry program comes from the other field of the same struct, and OBI ties it to the function's start address. The exit program goes only onto the `ReturnOffsets` that OBI collected from that same function.

Chapter 7 showed the rest. The kernel matches the PC at the moment of the trap against its registration table. It identifies which uprobe ran and runs only the program tied to it. The moment a `RET` fires, the kernel already knows which function's exit it is.

What remains is matching the entry and the exit of the same **call**. That decision happens at run time, and the goroutine is the key. The entry program leaves a record. The exit program looks up the record for the same goroutine, which pairs the start and the end of one call. How OBI identifies that goroutine is the subject of Hurdle 2.

Hurdle 1 is a case where a mechanism behind Go's efficiency, the movable stack, turns straight into a difficulty for instrumentation. OBI's answer is simple. It leaves the return address alone, finds every `RET` in the machine code, and places an ordinary uprobe on each one.