Hurdle 2: The Register-Based Calling Convention

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/45-hurdle2_abi.

We can now plant our hooks. The next problem is where the function’s arguments are at the moment a hook fires.

Unlike Hurdle 1, this problem does not cause a crash. If a generic eBPF tool assumes stack-based argument passing and reads arguments from a Go 1.17 or later binary, it reports unrelated values without any error. The incorrect values can still look plausible.

What is an ABI?

An ABI (Application Binary Interface) is the set of low-level agreements around a function call: where to put the arguments, where to put the return value, which registers each side must preserve, and which registers it may overwrite. The registers we saw in Chapter 2 get their roles here. Machine code compiled from the same Go source passes values according to these agreements.

The Go 1.17 shift

Before Go 1.17, all arguments were passed by pushing them onto the stack. That was easy to read from the outside: look at a fixed offset from the stack pointer at the function’s entry and you knew which argument you were looking at.

From Go 1.17 on, arguments are passed in CPU registers for better performance. This convention is called ABIInternal. On amd64, integer arguments are assigned to registers in the following order.

RAX, RBX, RCX, RDI, RSI, R8, R9, R10, R11

Only integer and pointer arguments use this sequence. Arguments beyond the ninth and large structs that do not fit in registers go on the stack, while floating-point arguments use the floating-point registers mentioned in Chapter 2. This book examines only the integer sequence because OBI reads pointers and integers. Its instrumentation targets functions in net/http and gRPC, which receive pointers to structs, strings, and integers. None of the OBI macros quoted later reads a floating-point register. Chapter 2 explained why both RAX and AX appear in the notation.

This change made Go function calls faster, but anyone reading arguments from the outside now has to know, for each architecture, which argument lands in which register. Generic eBPF tools are built on the assumption that arguments live on the stack, so pointed at a Go binary they read the fixed stack positions and return whatever unrelated values happen to be there as the arguments.

Figure 1: The arrows run from the argument’s storage location to the external reader, showing the information that reader needs. Argument passing moved from the stack to registers. Calls became faster, while external readers now need a register mapping table for each architecture.

Arguments in registers

The following function provides three arguments to inspect in the compiled result. //go:noinline preserves the call and its argument passing in machine code.

• Open in the Go Playground

package main

import "fmt"

//go:noinline
func Add3(a, b, c int) int {
	return a + b + c
}

func main() {
	fmt.Println(Add3(1, 2, 3))
}

Output the assembly with go build -gcflags=-S, and the function body is only three instructions.

$ go build -gcflags=-S -o /dev/null s3_abi.go
main.Add3 STEXT nosplit size=9 args=0x18 locals=0x0 funcid=0x0 align=0x0
	TEXT	main.Add3(SB), NOSPLIT|NOFRAME|ABIInternal, $0-24
	LEAQ	(BX)(AX*1), DX
	LEAQ	(CX)(DX*1), AX
	RET

The full output also includes FUNCDATA and PCDATA lines containing auxiliary information for the GC. The excerpt keeps the three relevant lines. TEXT declares the function; (SB) and NOSPLIT describe how the compiler generated it. ABIInternal identifies the calling convention used here.

a arrives in AX, b in BX, and c in CX, and the result is returned in AX. LEAQ is, as the name says, an instruction that computes an address, but (BX)(AX*1) means “BX plus AX times 1”, so here it serves as an addition tool. The whole function is nine bytes and never touches the stack. Before Go 1.17, this function would have read its arguments from the stack and written its result to the stack.

An external observer needs this mapping table to locate the arguments.

The code where OBI reads the registers

In OBI, that mapping table takes the form of macro definitions in bpf/bpfcore/utils.h.

#if defined(__TARGET_ARCH_x86)

#define GO_PARAM1(x) ((void *)(x)->ax)
#define GO_PARAM2(x) ((void *)(x)->bx)
#define GO_PARAM3(x) ((void *)(x)->cx)
#define GO_PARAM4(x) ((void *)(x)->di)
#define GO_PARAM5(x) ((void *)(x)->si)
#define GO_PARAM6(x) ((void *)(x)->r8)
#define GO_PARAM7(x) ((void *)(x)->r9)
#define GO_PARAM8(x) ((void *)(x)->r10)
#define GO_PARAM9(x) ((void *)(x)->r11)

// In x86, current goroutine is pointed by r14, according to
// https://go.googlesource.com/go/+/refs/heads/dev.regabi/src/cmd/compile/internal-abi.md#amd64-architecture
#define GOROUTINE_PTR(x) ((void *)(x)->r14)

#elif defined(__TARGET_ARCH_arm64)
// the arm64 variant is folded away below
#endif

#define GO_PARAM1(x) ((void *)((PT_REGS_ARM64 *)(x))->regs[0])
#define GO_PARAM2(x) ((void *)((PT_REGS_ARM64 *)(x))->regs[1])
#define GO_PARAM3(x) ((void *)((PT_REGS_ARM64 *)(x))->regs[2])
#define GO_PARAM4(x) ((void *)((PT_REGS_ARM64 *)(x))->regs[3])
#define GO_PARAM5(x) ((void *)((PT_REGS_ARM64 *)(x))->regs[4])
#define GO_PARAM6(x) ((void *)((PT_REGS_ARM64 *)(x))->regs[5])
#define GO_PARAM7(x) ((void *)((PT_REGS_ARM64 *)(x))->regs[6])
#define GO_PARAM8(x) ((void *)((PT_REGS_ARM64 *)(x))->regs[7])
#define GO_PARAM9(x) ((void *)((PT_REGS_ARM64 *)(x))->regs[8])

// In arm64, current goroutine is pointed by R28 according to
// https://github.com/golang/go/blob/master/src/cmd/compile/abi-internal.md#arm64-architecture
#define GOROUTINE_PTR(x) ((void *)((PT_REGS_ARM64 *)(x))->regs[28])

The sequence ax, bx, cx, di, si, r8, r9, r10, r11 corresponds to RAX, RBX, RCX, RDI, RSI, R8, R9, R10, R11. x captures the CPU registers when the uprobe fires, and each macro selects one register from that snapshot. OBI reads this state without a public Go API.

How to identify the current goroutine

Knowing which goroutine is currently running matters more than the arguments. One request spans multiple function calls, and instrumentation must connect calls made on the same goroutine.

As we saw in Chapter 6, the Go runtime has structs named g, m, and p. All this book needs is g, the runtime-internal struct that represents a single goroutine. At any given moment, the Go code executing on a CPU is associated with the g struct for the current goroutine.

The Go runtime keeps a pointer to this g struct resident in a dedicated register.

This is exactly the register the GOROUTINE_PTR macro reads. The CMPQ SP, 0x10(R14) we saw in Hurdle 1 was likewise reading byte 16 of the g struct (stackguard0) through the same R14 to check how much stack was left. R14 shows up all over the machine code of a program written in Go.

OBI makes a surprising choice here. The g struct contains goid, a serial number that could identify the goroutine. OBI does not read it. A repository-wide search finds no use of goid as a goroutine identifier¹.

OBI uses the address of the g struct itself. Its key pairs the pointer returned by GOROUTINE_PTR with the process ID.

typedef struct go_addr_key {
    u64 pid;  // PID of the process
    u64 addr; // Address of the goroutine
} go_addr_key_t;

The value of GOROUTINE_PTR(ctx) goes into addr.

OBI only needs to determine whether the current goroutine is the same one it observed earlier; it does not need a human-readable number. The address is sufficient for that comparison and avoids reading any fields inside the struct.

Reading goid would require its byte offset within g, which can change between Go versions. Using the struct’s address requires no field offset, and OBI’s offset table has no entry for runtime.g.

Figure 2: Solid arrows show the flow of data; the dashed line with the circle marks what is not read. OBI uses the value of R14 (R28 on arm64) directly as the goroutine identifier. Changes to the fields behind that pointer do not affect the identifier.

The moving stack and the stationary g struct

Hurdle 1 moved the goroutine’s stack. When it grows, the runtime copies it to a new region and changes its address.

The value that can serve as an identifier is the g struct. It is allocated on the heap and does not move while the goroutine lives. g records the bounds of its stack region in stack.lo and stack.hi; when the stack relocates, those fields are updated. The struct remains in place while the stack it references moves.

Figure 3: Arrows represent references. The g struct remains at one address while stack.lo and stack.hi change to point at the relocated stack. OBI uses the struct’s address as the identifier.

The g struct’s address is not on the stack, so it stays the same even when the stack relocates. That is why it can safely be used as a key.

Using an address as an identifier also requires OBI to handle what happens after the goroutine ends. The timeline below shows both the benefit and the cost.

Figure 4: Arrows represent time order. While processing continues, the same address means the same goroutine; after it ends, the address is handed to a different goroutine. The dashed part shows what happens when stale records aren’t deleted. This cleanup appears as real code in Hurdle 4.

Go application code has no official way to identify the current goroutine. Some code parses goid from the string returned by runtime.Stack, but the Go team leaves goid unexposed and directs applications to pass goroutine-local values through context.Context. OBI observes from outside and uses an address without changing that policy.

Hurdle 2 is that Go stores values in registers and inside runtime structs rather than in obvious stack locations. Reading them correctly requires architecture- and version-specific knowledge. OBI consults an architecture-specific register mapping table for arguments and uses the address of g directly to identify goroutines.

Using the address of g avoids reading its fields. Hurdle 3 covers structs whose contents OBI must read.