What the Go Runtime Manages

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/25-go_runtime.

A Go binary contains a runtime alongside the code you wrote. Rather than leaving scheduling and memory management to the OS, the runtime implements both itself. This design makes Go harder to observe from the outside.

One statically linked binary

In Chapter 4 we saw that linking comes in two flavors, static and dynamic. Go links statically by default. The standard library and the runtime are bundled into a single executable, and unless you use cgo, no shared libraries are needed. Run it through ldd and you get back not a dynamic executable.

Static linking makes a Go binary convenient to analyze. Uprobes also work with shared libraries: a uprobe identifies a file and a byte offset, and OBI uses them in OpenSSL to read TLS traffic. Go’s static binary offers a different advantage. Application code, libraries such as net/http, and the runtime all occupy fixed positions in one file. Analyzing /proc/<PID>/exe reveals every function available for instrumentation.

This is also why Chapter 1 said that Go is the only language with instrumentation that reaches down to the level of individual functions. In languages that run on a JIT compiler or an interpreter, the machine code for user code is generated at run time, so analyzing the executable tells you nothing about where the functions are. In Go, the position of every function is fixed at compile time and can be looked up through the symbol table or .gopclntab. That is what makes “place a hook at an address” instrumentation possible.

OS threads and goroutines

A goroutine is not an OS thread. It is a unit of execution managed by the runtime. Every go f() creates one with a small initial stack of a few KB. Creating tens of thousands of them is fine.

Inside the runtime, three structs divide up the roles. A g represents one goroutine, an m represents one OS thread, and a p provides the resources needed to run Go code together with a queue of waiting work. Many gs take turns running on a few ms through ps. The Go runtime, not the kernel, decides when to switch them. Only g appears again in the rest of this book, so you only need to remember the names m and p.

The kernel cannot see goroutines; its knowledge stops at threads. Goroutines from different requests can also run on the same OS thread, so a thread ID cannot connect two moments of work to the same request. Hurdle 4 starts from this mismatch.

Figure 1: Arrows show the “runs on” relationship: each item above runs on the item below it. The Go runtime, not the kernel, decides which goroutine runs when.

Goroutine stack growth

An OS thread’s stack is allocated at a substantial fixed size when the thread starts, and it never moves. A goroutine’s stack is different: it starts small and grows when it runs out.

Each function’s entry checks whether enough stack remains. If not, execution jumps to runtime.morestack, where the runtime allocates a larger area, copies the current contents, and rewrites pointers into the stack with their new addresses.

The address of a goroutine’s local variable can change while the program runs. The runtime fixes its pointers, hiding the relocation from Go code but not from an outside observer. This book calls a stack that can relocate during execution a movable stack. It causes Hurdle 1.

Figure 2: Arrows show the order of events in time, top to bottom. During the copy, the runtime walks the frames one by one, using return addresses as its guide. That the frames can be walked is a precondition for the move.

Go’s own calling convention

The agreement on where to put function arguments is another thing Go decides for itself. Since Go 1.17, arguments are passed in registers, not on the stack.

On amd64, the R14 register always points at the g struct of the currently running goroutine. It therefore appears throughout machine code compiled from Go source.

The details belong to Hurdle 2. For now, it is enough to hold onto one fact: Go’s function calls do not follow the platform’s standard convention.