How Function Calls Work

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/20-function_call.

The previous chapters covered the contents of an executable and how the OS loads it into memory as a process. This chapter examines one of the most common operations during execution: a function call. The return address in Hurdle 1 and the argument passing in Hurdle 2 both come straight out of this chapter.

Stack frames

When a function is called, a working area for that function is pushed onto the stack. This is called a stack frame. The frame holds local variables and values saved for later use, and it is discarded when the function returns. As we saw in Chapter 3, the stack grows from high addresses toward low ones, so the deeper the calls go, the lower the addresses where new frames are pushed.

A function has one location for its instructions and a separate stack frame for each call. The instructions live in .text and stay fixed. In the experiment below, four calls to the same function create four frames while sharing one set of instructions.

We can observe this with the pointers from Chapter 2. We recurse to deepen the call stack, printing the address of a local variable at each depth. The meaning of //go:noinline is explained at the end of this chapter.

• Run in the Go Playground

package main

import (
	"fmt"
	"unsafe"
)

//go:noinline
func descend(depth int) {
	var local [64]byte
	addr := uintptr(unsafe.Pointer(&local[0]))
	fmt.Printf("depth %d: address of local = %#x\n", depth, addr)
	if depth < 3 {
		descend(depth + 1)
	}
}

func main() {
	descend(0)
}

depth 0: address of local = 0x17ef0a08eeb8
depth 1: address of local = 0x17ef0a08ee08
depth 2: address of local = 0x17ef0a08ed58
depth 3: address of local = 0x17ef0a08eca8

Each time the depth increases by one, the address drops by exactly 176 bytes. That means the frame pushed by one call to descend is 176 bytes. When the function returns, the entire frame is discarded.

Figure 1: This figure has no arrows. Up and down correspond to high and low addresses; each deeper call to descend pushes a frame 176 bytes lower.

CALL and RET

The frame also holds the return address. On amd64, the CALL instruction pushes the address of the next instruction onto the stack before jumping. The RET instruction pops that value and jumps to it. The destination is a number in one stack slot.

Disassembling the calling side of the double example from Chapter 4 shows the real thing.

$ go tool objdump -s 'main\.main$' demo
  ...
  main.go:11   0x49e1b3   e8c8ffffff     CALL main.double(SB)   ← the call happens here
  main.go:11   0x49e1b8   440f117c2428   MOVUPS X15, 0x28(SP)   ← after returning, this runs next
  ...

The CALL instruction is at address 0x49e1b3, and the instruction after it is at 0x49e1b8. The moment this CALL executes, the number 0x49e1b8 is pushed onto the stack. The RET at the end of double pops that number off the stack and jumps to it.

The called side contains only the two instructions shown in Chapter 4.

$ go tool objdump -s 'main\.double$' demo
  main.go:7   0x49e180   4801c0   ADDQ AX, AX
  main.go:7   0x49e183   c3       RET            ← pops 0x49e1b8 off the stack and jumps there

The RET line does not specify where to return. It is a one-byte instruction, c3, with no operands. The destination is on the stack, so the instruction itself says only “return to whatever value is pushed there”.

Figure 2: Arrows show the order of events in time. CALL pushes “the address of the next instruction” onto the stack, and the called function’s RET pops it and jumps. The entire knowledge of where to return is a number written in one slot on the stack.

The uretprobe mechanism in Hurdle 1 has the kernel rewrite this number. Keep in mind that the return address being rewritten sits on the stack.

Calling conventions

Arguments are not necessarily placed inside the frame. Where does the caller put the values, and where does the callee read them from? This agreement is called a calling convention. Because it is an agreement, any location works, as long as both sides follow the same convention.

There are two main approaches: pushing arguments onto the stack, and putting them in registers. With stack-based passing, even an outside observer can read the arguments by looking at fixed positions in the frame. Register-based passing is faster because it skips memory, but you cannot read the arguments without knowing the mapping table of which argument goes in which register.

The earlier output has a live example. Look at the line just before CALL main.double(SB).

  main.go:11   0x49e1ae   b815000000   MOVL $0x15, AX   ← puts the argument 21 (=0x15) in AX
  main.go:11   0x49e1b3   e8c8ffffff   CALL main.double(SB)

Before the call, the caller puts the argument 21 from double(21) in the AX register. The callee’s ADDQ AX, AX reads it from the same register. That correspondence defines the convention.

Every platform has a standard calling convention, and on Linux amd64 all C programs follow it. External observation tools therefore assume that standard. Go does not follow it. That mismatch is the subject of Hurdle 2.

Figure 3: The same call f(a, b, c) puts its arguments in different places depending on the convention. To read arguments from the outside, you need to know which convention is in effect and which locations to look at.

Inlining

The examples so far assume that a function call survives in machine code as a CALL. That assumption does not always hold.

The compiler sometimes replaces a call to a small function with the function’s body expanded in place. This is called inlining. The round trip of CALL and RET and the argument passing disappear entirely, so execution gets faster. You can see the compiler’s decisions with -gcflags=-m.

$ go build -gcflags=-m main.go
./main.go:6:6: can inline double
./main.go:11:20: inlining call to double

//go:noinline is a directive that forbids this expansion and forces the call to remain as a CALL. The samples in this book use it repeatedly because inlining would remove the call we want to observe. You do not need it in ordinary applications.

The effect of inlining is not limited to observation. External instrumentation catches a function’s entry and exit by specifying its instruction address. An inlined function has no independent address to specify, so small accessors and wrappers can disappear from instrumentation. This is also why the Chapter 4 exercise found a function missing from the symbol table. How to place an observation point when an address does exist continues from Chapter 4: the destination is machine code on a read-only shared page, so the tool replaces it page by page via copy-on-write. Chapter 7 shows the actual insertion.

Figure 4: Arrows show the direction of transformation. Inlining erases the call instruction itself, so there is no address left to hook.

Key points for the hurdles

• The return address is just a number that CALL writes into one slot on the stack. The uretprobe in Hurdle 1 rewrites it.
• Where arguments live is decided by the calling convention. Register-based passing is fast, but reading it from the outside requires the mapping table. Hurdle 2 is the story of that table.
• An inlined function has no address to hook at all.

Exercises

1. In the descend experiment, if you change the array local from [64]byte to [128]byte, how will the spacing of the printed addresses change? Make a prediction, then check.
2. Remove //go:noinline from double, rebuild, and search the output of go tool objdump -s 'main\.main$' for CALL main.double. Why can’t you find it?