The OS and the Kernel

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/10-os_kernel.

The picture in the previous chapter had only one program in it. A real computer runs many programs on the same CPU and physical memory. The OS (operating system) isolates them by giving each program the illusion of having the machine to itself.

Processes and virtual memory create that illusion. The kernel, the core of the OS, maintains the boundary, and it manages execution in units called threads. eBPF runs inside the kernel, so Chapter 7 relies on these concepts.

Executable files and processes

The executable file sitting on disk and the running entity you get by launching it are two different things. The running entity is called a process. From one executable file you can launch any number of processes.

The OS assigns each process a number called a PID (process ID) to tell them apart. From Go you can see the PID with os.Getpid().

• Run on the Go Playground

package main

import (
	"fmt"
	"os"
)

func main() {
	x := 42
	fmt.Printf("PID: %d  address of x: %p\n", os.Getpid(), &x)
}

Run the same binary twice, and you get this.

PID: 25  address of x: 0x179411378070
PID: 31  address of x: 0x13456066e070

The same program and variable produce different PIDs and addresses between runs. You can confirm this by running the program twice on your machine. The Playground uses a single isolated environment, so its PID stays the same. The next section explains the different addresses.

Figure 1: Arrows show the launch relationship. Two processes launched from one executable file have separate PIDs and separate memory, and run independently of each other.

Virtual memory

The addresses a process sees are not the slot numbers of physical memory itself. They are a private sequence of addresses prepared for each process. These are called virtual addresses, and the kernel maintains a table that maps each virtual address to a location in physical memory.

As a result of this mechanism, the address 0x179411378070 in one process and the same address in another process point to unrelated places. A process cannot reach outside its own virtual address space, so it cannot read another process’s memory simply by specifying an address. The two runs in the previous section printed different addresses because each process has its own address space.

Figure 2: Arrows show correspondence. Each of the two processes sees its own private virtual address space, which is mapped through the kernel’s table to separate places in physical memory.

The regions of the address space

A process’s virtual address space is divided into regions by purpose: the text segment where machine code lives, the data segment for global variables, and then the heap and the stack.

The difference between the heap and the stack is lifetime and who manages them. The stack grows when a function is called and shrinks when it returns. You never have to clean it up, but values cannot remain there after the function returns. The heap is explicitly allocated and survives a function return. In exchange, someone must free it, or a mechanism such as Go’s garbage collector must reclaim it. In Go, the compiler decides which region holds a value. If a function returns the address of a local variable, the compiler places that variable on the heap.

Figure 3: Up and down represent higher and lower addresses; the stack grows from high addresses toward low, and the heap from low toward high.

Combine this with the registers from Chapter 2, and you can draw the full picture of a running process. SP points to the top of the stack in this figure, and PC points to the instruction currently executing, inside the text segment.

Figure 4: Arrows show where in memory each register points. The table on the right lists the memory regions covered in this section.

User space and kernel space

The CPU has privilege levels for execution, and application code runs on the restricted side. That side is called user space; the unrestricted side is kernel space. A user-space program cannot issue commands directly to the disk or the network card.

The kernel restricts hardware and mapping-table access to preserve process isolation. If any program could manipulate either one, it could bypass the boundaries between processes. By limiting the unrestricted side to the kernel, the OS keeps the illusion intact.

Figure 5: Arrows show the direction of requests. Solid lines are requests from the application to the kernel, dashed lines are results coming back, and red dashed lines mark what is not allowed.

System calls

When a program wants to access hardware, it asks the kernel through a system call. The kernel provides calls such as read to read a file, write to write one, and socket to open a communication endpoint. Go’s os.ReadFile ultimately invokes system calls too. The kernel takes the request, operates the hardware, and returns the result.

It looks like a function call, but what happens inside differs from an ordinary Go function call. An ordinary call simply jumps to the next instruction at the same privilege level. A system call switches over to the kernel side with a dedicated instruction, and control comes back after the kernel finishes the work. The switch makes it more expensive than an ordinary function call.

Linux’s strace command shows when control passes to the kernel by displaying the system calls a program issues. The following program reads one file and prints it.

Run on the Go Playground

package main

import (
	"fmt"
	"os"
)

func main() {
	data, err := os.ReadFile("/etc/hostname")
	if err != nil {
		panic(err)
	}
	fmt.Print(string(data))
}

Build this and run it under strace. The output includes every file the Go runtime reads at startup, so I filtered it down to just openat, read, and write, and kept only the tail.

$ strace -e trace=openat,read,write ./readfile
...
openat(AT_FDCWD, "/etc/hostname", O_RDONLY|O_CLOEXEC) = 4   ← open the file
read(4, "mymachine\n", 512)             = 10                ← read its contents
read(4, "", 502)                        = 0                 ← reached the end
write(1, "mymachine\n", 10)             = 10                ← write to the screen
+++ exited with 0 +++

The Go source contained only two calls: os.ReadFile and fmt.Print. Those functions invoked three kinds of system calls: open, read, and write. The = 4 is a number the kernel returned, and the subsequent reads refer to the file by that number.

eBPF programs run on the kernel side. Running user-written code inside the kernel would be dangerous without safeguards. Chapter 7 explains the risks and how eBPF controls them.

Threads

Inside one process, multiple flows of execution can run concurrently. Each of these flows is a thread.

Threads in the same process share the virtual address space. A value placed on the heap is visible from every thread. The stack alone is different: each thread has one dedicated stack of its own, because the pushing and popping of function calls is independent for each flow.

The kernel decides which thread gets the CPU and when; this assignment is called scheduling. Threads are the smallest units of execution that the kernel creates, numbers, and schedules. This limit becomes a problem later. As Chapter 6 shows, Go layers its own unit of execution (the goroutine) on top of threads, and goroutines are invisible to the kernel.

Figure 6: This figure has no arrows. One process contains multiple threads; each thread has its own stack, while the heap and the text segment are shared.

Key points for the hurdles

• A process has its own private virtual address space, and you cannot read another process’s memory from outside by specifying an address. That is why “observing from outside” needs special machinery.
• The stack is a region that grows and shrinks with function calls; the heap persists after a function returns.
• Each thread has exactly one dedicated stack; the heap is shared among threads.
• The unit of execution the kernel knows about stops at threads; beyond that, the kernel cannot see.

Exercises

1. If you write a function that returns a pointer to a local variable, what would happen if that variable stayed on the stack? How does the Go compiler solve this?
2. Why can’t os.ReadFile get by with an ordinary function call? Explain why it needs system calls using the vocabulary of this chapter.