Hurdle 4: Context Propagation Across Processes
Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-ebpf-primer/viewer/55-hurdle4_propagation.
The first three hurdles stay within one process. Context propagation crosses the process boundary, and a failure splits one operation into unrelated traces at a service boundary.
As we saw in Chapter 1, the work done in service A and the work done in service B become one trace by sharing the same trace ID. Normally this happens by attaching a traceparent header to the HTTP request, passing the trace ID and the sender’s span ID downstream. The downstream service continues the work in a new span, with that span ID as its parent.
The one line in SDK instrumentation
The instrumentation an ordinary Go developer writes takes just a few lines. With the OpenTelemetry SDK, you only swap out the HTTP client’s Transport.
client := &http.Client{
Transport: otelhttp.NewTransport(http.DefaultTransport),
}
The transport eventually calls this line.
otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))
It formats the trace ID and span ID held by ctx as a traceparent value and writes them into req.Header. The application can do this because it has direct access to both ctx and req.
Zero-code instrumentation must supply the same header from outside an application that has no traceparent logic. OBI has to reconstruct the trace context and write the resulting header into the outgoing request.
Figure 1: The arrows show who writes the header. With SDK instrumentation the app writes it itself; with zero-code instrumentation, OBI on the outside writes it instead.
Tracking across goroutines
Even within one process, work flows across goroutines. It’s common for the goroutine that receives a request and the goroutine that sends the downstream request to be different ones. Passing context.Context along as an argument is the Go way of handling this.
Hurdle 2 showed how to read one argument from a register when a uprobe fires. Tracking ctx across function calls and goroutines requires more than that snapshot.
Reading a register retrieves one value from a fixed location at the instant a uprobe fires. Using ctx, however, requires following the same unit of work as the context passes between functions and goroutines. Because ctx is an interface value, the register contains a reference to an object elsewhere rather than the context’s contents. Its internal structure changes as context.WithValue calls are nested, and its argument position differs between functions. Reading one pointer once is not enough to track it.
Chapter 6 showed that the kernel’s knowledge stops at threads. A thread ID cannot identify the work: goroutines from different requests share OS threads, and one request can hand work to several goroutines.
OBI therefore avoids decoding ctx and instead uses goroutine creation relationships as a proxy.
Figure 2: The arrows show what OBI can and cannot do. The dashed line ending in a circle is the one it cannot. The dotted lines list the reasons; they are not a flow of execution.
OBI hooks goroutine creation. The goroutine that executes go f() is the parent, and the new goroutine is the child. The runtime reaches runtime.newproc1, whose simplified signature looks like this.
// Creates and returns the child goroutine. callergp is the parent's g
func newproc1(fn *funcval, callergp *g, callerpc uintptr, parked bool, waitreason waitReason) *g
The second argument identifies the parent, and the return value identifies the child after the function finishes. OBI captures both entry and exit to form the pair.
Entry: note down the parent (the child doesn't exist yet)
Exit: pair the noted parent with the returned child, and record the pair
The following excerpt implements those two operations.
SEC("uprobe/runtime_newproc1")
int obi_uprobe_runtime_newproc1(struct pt_regs *ctx) {
void *creator_goroutine_addr = GOROUTINE_PTR(ctx);
new_func_invocation_t invocation = {.parent = (u64)GO_PARAM2(ctx)};
go_addr_key_t g_key = {};
go_addr_key_from_id(&g_key, creator_goroutine_addr);
// Save the registers on invocation to be able to fetch the arguments at return of newproc1
if (bpf_map_update_elem(&newproc1, &g_key, &invocation, BPF_ANY)) {
bpf_dbg_printk("can't update map element");
}
return 0;
}
At entry, OBI stores the parent from the second argument (GO_PARAM2, or BX) in a temporary map called newproc1. The creator variable identifies the goroutine executing newproc1. OBI uses it as the key for retrieving the stored parent at exit.
// The exit hook. Skeleton only
int obi_uprobe_runtime_newproc1_return(struct pt_regs *ctx) {
void *creator_goroutine_addr = GOROUTINE_PTR(ctx); // key to match up with the entry
void *goroutine_addr = (void *)GO_PARAM1(ctx); // return value: the address of the child's g
// Retrieve the parent noted down at the entry
new_func_invocation_t *invocation = bpf_map_lookup_elem(&newproc1, &c_key);
void *parent_goroutine = (void *)invocation->parent;
// Record "child -> parent" in the ongoing_goroutines map
goroutine_metadata metadata = {.timestamp = bpf_ktime_get_ns(), .parent = p_key};
bpf_map_update_elem(&ongoing_goroutines, &g_key, &metadata, BPF_ANY);
return 0;
}
At the exit, the return value (GO_PARAM1, that is, AX) holds the address of the child’s g. It gets paired with the parent noted at the entry, and the “child → parent” mapping is recorded in the ongoing_goroutines map. Now OBI can trace that “this goroutine is a child of the goroutine handling that request.”
Figure 3: The arrows show the order in time. At the entry only the parent is known, so it is noted down temporarily; at the exit, once the child’s address is known, the pair is recorded as “child → parent”.
The full implementation (including PID key construction, cycle avoidance, and stale-entry deletion)
SEC("uprobe/runtime_newproc1_return")
int obi_uprobe_runtime_newproc1_return(struct pt_regs *ctx) {
bpf_dbg_printk("=== uprobe/runtime_newproc1_return ===");
void *creator_goroutine_addr = GOROUTINE_PTR(ctx);
const u64 pid_tid = bpf_get_current_pid_tgid();
const u32 pid = pid_from_pid_tgid(pid_tid);
go_addr_key_t c_key = {.addr = (u64)creator_goroutine_addr, .pid = pid};
// The result of newproc1 is the new goroutine
void *goroutine_addr = (void *)GO_PARAM1(ctx);
go_addr_key_t g_key = {.addr = (u64)goroutine_addr, .pid = pid};
// Lookup the newproc1 invocation metadata
new_func_invocation_t *invocation = bpf_map_lookup_elem(&newproc1, &c_key);
if (invocation == NULL) {
bpf_dbg_printk("can't read newproc1 invocation metadata");
goto done;
}
// The parent goroutine is the second argument of newproc1
void *parent_goroutine = (void *)invocation->parent;
go_addr_key_t p_key = {.addr = (u64)parent_goroutine, .pid = pid};
goroutine_metadata *g_metadata =
(goroutine_metadata *)bpf_map_lookup_elem(&ongoing_goroutines, &p_key);
if (g_metadata) {
// Don't create cycles at one level on immediate goroutine reuse
if (g_metadata->parent.addr == (u64)goroutine_addr) {
bpf_dbg_printk("avoiding cycle %llx -> %llx", parent_goroutine, goroutine_addr);
goto done;
}
}
goroutine_metadata metadata = {
.timestamp = bpf_ktime_get_ns(),
.parent = p_key,
};
if (bpf_map_update_elem(&ongoing_goroutines, &g_key, &metadata, BPF_ANY)) {
bpf_dbg_printk("can't update active goroutine");
}
done:
// Delete any stale info on go_trace_map
bpf_map_delete_elem(&go_trace_map, &g_key);
bpf_map_delete_elem(&newproc1, &c_key);
return 0;
}
The full implementation also prevents a cycle when the runtime reuses a g address.
// Don't create cycles at one level on immediate goroutine reuse
if (g_metadata->parent.addr == (u64)goroutine_addr) {
bpf_dbg_printk("avoiding cycle %llx -> %llx", parent_goroutine, goroutine_addr);
goto done;
}
This is the cost of the decision in Hurdle 2 to use the address of the g struct as the goroutine identifier. When a goroutine exits and the runtime reuses its g, the same address identifies a different goroutine. An address previously recorded as a parent can later appear as a child. Without this check, the parent-child relationships could form a cycle and make the ancestor search in the next section loop forever.
For the same reason, this function deletes stale entries at the end.
done:
// Delete any stale info on go_trace_map
bpf_map_delete_elem(&go_trace_map, &g_key);
bpf_map_delete_elem(&newproc1, &c_key);
This keeps a reused address from carrying its previous owner’s information.
Using addresses avoids version-specific field tracking and makes OBI responsible for deleting state when those addresses are reused. Either choice carries a cost.
The _return suffix does not indicate a uretprobe. OBI disassembles runtime.newproc1, enumerates its RET instructions, and places an ordinary uprobe on each one, applying the Hurdle 1 workaround to this entry-and-exit pair.
Searching up to 6 ancestor levels
The hook that sends the downstream request uses the recorded parent-child relationships. When net/http.(*Transport).roundTrip or a gRPC client’s entry point fires, OBI searches from the current goroutine toward its ancestors, one parent at a time. It looks for the server-side receive where the work began; once it finds that receive, it can carry over the trace ID.
The search looks for an entry in go_trace_map. OBI also hooks server-side receives: at the entry of net/http.serverHandler.ServeHTTP for HTTP, or google.golang.org/grpc.(*Server).handleStream for gRPC, it writes the incoming request’s trace information into this map, keyed by the goroutine handling it. The receive side only writes the entry; the send side performs the ancestor search.
The code checks one level at a time: “does this goroutine have trace information?” It starts with the current goroutine, so if the receive and send happen on the same goroutine, the first lookup succeeds. Otherwise, it finds the parent in the parent-child map and repeats.
The ancestor-search code (find_parent_goroutine)
static __always_inline u64 find_parent_goroutine(go_addr_key_t *current) {
// ...
int attempts = 0;
do {
tp_info_t *p_inv = bpf_map_lookup_elem(&go_trace_map, parent);
if (!p_inv) { // not this goroutine running the server request processing
// Let's find the parent scope
goroutine_metadata *g_metadata =
(goroutine_metadata *)bpf_map_lookup_elem(&ongoing_goroutines, parent);
if (g_metadata) {
// Lookup now to see if the parent was a request
// Debug here commented out on purpose to avoid prints in loops.
// bpf_printk("lookup %llx -> %llx", r_addr, g_metadata->parent.addr);
r_addr = g_metadata->parent.addr;
parent = &g_metadata->parent;
} else {
break;
}
} else {
bpf_dbg_printk("Found parent, r_addr=%lx", r_addr);
return r_addr;
}
attempts++;
// We loop far back because some clients, e.g. Kafka Franz-Go really nest the
// client calls.
} while (attempts < 6); // Up to 6 levels of goroutine nesting allowed
return 0;
}
The source comments out bpf_printk inside the loop to avoid repeated debug-print overhead. Even one debug line’s overhead matters here.
The fixed upper bound is not a shortcut in the implementation. Chapter 7 showed that the verifier refuses to load a program whose loops lack an upper bound. OBI must express the search as “walk at most this many times” rather than “walk until you find the parent.”
The verifier does not require the specific value 6. OBI’s maintainers chose it to balance instruction count against the nesting depth needed by libraries such as the franz-go Kafka client, named in the source comment.
If the ancestor with trace information is more than 6 levels away, find_parent_goroutine returns 0. The send is still instrumented, but client_trace_parent generates a new random trace ID. The downstream request becomes a separate trace disconnected from the upstream one. This can be more misleading than a missing trace because one flow appears as two.
Figure 4: The arrows show the direction of following child-to-parent references. The verifier requires an upper bound, but the value 6 is an engineering choice. If the search reaches the limit, the send gets a fresh trace ID and becomes a separate trace.
Injecting the header into the outgoing request
After finding the trace context, OBI has to add it to the outgoing HTTP request.
Writing the Header field of http.Request would require manipulating the internals of a map[string][]string: computing the key hash, finding a bucket, and perhaps growing the storage. OBI avoids that unstable operation.
OBI writes just before net/http serializes the request into a byte string. The standard library can expose an HTTP/1.1 request at that stage.
package main
import (
"fmt"
"net/http"
"net/http/httputil"
)
func main() {
req, _ := http.NewRequest("GET", "http://service-b/items", nil)
req.Header.Set("Traceparent", "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01")
dump, _ := httputil.DumpRequestOut(req, false)
fmt.Printf("%q\n", dump)
}
"GET /items HTTP/1.1\r\nHost: service-b\r\nUser-Agent: Go-http-client/1.1\r\nTraceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01\r\nAccept-Encoding: gzip\r\n\r\n"
An HTTP/1.1 request is a byte string with one header per line, \r\n separators, and a blank line after the headers. SDK and zero-code instrumentation both need to insert one Traceparent: line into that string.
When net/http writes the headers, it passes through Header.writeSubset, which accumulates strings like the one above in a downstream bufio.Writer buffer. A bufio.Writer collects writes: buf stores the bytes, and n records how many bytes are in use. OBI hooks both the entry and return of this function. On return, it appends one line to the accumulated string. The probe registration lives on the Go side, in pkg/internal/ebpf/gotracer/gotracer.go.
Figure 5: The vertical arrows show the write-out path; the arrows from OBI show the write targets, and the dashed arrow with a blocked tip points at the target that cannot be written. The http.Header map cannot be written from outside, so OBI writes Traceparent into the bufio.Writer buffer just before serialization and advances n.
if p.headerPropagationEnabled() {
m["net/http.Header.writeSubset"] = []*ebpfcommon.ProbeDesc{{
Start: p.bpfObjects.ObiUprobeWriteSubset, // http 1.x context propagation
End: p.bpfObjects.ObiUprobeWriteSubsetReturns, // inject only if no traceparent present
}}
m["golang.org/x/net/http2.(*Framer).WriteHeaders"] = []*ebpfcommon.ProbeDesc{
{ // http2 context propagation
Start: p.bpfObjects.ObiUprobeGolangHttp2FramerWriteHeaders,
End: p.bpfObjects.ObiUprobeHttp2FramerWriteHeadersReturns,
},
The key names the symbol to instrument; Start identifies the entry eBPF program and End the exit program. Specifying End triggers the Hurdle 1 machinery, which enumerates every RET and places a uprobe on each one. The comment inject only if no traceparent present describes the guard against adding a second header when SDK instrumentation has already supplied one.
The return hook writes at the end of the bufio.Writer buffer.
unsigned char buf[k_traceparent_len];
make_tp_string(buf, &inv->tp);
if (len <
(size - TP_MAX_VAL_LENGTH - TP_MAX_KEY_LENGTH - 4)) { // 4 = strlen(":_")+strlen("\r\n")
char key[TP_MAX_KEY_LENGTH + 2] = "Traceparent: ";
char end[2] = "\r\n";
bpf_probe_write_user(buf_ptr + (len & 0x0ffff), key, sizeof(key));
len += TP_MAX_KEY_LENGTH + 2;
bpf_probe_write_user(buf_ptr + (len & 0x0ffff), buf, sizeof(buf));
len += TP_MAX_VAL_LENGTH;
bpf_probe_write_user(buf_ptr + (len & 0x0ffff), end, sizeof(end));
len += 2;
bpf_probe_write_user((void *)(io_writer_addr + io_writer_n_pos), &len, sizeof(len));
bpf_probe_write_user rewrites the target process’s user-space memory. Three calls append Traceparent: , the value, and \r\n; the fourth updates the bufio.Writer’s n. Without that update, the appended bytes would remain outside the used range and never be sent. The repeated (len & 0x0ffff) expression proves to the verifier that each index stays within a bound. The verifier checks index expressions as well as loop bounds.
Hurdle 3’s offsets.json contains buf, n, and wr because OBI rewrites these unexported fields of bufio.Writer. The variable io_writer_n_pos holds the offset of n.
Figure 6: The arrows show the direction of writes and the direction the bytes leave. Path 1 writes into a buffer in the app’s memory; path 2 inserts into the byte stream on its way out to the socket. Either way, what gets added is the same single line.
Environments that forbid the write, and the second path
bpf_probe_write_user collides with the OS’s security mechanisms. OBI’s support matrix says the following.
On Linux 5.10 and later, OBI requires effective
CAP_SYS_ADMINand kernel lockdown mode[none]to usebpf_probe_write_user.
Kernel lockdown and Secure Boot can make this helper unavailable. The flag g_bpf_probe_write_user_enabled controls whether OBI uses this path.
OBI can propagate context through a second path when user-space writes are unavailable. A nearby source comment describes the alternatives.
// For Go we support two types of HTTP context propagation for now.
// 1. The one that this code does, which uses the locked down bpf_probe_write_user.
// 2. By using a sock_msg program that will extend the packet.
// If this code ran, we should ensure that the second part doesn't run, therefore
// we remove the metadata setup in uprobe_persistConnRoundTrip(struct pt_regs *ctx), so
// that approach 2. skips this packet.
The second path uses an sk_msg program at the point where the kernel pushes data toward the socket. It extends the outgoing byte stream with a header without touching application memory.
The unit being handled here is the byte stream that TCP carries. The kernel decides where to cut it into packets, so there’s no guarantee that “one HTTP request equals one packet.” What sk_msg inserts into is the byte stream before those cuts are decided.
After the first path runs, OBI removes the map registration so the second path skips that send. This arbitration prevents the two paths from inserting the same header twice.
Context propagation is disabled by default. OTEL_EBPF_BPF_CONTEXT_PROPAGATION accepts headers, tcp, or all when the user chooses to enable memory or byte-stream writes. Because this feature rewrites process memory or the outgoing byte stream, the user decides whether to enable it.
OBI records parent-child relationships at goroutine creation, walks up to 6 ancestors when sending, and writes the trace ID into either the buffer just before serialization or the outgoing socket byte stream.
Figure 7: Solid arrows show the flow of processing; dashed lines show what gets disabled. The figure shows goroutine parent-child tracking (within a process) and traceparent injection (between processes). The tracking goes at most 6 levels; the injection has two paths, and when the kernel’s security mechanisms are active, only path 1 drops out.