> Source: https://www.ymotongpoo.com/blog/20260916-beyla-obi-diff/ :::message Test environment - Host: Ubuntu, kernel 7.0.0-31-generic, with `/sys/kernel/btf/vmlinux` - Instrumented target: `nginx:1.27-alpine` (port 80 exposed to the host) - Agents: `otel/ebpf-instrument:v0.13.0` (released 2026-09-04), `grafana/beyla:3.35.0` (released 2026-09-09, OBI submodule `861d907`) - Startup method: `docker run --privileged --pid=host --network=host`, with the Prometheus endpoint scraped by `curl` ::: ## Introduction I gave a presentation about [OpenTelemetry eBPF Instrumentation](https://opentelemetry.io/ja/docs/zero-code/obi/) (OBI) at the recent Go Conference and published the details in a [Zenn book](https://zenn.dev/ymotongpoo/books/go-ebpf-primer). I did not cover much about the fact that OBI originated from Beyla, which Grafana Labs developed. [Grafana Beyla](https://github.com/grafana/beyla) was [donated](https://grafana.com/blog/opentelemetry-ebpf-instrumentation-beyla-donation/) to OpenTelemetry in 2025. The CNCF community now develops it under the name OpenTelemetry eBPF Instrumentation. Beyla did not disappear. It remains as a downstream distribution of OBI. What changes when you replace one with the other today? I ran the latest OBI and Beyla container images available when I wrote this article on my Linux machine under the same conditions. I compared their output metrics and their configuration handling. ## TL;DR The telemetry core, including HTTP server metrics, route inference, protocol analysis, and eBPF probes, all comes from OBI and does not differ. The differences center on metric name prefixes, output paths for connecting to Grafana products, several extra features, and configuration file generations. ## Repository relationship The Beyla repository includes the OBI repository as the Git submodule `.obi-src` and replaces it in `go.mod` with `replace go.opentelemetry.io/obi => ./.obi-src`. Beyla is therefore a thin wrapper that vendors OBI as a library. The code volume reflects this relationship. | Item | OBI | Beyla | Difference | | --- | --- | --- | --- | | Go lines of code (excluding tests, vendor, and generated code) | 153,939 | 10,275 | Beyla is 6.7% | | Binary size (bytes) | 127,759,366 | 129,898,523 | 1.7% increase | Beyla 3.35.0 pins OBI to commit `861d907`, which is two commits after OBI tag `v0.13.0`. OBI always releases upstream. Most Beyla release notes describe OBI updates such as “Update OBI submodule to ...”. The Go packages that exist only on the Beyla side show the scope of its extra features. | Package | Role | | --- | --- | | `pkg/webhook` | A Kubernetes mutating webhook that injects the OpenTelemetry SDK into Pods (experimental feature) | | `pkg/export/otel` | OTLP configuration for Grafana Cloud, a separate output path for GenAI spans, and survey metrics | | `cmd` | Entry points for `beyla`, `beyla-schema`, and `k8s-cache` | | `pkg/internal/infraolly` | Collection of process CPU, memory, disk, and network metrics | | `pkg/beyla` | Conversion between Beyla configuration types and OBI configuration types | | `pkg/export/prom` | Prometheus output for process metrics | | `pkg/services` | Beyla-specific exclusion rules and survey selection conditions | | `pkg/export/alloy` | Integration with the Grafana Alloy trace receiver | ## Differences in metric names and identifiers I enabled only `features: [application]` for the same nginx instance and compared the metrics in Prometheus format. Metrics that follow semantic conventions have identical names. Only custom metrics outside those conventions have different prefixes. | Metric or attribute | OBI | Beyla | | --- | --- | --- | | HTTP server metrics | `http_server_request_duration_seconds` and others | Same | | Build information | `obi_build_info` | `beyla_build_info` | | Network flow | `obi_network_flow_bytes_total` | `beyla_network_flow_bytes_total` | | Flow attribute | `obi.ip` | `beyla.ip` | | `source` in `target_info` | `obi` | `beyla` | | `telemetry_sdk_name` in `target_info` | `opentelemetry` | `beyla` | | `telemetry_distro_version` in `target_info` | `v0.13.0` | `unset` | OBI exposes the prefixes through variables such as `attr.VendorPrefix`, so external code can replace them. Beyla overrides the prefix with `beyla` at startup [^version-info]. [^version-info]: The last difference does not seem intentional. In Beyla, `telemetry_distro_version` becomes `unset`, while `beyla_build_info` correctly contains `version=\"v3.35.0\"`. Build-time version embedding itself works. OBI copies `buildinfo.Version` into `TelemetryDistroVersion` when it initializes the package variable. Beyla overwrites that value later, inside `OverrideOBIGlobalConfig`. The copied initial value, `unset`, therefore remains. If your dashboards or alerts use this attribute, treat it as unset in Beyla. A [fix](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/3434) for this issue has been merged into OBI itself, but it was not included in the released OBI or in the OBI that Beyla included when I wrote this article. ## Features unique to Beyla ### Process metrics The `application_process` feature exists only in Beyla. When I passed the same value to OBI, OBI rejected startup and listed the valid feature names. ``` $ docker run --rm ... -e OTEL_EBPF_PROMETHEUS_FEATURES=application,application_process \ otel/ebpf-instrument:v0.13.0 level=ERROR msg="wrong configuration" error="... unknown metrics feature \"application_process\" (valid features: all, application, application_host, application_runtime, application_service_graph, application_span, application_span_otel, application_span_sizes, ebpf, network, network_flow_packets, network_inter_zone, stats, stats_tcp_failed_connections, stats_tcp_io, stats_tcp_retransmits, stats_tcp_rtt)" ``` When I enabled it in Beyla, six additional metrics appeared for each instrumented process. ``` process_cpu_time_seconds_total process_cpu_utilization_ratio process_disk_io_bytes_total process_memory_usage_bytes process_memory_virtual_bytes process_network_io_bytes_total ``` As their names indicate, these are operating system process metrics, not values collected by eBPF. Their scope covers only the PIDs of selected instrumented services, unlike an agent that covers the entire host. Take care with `process_network_io_bytes_total`. The source, `/proc//net/dev`, reports interface statistics per network namespace. Multiple processes in the same namespace therefore report the same value. ### Survey mode `discovery.survey` discovers processes and identifies their languages without instrumenting them. It lists the targets through the `survey_info` metric. The intended use is to provide an external auto-instrumentation system with an inventory of instrumentable targets. When I gave both tools the same YAML, Beyla produced `survey_info`, while OBI ignored this key and performed normal instrumentation. ``` # Beyla survey_info{...,job="nginx",service_name="nginx",source="beyla",...} 1 ``` ### Output paths for Grafana products Beyla provides a `grafana.otlp` section. You can write `cloud_zone`, `cloud_instance_id`, and `cloud_api_key`, and Beyla builds the Grafana Cloud OTLP endpoint and headers. OBI is published as a general-purpose eBPF instrumentation tool. You must configure its OTLP endpoint and authentication through the standard method: specify the OTLP endpoint URL and pass the constructed headers in the configuration. Beyla also includes a receiver integration that sends spans directly to [Grafana Alloy](https://grafana.com/docs/alloy/latest/) (`pkg/export/alloy`), connection spans for building a service graph across clusters on the [Tempo](https://grafana.com/docs/tempo/latest/) side (`BEYLA_TOPOLOGY_SPANS=inter_cluster`), and a path that extracts only GenAI spans and sends them to another OTLP destination (`BEYLA_GRAFANA_AI_*`). Each feature is implemented as an additional node that subscribes to the output queue of the OBI pipeline. None of them modifies OBI itself. ### SDK injection for Kubernetes `pkg/webhook` is the largest Beyla-specific package. It receives Pod creation requests as a mutating webhook, identifies the language, and injects the OpenTelemetry SDK. Beyla's own documentation does not describe it. The code explicitly labels it as experimental and says that it may be removed in the future. For now, it serves more as a way to study how Grafana plans to combine eBPF and SDK instrumentation than as a feature to evaluate for production use. ## Differences in defaults The same input can produce different results. One difference is the mode for inferring routes from HTTP paths. OBI defaults to `heuristic`. Beyla defaults to `low-cardinality`. In the latter mode, Beyla stores inferred routes in a tree for each process. When the number of segment types at the same level exceeds the default of 10, it replaces that level with a wildcard. I sent 26 types of `/shop/` requests and counted the `http_route` values that appeared in `http_server_request_duration_seconds_count`. ``` # OBI: The 26 types remain as they are /shop/alpha /shop/bravo /shop/charlie ... /shop/zulu # Beyla: Processes above the limit group them under /shop/* /shop/* /shop/alpha /shop/bravo ... ``` Both tools correctly grouped `/users/12345/orders/98` as `/users/*/orders/*`. They share the heuristic that turns numeric or hash-like segments into wildcards. The only difference is whether they stop when unknown string segments keep increasing. Beyla's default is safer in environments that must control cardinality. To get the same behavior in OBI, set `routes.unmatch: low-cardinality` explicitly. The default exclusion patterns for avoiding self-instrumentation also differ. OBI excludes `obi` and `otelcol*`. Beyla adds Grafana process names (`*beyla`, `*alloy`, and `*prometheus-config-reloader`), Kubernetes namespaces (`grafana-alloy`, `monitoring`, and others), and container names. The environment variable prefixes differ as well. OBI uses `OTEL_EBPF_*`, while Beyla uses `BEYLA_*`. Beyla also creates a corresponding variable that translates each variable beginning with `BEYLA_` to the same name beginning with `OTEL_EBPF_`. You can therefore use the variable names in the upstream documentation unchanged. ## Configuration file versions The largest current difference is the configuration file version. OBI has read Config v2, which follows OpenTelemetry declarative configuration and contains `file_format` and `extensions.obi`, since v0.11.0. It also includes subcommands for validation and migration. ``` $ docker run --rm -v /tmp/v2.yaml:/cfg.yaml otel/ebpf-instrument:v0.13.0 config validate /cfg.yaml configuration is valid $ docker run -d ... -e OTEL_EBPF_CONFIG_PATH=/cfg.yaml otel/ebpf-instrument:v0.13.0 level=INFO msg="configuration loaded" version=v2 ``` This is the v2 configuration that I tested. ```yaml file_format: "1.0" meter_provider: readers: - pull: exporter: prometheus/development: port: 9414 extensions: obi: version: "2.0" capture: policy: default_action: exclude rules: - action: include match: process: open_ports: "80" ``` When I gave the same file to Beyla, it tried to interpret it as v1 and failed. ``` $ docker run --rm ... -e BEYLA_CONFIG_PATH=/cfg.yaml grafana/beyla:3.35.0 level=ERROR msg="wrong Beyla configuration" error="missing application discovery section or network metrics configuration." ``` You can confirm the reason in Beyla's `cmd/beyla/main.go`. The v2 loader is in OBI's `internal/config/{schema,convert}`. Go's internal package rules cannot be bypassed by a local `replace`, so Beyla can read only v1 until OBI exposes a public loader under `pkg/`. Beyla also has no subcommand such as `beyla config validate`. ## OBI as an embedded component OBI has a `collector/` package that exposes a receiver factory for the `obi` component type. It can receive traces and metrics as part of an OpenTelemetry Collector pipeline. This lets it run as part of the Collector instead of as an independent agent process. Beyla's corresponding path is its Alloy receiver integration. The two projects also handle the telemetry contract differently. OBI publishes the metrics and spans that it emits as a Weaver-compatible schema registry under `site/schemas/obi/`. It places that URL in the resource's `schema_url`. The project records renames and removals as schema transformations. If you need to track changes to attribute or metric names mechanically, OBI is easier to manage because it provides this system. OBI also documents its stability policy. Its `VERSIONING.md` classifies the current state as Development. It says that a minor release within `v0` may introduce breaking changes to configuration, defaults, behavior, and output telemetry. It also says not to treat `latest` as a stable tag. The 2026 goals include 1.0, expanded protocol support, .NET support, and use alongside SDKs. ## Which one should you choose? The instrumentation engine is the same. Choose based on the destination and operational constraints. If you connect to Grafana Cloud, Alloy, or Tempo service graphs, Beyla is the natural choice because it provides the configuration paths and process metrics for those products. Use OBI directly when you want to embed it as an OpenTelemetry Collector receiver, adopt declarative configuration (Config v2), or track output changes through telemetry schemas. Regardless of your choice, report and fix bugs in the OBI repository. Beyla's README also asks contributors to send non-documentation pull requests upstream. When you investigate a problem found in Beyla, first check the commit that `.obi-src` points to, then search for the issue upstream. ## Conclusion Beyla is a distribution that includes OBI as a library and adds output paths and several extra features for connecting to Grafana products. OBI provides the instrumentation core. Beyla's additions remain around that core: connections to Grafana Cloud and Alloy, process metrics, survey mode, and SDK injection for Kubernetes. OBI is upstream, and Beyla follows it by updating the submodule. OBI is moving toward declarative configuration and expanded protocol support as it aims for 1.0 in 2026. Beyla still has areas that have not caught up, such as a configuration loader that reads only v1. For now, use OBI when you care about eBPF instrumentation itself. Choose Beyla when you combine it with Grafana products.