Understanding OpenTelemetry's Consistent Probability Sampling

Originally published in Japanese at https://zenn.dev/ymotongpoo/articles/20260717-cps.

At SRE NEXT 2026, I gave a talk titled “Sampling Is Statistics,” about how deciding a sampling rate requires working backwards from the acceptable margin of error to the required sample size.

In that talk I touched on Consistent Probability Sampling as the mechanism underpinning consistent trace sampling, but time constraints kept me from going into how it actually works. This article digs into that part.

The limits of TraceIdRatioBased

When you want to reduce the sampling rate of distributed traces, the first candidate in OpenTelemetry is the TraceIdRatioBased sampler. Because it decides deterministically from the trace ID, it always returns the same result for the same trace ID. If that’s the case, then even if multiple services each configure this sampler, no single trace should ever end up with missing pieces.

However, version 1.0 of the Tracing specification carried a warning to the effect that this sampler is only safe to use on root spans¹. There was no guarantee that the decision algorithm was consistent across language implementations, and the behavior when services chose different sampling rates was undefined. In other words, as of spec 1.0, performing probability sampling independently at multiple points in a distributed system simply wasn’t possible.

Consistent Probability Sampling (CPS from here on) is the answer to this long-unsolved problem. More than four years after the Tracing spec 1.0 was published, the set of related specifications and OTEPs came together in 2025, and implementations reached a usable state². This article explains what CPS solves and how the mechanism solves it.

What multi-stage sampling breaks

Why would you want to sample independently at multiple points in the first place?

Because in real systems, traffic volume differs by orders of magnitude between components. For example, consider wanting to sample the frontend at 100%, the cache behind it at 1/1000, and storage at 1/10. You want to keep everything from the low-traffic frontend, and aggressively thin out the cache that gets called orders of magnitude more often. The requirement itself is perfectly natural.

But if each service makes its decision with an unrelated random number, two things break.

The first is the trace itself. If the decisions scatter — keep at service A, drop at service B, keep at service C — the trace fragments. The causal chain you want to see during root cause analysis gets severed partway through, and the trace loses its value. Preventing fragmentation requires the relationship “any trace the 1/1000 sampler decides to keep must also be kept by the 1/10 sampler” — that is, the set kept at the lower probability must be a subset of the set kept at the higher probability.

The second is aggregation. To estimate population quantities from sampled data, you need a weight (adjusted count) that says how many original items each surviving item represents. A span kept at sampling rate r has weight 1/r, and summing each sample with this weight gives an unbiased estimate of the population total³. But if the SDK thins to 50% and the Collector then thins further to 20%, the final retention probability is the product: 10%. In a setup where each stage thins independently, this final value is written in no stage’s configuration file. Unless it knows the configuration of every stage, the backend cannot reconstruct the weights.

flowchart LR
    A["10,000 traces"] -->|"SDK: 50% pass with an independent random number"| B["5,000 traces"]
    B -->|"Collector: 20% pass with a different independent random number"| C["1,000 traces<br>retention probability = 50% × 20% = 10%"]
    C -.->|"the final 10% appears in<br>no configuration file"| D["Backend<br>cannot reconstruct weights"]

Using the ParentBased sampler to propagate the parent’s decision to children prevents fragmentation. But then the decision is made once, at the root. Per-stage adjustments — thinning just the cache more aggressively, or trimming further at the Collector — are impossible.

To sum up, two things are needed: keeping traces intact even when each stage chooses a different probability, and conveying the final retention probability downstream so they can be counted correctly. CPS achieves both with a single mechanism.

Comparing the same random number against thresholds

The CPS mechanism boils down to two 56-bit values.

• Random value rv: a random number determined once per trace. As a rule it’s the low 7 bytes (56 bits) of the trace ID, used as-is, and it never changes at any stage
• Threshold th: a rejection threshold in the 56-bit space. Only traces satisfying rv >= th are kept

The relationship between retention probability $p$ and the threshold is:

$$ th = (1 - p) \times 2^{56} $$

If $th = 0$, every trace satisfies the condition, so retention is 100%; the larger $th$ gets, the wider the rejection region and the lower the retention probability.

The crucial point is that no stage’s sampler re-rolls the random number. Everyone looks at the same rv and merely compares it against their own threshold. As a result, any trace kept by a stricter (larger) threshold automatically satisfies the condition of a looser threshold. The subset relationship we said was needed earlier follows directly from the decision method itself.

So what is the final retention probability of a trace that passes through multiple stages?

With independent random numbers, passing 50% and then 25% gave a retention probability of 12.5%. CPS is different. The decision condition is “rv is at or above both thresholds,” which is the same as “rv is at or above the larger threshold.” Thresholds compose by max, not by multiplication: passing 50% (th:8) and then 25% (th:c) yields a final retention probability of 25%.

flowchart LR
    A["10,000 traces<br>share a common rv"] -->|"SDK: compare against th:8 (50%)"| B["5,000 traces<br>satisfying rv >= th:8"]
    B -->|"Collector: compare the same rv against th:c (25%)"| C["2,500 traces<br>also satisfying rv >= th:c<br>retention probability = max(50%, 25%) = 25%"]

This property radically simplifies the aggregation problem. Since the final probability is determined solely by the strictest threshold, the backend doesn’t need to multiply out the history of every stage. Reading the last $th$ recorded on the span, the weight can be reconstructed with:

$$ \text{adjusted_count} = \frac{2^{56}}{2^{56} - th} $$

It also matters that the stages never need to know each other’s configuration. The SDK writes the threshold it used into the data. The Collector reads the incoming threshold and, if necessary, raises it and writes it back. The retention probability up to the previous stage travels with the data, not through shared configuration.

Representation in tracestate

So where are th and rv written?

In the W3C Trace Context tracestate header. OpenTelemetry places the values as colon-delimited subkeys under the key ot⁴.

tracestate: ot=th:c;rv:6e6d1a75832a2f

th is 1 to 14 lowercase hex digits, with trailing zeros omittable. It’s zero-padded to 14 digits and read as a 56-bit integer, so th:8 is 0x80000000000000 — the upper half is rejected, giving a retention probability of 50%. Likewise th:c represents 25% and th:0 represents 100%. For power-of-two probabilities, one or two characters suffice — a compact representation.

You might wonder why a hex threshold rather than a decimal probability. It’s because rv is a literal substring of the trace ID’s hex representation. With both in hex, they can be compared as strings without conversion, and the retention probability can be recovered from th reversibly, without rounding error. The design prioritizes simplicity of the decision and reversibility of the probability.

rv, on the other hand, actually doesn’t appear in tracestate in most cases. If the low 56 bits of the trace ID can be trusted to be random, they can be used directly as rv. The problem is that W3C Trace Context only requires uniqueness of trace IDs, not randomness. IDs that embed timestamps, or 64-bit IDs zero-padded to size, do exist in the wild. That’s why W3C Trace Context Level 2 added a traceparent flag declaring “the low 7 bytes are random” (Random Trace ID Flag). Only when randomness cannot be guaranteed — for instance, when this flag is not set — does the root generate a random number and propagate it as an explicit rv in tracestate. In other words, the explicit rv is a fallback for compatibility with ecosystems whose trace IDs aren’t random.

Where the spec and ecosystem stand

CPS was specified through four OTEPs: propagation of sampling information (OTEP 0168), the definition of sampling probability (OTEP 0170), recording the threshold in tracestate (OTEP 0235), and composite samplers (OTEP 0250). These crystallized into specifications such as Trace Probability Sampling, while the prerequisite trace ID randomness was taken up by W3C Trace Context Level 2.

Collector-side implementation is progressing too. The probabilisticsampler processor in collector-contrib complies with OTEP 0235 and reads and rewrites incoming th values. It has an equalizing mode that pins the final probability to the configured value, and a proportional mode that recomputes the threshold so the resulting probability is the incoming probability multiplied by the configured value. For data arriving from CPS-unaware upstream stages, it takes care not to break existing decisions by synthesizing an rv-equivalent value from a hash.

Caveats when adopting it

CPS only works when adopted across the entire pipeline. If any stage in the middle ignores the threshold and thins with an independent random number, both the subset relationship and the correctness of the final th collapse. You should take stock of where sampling happens — SDK, Collector, backend — before adopting it.

Another point: the situations where the weights matter are limited. adjusted_count is needed only when asking quantitative questions of stored traces, like “how many times did this failure pattern actually occur?” — reading individual traces one by one for investigation needs no weights. And for SLIs like error rates and latency distributions, it’s more reliable to generate them from the full span stream upstream of sampling, using something like the spanmetrics connector. CPS’s weight reconstruction is best positioned as insurance for answering unanticipated quantitative questions that couldn’t be baked into metrics in advance.

Conclusion

Back to the warning from the beginning. TraceIdRatioBased was unsafe anywhere but the root because there was no random number the stages could share, and no common yardstick against which decisions could be compared. CPS provides both: the 56-bit random value rv derived from the trace ID is shared by every stage, and the rejection threshold th is handed down through tracestate. As a result, traces don’t fragment even when each stage chooses a different probability, and each trace’s weight can be reconstructed from the single final th.

Sampling discards data, and at the same time it changes the meaning of the data that remains. Each surviving item is one that represents some number of items in the population, and without metadata conveying that representativeness downstream, correct counting is impossible. CPS is the standard way to carry that metadata.

The remaining question is how far backends and query layers will actually make use of this th. The specifications and the propagation machinery are in place. Whether weighted aggregation over stored traces becomes the norm now depends on each implementation going forward.