Telemetry for AI Workloads

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/observability-platform-with-otel/viewer/60-ai_workload_telemetry.

Suppose that a team adds a summarization feature that uses an LLM to its product. A month later, accounting asks for a breakdown of the API charges by feature, and SRE asks for a latency SLO for the summarization feature. The existing dashboards show HTTP metrics, but nothing records token usage or latency per model.

You can observe AI workloads with the same SDK distribution, Collector, and semantic conventions. AI workloads record different information from traditional web services, though, so you extend the conventions and the instrumentation.

What to observe in AI workloads

Workloads that use LLMs have properties that traditional web services do not have.

• The output is nondeterministic. The same input can give a different result, so replay tests alone cannot guarantee that the system “works correctly.” You need to keep monitoring the quality through production telemetry
• The unit of cost is the token. The number of input and output tokens sets the bill, not the number of requests. Cost management therefore depends on measuring token usage
• LLM calls dominate the latency. Calls of several seconds to tens of seconds mix with operations that take tens of milliseconds. For streaming responses, a new indicator becomes meaningful: the “time to the first token”
• The error rate does not measure quality. A summary that misses the point but comes back with HTTP 200 is invisible to the traditional golden signals

To instrument these properties, define how to record the model, the token counts, the processing stages, the quality evaluations, and similar data. The registry from Chapter 6 manages the names, types, and recording conditions of the attributes.

The state of the GenAI semantic conventions

OpenTelemetry has semantic conventions for generative AI (gen_ai.*). To adopt them as an organizational standard, though, you cannot treat them the same way as the official HTTP conventions. Their stability is low, and the way that the project distributes them is in the middle of a transition. First, look at their current state.

The semconv v1.42.0 release, published in June 2026, reflects the move of the GenAI conventions out of the main semantic conventions and into a dedicated semantic-conventions-genai repository. That repository also manages the MCP-related mcp.* conventions in addition to gen_ai.*. As of September 2026, all GenAI conventions are in the Development stage, and none of the definitions is stable. The dedicated repository has no versioned releases either, so you cannot declare compliance with a specific version.

In fact, the attribute names have changed many times. The major renames alone are these.

Assume this rate of change, and bring the gen_ai attributes into the internal registry as a dependency. When upstream renames an attribute, detect the difference with diff. Then update the generated code and the transformation configuration of the gateway as one change.

Suppose that you depend on main of a repository that has no releases. If you resolve the same internal registry at a later date, you get a different result. Pin the dependency to a commit SHA. Record the adopted SHA, the adoption date, and the upstream schema_url in the internal registry. Take in updates through periodic PRs and review the diff. After upstream starts to make releases, move to pinning by tag.

The current conventions split spans into model calls and agent operations. The span name combines the operation name and the model name, as in chat gpt-4. Record the kind of operation in gen_ai.operation.name and the provider in gen_ai.provider.name. Record the requested model and the responding model in gen_ai.request.model and gen_ai.response.model. Record the token usage in gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.

The metrics include a histogram of token usage (gen_ai.client.token.usage), the operation latency (gen_ai.client.operation.duration), and the time until a streaming response delivers its first chunk. Combine these metrics with internal attributes such as the feature name. Then you can compute the cost breakdown and the latency SLO from the start of this chapter.

The trade-offs of recording prompts

In GenAI observability, you decide whether to record the text of prompts and responses.

If you record the text, you can investigate “why this summary came out” directly from the input. The text also serves as material for quality evaluation. On the other hand, the text is high-risk data that can contain personal information or sensitive information. It is also large, so it pushes up the cost of telemetry.

The conventions mark the attributes that carry the text, such as gen_ai.input.messages and gen_ai.output.messages, as Opt-In. The instrumentation does not record them by default. You enable them explicitly in the settings of the instrumentation¹.

The text contains natural-language messages, tool arguments, and references to external resources. The transformation rules of the gateway alone therefore cannot identify all personally identifiable information (PII) and sensitive information. Protect the data in several stages.

1. Do not record the text by default
2. A service that enables recording first registers the classification of the data that it handles and the purpose of use. It then records only the fields that the SDK side allows
3. Use the redaction at the gateway (Chapter 4) as an additional defense against known patterns
4. Send telemetry that contains the text to a dedicated pipeline, with a short retention period and restricted read access
5. Check with live-check (Chapter 6) and integration tests that no known data that you must not record flows through

The platform team can control the storage location, the access permissions, the retention period, and the removal of known patterns. The platform team cannot guarantee the complete removal of sensitive information from arbitrary natural language. Development teams enable text recording with this limit in mind.

Instrumenting LLM calls in Go

As of September 2026, the official instrumentation for Python is ahead. A dedicated repository provides instrumentation packages for OpenAI, Anthropic, LangChain, and others. The official project for GenAI instrumentation targets Python and JavaScript. Go has no official GenAI instrumentation library that you import. However, otelc from Chapter 3 has compile-time instrumentation for the OpenAI and Anthropic Go SDKs. This chapter shows manual instrumentation, which lets you control the business attributes and the scope of recording explicitly.

If the distribution from Chapter 2 initializes the SDK, manual instrumentation only adds a span around the LLM call. The following example is an excerpt that covers traces only. It omits metrics such as the histogram of token usage. For the attribute names, it uses constants from the semconv package that the registry generates.

func (c *SummaryClient) Summarize(ctx context.Context, doc string) (string, error) {
	ctx, span := c.tracer.Start(ctx, "chat "+c.model,
		// SpanKind: the kind that marks this span as the caller of an external service
		trace.WithSpanKind(trace.SpanKindClient),
		trace.WithAttributes(
			semconv.GenAIOperationNameChat,
			semconv.GenAIProviderNameKey.String(c.provider),
			semconv.GenAIRequestModelKey.String(c.model),
		),
	)
	defer span.End()

	resp, err := c.llm.Chat(ctx, buildPrompt(doc))
	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		span.SetAttributes(semconv.ErrorTypeKey.String(errType(err)))
		return "", err
	}
	span.SetAttributes(
		semconv.GenAIResponseModelKey.String(resp.Model),
		semconv.GenAIUsageInputTokensKey.Int(resp.Usage.InputTokens),
		semconv.GenAIUsageOutputTokensKey.Int(resp.Usage.OutputTokens),
	)
	return resp.Text, nil
}

The current conventions conditionally require the error.type attribute on a call that ends in an error. This example records the exception and the attribute in addition to the status. errType is a helper in the distribution that returns a string for the kind of error. A real distribution provides this whole boilerplate as a helper. When you regenerate the attribute constants and a name changes, the compile errors show you the places that need migration.

The trace structure of agents

An agent-style application uses tools to process a task in several steps. Record each step as a span in a parent-child relationship. The conventions define operations such as invoke_agent for agent execution and execute_tool for tool execution.

Figure 1: The arrows show the parent-child relationships between spans. The child spans of the agent execution run from left to right in time order. The gen_ai attributes attach to the purple spans.

When a tool calls an internal API, an ordinary distributed trace continues below the tool span. You can follow the work of the agent and the work of the microservices as one trace, so you can use the existing observability platform. To correlate the traces that belong to one conversation, use the gen_ai.conversation.id attribute.

AI development tools such as Claude Code and Codex emit their own telemetry in the OpenTelemetry format. I explain how to collect usage data for AI tools in a separate article (in Japanese).

Changes to the three pillars

Implement the changes for AI workloads across the three pillars.

In the SDK distribution (Chapter 2), add helpers to instrument LLM calls, and add default instrumentation for token usage metrics. Go has no official GenAI instrumentation to import today, so the distribution fills that gap.

In the Collector layer (Chapter 4), add governance for AI. The gateway takes on four jobs: a separate pipeline for telemetry that contains text, the redaction of known patterns, cost aggregation from token usage, and size limits on large spans.

In the semantic convention governance (Chapter 6), bring in the gen_ai attributes as a dependency pinned to a SHA. Define the internal attributes specific to AI under com.example.*, such as the feature name and the prompt version. The conventions are still in the Development stage. For that reason, automate how you follow renames, with the diff and the generation of the registry.

Development teams decide whether to record the text. The platform team manages the storage location, the permissions, and the retention period of the recorded text. The platform does not pull the decision itself to the center. Instead, it provides a boundary that keeps the outcome of the decision safe. Do not add a platform dedicated to AI workloads. Extend the existing distribution and governance.