Conclusion

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/observability-platform-with-otel/viewer/90-conclusion.

This book started with an inventory service whose traces broke, and with an example of three values: production, prod, and prd. I did not blame these problems on carelessness in each team. Instead, I looked at ways to hand those decisions over to mechanisms.

What the three pillars solved

The first pillar, the SDK distribution, moved initialization decisions into a library. You prevent a missing propagator setting by distributing something that already contains the correct settings, not by asking each team to write the settings correctly. For services whose code you cannot change, zero-code instrumentation delivered a minimum level of instrumentation.

The second pillar, the Collector layer, moved the destination and the processing of telemetry out of the application. When you separate the distribution of configuration from deployment, you can change the configuration more often without restarting every Pod. However, the OpAMP implementation is in alpha. I also measured and confirmed that automatic rollback alone cannot protect the fleet.

The third pillar, the governance of semantic conventions, gathered the meaning of attributes into a registry. You prevent production and prod from coexisting by making such data fail a check, not by distributing the conventions as documents.

The three pillars share one form. They take the decisions that the organization shares and turn them into a system of executable defaults, distributed artifacts, and checks. That is what self-service means in this book. The goal is not to reduce the requests to the central team. The goal is to let teams start from a correct state without any request.

The assumptions behind AI agents reading telemetry

Chapters 7 and 8 covered observing AI workloads and incident investigation by AI agents. Both extend the three pillars. I did not build a new platform. I extended the existing distribution and governance.

A hypothesis still remains here: if you govern the schema and make it available to machines, you can reduce incorrect queries from AI agents. This book does not measure that effect. Even so, the work also helps humans with incident response, and it is well worth doing. What is still needed is an evaluation that uses the same incident data and changes whether a registry exists and how strictly you govern the schema. It then compares the findings and the queries that the generative AI issued.

Working with the parts that are not stable

As I noted at the start, many of the OpenTelemetry features in this book are not stable yet. OpAMP Supervisor is in alpha, Weaver is before 1.0, and the GenAI conventions are at the Development stage. By the time this book is published, attribute names or command arguments may already have changed.

Even so, you do not need to stop your design work. The distinction between the decisions that the central team owns and the decisions that stay with development teams does not depend on tool versions. When a stable release comes out, you will want to replace the distributed artifacts and checks. Keep your decisions as code and configuration, not as documents. That practice is itself the preparation for the migration.

To prepare for breaking changes, you can reuse the mechanisms from the third pillar as they are. A registry diff detects renamed attributes. You then regenerate the generated artifacts and use live-check to see the differences from the actual data. Assume that changes will come, and automate how you follow them.

Start small

As the incremental adoption checklist in Chapter 9 showed, you do not need to build the three pillars at the same time. The distributed artifacts and checks that you introduce at each stage are useful without waiting for the later stages.

A practical first step is the distribution from Chapter 2. Move the initialization code of one service into a library, and then try that library in another service. At this point, missing propagator settings disappear from the organization. Next, define one attribute in the registry and run the check in CI. You get results sooner if you add distributed artifacts and checks one at a time than if you design the complete platform before you start.

The reference implementation (otel-platform-blueprint) lets you run the whole system locally. Use it as a starting point to try the designs in this book.

If you notice anything about the content of this book, or if you introduce these ideas in your organization and have results to share, please let me know. You can use the repository’s Issues or social media (X: @ymotongpoo, Bluesky: @ymotongpoo).