# How Much Can We Trust Code We Have Not Read?

> Source: https://www.ymotongpoo.com/blog/20261030-how-we-trust-ai/


## Introduction

I have promoted SRE[^1] since around 2018 to the community. Observability in production is essential to SRE, and I have long argued for its importance. This background also led me to translate several SRE books into Japanese. They include [*Implementing Service Level Objectives*](https://www.amazon.co.jp/dp/4814400349?tag=ymotongpoo-22) and [*Observability Engineering*](https://www.amazon.co.jp/dp/4814400128?tag=ymotongpoo-22).

Over the past year, each proof of concept that I built with generative AI coding agents raised the same concern. How much could I trust code that passed its tests? I do not read every line that the agents write. Throughout 2026, developers and people in operations continued to discuss the same question. I want to draw on those discussions to explain my current thinking. These are my personal views, not those of my employer.

## TL;DR

Even as AI gets better at writing tests, the specifications and properties that determine correctness still depend on human understanding. I think we should use tests to search for counterexamples and limits, to expand our understanding of specifications. We should also check technical specifications and nonfunctional requirements in production. Release reversible changes to a limited scope, then use SLOs and observations to decide whether to expand the rollout. As we work with more code that we have not read, observability deserves our attention now as the basis for those decisions.

## What happened to AI that cheats on tests?

In June 2025, Kent Beck[^2] reflected on development with AI agents in [“Augmented Coding: Beyond the Vibes.”](https://newsletter.kentbeck.com/p/augmented-coding-beyond-the-vibes) He called his agent a “genie” and listed this sign that it was going off track:

> Any indication that the genie was cheating, for example by disabling or deleting tests.

In an [interview that month](https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent), Beck said that he still could not stop agents from deleting tests to make them “pass.” Even so, he described TDD as a “superpower” when working with agents. I have also repeatedly seen agents rewrite failing tests to make them pass in my own proof of concepts.

A little over six months later, the situation had changed. In February 2026, Martin Fowler[^3] hosted a Thoughtworks retreat.[^4] Its [report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf) states that TDD produces “dramatically better” results from AI coding agents.[^5] When tests precede the code, an agent cannot write them afterward to endorse its own incorrect implementation. The [report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf) even calls TDD “the strongest form of prompt engineering.”[^6]

Practices such as TDD make agent errors easier to address, while the models that generate code also improve. Google Cloud's DORA team published [*The ROI of AI-assisted Software Development*](https://services.google.com/fh/files/misc/dora-roi-of-ai-assisted-software-development-2026.pdf#page=10) in April 2026. It describes AI capabilities as advancing through generations on a monthly timescale. Yet it does not claim that the problems have disappeared:[^7]

> For an individual developer, AI does not necessarily make friction vanish; rather, the friction moves.

That friction has moved to verification. The report says that a “verification tax” replaces the effort of writing routine code by hand. This tax is the cognitive load of rigorously auditing AI-generated code that looks almost indistinguishable from correct code.

## Production is missing from the destinations for rigor

The retreat report[^8] identifies “Where does the rigor go?” as the question that received the most discussion. Its answer is that AI does not eliminate the rigor once applied to writing and reviewing code. That rigor moves elsewhere. The report lists five destinations: specification reviews, test suites, types and constraints, verification tiers based on risk, and continuous understanding.

DORA offers similar prescriptions. Its [2025 report](https://services.google.com/fh/files/misc/2025_state_of_ai_assisted_software_development.pdf) found that AI adoption was associated with both higher throughput and greater delivery instability. In the 2026 report,[^9] individual task efficiency shows the largest effect from AI adoption, followed by delivery instability. The proposed responses include automated tests, continuous integration, version control, small batches, checks before commits, and AI-assisted reviews.[^10]

Reading these reports from an SRE perspective, I cannot find production among the destinations for rigor. The body of the [2026 DORA report](https://services.google.com/fh/files/misc/dora-roi-of-ai-assisted-software-development-2026.pdf) never uses the word “observability.”[^11] The retreat report[^12] mentions better rollbacks, feature flags, and observability only as prerequisites for self-healing systems. It does not discuss concretely what to check in production or how to check it.

The retreat report[^13] also leaves several questions unresolved. One is: “What would need to be true for organizations to stop reviewing AI-generated code entirely?”

The same question has emerged from operations. In an [August 2026 interview](https://newsletter.pragmaticengineer.com/p/stop-being-skeptical-about-ai-for), Charity Majors[^14] urged us to consider the conditions that would make shipping unread code comfortable. She wanted to move past debating whether or when it would happen.[^15] Developers and people in operations arrived at the same question in the same year. I will frame it here as: how much can we trust code we have not read?

## The limits of tests and specification understanding

Suppose AI continues to improve at writing tests until it can check everything in a specification without omissions. Would passing those tests justify trusting code that we have not read?

I do not think so. Correctness depends on what we measure it against. Software engineering distinguishes **verification**, checking that we built something according to its specification, from **validation**, checking that we built what people actually needed.[^16] Better AI accuracy narrows the first gap. Passing tests does not guarantee that the specification itself matches intent or reality.

I believe the gap comes from human **specification understanding**. By this, I mean how much of the system's detailed behavior people understand. TDD assumes that developers know the specification well enough to write tests first. Yet AI implements details such as error handling, retries, boundary values, and sorting at a finer level than people have consciously specified. Tests that enumerate individual examples can miss conditions and behavior that nobody thought of.

Asking AI to write the tests does not necessarily remove those blind spots. [*Observability Engineering, 2nd Edition*](https://www.amazon.co.jp/dp/1098179927?tag=ymotongpoo-22), published in June 2026 and abbreviated here as OE2e,[^17] makes this point in [Chapter 2](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch02.html):

> AI agents writing tests for code they just wrote will blithely apply the same assumptions to both. Do your tests validate developer intent, or merely implementation?

[Chapter 2](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch02.html) continues: even an exemplary test suite that passes only assures us that errors found in production will be new errors.

The world that our tests assume also has limits. Andre Hora and Romain Robbes published [“Are Coding Agents Generating Over-Mocked Tests? An Empirical Study”](https://arxiv.org/abs/2602.00409) in January 2026. Among commits that changed tests, 36% of agent commits added mocks, compared with 26% of non-agent commits.[^18] A mock embodies an assumption about how a dependency should behave. However accurate tests become, they check correctness only within that assumed world.

As AI becomes more accurate, people will probably have fewer occasions to read code. The retreat report[^19] describes technical debt turning into **cognitive debt**: the gap between a system's complexity and human understanding. When specification understanding cannot keep pace with implementation details, that gap accumulates as cognitive debt. Test blind spots widen for the same reason.

Cognitive psychologist Lisanne Bainbridge[^20] described a consequence of greater automation in her 1983 paper “Ironies of Automation.”[^21] The tasks left to people become harder, while their ability to perform those tasks deteriorates. Her observation is over 40 years old, but it applies directly to developers who stop reading code. Takuto Wada, known as t-wada, also discussed cognitive debt and Bainbridge's argument in his [July 2026 presentation](https://speakerdeck.com/twada/agentic-software-engineering-2026-07-findy-edition).[^22]

## Business specifications and technical specifications

Test blind spots grow differently depending on the type of specification. I distinguish business specifications from technical specifications and nonfunctional requirements.

The business defines business specifications, such as pricing calculations and approval flows. Because people define them, they fall within human specification understanding. TDD works here as it always has. As the [retreat report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf#page=4) describes, writing tests first lets AI write code that follows the business specifications.

Technical specifications and nonfunctional requirements present a different situation. AI increasingly proposes timeout values, cache designs, and concurrency controls. If we accept each proposal without checking its validity, we can unknowingly delegate more decisions to AI. This is where the gap between human specification understanding and implementation detail grows widest. Actual cache behavior and latency under load also depend on real traffic, data, and dependencies. Without them, we cannot fully check those behaviors.

| | Business specifications | Technical specifications and nonfunctional requirements |
|---|---|---|
| Who decides | People on the business side | People assess AI proposals. Without review, more decisions fall to AI |
| Human specification understanding | Within its scope | The gap with implementation detail grows |
| Main place for verification | Tests: TDD and acceptance tests | Production: observability |

One objection is that spec-driven development[^23] can bring technical specifications within human understanding by describing them in enough detail. [The retreat also listed specification reviews first among the destinations for rigor.](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf#page=4) I agree with this direction. But more detailed specifications also place a greater burden on the people who must read them and judge their correctness. The gap in specification understanding may shrink, but it will probably remain.

OE2e's [Chapter 2](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch02.html) welcomes the move to express developer intent in specifications and architectural models. It also says that, for now, instrumentation in production is the “living spec.” This records what the system was asked to do and how it actually responded.

## Tests that search beyond our assumptions

We can search for unknown behavior without writing every concrete example ourselves. I translated [*Property-Based Testing with PropEr, Erlang, and Elixir* into Japanese](https://www.amazon.co.jp/dp/4908686181?tag=ymotongpoo-22) because I consider this kind of exploration important.

**Property-based testing** (PBT) defines properties that must hold and a range of inputs to generate, then searches for counterexamples. For example, we can generate varied inputs to check the property that saving and reading a value returns the original value. This can reveal failures for inputs that we would not have thought of individually. **Fuzzing** also varies inputs and repeatedly executes the program to find crashes or property violations.[^24] People define the criteria; machines can then search for counterexamples.

Load testing lets us explore system limits by varying request volume and duration. [k6](https://grafana.com/docs/k6/latest/), from my employer Grafana Labs, is a load testing tool for this purpose. As load increases in a test environment, latency may rise sharply beyond a certain point. Resource exhaustion may appear only after a long run.[^25] Recording execution paths and resource usage also lets us investigate where the degradation began.

The properties, inputs, and load models that we choose define the scope of that exploration. Real users, data, and dependencies can behave outside those assumptions. I therefore think we should continue exploring in production, with the instrumentation that we prepared for load testing. Add discovered counterexamples to regression tests, and measure performance limits again after changes. This creates a loop in which test results and observations expand human specification understanding.

## Criteria for releasing code we have not read

For technical specifications and nonfunctional requirements, we need criteria for releasing unread code to production. At its first release, we have no production history showing that it behaves as expected. Tests must first confirm that it follows the specifications we understand. Beyond that, I think release decisions should depend on preparations to limit the impact if the code is wrong. For reversible changes, release to a limited scope, observe behavior, and roll back if you notice degradation. Use the results from that scope to decide whether to expand the rollout.

SRE offers a method for this assessment. Define the service quality that users need, measure actual performance against that target, and use the results to decide whether to proceed with changes.

That service quality target is an **SLO**, or service-level objective.[^26] For example: “99.9% of payment requests over the past 30 days succeed within one second.” An SLO defines how reliable a service should be for its users. The target does not rise or fall because a human or an AI wrote the code. Even without knowing every code detail, we can define the outcomes that users need and compare them with actual behavior.

An SLO also determines how much failure we can tolerate during its evaluation window. Google's [*Site Reliability Engineering*](https://www.amazon.co.jp/dp/149192912X?tag=ymotongpoo-22)[^27] argues that 100% is the wrong reliability target for almost everything, and derives error budgets from that premise.[^28] An **error budget** is the amount of failure that an SLO permits. In the example above, it corresponds to 0.1% of the relevant requests.

SRE uses the consumption of that budget to guide changes. Teams agree on policies in advance, such as slowing releases and prioritizing reliability when little budget remains. The remaining error budget comes from service performance; it does not quantify our confidence in the code. The retreat report[^29] recommends investment in verification in proportion to the impact if the code is wrong. Combining that impact assessment with the remaining error budget helps us decide how cautiously to proceed.

Trust in code, meanwhile, grows through repeated experience of it behaving as expected in production.[^30] OE2e's final chapter, [Chapter 32, “Where Do We Go From Here?”](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch32.html), includes these words from Charity Majors:

> There is no test suite on earth that could make me trust a new chunk of code more than I trust the code that’s been running in production for two years. That’s how trust works; it takes time.

SLOs alone cannot reveal every blind spot. OE2e's [Chapter 2](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch02.html) explains that predefined detection methods, such as SLOs, find major problems but miss subtle behavioral changes. This is where **observability** becomes necessary.[^31]

We might start investigating when we notice faster error budget consumption.[^32] From there, we can compare users on new and old builds, then narrow the comparison to a particular endpoint or subscription plan. This can reveal degradation affecting only 1% of traffic that aggregate metrics do not show. Because we can narrow down the affected users or operations, we do not need to know every detailed specification in advance. When we release unread code to a limited scope, comparing new and old builds helps us decide whether to expand the rollout.

In my 2024 presentation, [“Observability from Macro to Micro,”](https://speakerdeck.com/ymotongpoo/observability-from-macro-to-micro) I separated two perspectives.[^33] The macro perspective starts from SLOs and traces problems back to their causes. The micro perspective checks the performance required of individual services. Here, we can connect them by investigating individual operations from service-wide symptoms, then using what we learn to improve implementation and tests.

This argument does not reject TDD. Charity Majors has advocated testing in production in articles such as [“I test in prod”](https://increment.com/testing/i-test-in-production/) from 2019. OE2e's [Chapter 2](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch02.html) explicitly describes this as a complement to tests before deployment, “not as a replacement.” Following OE2e's distinction, TDD checks correctness against specifications, while observability checks behavior against reality. I see observability as an extension of the TDD feedback loop into production.

## Discover in production, then add tests

Production observations can also help with business specifications. AI fills gaps in implementation with business decisions that people never specified or confirmed. Examples include rounding monetary values, handling dates at time zone boundaries, ordering lists, and choosing defaults for empty fields. These are business decisions, so they belong in business specifications and tests. Yet the people involved may not realize that anyone made those decisions. The code can still pass tests if those choices do not violate the properties that we wrote.

OE2e's [Chapter 2](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch02.html) describes what happens to code's behavior after it reaches production:

> Its behaviors turn into expectations, intended or not, and user expectations should be honored.

This connects to [Hyrum's Law](https://www.hyrumslaw.com/). With enough users of an API, someone will depend on every observable behavior, regardless of what its contract promises. A rounding rule that AI chose without asking anyone can become behavior that users depend on once it runs in production. Production observations will often provide the first clue to that dependency. We might see an increase in support inquiries or business metrics that skew only under particular conditions.

I therefore think we should treat tests and production observations as one loop. When we discover an implicit business decision in production, document it as a business specification and add a test. That test can detect a later change if AI rewrites the decision. Production observations help us discover what belongs in tests, while those tests continue to do their job. Responsibility lies with business specifications, even when production is where we discover the issue.

## What decision-makers can change

Organizations need changes that let them assess service quality as they release code that people have not read. OE2e's [Chapter 23, “Organizational Learning Speed Is Now Your Biggest Constraint: An Open Letter to CTOs,”](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch23.html) identifies the constraint on delivery in the AI era. It is the speed of verification, understanding, and learning, rather than the speed of writing code. I think decision-makers can make three changes to improve that speed:

- Include instrumentation and SLOs in the definition of done. A change without instrumentation is a change that we cannot verify in production. AI can now write instrumentation that follows [OpenTelemetry](https://opentelemetry.io/ja/) conventions,[^34] so this takes less effort than it once did.[^35]
- Allocate verification time and people according to the impact if a change is wrong. Give payment changes and changes affecting many users more thorough tests and human review. If little error budget remains, limit the scope or pace of releases and prioritize reliability. Use the potential impact to estimate the verification effort, and the remaining budget to decide whether releases can proceed.
- In human reviews, first validate the business specifications and SLOs. Before generating code, agree on what AI should implement and what outcomes to promise users. Assess test results and production observations against that agreement. For operations with a large potential impact, review the code diff as well.

## Changes we cannot try in production

Observing production is not enough for every change. Database migrations and schema changes can be irreversible if they go wrong. The same applies to operations, such as payments, that cannot be undone once they occur. OE2e's [Chapter 2](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch02.html) calls for greater human understanding when changing persistence layers. It also recommends more cautious, gradual releases and more observations before and after the change.

Safe verification in production requires mechanisms that limit impact alongside observability. These include feature flags, canary releases, gradual rollouts, and automated rollbacks.[^36] They overlap with the prerequisites for self-healing in the retreat report.[^37] SRE described these practices long before generative AI became widespread.[^38]

There is a precedent for this combination. In the [interview mentioned earlier](https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent),[^39] Kent Beck recalled his surprise at the absence of unit tests when he joined Facebook in 2011. The service remained stable because developers took responsibility for their code. They put risky changes behind feature flags and rolled them out gradually, starting in smaller markets such as New Zealand. Beck, a strong advocate of tests, saw an organization where other layers supplemented verification.

## Closing thoughts

Putting TDD first has helped contain the problem of AI cheating on tests. What remains is behavior beyond human specification understanding. We can expand that understanding by searching for counterexamples and limits in test environments, then continuing to observe in production. SRE already gives us ways to guide changes through service quality targets and actual results.

How much can we trust code we have not read? My current answer is to ground that trust in test results and experience of the code behaving as expected in production. When code includes behavior that we have not yet checked, release reversible changes to a limited scope and use observations to guide further rollout. We need a loop that keeps TDD checks in place while observing technical specifications and nonfunctional behavior in production. The implicit business decisions that we discover there must return to specifications and tests.

In this approach, observations in production become a condition for deciding whether to expand a rollout. As unread code increases, I expect more behavior that we encounter for the first time in production. AI can now also write the instrumentation that we need to observe it. Now is a good time to invest in observability and make production one of the places where we apply rigor. By sharing our practices as a community, we can gradually make the conditions for trusting unread code more concrete.

## References

### Reports

- Thoughtworks, [*Future of Software Development Retreat: Key Takeaways*](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf), 2026
- DORA, [*The ROI of AI-assisted Software Development*](https://services.google.com/fh/files/misc/dora-roi-of-ai-assisted-software-development-2026.pdf), Google Cloud, 2026
- DORA, [*State of AI-assisted Software Development*](https://services.google.com/fh/files/misc/2025_state_of_ai_assisted_software_development.pdf), Google Cloud, 2025

### Papers

- Andre Hora and Romain Robbes, [“Are Coding Agents Generating Over-Mocked Tests? An Empirical Study”](https://arxiv.org/abs/2602.00409), arXiv:2602.00409v1, 2026; [full text](https://arxiv.org/html/2602.00409v1)
- Lisanne Bainbridge, [“Ironies of Automation”](https://www.sciencedirect.com/science/article/abs/pii/0005109883900468), *Automatica*, 19(6), pp. 775–779, 1983; [archived PDF](https://web.archive.org/web/20200717054958if_/https://www.ise.ncsu.edu/wp-content/uploads/2017/02/Bainbridge_1983_Automatica.pdf)

### Books

- Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds., [*Site Reliability Engineering*](https://www.amazon.co.jp/dp/149192912X?tag=ymotongpoo-22), O'Reilly Media, 2016
- Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds., [*Site Reliability Engineering*, Japanese edition](https://www.amazon.co.jp/dp/4873117917?tag=ymotongpoo-22), O'Reilly Japan, 2017; translated by Ryuji Tamagawa, with translation supervision by Takeo Sawada, Tatsuo Sekine, Kazushige Hosokawa, and Daisuke Yabuki
- Alex Hidalgo, [*Implementing Service Level Objectives*, Japanese edition](https://www.amazon.co.jp/dp/4814400349?tag=ymotongpoo-22), O'Reilly Japan, 2023; translated by Yoshi Yamaguchi and Shoji Narita, with translation supervision by Yoshi Yamaguchi
- Charity Majors, Liz Fong-Jones, and George Miranda, [*Observability Engineering*, Japanese edition](https://www.amazon.co.jp/dp/4814400128?tag=ymotongpoo-22), O'Reilly Japan, 2023; translated by Kazunori Otani and Yoshi Yamaguchi
- Charity Majors, Liz Fong-Jones, George Miranda, and Austin Parker, [*Observability Engineering, 2nd Edition*](https://www.amazon.co.jp/dp/1098179927?tag=ymotongpoo-22), O'Reilly Media, 2026
- Fred Hebert, [*Property-Based Testing with PropEr, Erlang, and Elixir*, Japanese edition](https://www.amazon.co.jp/dp/4908686181?tag=ymotongpoo-22), Lambda Note, 2023; translated by Yoshi Yamaguchi
- Stephen R. Covey, [*The 7 Habits of Highly Effective People*, Japanese edition](https://www.amazon.co.jp/dp/4863940246?tag=ymotongpoo-22), King Bear Publishing, 2013; translated by FranklinCovey Japan
- Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara, and Stephen Thorne, eds., [*The Site Reliability Workbook*](https://www.amazon.co.jp/dp/1492029505?tag=ymotongpoo-22), O'Reilly Media, 2018

### Personal websites

- [Kent Beck](https://kentbeck.com/)
- [Martin Fowler](https://martinfowler.com/)
- [Charity Majors](https://charity.wtf/)

### Wikipedia

- [Chatham House Rule](https://ja.wikipedia.org/wiki/%E3%83%81%E3%83%A3%E3%82%BF%E3%83%A0%E3%83%8F%E3%82%A6%E3%82%B9%E3%83%AB%E3%83%BC%E3%83%AB) (in Japanese)
- [Ironies of Automation](https://en.wikipedia.org/wiki/Ironies_of_Automation)

### Other sources

- [Augmented Coding: Beyond the Vibes](https://newsletter.kentbeck.com/p/augmented-coding-beyond-the-vibes)
- [TDD, AI agents, and coding with Kent Beck](https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent)
- Thoughtworks, [Future of Software Development Retreat](https://www.thoughtworks.com/en-gb/about-us/events/the-future-of-software-development), event information, 2026
- [Stop being skeptical about AI for development with Charity Majors](https://newsletter.pragmaticengineer.com/p/stop-being-skeptical-about-ai-for)
- NASA, [IV&V Overview](https://www.nasa.gov/ivv-overview/)
- Takuto Wada, [Thinking About Software Development in 2026 (July 2026 Edition)](https://speakerdeck.com/twada/agentic-software-engineering-2026-07-findy-edition), 2026 (in Japanese)
- [Socrates Goes to the Gym](https://note.com/cap120/n/nd766cccccade) (in Japanese)
- [From chat to specs: a deep dive into AI-assisted development with Kiro](https://kiro.dev/blog/from-chat-to-specs-deep-dive/)
- GitHub, [Spec Kit](https://github.com/github/spec-kit)
- Takuto Wada, [Thinking About Software Development in the AI Era (July 2025 Edition)](https://speakerdeck.com/twada/agentic-software-engineering-findy-2025-07-edition), 2025 (in Japanese)
- Go, [Go Fuzzing](https://go.dev/doc/security/fuzz/)
- Go, [Tutorial: Getting started with fuzzing](https://go.dev/doc/tutorial/fuzz)
- [Grafana k6 documentation](https://grafana.com/docs/k6/latest/)
- Grafana Labs, [Breakpoint testing](https://grafana.com/docs/k6/latest/testing-guides/test-types/breakpoint-testing/)
- Grafana Labs, [Soak testing](https://grafana.com/docs/k6/latest/testing-guides/test-types/soak-testing/)
- [To Earn Your Team's Trust, First Build a Balance in Your Emotional Bank Account](https://www.franklincovey.co.jp/blog/?p=953) (in Japanese)
- Yoshi Yamaguchi, [Observability from Macro to Micro](https://speakerdeck.com/ymotongpoo/observability-from-macro-to-micro), 2024 (in Japanese)
- [I test in prod](https://increment.com/testing/i-test-in-production/)
- Hyrum Wright, [Hyrum's Law](https://www.hyrumslaw.com/)
- [OpenTelemetry documentation in Japanese](https://opentelemetry.io/ja/)

[^1]: SRE stands for site reliability engineering. It originated at Google and applies software engineering methods to operating reliable services. See [Google's introduction to SRE](https://sre.google/sre-book/introduction/).

[^2]: Beck introduced Extreme Programming (XP) and test-driven development (TDD), and was one of the authors of the Manifesto for Agile Software Development. See [his website](https://kentbeck.com/).

[^3]: Fowler is a software engineer known for his work on refactoring and agile development. At the time of writing, he is Chief Scientist at Thoughtworks. See [his website](https://martinfowler.com/).

[^4]: The [Future of Software Development Retreat](https://www.thoughtworks.com/en-gb/about-us/events/the-future-of-software-development) took place in Park City, Utah, on February 1–3, 2026. It marked 25 years since the Manifesto for Agile Software Development. The event followed the [Chatham House Rule](https://ja.wikipedia.org/wiki/%E3%83%81%E3%83%A3%E3%82%BF%E3%83%A0%E3%83%8F%E3%82%A6%E3%82%B9%E3%83%AB%E3%83%BC%E3%83%AB), so the report does not name participants.

[^5]: [Page 4 of the report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf#page=4): “One of the retreat's most shareable insights was that test-driven development produces dramatically better results from AI coding agents.”

[^6]: See [pages 16–17 of the report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf#page=16).

[^7]: See [page 20 of the report](https://services.google.com/fh/files/misc/dora-roi-of-ai-assisted-software-development-2026.pdf#page=20).

[^8]: See [pages 4–5 of the report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf#page=4).

[^9]: See [Figure 4 on page 20 of the report](https://services.google.com/fh/files/misc/dora-roi-of-ai-assisted-software-development-2026.pdf#page=20).

[^10]: See [page 19](https://services.google.com/fh/files/misc/dora-roi-of-ai-assisted-software-development-2026.pdf#page=19) and [page 33](https://services.google.com/fh/files/misc/dora-roi-of-ai-assisted-software-development-2026.pdf#page=33) of the report.

[^11]: [Page 91 of the 2025 report](https://services.google.com/fh/files/misc/2025_state_of_ai_assisted_software_development.pdf#page=91) uses the term to describe observability into developers' toolchains.

[^12]: See [page 9 of the report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf#page=9).

[^13]: See [page 16 of the report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf#page=16).

[^14]: Majors is an engineer who has helped spread observability practices. At the time of writing, she is co-founder and CTO of Honeycomb. See [her website](https://charity.wtf/).

[^15]: Around 11:27–11:41 in “Stop being skeptical about AI for development with Charity Majors,” released on August 12, 2026. Majors asks us to consider the conditions for releasing code without reading and understanding it, rather than debating feasibility or timing. The article's framing in terms of whether versus when is interviewer Gergely Orosz's summary, not a direct quotation from Majors.

[^16]: See [NASA's explanation of software verification and validation](https://www.nasa.gov/ivv-overview/).

[^17]: Charity Majors, Liz Fong-Jones, George Miranda, and Austin Parker, O'Reilly Media, 2026, second edition. Kazunori Otani and I translated the [first edition into Japanese](https://www.amazon.co.jp/dp/4814400128?tag=ymotongpoo-22).

[^18]: Andre Hora and Romain Robbes, “Are Coding Agents Generating Over-Mocked Tests? An Empirical Study,” arXiv:2602.00409v1, January 30, 2026. See [Section 3.2.1 and Table 7](https://arxiv.org/html/2602.00409v1#S3.SS2.SSS1). The study analyzed more than 1,200,000 commits from 2025, including 48,563 agent commits. The denominator for these percentages is commits that changed tests: 3,934 / 11,035 for agents and 40,966 / 158,326 for non-agents.

[^19]: See [page 3 of the report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf#page=3).

[^20]: A British cognitive psychologist who studied the relationship between people and automated systems.

[^21]: See the [Wikipedia explanation](https://en.wikipedia.org/wiki/Ironies_of_Automation) and the [archived PDF of the paper](https://web.archive.org/web/20200717054958if_/https://www.ise.ncsu.edu/wp-content/uploads/2017/02/Bainbridge_1983_Automatica.pdf) linked from it. Bibliographic details: Lisanne Bainbridge, “Ironies of Automation,” *Automatica*, 19(6), pp. 775–779, 1983. The [publisher's page](https://www.sciencedirect.com/science/article/abs/pii/0005109883900468) also provides these details.

[^22]: See [slide 22](https://speakerdeck.com/twada/agentic-software-engineering-2026-07-findy-edition?slide=22) and [slide 25](https://speakerdeck.com/twada/agentic-software-engineering-2026-07-findy-edition?slide=25).

[^23]: Spec-driven development starts with detailed specifications, then asks AI agents to generate code from them. A [Kiro blog post from July 15, 2025](https://kiro.dev/blog/from-chat-to-specs-deep-dive/) describes this workflow in Kiro, an IDE with AI capabilities. Requirements, designs, and implementation tasks are documented in sequence, with human review before implementation. GitHub's [Spec Kit](https://github.com/github/spec-kit) is another tool for this approach.

[^24]: See [Go's explanation of fuzzing](https://go.dev/doc/security/fuzz/). Its [tutorial](https://go.dev/doc/tutorial/fuzz) checks properties of string reversal and saves failing inputs so that ordinary test runs can execute them again.

[^25]: See the k6 documentation on [breakpoint testing](https://grafana.com/docs/k6/latest/testing-guides/test-types/breakpoint-testing/) and [soak testing](https://grafana.com/docs/k6/latest/testing-guides/test-types/soak-testing/).

[^26]: See [Chapter 4 of Google's SRE book](https://sre.google/sre-book/service-level-objectives/). [*Implementing Service Level Objectives*, which I translated into Japanese](https://www.amazon.co.jp/dp/4814400349?tag=ymotongpoo-22), also explains SLO concepts and adoption in detail.

[^27]: Edited by Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, O'Reilly Media, 2016. The [Japanese edition](https://www.amazon.co.jp/dp/4873117917?tag=ymotongpoo-22) has the same editors, with translation by Ryuji Tamagawa of Sky Co., Ltd., O'Reilly Japan, 2017. Takeo Sawada, Tatsuo Sekine, Kazushige Hosokawa, and Daisuke Yabuki supervised the translation.

[^28]: See [the explanation of error budgets in Chapter 1](https://sre.google/sre-book/introduction/).

[^29]: See [page 5 of the report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf#page=5).

[^30]: The “Emotional Bank Account” metaphor in [*The 7 Habits of Highly Effective People*, Japanese edition](https://www.amazon.co.jp/dp/4863940246?tag=ymotongpoo-22), also describes trust accumulating through actions such as keeping promises. See [FranklinCovey Japan's explanation](https://www.franklincovey.co.jp/blog/?p=953). The metaphor concerns relationships between people, but it shares the idea that trust grows through experience.

[^31]: OE2e's [Chapter 1](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch01.html) treats observability as a property of dependability. It defines observability as the degree to which we can understand and debug the state of an arbitrary system.

[^32]: Burn rate measures how fast an error budget is consumed. A rate of 1 would consume exactly the whole budget over the SLO's evaluation window. For a 30-day SLO, a sustained burn rate of 2 would consume the entire window's budget in 15 days. For request-based SLOs, this calculation assumes a constant request rate. See [Chapter 5 of Google's SRE workbook](https://sre.google/workbook/alerting-on-slos/#4-alert-on-burn-rate).

[^33]: See [slide 11](https://speakerdeck.com/ymotongpoo/observability-from-macro-to-micro?slide=11) and [slide 22](https://speakerdeck.com/ymotongpoo/observability-from-macro-to-micro?slide=22).

[^34]: OpenTelemetry is a collection of vendor-neutral specifications and libraries for collecting traces, metrics, and logs from applications.

[^35]: See “Practice 3: Instrument Your Code and Validate in Production” in OE2e's [Chapter 2](https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915/ch02.html).

[^36]: Feature flags let you enable or disable functionality in deployed code through configuration. A canary release sends a new version to a subset of users or servers first. You compare it with the existing version before expanding the rollout to everyone.

[^37]: See [page 9 of the report](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf#page=9).

[^38]: [Chapter 8](https://sre.google/sre-book/release-engineering/) of [*Site Reliability Engineering*](https://www.amazon.co.jp/dp/149192912X?tag=ymotongpoo-22) (2016) covers controlling features with flags and canary deployments. [Chapter 16](https://sre.google/workbook/canarying-releases/) of [*The Site Reliability Workbook*](https://www.amazon.co.jp/dp/1492029505?tag=ymotongpoo-22) (2018) also discusses gradual rollouts and rollback when problems appear.

[^39]: Beck discusses his Facebook experience from 54:26 onward. The New Zealand example appears around 1:03:23.