Why IaC Stays Even When the Cloud Tracks Its Own State

Originally published in Japanese at https://ymotongpoo.hatenablog.com/entry/2026/11/02/093500.

Introduction

On October 31, 2026, my former colleague James Ward¹ posted “IaC is dead” on X. IaC² means managing infrastructure configuration as code, and his post argued that AI makes it unnecessary. We exchanged several replies about the idea afterward.

The exchange made me think about a question that becomes more pressing as generative AI starts to write infrastructure changes. Someone has to decide how to check those changes, approve them, and roll them back. I think we can answer that question if we look back at what IaC has done so far. Suppose that the cloud could track and control its own state correctly. I ask which roles of IaC would then disappear and which would remain, and I look at what Kubernetes changed along the way. These are my personal views, not those of my employer.

TL;DR

IaC has played two roles. It records intent by keeping the desired state as code, and it acts as an execution layer that holds state, computes diffs, and applies changes. Inside the cluster, Kubernetes built the execution layer into the platform, yet users still write manifests and manage them in Git. Even if the whole cloud tracked its state and kept converging toward the declarations, the platform could take over only the execution layer. The role of recording intent would remain. Even when AI writes the declarations, you still need to keep them as artifacts that you can approve, reproduce, and roll back.

The “IaC is dead” post

The first three paragraphs of James’s post read as follows:

IaC is dead.

Sorry DevOps / SREs / Cloud Engineers but you’re in for the same roller coaster as Devs. AI can manage infra change better than you.

Unfortunately for IaC, we don’t need code because with the actual system source code and the infra, AIs have everything needed to determine change sets / approvals and dev env lensing.

James argues that AI can read the system source code and the actual infrastructure, then decide what to change and whether to approve it. In that case, nobody needs to write the desired configuration as code.

Later the same day, James added a follow-up:

For a glimpse of what I’m talking about, check out Alchemy Effect and how system code can express infra dependencies.

Alchemy³ is a framework that lets you declare buckets and permissions inside the application code that uses them. Its documentation describes Alchemy itself as IaC:

Alchemy is Infrastructure as Code built in pure Effect, with Infrastructure as Effects on top.

The same page puts Alchemy in the same family as Terraform, Pulumi, CloudFormation, and the AWS CDK. On each deploy, Alchemy reads the current state and computes the diff against the program. It shows a plan, applies the changes in dependency order, and saves the result to a state store. Alchemy still has the three parts of IaC: a declaration, a state, and a diff. What changes is where you write the declaration, which is now the application code.

In the replies that followed, I pointed out that Alchemy is IaC, and James explained his view further. He wrote that Alchemy Effect builds Infrastructure as Effects on IaC, but that IaC “isn’t actually necessary.” He then split the idea into two parts. System code expresses the bindings between the code and the resources that it uses. AI agents apply the infrastructure changes that those bindings need. I asked what would delete a resource that someone removed from the code, and in what order. James answered: “An agent can easily figure this out from infra APIs and code.”

His final position raises a question that needs a closer look. Can we keep declarations in code and leave state tracking and change application to AI agents? For the answer to be yes, several conditions must hold. They closely match the design of Kubernetes and Terraform. Before I list those conditions, I look back at infrastructure automation before Kubernetes, at what Kubernetes changed, and at IaC in general-purpose languages.

Infrastructure automation before Kubernetes

There are two styles for describing infrastructure. The declarative style describes the final state. The procedural style lists the operations that lead to it. Declarative thinking existed before Kubernetes. Convergence means moving the current state toward a declared desired state, again and again. The idea goes back to CFEngine, which Mark Burgess⁴ developed in 1993. Later configuration management tools⁵ such as Chef, Puppet, and Ansible also let you declare the desired state of packages and files on each server.

However, these tools described the contents of individual servers. The configuration of the whole system came from procedural scripts or manual work in a console. That configuration covered how many servers to run, where to place them, and which networks connected them.

Procedural automation has weaknesses that grow with scale. Idempotency⁶ depends on the care of whoever writes the script. The writer must also handle the order of operations, the dependencies, and the rollback after a failure partway through. And no record remains of how the system should look. Only the running system remains. In that system, the intended configuration, console changes, and temporary incident fixes pile up together, and nothing tells them apart. In the “pets vs. cattle” analogy⁷ of that time, each server was a pet. Each had its own history of changes, and nobody could rebuild it with the same configuration.

Many cloud APIs are also imperative (procedural). Users had to decide how to combine operations such as “create an instance” and “change a rule.” This was true for both ClickOps⁸ and scripted automation. State was scattered across services, and no mechanism compared it with the desired state.

Incident response exposes this problem. Suppose that during a late-night incident, an engineer restores service by changing a setting in the console. If nobody records that change, an automation script that runs months later overwrites it, and the same incident happens again. If the automation instead accepts the current state as correct, the temporary fix becomes part of the design. Without a record of intent, nobody can even ask whether a change was intended.

For the contents of servers, people also tried to write the intended state as tests and check the actual state against them. In 2013, Gosuke Miyashita (mizzy)⁹ released Serverspec.¹⁰ It describes the desired state of a server in the style of RSpec, a Ruby testing framework. Examples include “this package is installed” and “this port is listening.” When you run the tests, Serverspec runs commands on the actual server over SSH or a similar transport, and checks whether the server matches. It works no matter which tool configured the server, such as Puppet, Ansible, or CFEngine. The official site says that “the true aim of Serverspec is to help refactoring infrastructure code.” If you write infrastructure as code, you need tests to confirm that a rewrite kept the same behavior, as you do with application code.

Serverspec tests describe the intended state separately from the code that changes the configuration. With such a description, a failed test can reveal that a change outside IaC moved the actual state away from the intent.¹¹

Declarative IaC tools filled the remaining gap: a declaration of the whole system that the tool compares with its actual state. In 2011, AWS released AWS CloudFormation, which let users declare and manage AWS resources in templates. In 2014, Terraform¹² arrived with support for multiple clouds. Terraform describes the desired state as code and keeps the actual state in a state file. Before it applies changes, it computes the diff with plan. Declaration, state, and diff were now all in place.

Here IaC plays two roles. The first is recording intent: you write the desired state as code, review it, and keep its history. The second is serving as an execution layer: the tool holds the actual state, computes diffs, and applies changes in dependency order. In my view, the cloud itself lacked an execution layer, and IaC tools added one from outside the cloud.

What Kubernetes changed

Google released Kubernetes¹³ in 2014, the same year as Terraform.¹⁴ Google had run Borg,¹⁵ its internal cluster management system, in production for more than a decade. Kubernetes drew on the experience with Borg and its successor, Omega.¹⁶ Google described this history in the 2015 Borg paper and in the 2016 article “Borg, Omega, and Kubernetes.”

Kubernetes is a platform that runs containers. It also moved the mechanisms for declaration, state, and diff into the platform itself. Before Kubernetes, IaC had kept those mechanisms outside the cloud. I think this is what made Kubernetes groundbreaking.

Declarative APIs

Kubernetes users submit a desired state (the spec), written in the declarative style, instead of a procedure. You do not command “start three containers.” You declare “keep three replicas of this workload,” and Kubernetes decides how to achieve it. The spec does more than sit in a local configuration file. The cluster registers it as an API object and stores it together with the current state (the status).

Consider what happens when you create one Deployment.¹⁷ You submit the Deployment spec to the API server,¹⁸ and the API server stores it in etcd.¹⁹ The Deployment controller²⁰ reads the spec and creates a ReplicaSet.²¹ The ReplicaSet controller then creates Pods.²² The scheduler picks a node for each Pod. The kubelet²³ on that node starts the containers and writes the status back. Nowhere in this flow does anyone write a procedure that orders the operations. The flow is a chain of small loops. Each loop compares the desired state with the current state at its own stage.

Reconciliation loops and level triggering

Kubernetes calls the process that a controller repeats a reconciliation loop. The controller keeps observing the current state and acts to close the gap with the spec. If a Pod stops, the controller recreates it. If a node disappears, Kubernetes places the Pod on another node. The behavior known as self-healing is the result of this simple loop. The Kubernetes documentation compares a controller to a thermostat in a room. You set the temperature, which is the desired state. The thermostat watches the room temperature and turns the equipment on or off to bring the room to the setting.

The reconciliation loop is level-triggered:²⁴ it reacts to the state itself, not to notifications of change. If a controller misses a notification, it still keeps working as long as the current state differs from the spec. With this design, automation keeps the system converging, instead of applying a change once and stopping.

Declarations and reconciliation loops work better than procedural automation at scale because of how distributed systems behave. Procedural automation assumes that nothing around it changes until it finishes every operation. In practice, networks drop, processes stop, and people interrupt. When procedural automation fails partway, nobody knows how far it got. A declaration specifies only the final state, and the loop closes only the remaining gap. So the system converges to the same state no matter how many times something interrupts it. Idempotency becomes a property of the design instead of a matter of the writer’s care.

Stored state and a single entry point

Kubernetes keeps the state of every resource in etcd. The cluster holds data that describes its own entire state. Because of this data, Kubernetes can compute diffs, the scheduler can decide where to place Pods, and Kubernetes can move Pods after a failure.

Also, only the API server reads and writes that state. Every change passes through the API server, so Kubernetes can apply authentication and authorization and admission control²⁵ in one place. Kubernetes can enforce who may change which resources, and in what ways, because all changes take one path. The Borg paper explains that the Kubernetes API server only processes requests and manipulates state. Cluster management logic, such as the controller that keeps the replica count, runs as a client of the API server.

A uniform API model and extensions

Every Kubernetes resource has the common fields apiVersion, kind, and metadata. Most resources also pair a spec with a status to express the desired state and the current state. With a Custom Resource Definition (CRD)²⁶ and an Operator,²⁷ you can add new kinds of resources and the controllers that manage them. The new resources follow the same structure as the built-in ones. Kubernetes became both a product and a foundation for building declarative systems. Because of this extensibility, many projects grew on top of Kubernetes.

Problems that remain in Kubernetes

Kubernetes still has many problems. etcd holds all the state, so if you lose etcd, you lose the entire state of the cluster. Backup and restore put a heavy burden on operators. Also, a healthy status on an object does not mean that the application works correctly.

Even in a cluster that you manage declaratively, objects that someone edits directly with kubectl slowly drift from the manifests²⁸ in Git. That is why GitOps²⁹ tools such as Argo CD and Flux keep detecting differences. Teams must also keep up with version upgrades and API deprecations. Many organizations run a platform team just to operate Kubernetes.

Even so, Kubernetes changed how automation works. It moved the execution layer, which IaC had provided from outside the cloud, into the center of the platform. The center of automation shifted from procedural scripts to systems that keep their own state. Yet even with this much automation, the Kubernetes ecosystem rests on declarations written in large amounts of YAML. The role of recording intent stayed with the users, and the declarations grew in number instead of disappearing.

IaC in general-purpose languages

After Kubernetes appeared, the languages for writing IaC also changed. In 2019, AWS announced the general availability of the AWS Cloud Development Kit (CDK). With the CDK, you define infrastructure in general-purpose programming languages such as TypeScript, Python, Java, C#, and Go. The CDK converts the code into CloudFormation templates and deploys them. Pulumi is another IaC tool that defines cloud resources in general-purpose languages.

Types are one advantage of a general-purpose language. The CDK provides a class for each kind of resource, and each class defines its settings and their value types. In TypeScript, the compiler catches unknown setting names, values of the wrong type, and missing required settings before deployment. The editor also suggests the settings that you can enter. With templates in YAML or JSON, many of these errors stay hidden until you run a separate checker or deploy.

Types also let code express relationships between resources. In the CDK, you can pass a Lambda function to the grantRead method of a bucket object. The CDK then generates the IAM policy that the function needs to read the bucket.³⁰ The CDK documentation lists one advantage: you can define infrastructure and application logic in the same programming language. Alchemy expresses infrastructure dependencies in system code, and the CDK already took the same direction in 2019.

Still, the CDK did not remove the declarations. When you run CDK code, it generates a CloudFormation template, which is a declaration, and a CloudFormation stack holds the state. With cdk diff, you can check the diff before you deploy. General-purpose languages and types improved one part of recording intent: how you write declarations and how you check them. The structure of declaration, state, and diff stayed the same.

Conditions for handing changes to AI

Now back to James’s idea. Suppose that AI derives a change set without code, or from bindings alone, and also decides on approval. I think that this requires at least the four conditions below. The platform itself can meet the first two. The other two require a written record of the desired state.

Knowing the current state

The first condition is that the system keeps an accurate record of its full current state. The recorded state must match the actual state, including changes outside IaC, such as direct edits in a console. If AI sees only part of the actual state, a change set that it derives can damage the part that it cannot see.

Reading the current state through APIs is an observation, and an observation does not give you a complete record. It is a sample at one point in time. It misses changes that eventual consistency has not reflected yet, resources outside your permissions, and resources that nobody manages. Clouds do have services that collect the current state. AWS Config and Cloud Asset Inventory record resource configurations and their change history. However, each service covers only the resource types that it supports, and its records lag behind the changes.

In Kubernetes, the API server accepts every change, and the data in etcd is the cluster state. In the cloud, each service keeps its own state, and a separate system assembles a combined view later. No matter how fast AI reads the APIs, it cannot read resources outside that system’s coverage, or changes that the system has not recorded yet.

Terraform tried to derive state from APIs and code in its early days. According to the Terraform documentation, early prototypes had no state file and used AWS tags to map code to real resources. The documentation says that they quickly ran into problems, because not all resources and clouds support tags. Code also cannot tell you the order in which to delete a resource that you removed from it. That is why Terraform keeps the latest dependencies in its state.

Convergence that tolerates partial failure

The second condition is a mechanism that converges the current state toward the desired state while it tolerates partial failure. The mechanism must order work by dependencies, prevent conflicts between concurrent changes, and handle an apply that stopped partway. Without it, a change set is only a list of changes with no guarantee of execution. Terraform state locking³¹ and Kubernetes optimistic concurrency control³² exist to solve this problem.

A verifiable desired state

The third condition is that the desired state exists in a form that you can verify, store, and diff. A vague intent such as “roughly this kind of configuration” is not enough to derive a change set. Even a desired state that AI comes up with must exist in written form so that someone can check it. And a written desired state is IaC that records intent.

Moreover, you cannot recover intent from the actual system. James says that the system source code and the actual infrastructure are enough. But bindings express only the links between the code and the resources that it uses. Source code does not contain availability targets, or the legal requirements that set data retention periods. The same goes for cost trade-offs and for recovery objectives (RPO and RTO).³³ The organization decides these, and nobody can refer to them until someone writes them down. If AI infers intent from the actual system, it can treat temporary incident fixes and security flaws as intended design.

Diffs and approval before apply

The fourth condition is that you can show a diff before you apply a change. The process must also hold up to human approval and later audits. If nobody can see what will change in advance, approval becomes a formality. If nobody can trace later what changed, when, and with whose approval, you cannot do incident response or compliance checks. A rollback also needs the previous desired state as an artifact. A Kubernetes Deployment can roll back because ReplicaSets with earlier Pod templates remain as history.³⁴ Without the artifact, you have nothing to roll back to.

If AI decides on approval, another problem appears. Approval does two things: it evaluates a change, and it makes a specific person responsible for that change. If the AI that derived a change set also approves it, the proposer approves its own proposal. Under the principle of segregation of duties,³⁵ that is not a control. Change management separates the proposer from the approver for this reason. Even if AI can produce diffs, technology cannot settle who is responsible for a change.

The distance between Kubernetes and the cloud

Together, the four conditions describe the design of Kubernetes and Terraform. However, Kubernetes as a platform meets only the first two. The spec in etcd is only the current desired state. It keeps no history of reviews that shows who changed it and why. To rebuild the same configuration on another cluster, you also need a separate description without cluster-specific values. Only the manifests that users write and manage in Git satisfy the other two conditions.

Kubernetes also knows the state only inside the cluster. Many cloud resources, such as IAM, DNS, networks, and databases, live outside it. That is why many teams manage the cluster and its surroundings with Terraform, and the inside of the cluster with manifests.

Today’s clouds do not meet even the first two conditions. Many APIs are imperative, state is split across services, and no standard mechanism computes a diff for the whole system. The cloud may record a change outside IaC, but nothing returns it to the desired state. CloudFormation drift detection shows where resources differ from the template, but you have to fix them yourself. Tools such as Terraform must keep a state file and a plan outside the cloud. I take that as evidence that the cloud does not meet those conditions yet.

Cloud providers recognized early that users needed a declarative layer. Fifteen years have passed since AWS released CloudFormation. However, CloudFormation manages the resources that you define in a stack, not the state of the whole cloud. Even a cloud provider, working on its own services, has not gone beyond the scope of a stack.

Tools for managing cloud resources are moving toward building declarations into the platform, not toward removing them. Crossplane, AWS Controllers for Kubernetes (ACK), and Google Cloud’s Config Connector manage cloud resources with the Kubernetes API model. You declare cloud resources as Kubernetes resources. The controllers of each tool then call the cloud APIs and keep the real resources in line with the declarations.

When AI writes the declarations

AI may well write the declarations, and it already does. But a desired state that AI generates still has to remain as an artifact that you can check. Only then can you approve it, reproduce it, and roll it back. In application development, we review AI-generated code, commit it to Git, and keep its history. Generated declarations deserve the same treatment as code. The faster AI generates them, the more diffs people must review, and the more important it becomes to keep the declarations as records.

Closing thoughts

I looked closely at the premise of “IaC is dead.” IaC has two roles, and Kubernetes moved only one of them into the platform 12 years ago: the execution layer. If the whole cloud tracked its state and converged toward declarations as Kubernetes does, the platform could absorb IaC as an execution layer. Even then, the organization that uses the cloud decides the desired state, and the cloud cannot decide it for them. IaC would remain in the role of recording that intent, reviewing it, and keeping its history.

For now, even Kubernetes does not know the state outside the cluster. When you let AI write declarations, you should still review the generated declarations as code. Keep their history, and make sure that you can check the diff.

References

Papers and articles

• Abhishek Verma et al., “Large-scale cluster management at Google with Borg”, EuroSys 2015
• Malte Schwarzkopf et al., “Omega: flexible, scalable schedulers for large compute clusters”, EuroSys 2013
• Brendan Burns et al., “Borg, Omega, and Kubernetes”, ACM Queue, 2016
• Randy Bias, “The History of Pets vs Cattle and How to Use the Analogy Properly”
• Kubernetes, “10 Years of Kubernetes”, 2024

Documentation

• Kubernetes Documentation
  • Controllers
  • Deployments
  • Custom Resources
  • API Conventions
• Terraform Documentation
  • State
  • Purpose of Terraform State
  • terraform plan command
• AWS, AWS CloudFormation
• AWS, What is the AWS CDK?
• AWS, What Is AWS Config?
• AWS, AWS Controllers for Kubernetes
• Google Cloud, Cloud Asset Inventory overview
• Google Cloud, Config Connector overview
• Crossplane

Other sources

• Alchemy, “What is Alchemy?”
• CFEngine
• Serverspec
• Gosuke Miyashita, Serverspec (in Japanese), O’Reilly Japan, 2015