Terraform Applied Successfully. Did the AI Agent's Change Actually Work?

Part 2 of 3 in "Terraform with an AI co-worker: access, testing, review".

The task looked small: create an S3 bucket, two SQS queues and a service user across four test environments and production. An AI agent was doing the work.

Before deploying, it ran Terraform plans. In two test environments, those plans included other teams' access keys, permission changes and the deletion of two empty S3 folder markers. None of that belonged to our task.

That was the first useful result of testing, before we had deployed anything. Later, the agent would upload real objects, deliberately trigger access denials, and discover that some of its own checks were wrong.

In Part 1, I covered how we restricted the agent's AWS access. This part is about the evidence I wanted before accepting its work as done.

A Terraform plan previews proposed changes. It doesn't apply them, which makes it a useful first check. It also only covers what the configuration and state let Terraform see, so it isn't a complete inventory of an AWS account.

In our case, a colleague's cleanup had been merged into the repository but hadn't been deployed. A full apply would have removed two empty folder markers, created access keys and permission policies for other teams, and changed existing permission policies.

The folder-marker deletions were intended by their owner. This wasn't a confirmed data-loss scenario. The problem was that nobody deploying our change had reviewed or approved those unrelated changes for this rollout.

They would have gone live as a side effect of our task, at a time their owners hadn't chosen.

We used -target=module.<name> to limit this deployment, then checked the resulting plan against the expected resource addresses and actions. Anything outside that scope, including any deletion, stopped the deployment before approval. The unrelated changes were reported separately.

Targeting needs a caveat: it includes dependencies, and HashiCorp recommends it only for exceptional situations. It doesn't mean "touch exactly this module and nothing else." The resulting plan still needs inspection, and a subsequent full plan is needed to see what remains pending.

There is another distinction worth making. We used terraform plan -lock=false for the initial inspection with read-only credentials. The credentials restricted writes; -lock=false only disabled state locking. It isn't a general safety option. Keep locking enabled for normal deployment work, and don't use an unlocked inspection while another operation may be changing the same state.

The bucket had a rule that mattered to the application.

Each hour, an upstream system uploads data files, followed by a small marker file saying the upload is finished. The marker triggers an S3 notification to an SQS queue. A retry must not overwrite that marker and trigger another object-created event for the same hour.

We enforced conditional writes for markers through a bucket policy. The uploader had to send If-None-Match: *, meaning "create this object only if the key doesn't already exist." Ordinary data files didn't need that header.

Reading the policy back would show that it had been saved. I wanted to know whether it actually enforced those rules.

Before deploying to the real environments, the agent created a throwaway bucket and queue in a test account and exercised the same policy:

The agent then removed the resources it had created for the test.

There is an important limit here: preventing marker overwrites doesn't guarantee exactly-once processing. S3 notifications and standard SQS queues can deliver duplicates. Seeing one message in a test doesn't change that contract. The downstream consumer still needs to be idempotent, and we passed that requirement to its developers.

After each deployment, the agent ran a smoke test using the application's service-user credentials. A successful upload with a privileged deployment identity wouldn't tell us whether the application could do the same thing.

Here is the production output, with names shortened:

identity: user/<app-user>-prod content folders: input/content/ output/content/ 1 put without header (expect AccessDenied): AccessDenied 2 put with If-None-Match (expect OK): OK 3 duplicate put (expect PreconditionFailed): PreconditionFailed 4 parquet without header (expect OK): OK 5 other bucket (expect AccessDenied): AccessDenied 6 input queue: smoke-test events=1 (expect 1) 7 output queue: non-test events=0 (expect 0) cleanup: OK Enter fullscreen mode Exit fullscreen mode Line 5 mattered as much as the successful upload: the identity couldn't access the other bucket we tested. Line 7 checked that the test hadn't produced an event in the wrong queue during the observation period.

The production queues were dedicated to this application, brand new and not yet in use. That made it possible to inspect the test messages without competing with a live consumer. This procedure would need a different design for an active queue.

The test objects used a year-2000 date to make them recognizable, and cleanup removed only the objects created by the test. The fake date was a naming convention, not protection against real processing. The fact that the queues weren't in use was the relevant condition here.

In the first test environment, uploads initially produced no visible messages. The configuration looked correct, but it took roughly five to eight minutes before we observed notifications flowing.

That was our observation, not proof of a particular AWS root cause or a delay to expect every time. S3 notifications can be delayed, and SQS message counts are approximate. The useful response was bounded waiting and checking the actual messages, rather than immediately changing a configuration that might already be correct.

The agent also produced two false alarms in its safety checks. One treated an empty response as a failure; another expected a field that its query hadn't returned. There were smaller script bugs too: a check inspected the wrong part of the data, and extra command output broke a later step.

A guard that stops on uncertainty can be useful. But if the guard itself is broken, "the deployment was blocked" doesn't tell you much.

We needed to verify that it recognized valid results as well as unexpected ones.

Those script and checking bugs were fixed before the production rollout. AWS timing behaviour remained something the tests had to accommodate.

For this change, the evidence was fairly concrete:

These checks can live in scripts or a testing framework. They don't need an LLM to decide whether AccessDenied matches an expected denial. The agent was useful for investigating the environment, preparing the tests and working through unexpected results. The repeatable comparisons should remain explicit.

The most useful part wasn't the final successful apply. It was discovering what that apply would have included, and then checking that the deployed resources behaved as the application required.

That's the evidence I want alongside an agent's summary.

Next, in Part 3: reviewing the agent's work with independent agents, what those reviewers caught, and why we stop after three rounds.

Technical references: Terraform plan and targeting, S3 conditional-write enforcement, S3 notification delivery, and SQS approximate message counts.