Skip to main content

What four rounds of adversarial AI review caught in auth infrastructure code

· 5 min read
Andrew Cates
Maintainer, next-starters

Each round found something real. The findings got smaller every time — until they stopped.

The problem​

We had a proven pattern: AWS email infrastructure, provisioned by Terraform, synced into a hosted auth provider's config, with branded transactional emails compiled from templates and pushed the same way. It worked, in production, across more than a dozen internal projects. It had just never been turned into something a new project could pick up and use — it was tribal knowledge, copy-pasted and drifting.

The task: package it as a reusable, documented pattern. Simple enough to describe in a sentence. Nowhere near simple enough to get right on the first pass — because the pattern touches Terraform-provisioned AWS credentials, a hosted auth provider's SMTP/captcha config, and an HMAC-signed internal email-sending endpoint. Get any of those subtly wrong and you don't get a build failure. You get a working-looking scaffold that quietly mishandles a secret.

What I tried​

The approach that mattered wasn't the architecture — it was the review loop around it. A plan went through four rounds of an adversarial AI critic before implementation even started, each round explicitly told to verify claims against real files on disk, not trust the plan's prose. Then, after a team of parallel agents implemented it, the implementation went through four more rounds of the same critic, re-reading source files and re-running the test suite independently each time.

The rounds narrowed in a way worth noticing:

  • Round 1 found an architectural flaw: the plan split the system along the wrong axis. Two "clean" halves shared no real connection — one produced data the other needed, but nothing wired them together. The fix wasn't a patch, it was re-cutting the boundary around the actual data flow between them.
  • Round 2, after the re-cut, found execution-level gaps: a drift-check test that would always fail against its own supporting files, and generated documentation that pointed at the wrong tool for one specific step.
  • Round 3 found something sharper: an idempotency rule meant to keep a secret-sync script from re-sending credentials on every run was too effective — it also silently swallowed a legitimate credential rotation. The script would report success while quietly not rotating a secret that had actually changed upstream.
  • Round 4 found one narrow leftover: a comment in generated output still named the internal tool the new script had replaced.

By round 4, the reviewer's own words were "no architectural replanning is otherwise required." That's the signal that the loop had converged — not a fixed number of rounds, but the point where findings stop being about the design and start being about a single stale string.

Round 1: wrong system boundary          (architecture)
Round 2: test scope, doc accuracy       (contract completeness)
Round 3: idempotency vs. secret rotation (correctness)
Round 4: one stale reference             (polish)
→ APPROVE

The same shape repeated after implementation: four more rounds, same narrowing pattern, same stop condition.

What I learned​

  • A single review pass is not enough for secret-handling code. The rotation bug in round 3 is exactly the kind of thing that passes every functional test — the script behaves correctly on every input except "the secret changed and nothing else did," which is precisely the input a real credential rotation produces. It took a reviewer specifically looking for it.
  • Split systems along the data dependency, not along the technology. The instinct to divide "infrastructure code" from "application code" felt clean and was wrong — it ignored that infrastructure output was the literal input the application half needed. The dependency should decide the boundary, and the boundary should carry a concrete, versioned artifact across it — not an assumption.
  • Track when a review loop is actually converging. Round-over-round, each finding got narrower and less architectural. That's the signal to keep going one more round, not stop — and the signal to actually stop is an explicit "no architectural replanning needed," not a fixed budget.
  • Multi-agent implementation surfaces its own race conditions. With several agents editing a shared codebase in parallel, a test failure during active concurrent work isn't automatically a bug — it can be a read caught mid-write by another agent. Re-run before diagnosing; don't let a transient read become a false story about what broke.

Takeaways​

  • For anything touching credentials, secrets, or auth config: budget for 3+ independent review rounds, not 1. The rounds get cheaper and narrower as they go — that's the tell they're working, not a sign to stop early.
  • When splitting a system into independently-usable pieces, ask what data actually has to cross the boundary before deciding where the boundary goes.
  • The single most valuable instruction to give a review pass isn't "check this" — it's "verify this against the real files, don't trust the description." Every round that mattered here found something because it re-read source, not because it re-read the plan.