Skip to main content

Three verified bugs that only showed up when I actually ran the code

· 4 min read
Andrew Cates
Maintainer, next-starters

Adversarial code review that runs the code beats adversarial code review that reads the code.

The problem​

I had a working plan for distributing small content packages (call them "skills" — self-contained bundles of docs + reference files an AI coding assistant can load) three ways at once: an existing symlink-based install, a plugin marketplace, and versioned npm packages. The plan went through several rounds of review before any code got written — a planner drafts, an architect and a critic both try to find holes, and the planner revises until nobody can find another one.

Three of those review rounds each found a real bug. Not a style nit, not a "have you considered" — an actual, reproducible defect that would have shipped broken artifacts to a public package registry. What made them worth trusting wasn't that the reviewers were thorough on paper. It's that they stopped taking the plan's word for anything and started running it.

What I tried​

The review discipline was simple to state and expensive to follow: every claim about how a third-party tool behaves gets verified against the real tool, in a scratch environment, before it's allowed into the plan.

Round one: the plan assumed a version-bump-and-publish tool would shell out to one specific packaging command regardless of context. A reviewer built a real workspace, ran the real publish path, and watched it call a different command than the plan assumed — one with different file-exclusion rules. The plan's own validation step would have silently passed while the real artifact shipped broken.

Round two: the plan piped a multi-command shell chain into a CI action's input string. A reviewer traced the action's source, found it spawns processes without a shell, and reproduced the exact failure — the chain silently truncated after the first command and the job reported success anyway.

# what the plan assumed (works in a terminal):
"version && sync-metadata && install --lockfile-only"

# what actually happens with no shell — everything after && is dropped,
# and the tool that received it still exits 0

Round three: a version-consistency rule was supposed to stop private packages from silently drifting ahead of what's actually published. It worked — until a reviewer noticed the fix itself had a timing bug: it wrote the corrected version to source control one step after the artifact was already packed and pushed to the registry. Structurally correct, sequenced wrong.

None of these were things you'd catch by reading the diff carefully. All three needed someone to actually spin up the dependency, feed it the exact input the plan specified, and watch what came back.

What I learned​

  • "I read the source and I'm confident" is a different claim from "I ran it and confirmed." The gap between them is exactly where these three bugs lived.
  • A review that only critiques the plan's logic will pass a plan whose logic is sound and whose premises are wrong. All three bugs were premise bugs — "this tool behaves like X" — not reasoning bugs.
  • Give reviewers permission to be adversarial, and give them a scratch environment to be adversarial in. A review that can only read text argues about what's plausible. A review that can spin up the real dependency argues about what's true.
  • The fix for a caught bug deserves the same scrutiny as the original plan. The version-timing bug above was found on the second review pass, in a fix for something the first pass had already caught — verification has to be recursive, not one-and-done.
  • Rolling the same pattern out to a second, third, and fourth real environment surfaces the assumptions a single environment let you get away with. The first repo this pattern shipped to was clean. The next several each had at least one thing the first one never exercised — a different tool version, an existing pipeline with conflicting conventions, content that had never been validated before. None of those were bugs in the pattern; they were gaps in how much of the real world the first pass had touched.

Takeaways​

  • Treat "I'm confident this is how the tool works" as a hypothesis, not a fact — verify it against the real tool before it becomes a load-bearing assumption in a plan.
  • When you can afford it, give a review pass a disposable environment and explicit permission to try to break the plan for real, not just to critique it on paper.
  • A bug found in round N doesn't retire the need for round N+1 to re-verify — fixes have premises too.
  • The cheapest time to find a wrong assumption about a third-party tool is before you've built five things on top of it, not after.