In 2025, METR ran a randomized trial on 16 experienced open-source developers across 246 real tasks on codebases they already knew well. Before starting, they predicted AI tools would make them 24% faster. The tools made them 19% slower. Afterwards, they still believed they’d been about 20% faster.
Not “AI is bad” — the study is narrow: experienced devs, mature codebases, early-2025 tooling. The finding that matters is smaller and worse. They could not feel the difference. Speed and the illusion of speed are indistinguishable from the inside.
That’s the argument for gates. Not vibes. Gates.
I build with Claude Code, Codex, and whatever model fits the job. They carry my speed. The architecture, the review, and the accountability are mine.
| The agent | Me |
|---|---|
| Drafts code from a written spec | Writes the spec, decides the architecture |
| Refactors, writes the first tests | Reviews every diff, adds the tests that matter |
| Works at 2 a.m. | Answers the email at 9 a.m. |
What actually breaks
The tests agree with the code instead of the requirement. Ask an agent to write tests for a function and it writes tests that describe that function. They pass. They’ll keep passing while the function does the wrong thing, because they were derived from it. Kent Beck, who invented TDD, has described agents going further and simply deleting a failing test to get green. Green suite, zero information. It’s the most misleading artifact in this line of work.
It works great in the room it was built in. A fire-inspection prototype ran perfectly inside a Claude.ai Artifact and died every time as a plain file in a browser: its extraction call hit Anthropic’s API with no key of its own, which only works while Claude’s environment quietly proxies the request. The code wasn’t wrong, exactly. It was written by something that could see one room. Run it where the user will run it. A screenshot from the agent is not evidence.
It reaches for the model when plain code is cheaper. Rebuilding that tool, I opened the PDFs. They already carried clean text layers. Extraction runs through a deterministic parser now, with a model behind it as a fallback for scanned paper only: under a second per report, 32 of 32 tests passing, zero dollars a month. An agent reaches for the model because your prompt said “AI.” Ask what the data actually is first.
The gates
Google’s 2025 DORA report found AI adoption now correlates positively with delivery throughput — and still negatively with delivery stability. Lead author Nathen Harvey’s summary: “AI is an amplifier.” It multiplies whatever discipline you already had. So the discipline has to be mechanical, not aspirational.
Failing tests as the spec. On a rules engine I’m building for a licensed California contractor, the first file in the repo is a regression test for every known error in the report it replaces:
"""Each test would have caught one error that shipped under a contractor's license.
They fail until the rules exist. That is intentional. This file is the spec."""
pytestmark = pytest.mark.xfail(reason="rules not implemented yet", strict=False)
@pytest.mark.parametrize(("primary_sqft", "expected_max"),
[(1736, 868), (1400, 850), (1000, 850), (2400, 1200)])
def test_attached_adu_size_has_an_850_floor(primary_sqft, expected_max): ...Write the assertion before the implementation exists and it can’t be derived from the implementation. The 1,736 case passes either way — the 1,400 case is the one that catches the missing floor. That’s the whole trick.
Gates that fail the build, not warn in a log. In Rudy, the Cal Poly EHS safety assistant, a build-time check kills the deploy outright if any emergency phone number can’t be verified against source documents. On this site, a claims checker fails the build if a page carries a claim I can’t point at. A warning gets skimmed. A red build does not.
Check it yourself, in a real browser. The agent’s account of what it built is a hypothesis, not a result.
What I don’t let it do
Decide the architecture. Own auth or payments without a written spec and tests that fail closed. Invent a number. Choose what ships. Be the only thing that has read the diff.
The model can explain a decision. It doesn’t get to make the rules.
What this is not
Not a claim that agent review replaces a second engineer: reviewer passes share blind spots, and none of them own the outcome. Not a claim that I catch everything either — I ship things with open items and write them down. And not universal. This is solo work on tools I can hold in my head.
But the METR number is the one I keep coming back to. The speed is real. You just can’t feel whether you’re spending it well, which is why the boring gates aren’t optional.
I take on a small number of builds like these alongside my day job, and each one ships with the tests, the docs, and the accounts in the client’s name.