A better tool doesn’t know what to build.

The study nobody expected

METR, a research group that tests AI systems, ran a controlled trial in early 2025. Sixteen experienced developers worked through 246 real tasks on code they knew well, some with AI tools and some without.

Before starting, they predicted AI would make them 24% faster. The measurements showed they took 19% longer. And when it was over, they estimated AI had made them about 20% faster.

That is the part worth sitting with. It’s not that the tools were slow. It’s that feeling fast and being further along turned out to be different things, and nobody in the room could tell them apart.

To be fair, the tools have moved on. METR’s own update in February 2026 says developers are probably sped up now, while cautioning that its new data is weak evidence of by how much. We use these tools every day and they do save time. But the gap between how fast it feels and how far you’ve actually got hasn’t gone away.

The same pattern, at company scale

In mid-2025, MIT’s NANDA initiative looked at how businesses were getting on with generative AI. Its report, The GenAI Divide, estimated that companies had put $30–40 billion into it, and that about 95% of organisations had no measurable effect on profit and loss to show for it.

The report didn’t blame the models. It pointed to what it called a learning gap: most systems don’t retain feedback, adapt to the business’s context or improve over time. In other words, the tool worked. Nobody had built it into how the business actually runs.

Where the speed actually comes from

A new driver won’t fix a slice. Neither will a faster way of writing code fix a system nobody has thought through.

The speed in a software project comes from knowing what to build:

• Understanding the operation well enough to know which problem is worth solving.
• Getting the data model right before the first screen is drawn.
• Knowing which systems it has to talk to, and what happens when one of them is wrong.
• Knowing what a failure costs, so the right things get tested.

AI can’t supply any of that. It produces what you describe, quickly and confidently. If what you describe is incomplete, you get an incomplete system faster.

How we use it

We use AI on every project. The difference is where it sits.

• Senior engineers design the foundation: the data model, the architecture and the integrations.
• AI speeds up the routine work, under their direction.
• Every release is tested and reviewed by a senior developer before it goes live.
• You get a written scope, with the cost, before any build starts.

The tool makes us quicker. It doesn’t decide what gets built.