The better the models get, the harder it is to tell whether they did the job, and the less practice anyone has at telling. The problem isn’t new either: it has a name, and it’s from 1983.
When I finished my novel, I put two models from different vendors on the last four unedited chapters, one as editor and one as independent reviewer. Both reported no errors. I read the chapters myself and found that something happens to a supporting character near the end while the main characters, who have strong ties to them, don’t react at all. Two models, zero flags.
Ambition needed
The advice I’ve seen going round developer circles right now is to be more ambitious. A recent Hacker News thread about the newest models kept coming back to it: you have to push the ambition of your prompt to see what they can do. Dan Luu makes the case properly, and it’s a good case. Doing what you did five years ago, but with AI, gives you a modest speedup. The big gains are in work that used to be too expensive to attempt (he built a regex engine, a ripgrep fork and a Rust interpreter in a few hours each, at “works for me” quality).
He also prints the strongest objection inside his own essay. He quotes Yossi Kreinin, who argues that most people should be less ambitious, because most problems have no clean pass or fail, and most people can’t reason about a black box they can only constrain. One reply in that HN thread put the bargain more bluntly: the strongest models now produce more than you can review in the time it took them to write it, so if you want to understand the result, use a weaker model on a smaller task.
Bainbridge, 1983
In 1983 the psychologist Lisanne Bainbridge published a short paper called Ironies of Automation, about control rooms in process plants. Automate the routine work, she pointed out, and the human is left with two jobs: watching the system, and taking over when it fails. People are bad at the first, since attention drops off within about half an hour, and she called that “an impossible task”. The second needs skills that fade precisely because the machine now does the work that kept them sharp. So the operator who has to step in during an abnormal situation is the least practised person in the building at the moment practice matters most, and the more reliable the system, the rarer that practice gets.
Swap the control room for a report, a codebase or a tender and nothing in the argument changes, except that it now hits from both ends. Juniors who start with the model never build the skills in the first place, while seniors who built them years ago lose them one handed-over task at a time, and it’s usually the seniors doing the signing.
The counterfeit gets better
Luu’s sharpest point is in a footnote. Right now, he says, the people who get the most out of models are the ones who already have the skills the models are replacing. He borrows a framing from Max Bittker: models are very good at counterfeiting, and the expertise it takes to tell a counterfeit from the real thing goes up as the models improve. Luu adds that with every more capable model he ends up further from “it just works”, because the gap between what a model can build and what it can verify keeps widening. (He’s careful to say this isn’t a claim about the future, which is fair, but it is a claim about now, and now is when the reports are being signed.)
Put Bainbridge and Bittker side by side and you get what I’ll call the signature gap. Ambition means handing over more. Handing over more means practising less. The counterfeit gets more convincing in the meantime. And whoever puts their name on the result needs the most judgement at exactly the point they’ve had the least practice using it.
Works for me
None of this is an argument against using the tools. A Postgres developer in Luu’s essay had a model build a fiendishly specific code-formatting function for his editor and admits he has “only a vague understanding of how all this works”. That’s fine, because it formats his code and if it breaks he’s the only one inconvenienced.
The trouble starts when “works for me” gets a signature on it and goes to somebody else: a client, a court, a patient, a board. Services have never really been paid for the knowledge. They’re paid for the person who signs, adapts the advice to the situation and picks up the phone when it goes wrong. A model can produce the document. It can’t be the one who gets called.
Mitchell Hashimoto has a decent test for where that line sits, which he calls the “whiteboard defense”: anyone should be able to pull you aside and ask you to explain any customer-facing system you’ve shipped, how it works and why you made the decisions you made. It’s a good benchmark for today, though Bainbridge’s point is that passing it this year says little about passing it in three, once the model has been doing the work in between.
Gartner, never shy of a forecast, now expects more than 40% of agentic AI projects to be cancelled by the end of 2027, for cost, unclear value and weak risk controls. I’d guess a good share of them will be the ones where nobody could say, with a straight face, what they’d signed.
I found the problem in my novel because I read the chapter. With better models next year, I might not bother, and then I wouldn’t find it. That’s something I don’t have an answer to yet.