Errors That Look Like Competence
A bricklayer putting a wall five centimetres out is visible. An AI fabricating a legal citation is not, and that’s the part that compounds
In May 2025, the High Court of England and Wales ruled on a case called Ayinde v London Borough of Haringey. The headline wasn’t the dispute. The headline was that a witness statement in the proceedings contained 18 fabricated legal citations out of 45. The citations looked real. They had case names, court details, citation formats that matched proper legal conventions. They just didn’t exist. An AI had made them up, and the person submitting the statement hadn’t checked.
By the end of November 2025, the UK had recorded 24 separate instances of AI-fabricated legal citations ending up in court proceedings. The global total was over 600. In July 2025 alone, more than 50 such cases were publicly reported across jurisdictions.
Judges have started imposing costs orders, referring solicitors to regulators, and in some cases warning of criminal prosecution for contempt. The Bar is producing guidance documents. Law societies are scrambling. None of this is solving the core problem, which is not that lawyers are using AI. It’s that AI is producing output that looks exactly right when it’s completely wrong, and humans cannot reliably detect the difference at speed.
This is the distinguishing feature of AI error. It’s not that errors happen faster. It’s that they look like competence.
Why this is different from previous technology errors
Every tool humans have ever invented has produced errors. Calculators get entered wrong. Spreadsheets get formula errors. Word processors have spell-check limits. Automation in manufacturing introduces systematic defects when the setup is wrong.
What’s different about AI error is the combination of three things that previously only occurred one at a time.
First, AI errors are fluent. A calculator error returns the wrong number. You can tell it’s a number. You can check it against what you expected. An AI error produces a fluent paragraph, a credible-looking citation, a recommendation that matches the surface pattern of a good recommendation. The error is wrapped in the linguistic signals of competence.
Second, AI errors are confident. There’s no uncertainty signal. The system produces the wrong answer with the same tone it produces the right answer. In earlier automation, a lot of the signalling came from hesitation, partial completion, or visible edge cases. The machine would stop, throw an error, flag the anomaly. AI systems don’t stop. They produce, and they produce fluently.
Third, AI errors are at scale. One person can generate the volume of output that used to require a team. The rate of production outruns the rate at which any human can plausibly check the work. Even if the error rate per output is low, the absolute volume of errors is high, and the proportion that escapes detection rises as volume rises.
Rowan Jackson’s framework for errors, rework, and waste identifies the same pattern in industrial settings. When processes produce outputs faster than the quality system can verify them, errors accumulate and propagate. In a factory, the errors eventually become visible because the product is physical. In knowledge work, the errors stay invisible until someone downstream acts on them and something breaks.
The Next Rung is a book on how AI is quietly dismantling the middle of knowledge work, and what you can do about it before the market decides for you: pre-order it before it publishes in January.
The bricklayer analogy
Rowan used this example in correspondence with me, and it’s worth sitting with. If a bricklayer places a wall five centimetres out of alignment, the error is immediate and physical. Someone walks past, notices, measures. The wall has to come down and be rebuilt. The cost is painful but bounded. You pay for the bricks twice, the labour twice, the time twice.
If an AI generates a legal clause that contradicts another clause in the same contract, the error is invisible. The document looks well-formed. The clauses read clearly on their own. The contradiction only surfaces when something triggers the two clauses at the same time, which might be years after signing, and might involve a dispute where the cost is orders of magnitude larger than the original drafting fee.
Rowan mentioned directly that his team has corrected clients’ contracts that had clauses directly contradicting each other. These weren’t AI-generated. They were the result of humans copy-pasting from templates without reading carefully, which is the same failure mode AI is now industrialising. The human version was rare. The AI version will be common, because the production speed is higher and the verification pressure is lower.
The cost of the bricklayer’s five-centimetre error is visible in the rework budget. The cost of the contract error is hidden until it activates. By the time it activates, the person who drafted it has moved on, the firm that employed them has billed the fee, and the client is the one holding the liability. This is not a technical problem. It’s a structural one.
The Air Canada precedent
In February 2024, the British Columbia Civil Resolution Tribunal ruled in Moffatt v Air Canada. The facts are simple. A customer asked Air Canada’s website chatbot about bereavement fares for a grandmother’s funeral. The chatbot gave incorrect information. The customer relied on it and booked a full-fare ticket. Air Canada then refused to honour the bereavement rate the chatbot had promised.
Air Canada’s defence was remarkable. It argued that the chatbot was a separate entity whose misrepresentations were not the airline’s responsibility. The tribunal rejected this, ruled the airline liable, and awarded around $650 plus fees.
The damages are trivial. The precedent is not. The case established that a company is liable for the misrepresentations made by its AI, and that “the AI said it” is not a defence. Every deployment of customer-facing AI since Moffatt sits in this legal context, whether the deploying company realises it or not.
More importantly, the case demonstrated that a fluent-sounding AI error can produce a legally binding commitment. The customer relied on it. The tribunal found reasonable reliance was warranted because the information appeared competent. The fluency was the problem. If the chatbot had stammered or said “I don’t know”, the customer would have checked. Because the chatbot sounded like it knew, the customer acted on it.
Multiply this across financial services, insurance, healthcare, legal services, government. Every AI interface that produces output in natural language is now a potential source of binding commitments by the deploying organisation, whether the output is accurate or not.
The detectability gap
Here’s where the problem gets structural. Consider how humans check each other’s work.
A junior associate’s contract draft is reviewed by a senior associate. The senior associate has years of pattern recognition. She’s seen enough bad drafts to know what a weak clause looks like, what a common miswording is, what pattern of structure suggests the junior didn’t understand the underlying transaction. The review is possible because bad human work tends to carry signals of its badness, at least to a trained reader.
A junior associate’s AI-generated contract draft, handed to the same senior associate, carries different signals. The clauses are grammatically clean. The structure matches template conventions. The terminology is consistent. The language signals of bad work are absent. What’s left is substantive error, which requires substantive review. That takes approximately the same time as writing the contract from scratch.
This is the core problem with quality control at scale for AI output. The surface signals that used to help reviewers triage bad work have been erased by the AI. Every piece of output now demands substantive checking, because the superficial checks no longer discriminate. Either reviewers slow down to do substantive review on everything, in which case the productivity gain evaporates, or they speed up and let substantive errors through, in which case the rework cost shows up later somewhere else.
There’s no middle path. The old quality control system was calibrated to a world where poor work looked poor. That world is gone.
The Next Rung is a book on how AI is quietly dismantling the middle of knowledge work, and what you can do about it before the market decides for you: pre-order it before it publishes in January.
The hallucination metric problem
Vendors of AI systems report hallucination rates. OpenAI, Anthropic, and Google all publish benchmarks. The numbers have been coming down. State-of-the-art systems in 2025 are reporting single-digit hallucination rates on standard evaluation sets.
This sounds reassuring. It isn’t.
A single-digit hallucination rate means roughly one in every 15 to 25 outputs contains a fabrication. If you’re producing legal documents, medical summaries, or financial analyses at a rate of hundreds per week, you’re producing dozens of hallucinated outputs per week. Even if each one has only a small probability of causing material harm, the portfolio-level risk is substantial.
More importantly, benchmark hallucination rates are measured on evaluation tasks where the ground truth is known. The hallucination rate on novel, open-ended, commercially relevant tasks is generally higher and is not measured in any systematic way. The vendor reports the benchmark number. The user acts on the novel task. The gap between the two is invisible.
And the benchmark itself is a lagging indicator. Every time a category of hallucination becomes publicly embarrassing, the next model version is tuned to address it. This creates the impression that hallucinations are being solved, when what’s actually happening is that the hallucinations are being redistributed into categories that haven’t yet been measured.
What this does to the next rung argument
The original Next Rung thesis argued that AI is automating cognitive work across the board, leaving displaced workers without a clear next job to move into. One of the standard objections to this argument is that humans will still be needed for quality control. Humans will review the AI output, catch the errors, ensure quality. The displaced workers can become AI auditors.
This objection looks much weaker once you understand the detectability gap. AI errors don’t announce themselves. They require substantive expert review to catch, which takes roughly the same time as doing the original work. You don’t get a productivity gain if you have to substantively review every output. You just shift where the labour happens.
And substantive review scales worse than production. A senior lawyer can draft one contract at a time. She can’t review ten AI-generated contracts at a time with the same rigour. The production side scales with compute. The review side scales with human expertise, which is rare, expensive, and slow to develop.
The practical outcome is not a new category of AI auditor jobs. It’s a quality degradation spread across the economy, where the output volume goes up and the average quality goes down, with the losses absorbed by customers, patients, clients, and downstream processes that can’t detect the errors until they’ve already acted on them.
What to do
For any organisation deploying AI into a workflow where errors matter, three actions are non-negotiable.
First, assume the AI is producing some proportion of fluent-but-wrong outputs, and design the review process around substantive verification rather than surface checking. The days of skim-reading junior output are over. Every piece of AI-generated work that goes external needs line-by-line review by someone qualified to spot substantive error.
Second, track error rates downstream, not just at the point of production. If your customer service AI is giving wrong answers, the metric that matters is customer complaint volume and churn, not the chatbot’s internal confidence score. The error rate at the point of production is often low. The error rate at the point of consequence is higher.
Third, accept that the productivity gain from AI in high-stakes work is smaller than the vendor claims, because most of the saved time on production will be spent on verification. If you’re going to deploy AI in legal, medical, financial, or safety-critical contexts, budget for the verification overhead from day one. If the business case doesn’t work once you include it, the deployment probably shouldn’t happen.
The speed is real. The scale is real. The quality signalling, for the first time in the history of technology, has been decoupled from the quality itself. That’s what’s new. That’s what matters.
With thanks to Rowan Jackson, whose observation that contract drafting errors are a particularly hidden category of rework, and whose example of clauses that directly contradict each other within the same document, sharpened the legal section of this piece.
Primary sources: Ayinde v London Borough of Haringey and Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin); Moffatt v Air Canada, 2024 BCCRT 149; Damien Charlotin’s AI Hallucination Cases Database; the Bar of England and Wales guidance on AI use, 2025; Vectara hallucination leaderboard; Matthew Syed, Black Box Thinking.