When Faster Means Worse
Speed is a virtue in systems where errors don’t compound. In systems where they do, speed is the problem
On 1 August 2012, Knight Capital Group deployed a software update to its trading system. The update contained a bug. The system began executing trades at a rate of several thousand per second, in patterns that had no connection to the strategy it was meant to run. By the time someone pulled the plug, 45 minutes later, the firm had lost around $440 million. Knight Capital, which had been a top-tier market maker on the New York Stock Exchange, had to be rescued within days. It never recovered its independence.
The bug wasn’t catastrophic on its own. It was catastrophic because it executed at speed. A slower system would have flagged anomalous behaviour before the losses compounded. A human trader making bad decisions would have run out of positions or capital within minutes. The software, operating at machine speed with machine confidence, did enough damage to bankrupt the firm before the control mechanisms could respond.
This is what “faster means worse” looks like in a real system. Speed without correction is not a productivity gain. It’s a risk multiplier. Every AI deployment that operates faster than its verification layer is running the same experiment, usually without realising it.
The conditions that make speed dangerous
Speed is helpful in systems that have three properties. Errors are local and bounded. Feedback is fast. Correction is cheap. A calculator is a good example. It runs at speed, any mistake affects one computation, you see the result immediately, you can redo it in seconds if it’s wrong.
Speed is dangerous in systems where any of those three conditions fails. In a trading system, errors propagate across positions and counterparties. In a content system, errors reach audiences and get repeated. In a clinical system, errors affect patients and may not become visible for weeks. In a legal system, errors get signed into contracts and only surface during disputes. The faster the production, the more error is produced before anyone notices.
Rowan Jackson’s framework on errors, rework, and waste makes the point directly. “Technology does not replace human error. It just makes it faster. In some cases more consequential.” The second sentence is the important one. Most of the damaging deployments of technology aren’t failing because the error rate went up. They’re failing because the error rate stayed roughly the same while the production rate went up an order of magnitude. The absolute volume of error rises with volume. Beyond a certain threshold, the verification layer can’t keep up.
AI sits exactly at the point where this threshold is being crossed in domain after domain.
The Next Rung is a book on how AI is quietly dismantling the middle of knowledge work, and what you can do about it before the market decides for you: pre-order it before it publishes in January.
The 737 MAX, a different kind of speed
Boeing’s 737 MAX is a different shape of the same problem. It wasn’t a trading algorithm running too fast. It was a development programme running too fast. Boeing was under commercial pressure to match Airbus’s A320neo, and it compressed the development and certification timeline on an aircraft that, to avoid expensive re-engineering, was fitted with a flight control system called MCAS that pushed the nose down in certain high-angle conditions.
MCAS relied on a single angle-of-attack sensor. Pilots weren’t told it existed in most training materials. When the sensor failed, the system activated without warning, and the pilots didn’t know what was happening or how to disable it. Two crashes, 346 deaths, the aircraft grounded globally for almost two years.
The speed that caused the 737 MAX disaster wasn’t operational speed. It was decision speed. Boeing moved faster than its safety culture could verify. The regulator moved faster than its own inspection protocols. The airlines moved faster than their training programmes. Every individual step looked reasonable in isolation. The system as a whole outran its own verification capacity.
This is the shape of the AI deployment risk right now. Not that any individual AI system is catastrophically broken. That the collective deployment pace exceeds the capacity of regulators, auditors, firms, and users to verify what the systems are actually doing. The analogous failure is not a single chatbot telling a single customer the wrong thing. It’s the distributed, cumulative effect of thousands of AI-driven decisions that each look defensible and collectively produce systemic outcomes nobody chose.
The flash crash class
Financial markets have given us a library of “faster means worse” failures. The 6 May 2010 flash crash in US equities wiped almost a trillion dollars off US stock values in minutes before the market recovered. The 15 January 2015 Swiss franc move, when the Swiss National Bank removed its peg, destroyed several retail forex brokers and caused tens of millions in losses for their clients before anyone could react. The 23 September 2022 sterling crash during the Truss mini-budget was partly driven by automated deleveraging in pension funds’ liability-driven investment strategies, which sold gilts into a falling market and triggered further falls.
Each of these events shares the same structure. Automated systems operating at speed, making decisions that are individually rational but collectively destabilising, running faster than the humans who designed the systems could intervene. The systems don’t fail because they’re broken. They fail because they’re working exactly as designed in a market state the designers didn’t anticipate.
AI is now being deployed into an economy full of systems with this property. Supply chains. Pricing engines. Underwriting. Content recommendation. Fraud detection. Each system individually looks fine. Each system makes decisions at speed. Each system is supervised by humans who were told the AI would help them make better decisions faster.
The question nobody is asking is what happens when several of these systems interact in ways their designers didn’t model. The history of financial markets suggests the answer is: occasional catastrophic failures that nobody planned for and nobody can stop in time.
Normal accident theory, briefly
Charles Perrow published Normal Accidents in 1984, analysing why complex, tightly-coupled systems produce catastrophic failures with a regularity that surprises their designers. The argument is that in systems with many interacting components, sooner or later some combination of events will occur that nobody modelled. The failure looks bizarre after the fact, but the probability of some failure of some shape is close to one, given enough time.
Three Mile Island, Chernobyl, the Challenger explosion, the Deepwater Horizon spill. These are the textbook cases. Perrow’s point is that you can’t fully prevent them through better engineering. The complexity itself generates failure modes that outpace the ability to anticipate them.
AI deployment is rapidly producing systems that match Perrow’s criteria. Many components. Tight coupling between them. Operators who don’t fully understand the internal logic. Feedback loops that aren’t visible until they trigger. Decision speeds that exceed human intervention capacity. The conditions for a normal accident are in place. The question is what the specific form looks like.
It probably won’t be a Skynet scenario. It’s more likely to be a sequence of compounding errors across an industry, at a time nobody anticipated, producing a systemic outcome that looks obvious in hindsight and was unpredictable in prospect. The 2010 flash crash is a better model than the AI-apocalypse fiction. Something goes wrong, the system amplifies it, by the time humans intervene there’s substantial damage.
The Next Rung is a book on how AI is quietly dismantling the middle of knowledge work, and what you can do about it before the market decides for you: pre-order it before it publishes in January.
The ordinary version
This isn’t only about rare catastrophic events. The more common version is the steady, compounding degradation of quality in systems where AI deployment has outrun the verification layer.
Every piece of AI-generated marketing content that goes out without proper review is a minor instance of this. Every AI-drafted legal clause that slides through a busy deal cycle. Every AI-summarised medical note. Every AI-assisted underwriting decision. The individual errors are small. The aggregate effect is a lowering of the quality of everything the AI touches, slowly enough that it doesn’t trigger alarms.
Rowan’s 30 per cent ERW figure for construction is the analogue. In construction, rework has been running at around 20-30 per cent of project value for years. Nobody treats it as a crisis, because the cost is distributed and absorbed. The buildings still get built, eventually, at higher cost than advertised. The industry operates at a persistent productivity discount to what it could achieve.
AI is about to create the equivalent ERW overhang in every sector it touches. The costs will be absorbed in cost of sales, in customer complaints, in regulatory fines, in lost contracts, in the slow erosion of institutional credibility. No single event will trigger a reckoning. The cost will just sit there, spread across the economy, invisible on any individual balance sheet.
The NHS orthopaedic consultant Rowan spoke to put clinical rework at 60 per cent. That’s not a scandal that breaks. It’s a background rate that persists. AI deployment into clinical workflows, without proper verification, will push that number up before it pushes it down.
What “fast” should actually mean
Speed is useful when it’s matched to the system’s capacity to verify and correct. The virtue of a calculator isn’t that it’s fast. It’s that it’s fast and accurate and the user can check the output in the same cognitive register. The virtue of a well-designed automation isn’t speed either. It’s speed paired with observability, auditability, and a correction mechanism that runs at comparable speed to the production mechanism.
AI systems, as currently deployed, routinely fail this pairing test. The production speed is high. The verification speed is human-paced, because verification requires substantive understanding and that’s slow. The gap between production and verification is where the errors live.
A useful framing for any AI deployment: what’s the ratio of production speed to verification speed? If it’s 10:1 or higher, the deployment is running with a verification deficit. Either you accept the error rate and manage the consequences, or you invest in substantive human review and lose the speed advantage, or you build automated verification that operates at comparable speed to production.
Most deployments take option one, quietly, because options two and three are expensive. This is the setting in which faster means worse. Not always dramatically. Sometimes catastrophically, as in Knight Capital or 737 MAX. Usually just as persistent quality degradation, absorbed as the cost of doing business, showing up later in places the accounting doesn’t track.
Back to the Next Rung
Speed is the economic reason AI deployments displace workers. If the AI weren’t faster, the displacement wouldn’t happen. The productivity case depends entirely on the speed advantage.
The piece I want to keep making is that speed without verification is not productivity. It’s liability, booked to future periods. The displacement happens now. The rework cost comes later. The gap between them is where the optimists live, and where the workers lose their jobs.
If you want a clean framing for any AI deployment decision, it’s this: are we moving faster than we can verify? If the answer is yes, the deployment isn’t solving a problem. It’s storing one up. And in the meantime, it’s eliminating the middle-skill jobs that would have provided the verification if we’d kept them.
The Next Rung argument doesn’t require AI to fail catastrophically to land. It only requires AI to do what it’s clearly already doing: displace the middle layer of workers faster than the economy can absorb them, while producing output faster than the quality layer can verify it. Both effects compound. Both effects happen quietly. Both effects will eventually surface in ways nobody currently wants to talk about.
With thanks to Rowan Jackson, whose framing that “technology does not replace human error, it just makes it faster, in some cases more consequential” sits at the centre of this argument.
Primary sources: SEC report on the Knight Capital incident (2013); US House Committee on Transportation & Infrastructure, The Design, Development & Certification of the Boeing 737 MAX (2020); SEC-CFTC joint report on the 6 May 2010 flash crash; Bank of England reporting on the September 2022 gilt market events; Charles Perrow, Normal Accidents: Living with High-Risk Technologies (1984); Rowan Jackson, ERW paper, December 2024.