Reasoning Fragility
The reasoning is a story about an answer, not the process that produced it.
This page covers one specific way AI gets things wrong at work, and what to do about it.
It runs in order. What goes wrong, why it happens, where you'd notice it on an ordinary day, who takes the blame, roughly what it costs, and the check that catches it. Then one thing to try this week.
The dollar figures are estimates, not measurements. The assumptions behind each one are printed right there, so you can swap in numbers that fit your job. Anything actually measured carries an OBSERVED tag.
What is reasoning fragility?
Step-by-step generation genuinely improves accuracy on many tasks — this is well established and worth using. But the produced explanation is not a trace of an internal computation. It is more text, generated under the same pressures as everything else: plausibility and fluency.
Two failures follow:
1. Unfaithful reasoning. The stated steps do not reflect what actually drove the output. Models can be influenced by a feature they never mention, then produce a clean rationale that omits it entirely. 2. Error propagation. In a long chain, an early mistake is not caught downstream. It is built upon — each subsequent step treats it as established, and the chain grows more confident as it grows longer.
Self-correction does not reliably fix this. Asking a model to check its own reasoning often produces a confident confirmation of the original error.
What do people assume?
That when a model shows its work step by step, the steps caused the conclusion — that visible reasoning is auditable reasoning.
The steps are generated text. They frequently correlate with the answer. They do not reliably determine it, and a correct-looking chain can accompany a wrong conclusion while a flawed chain accompanies a right one.
Where does it show up at work?
An FP&A analyst asks for a build-up of a segment forecast. The model produces eight steps of clean arithmetic and logic. Step 3 applies a growth rate to a base that already included that growth. Steps 4 through 8 are internally consistent with step 3.
The output is coherent, well-formatted, and wrong by a compounding margin that grows with every step.
Who carries the downside?
Vendor: none. Executive: allocates against the forecast. Manager: owns the plan. You: built the model. Coherence is what made it through review.
What does it cost?
[MODELED — not reported]
ASSUMPTIONS Multi-step analyses / year: 80 Rate w/ an early undetected error: 12% (~10 / year) Rate caught in review: 60% Errors reaching a decision: ~4 / year Cost per error: $8,000 – $60,000 (misallocation, missed target, rework, credibility)
Annualized exposure: ~$32,000 – $240,000
How do you control for it?
Independent recomputation of the highest-exposure step. Not a review of the chain — a separate calculation of the single number with the most riding on it, performed without reference to the model's work.
Reviewing a chain for coherence tests the wrong property. The chain is always coherent. That's the mode.
CONTROL COST Multi-step analyses: 80 / year Recomputation: 20 minutes each Annual: 27 hours Fully loaded rate: $75 / hour
Annualized control cost: $2,025
What should you do this week?
RECOMMENDATION
For your next AI-assisted analysis, identify the pivot number — the one input that, if wrong, changes the decision. Recompute it independently. Write down both figures.
Do this five times and you will have a defensible personal error rate for AI-assisted analysis in your function. Almost nobody in your company has this number. It converts an argument about AI into a measurement, and whoever holds the measurement controls the argument.
Evidence
RESEARCH Chain-of-thought improving task accuracy.
RESEARCH Unfaithfulness — stated reasoning not reflecting the actual drivers of output.
RESEARCH Limits of unaided self-correction.
This is one of 24 failure modes. The book gives you all of them — plus the controls that catch each one and a 90-day plan to prove you ran them.
Preorder the Book