Mathematical Fragility
It is not calculating. It is predicting what a calculation looks like.
This page covers one specific way AI gets things wrong at work, and what to do about it.
It runs in order. What goes wrong, why it happens, where you'd notice it on an ordinary day, who takes the blame, roughly what it costs, and the check that catches it. Then one thing to try this week.
The dollar figures are estimates, not measurements. The assumptions behind each one are printed right there, so you can swap in numbers that fit your job. Anything actually measured carries an OBSERVED tag.
What is mathematical fragility?
A language model predicts tokens. It does not execute operations. Correct answers on common arithmetic reflect the frequency of those patterns in training, not computation.
Reliability degrades along predictable axes: more digits, more steps, unusual number combinations, multi-step problems requiring intermediate state. Compositional tasks — where the output of one operation feeds another — degrade fastest.
Tool use changes the failure surface, not the failure rate. When a model calls a calculator or writes code, the arithmetic becomes correct. What remains fallible is everything around it: whether the right numbers were extracted, the right operation selected, the right units applied, the result interpreted correctly. So errors migrate from computation to setup — and setup errors are much harder to spot, because the arithmetic now checks out perfectly.
What do people assume?
That arithmetic is the safest thing to delegate — deterministic, checkable, unambiguous. That if anything is reliable, it's the math.
Arithmetic is the thing a language model is structurally least suited to, and modern tool use has made this harder to detect rather than easier.
Where does it show up at work?
A pricing analyst asks for margin impact across a tiered discount structure. The model calls a code tool. The code runs correctly.
It applied the tier-3 discount rate to the full volume rather than the incremental volume above the tier-3 threshold. Every number is arithmetically correct. The model is wrong.
Checking the math confirms the math. That is exactly why it survives review.
Who carries the downside?
Vendor: none. Executive: approves the structure. Manager: owns the margin. You: built the analysis. And "the calculation was right" will not help you, because the calculation was right.
What does it cost?
[MODELED — not reported]
ASSUMPTIONS Quantitative analyses / year: 120 Rate w/ setup or interpretation error surviving review: 8% (~10 / year) Rate caught downstream: 55% Errors reaching a decision: ~4 / year Cost per error: $10,000 – $80,000
Annualized exposure: ~$40,000 – $320,000
How do you control for it?
Verify the setup, not the arithmetic. Three checks, in this order:
1. Units and base. What is each number measured in, and against what denominator? 2. Operation selection. Is this the right calculation for the question asked? 3. Order-of-magnitude sanity. Does the result sit in a plausible range given what you know?
Then independently recompute one pivot cell — the number with the largest exposure — by hand.
CONTROL COST Quantitative analyses: 120 / year Setup verification: 12 minutes each Annual: 24 hours Fully loaded rate: $80 / hour
Annualized control cost: $1,920
What should you do this week?
RECOMMENDATION
On your next AI-assisted quantitative analysis, write down the three setup assumptions before you look at the answer: what each number measures, what operation applies, and what range the result should fall in.
Then compare. You are pre-registering your expectation, which is the single most effective defense against a plausible wrong number — and it is a discipline that transfers to every analytical task you will ever do, with or without AI.
Evidence
RESEARCH Degradation on compositional and multi-step arithmetic.
ANALYSIS The claim that tool use migrates errors from computation to setup is the author's, and it is the most important argument in this chapter. Verify or clearly label — do not present as research.
This is one of 24 failure modes. The book gives you all of them — plus the controls that catch each one and a 90-day plan to prove you ran them.
Preorder the Book