Hallucination
The model does not know the difference between remembering and inventing. Neither does the output.
This page covers one specific way AI gets things wrong at work, and what to do about it.
It runs in order. What goes wrong, why it happens, where you'd notice it on an ordinary day, who takes the blame, roughly what it costs, and the check that catches it. Then one thing to try this week.
The dollar figures are estimates, not measurements. The assumptions behind each one are printed right there, so you can swap in numbers that fit your job. Anything actually measured carries an OBSERVED tag.
What is hallucination?
A language model produces tokens by predicting what is statistically likely to come next given everything before it. It has no internal flag distinguishing "this sequence corresponds to something true" from "this sequence is well-formed." A fabricated legal citation has the same shape as a real one — volume number, reporter abbreviation, page, year — and shape is what the model optimizes for.
This gets worse in exactly the places you'd want it to get better. Ask about a niche topic with thin training coverage and the model doesn't fall silent; it interpolates from the pattern of what such an answer usually looks like. Low knowledge density produces more confident-sounding invention, not less. The failure is not that the model lies. It is that the model has no mechanism for the concept of lying, and therefore none for its opposite.
What do people assume?
That a model producing a specific, formatted, confident-sounding fact — a case citation, a statistic, a page number, a product SKU — retrieved that fact from somewhere. The specificity feels like evidence. A vague answer feels like a guess; a precise one feels like a lookup.
The specificity is generated the same way the vagueness would have been.
Where does it show up at work?
A market analyst asks an assistant to pull three competitor pricing data points for a board deck. The model returns three, each with a company name, a figure, and a source publication. Two are accurate. One is a plausible number attached to a publication that never ran the story.
The deck goes to the board. Nobody checks the third one, because the first two checked out.
Who carries the downside?
Vendor: none. Terms of service disclaim output accuracy. Executive: reputational, if challenged publicly. Manager: owns the deck. You: you built the slide. When the board asks where the number came from, the question routes to your name — not the model's.
This is the pattern for the entire book. The vendor sells capability and disclaims consequence. The consequence lands on the last human who touched the file.
What does it cost?
[MODELED — not reported]
ASSUMPTIONS Analyst-produced external claims: 40 / month Fabrication rate, unverified: 2% (~1 claim / month) Detection rate before publication: 50% Claims reaching an external audience: ~6 / year Cost per retraction / correction: $8,000 – $30,000 (rework, legal review, relationship repair, credibility discount on future analysis)
Annualized exposure: ~$48,000 – $180,000
The wide range is the honest answer. A corrected internal memo costs rework. A corrected claim in front of a regulator, an acquirer, or a board costs something you can't fully price — which is precisely why you should present the range rather than a fake midpoint.
How do you control for it?
Source verification on every external factual claim. Open the source. Find the sentence. Confirm the number matches. Not "does a source exist" — does this source say this thing.
CONTROL COST Claims requiring verification: 40 / month Time per verification: 6 minutes Monthly: 4 hours Fully loaded rate: $65 / hour Monthly: $260
Annualized control cost: $3,120
What should you do this week?
RECOMMENDATION
Take the last ten AI-assisted claims that left your team — a deck, a memo, a customer email, a report. Verify each one to the sentence. Not to the source; to the sentence.
Write down two numbers: how many you couldn't verify, and how long verification took.
That's your first entry in the AI Error Log (Part V) and the opening line of a conversation with your manager that does not sound like fear. It sounds like an audit, because it is one.
Evidence
REPORTED Mata v. Avianca, Inc., S.D.N.Y. 2023 — attorneys sanctioned after filing a brief containing judicial citations produced by ChatGPT that did not exist. The model also affirmed the cases were real when asked directly.
RESEARCH Hallucination as an inherent property of next-token prediction rather than a correctable bug.
ANALYSIS The claim that thin training coverage increases fabrication confidence is the author's reading of observed behavior.
This is one of 24 failure modes. The book gives you all of them — plus the controls that catch each one and a 90-day plan to prove you ran them.
Preorder the Book