No Real-World Verification
It can tell you what the invoice should say. It cannot tell you what the invoice says.
This page covers one specific way AI gets things wrong at work, and what to do about it.
It runs in order. What goes wrong, why it happens, where you'd notice it on an ordinary day, who takes the blame, roughly what it costs, and the check that catches it. Then one thing to try this week.
The dollar figures are estimates, not measurements. The assumptions behind each one are printed right there, so you can swap in numbers that fit your job. Anything actually measured carries an OBSERVED tag.
What is no real-world verification?
A model operates on text. Retrieval brings more text. Neither operation observes the world; both observe records about the world, and records diverge from reality constantly — a status field not yet updated, a record for a similarly-named account, a system that reconciles nightly and it is currently 2pm.
The model has no way to detect divergence, because it has nothing to compare the record against. From the inside, a stale record and a current one are identical.
This compounds with authority confusion: when two systems disagree, the model has no principled basis for preferring one. It will typically prefer whichever text is more prominent in its context, which is an artifact of retrieval order, not of correctness.
What do people assume?
That a system with search or database access is checking reality — that "connected" means "verified against the world."
Connection provides access to a representation. The representation may be stale, partial, wrong, or about a different entity than the one you meant.
Where does it show up at work?
A customer success manager asks an assistant to confirm an account's entitlement before promising a feature. The assistant checks the CRM, which reflects the signed order. The provisioning system — the one that actually controls access — was never updated after a downgrade.
The promise goes out. The feature isn't there.
Who carries the downside?
Vendor: none. Executive: none. Manager: owns the escalation. You: made the commitment to the customer in writing.
What does it cost?
[MODELED — not reported]
ASSUMPTIONS Customer commitments made from system-of-record lookups: 600 / year Rate where the record diverges from operational reality: 4% (~24 / year) Rate caught before commitment: 70% Bad commitments: ~7 / year Cost per incident: $2,000 – $20,000 (remediation, credits, churn risk)
Annualized exposure: ~$14,000 – $140,000
How do you control for it?
Name the system of record for each fact class, in writing, and verify against that system — not against whichever system answered first. This is the Authoritative Source Register (Part V) applied to internal systems rather than external sources.
For high-consequence commitments: confirm in the controlling system, not the reporting one.
CONTROL COST High-consequence commitments: 200 / year Verification: 5 minutes each Annual: 17 hours Fully loaded rate: $60 / hour
Annualized control cost: $1,020
What should you do this week?
RECOMMENDATION
List the ten facts your function commits to customers or executives most often. Next to each, name the system that actually controls it — not the one you usually look at. Those are different in more cases than you expect.
Where they differ, you have found a control gap that predates AI by years. AI didn't create it; AI just increased the rate at which you query the wrong system. Being the person who mapped this is worth more than being the person who complains about data quality.
Evidence
ANALYSIS Entire mechanism section is architectural reasoning, not a research finding. Label accordingly — do not dress this as empirical.
ANALYSIS Retrieval-order effects on authority preference.
This is one of 24 failure modes. The book gives you all of them — plus the controls that catch each one and a 90-day plan to prove you ran them.
Preorder the Book