Training-Data Bias
The model learned how decisions have been made. You are asking it how they should be made.
This page covers one specific way AI gets things wrong at work, and what to do about it.
It runs in order. What goes wrong, why it happens, where you'd notice it on an ordinary day, who takes the blame, roughly what it costs, and the check that catches it. Then one thing to try this week.
The dollar figures are estimates, not measurements. The assumptions behind each one are printed right there, so you can swap in numbers that fit your job. Anything actually measured carries an OBSERVED tag.
What is training-data bias?
Training data is a record of human output, including a long record of human decisions with their historical patterns intact. The model learns the correlations present in that record — including correlations between demographic proxies and outcomes that were never legitimate criteria.
It does not learn these as rules it can state. It learns them as weightings that shift probability. So the output arrives as a fluent professional judgment with no visible reasoning chain to inspect. The word "bias" is nowhere in it.
Proxies are the hard part. Remove the protected attribute and correlated features remain — school, zip code, employment gap, name, phrasing. Removing the variable does not remove the signal.
What do people assume?
That bias means slurs, stereotypes, or overtly offensive output — the things content filters catch. That a system producing professional, neutral-sounding language is therefore operating neutrally.
The expensive bias is the kind that reads as reasonable.
Where does it show up at work?
A recruiter uses an assistant to rank 200 applications against a job description. The rankings look defensible. Every summary cites job-relevant reasons.
Candidates with employment gaps rank systematically lower, including where the gap is legally protected leave. The stated reason is always "less continuous recent experience" — which sounds like a real criterion and functions as a proxy.
Who carries the downside?
Vendor: increasingly some, under emerging AI-hiring regulation. Not enough to help you. Executive: named in the complaint. Manager: owns the hiring process. You: ran the screen. Discovery asks how the ranking was produced.
What does it cost?
[MODELED — not reported]
ASSUMPTIONS Roles screened with AI ranking: 40 / year Candidates per role: 150 Probability of a claim in a given year: 3% – 8% Defense + settlement cost: $75,000 – $400,000 Regulatory penalty exposure (jurisdiction-dependent): adds materially
Annualized exposure: ~$2,250 – $32,000 expected value Note: expected value understates this badly. The distribution is not symmetric — the modal outcome is zero and the tail is severe. Present both numbers, never just the average.
How do you control for it?
Adverse-impact testing on the output distribution, not review of the reasoning text. The reasoning text will always sound fine — that is the mode. Compare selection rates across groups; investigate divergence.
CONTROL COST Quarterly impact analysis: 4 / year Time per analysis: 8 hours Annual: 32 hours Fully loaded rate: $95 / hour (analyst + counsel review)
Annualized control cost: $3,040
What should you do this week?
RECOMMENDATION
If AI touches any evaluative decision about people in your function — hiring, promotion, performance, credit, pricing by segment — ask one question in writing: has anyone tested the output distribution for disparate impact?
In writing. Email, not hallway.
If the answer is no, you have identified an unowned control in a regulated area, and you have a timestamped record of having raised it. That is the single highest-leverage sentence in this chapter, and it takes four minutes.
Evidence
RESEARCH Bias transmission from training corpora to model outputs.
RESEARCH Proxy variables preserving disparate impact after protected-attribute removal.
REPORTED Regulatory action on automated employment decision tools — NYC Local Law 144, EEOC guidance, EU AI Act high-risk classification.
ANALYSIS The expected-value-versus-tail argument is the author's. Retain the label.
This is one of 24 failure modes. The book gives you all of them — plus the controls that catch each one and a 90-day plan to prove you ran them.
Preorder the Book