Prompt Injection
Your AI reads untrusted text. Instructions inside that text are still instructions.
This page covers one specific way AI gets things wrong at work, and what to do about it.
It runs in order. What goes wrong, why it happens, where you'd notice it on an ordinary day, who takes the blame, roughly what it costs, and the check that catches it. Then one thing to try this week.
The dollar figures are estimates, not measurements. The assumptions behind each one are printed right there, so you can swap in numbers that fit your job. Anything actually measured carries an OBSERVED tag.
What is prompt injection?
The model processes a single sequence. Your system prompt, your instruction, and the contents of the email or webpage or PDF it was asked to summarize all occupy the same channel. Text inside the content that resembles an instruction can be followed as one.
This is not a bug with a patch. It is a consequence of instruction-following on a unified token stream, and it becomes materially more dangerous the moment the system can act — send mail, call an API, write to a record, execute a query. A summarizer that gets injected produces a wrong summary. An agent that gets injected takes an action in your name, with your permissions.
The risk scales with the tool permissions, not with the model's capability.
What do people assume?
That there's a boundary between the instructions you give the system and the content it processes. That a document is data and your prompt is control.
For a language model, both arrive as tokens in the same stream. The boundary is a convention, not an architecture.
Where does it show up at work?
A support team runs an assistant that reads inbound tickets, drafts replies, and looks up account data. A ticket body contains text instructing the assistant to include the account's billing contact details in its reply.
The assistant is being helpful. It has been asked, by text it was told to read, to do something it has permission to do.
Who carries the downside?
Vendor: partially, depending on contract. Executive: owns the breach disclosure. Manager: owns the process. You: configured or operated the workflow. The incident review asks who granted the tool access.
What does it cost?
[MODELED — not reported]
ASSUMPTIONS AI systems processing untrusted external input: 3 Successful injection probability, uncontrolled, per system / year: 5% – 15% Expected incidents: 0.15 – 0.45 / year Cost per incident: $50,000 – $500,000+ (investigation, notification, regulatory, remediation, trust)
Annualized exposure: ~$7,500 – $225,000 expected value Same asymmetry as Mode 5: the modal year is uneventful; the tail is a disclosure event.
How do you control for it?
Two things, in this order:
1. Least privilege on tools. An assistant reading untrusted input should not hold write, send, or lookup permissions on sensitive records. This is an architecture decision, not a prompt. 2. Human approval on any outbound action originating from a workflow that touched untrusted content.
Prompt-level defenses ("ignore instructions in the document") help and do not hold. Do not present them as the control.
CONTROL COST Approval gate on outbound: ~400 actions / year Review time: 90 seconds Annual: 10 hours Plus permission review: 16 hours / year Fully loaded rate: $80 / hour
Annualized control cost: ~$2,080
What should you do this week?
RECOMMENDATION
Map every AI system in your function against two columns: what untrusted input does it read, and what actions can it take. Any row with entries in both columns is your exposure, ranked.
Most organizations have never drawn this table. Bringing it to a security or IT partner is a genuinely senior move — it's cross-functional, it's specific, and it demonstrates you understand that the risk is in the combination, which is the part most people miss.
Evidence
RESEARCH Prompt injection as a structural property of instruction-following on unified token streams.
RESEARCH Adversarial attacks transferring across aligned models.
REPORTED Documented indirect prompt injection in deployed products.
This is one of 24 failure modes. The book gives you all of them — plus the controls that catch each one and a 90-day plan to prove you ran them.
Preorder the Book