The LLM threat model

Security work starts by asking: what are we protecting, from whom, and where can they influence the system? LLM applications add one fundamental twist to that question, and the rest of this course follows from it.

#The core problem

Classic software separates code from data. An LLM receives one stream of text containing your instructions, the user's message, retrieved documents and tool outputs, and it cannot reliably tell which parts are instructions it should follow. Anything that can put text in front of the model can potentially steer it.

Treat every model output as untrusted input produced by a system that may have been manipulated. That single rule explains most of what follows: prompt injection, data leakage through outputs, insecure output handling and excessive agency are all consequences of it.

#Assets worth protecting

Confidential data the model can see: system prompts, documents, user conversations, secrets in context.
Actions the model can trigger: sending email, running code, writing to databases, spending money.
The service itself: GPU time is expensive and easy to exhaust.
Your model and prompts (intellectual property).
Users: harmful or deceptive output delivered under your name.
Your reputation and legal position: leaks, bias and wrong advice have consequences (compliance).

#Attackers and their entry points

Attacker	Goal	Entry point
Malicious user	Bypass policies, extract the prompt, abuse free compute	The chat box, API parameters, uploaded files
Third party	Steal data or trigger actions through your app	A web page, email, PDF or ticket your app reads (indirect injection)
Insider or contractor	Access data they should not	Prompts, logs, fine-tuning data, admin tools
Supply-chain attacker	Run code or plant a backdoor	Model files, packages, plugins, datasets
Curious but careless user	Not malicious: pastes secrets or personal data	The prompt
Competitor	Copy your prompts, model or outputs at scale	The API
Opportunistic scanner	Free GPU time or data	Exposed ports and weak auth

Assume attackers are patient and cheap: LLM attacks are mostly text, so a failed attempt costs nothing, and successful ones are easy to share.

#Trust boundaries in a typical app

text

User  ->  App  ->  [ system prompt + user text + retrieved docs + tool results ]  ->  Model
                                                                                      |
                              Tools / APIs / database / browser  <-  model output  <---+

Mark each arrow where data from a less trusted party flows toward something more privileged. Those are your review points. Typical boundaries:

Boundary	Why it matters
Internet user → your API	Authentication, rate limits, abuse
User text → model prompt	Direct prompt injection
Retrieved or fetched content → model prompt	Indirect injection (the dangerous one)
Model output → UI	XSS and link or image exfiltration
Model output → tool or database	Injection into your own systems
Tool result → model	Another injection path
Tenant A data → tenant B context	Cross-tenant leakage
Your servers → model weights and packages	Supply chain
App → logs and analytics	Privacy and secrets

#The OWASP Top 10 for LLM Applications

OWASP publishes a community-maintained list that is a good checklist. The 2025 edition covers roughly: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. Check the current list at owasp.org since it is revised. This course covers each theme:

Theme	Where
Prompt injection	Prompt injection
Sensitive information disclosure, system prompt leakage, vector weaknesses	Data leakage
Improper output handling, excessive agency	Output handling and excessive agency
Supply chain, data and model poisoning	Supply chain and stack security
Unbounded consumption	Reliability patterns (Production course)
Misinformation	The LLM Evaluation course

#The lethal trifecta

The dangerous combination is an agent that has:

Access to private data,
Exposure to untrusted content, and
A way to send data out (email, HTTP request, a rendered link or image).

If all three are present, assume an attacker can steal the data by planting instructions in the untrusted content. Remove at least one: no private data in that agent, no untrusted content, or no outbound channel (or a gated one). Many serious incidents trace back to exactly this combination.

#A worked threat model: a RAG support bot

System: a customer-support chatbot. It retrieves from a knowledge base (which staff edit), reads the customer's order history through a tool, and can create support tickets. It logs conversations.

#	Entry point	Capability	Worst case	Likelihood	Impact
1	Customer types text	Model + order tool	Customer A reads customer B's orders by asking cleverly	Medium	High
2	A knowledge-base article is edited by a compromised staff account	Model output to all customers	Instructions planted to mislead or phish customers	Low	High
3	Customer pastes a card number	Logs	Card data stored in plain text logs	High	High
4	A customer uploads a PDF to a ticket	Model reads it	Hidden text instructs the bot to reveal other tickets	Medium	Medium
5	Anonymous scanner finds the model port	GPU	Free compute, data exposure	Medium	Medium
6	Chat UI renders model markdown	Browser	Image URL leaks conversation to a third party	Low	Medium
7	Very long prompts	GPU	Cost and outage	High	Low to medium

Mitigations (each ties to a later lesson):

#1: the order tool authorises by the signed-in customer's identity on the server, never from model arguments (output handling and agency).
#2: review and sign knowledge-base changes; sanitise retrieved text as data (prompt injection).
#3: redact card and ID patterns before logging; retention limits (PII).
#4: process uploads in a tool-less call, pass a validated summary.
#5: bind to localhost, gateway with auth (stack security).
#6: sanitise markdown, allowlist image domains (data leakage).
#7: token and size limits (reliability patterns).

#Ranking risks

You cannot fix everything at once. A simple scoring scheme works well:

text

risk = likelihood (1–3) × impact (1–3)       → 1–9

Score	Action
7–9	Fix before launch
4–6	Fix soon; add monitoring now
1–3	Accept or schedule

Adjust for irreversibility (data theft cannot be undone), blast radius (one user versus all) and detectability (can you tell it happened?). Low-likelihood, high-impact items often rank first when the damage is irreversible.

#How to run a threat-modelling session

Draw the system on a whiteboard: users, app, model, tools, data stores, third parties. 20 minutes.
Mark trust boundaries and where text from outside enters.
List capabilities: what can the model read, call and change?
Brainstorm abuse with a mixed group (engineering, product, security, support): for each entry point and capability, "what is the worst thing that could happen?" 30 minutes.
Rank with the scheme above.
Assign mitigations and owners, and note the unmitigated residual risk explicitly.
Write it down (one or two pages) and revisit when you add a tool, a data source or a new user group.

#A one-page threat model template

For your own app, write:

Assets (data, actions, money, reputation)
Entry points where outside text reaches the model
Capabilities the model has (tools, data access)
For each entry point × capability pair: what is the worst thing that could happen?
Existing controls and their type (deterministic or probabilistic)
Mitigations to add, owners and dates
Residual risk you are knowingly accepting

#Misconceptions to drop early

"The model is aligned, so it will refuse." Alignment reduces, not eliminates, harmful behaviour, and says nothing about obeying injected instructions.
"Our system prompt forbids it." A prompt is a request, not an access control.
"We only use a local model, so we are safe." Local removes a vendor; it does not remove injection, leakage or tool abuse.
"Nobody would attack our small app." Automated scanners and opportunists do not care how small you are.
"We will add security later." The trifecta and permissions are architectural; they are cheap early and expensive late.

#Exercise

An internal assistant reads a shared mailbox (including emails from outside), can look up employee records and can send emails on behalf of staff.

Which trifecta elements are present?
Name three mitigations.

Answers:

All three: private data (employee records), untrusted content (external emails) and an outbound channel (send email).
Process incoming emails in a tool-less step that outputs a validated summary; require a human confirmation showing recipient and body for every outgoing email; give the assistant read access only to the records the requesting employee may see, enforced on the server, and restrict recipients to an allowlist.