The Confused Deputy at Scale: Threat Modeling and Defending Enterprise RAG Systems Against Prompt Injection and Data Exfiltration

Introduction

Retrieval-augmented generation (RAG) has become the default architecture for enterprises deploying large language models against proprietary knowledge: internal wikis, customer records, contracts, codebases, and meeting notes. By grounding model outputs in retrieved documents, RAG promises accuracy, freshness, and relevance. But it also creates something enterprise security teams have never had to defend before: a system that automatically ingests semi-trusted content from across the organization, processes it alongside privileged data, and can act on the result through connected tools and APIs.

The security consequences are severe and increasingly well documented. Prompt injection now ranks as the #1 threat in the OWASP Top 10 for LLM Applications 2025, and researchers widely describe it as analogous to the early days of SQL injection--except without an equivalent of parameterized queries ^[1]. The empirical numbers are sobering: indirect prompt injection delivered via retrieved documents achieves roughly 90% attack success rates when just five poisoned texts are planted among millions in a corpus (the PoisonedRAG research, published at USENIX Security 2025), while tool-hijacking attacks against agents using the Model Context Protocol have succeeded more than 72% of the time across 20 evaluated LLM agents ^[1].

This article provides a structured analysis of the threat classes facing production RAG systems, a practical threat modeling methodology tailored to LLM architectures, and a defense-in-depth blueprint that combines ingestion-time controls, retrieval-time authorization, guardrails, and architectural isolation. The goal is not to offer a silver bullet--none currently exists--but to help security engineering teams reason systematically about where their systems will fail and how to limit the blast radius when they do.

The Threat Landscape: Why RAG Systems Are Uniquely Exposed

The Lethal Trifecta

The most useful mental model for RAG security comes from Simon Willison's "Lethal Trifecta" framework (June 2025), which identifies the vulnerability condition precisely: any system that combines (1) access to private data, (2) exposure to untrusted content, and (3) an exfiltration vector is vulnerable to data theft ^[1]. Enterprise RAG systems hit all three conditions by design. They retrieve private documents, they ingest content that users and third parties can influence, and they typically connect to tools--email, search, code execution, APIs--that can move data out of the trust boundary.

This framing matters because it shifts the conversation from "how do we stop prompt injection?" to "how do we design so that a successful injection cannot complete an exfiltration?" Security practitioners applying this framework to threat modeling treat the presence of all three conditions as a trigger for maximum scrutiny, assuming that prompt injection will eventually succeed and building defenses accordingly ^[2].

The Root Cause: No Separation of Code and Data

Prompt injection is possible because LLMs process instructions and data in the same channel. There is no hardware-enforced or architecturally enforced separation between "code"--system prompts, developer instructions--and "data"--user input, retrieved documents, tool outputs. To the model, everything is tokens ^[3]. This is the fundamental architectural flaw that no current defense fully solves, and it explains why prompt injection remains an open problem even as models improve ^[1].

The Regulatory and Standards Context

The threat landscape is now codified in industry frameworks. The OWASP LLM Top 10 (2025) maps directly onto RAG-specific risks: prompt injection (LLM01), sensitive information disclosure (LLM02), excessive agency (LLM06), system prompt leakage (LLM07), and vector/embedding weaknesses (LLM08) are the five most relevant entries for RAG deployments ^[1]. Data and model poisoning (LLM04:2025) has evolved to explicitly include RAG knowledge base corruption, while improper output handling (LLM05:2025) covers the downstream vulnerabilities that arise when LLM-generated content is executed in other systems without validation ^[2]. On the regulatory side, NIST AI 600-1 (July 2024), the Generative AI profile of the AI Risk Management Framework, identifies information security as one of 12 risk categories and explicitly calls out data poisoning and prompt injection ^[1].

Prompt Injection: Direct and Indirect Vectors

Direct Injection: The Attacker Is the User

Direct prompt injection targets the user-facing interface. An attacker crafts inputs designed to override or extend the system prompt--extracting proprietary instructions, bypassing content policies, or coercing the model into harmful behavior. One of the most documented production attack patterns is system prompt exfiltration: convincing a deployed chatbot to reveal its system prompt, which often contains business logic, persona definitions, tool configurations, and sometimes hardcoded credentials or API keys ^[3]. Defenses that rely solely on wording inside the system prompt--"never reveal these instructions"--are widely considered ineffective, yet they remain one of the most common failure patterns found in security audits ^[4].

Indirect Injection: The Attack Comes Through the Pipeline

Indirect prompt injection is far more dangerous in production. Here, malicious instructions are embedded in content the model retrieves or processes: web pages, documents, emails, database records, or API responses. When the LLM reads this data as part of a RAG pipeline or agentic workflow, the injected instructions execute silently ^[3]. An employee asks the assistant to summarize a vendor's PDF, and hidden text in that PDF instructs the model to search the corpus for confidential contracts and email them to an external address. No user does anything wrong; the system simply follows instructions it cannot distinguish from legitimate ones.

The PoisonedRAG research makes the scalability of this threat concrete: five carefully crafted poisoned documents among millions achieve approximately 90% attack success rates ^[1][2]. This is precisely the threat model for enterprise RAG deployments that draw on partially untrusted knowledge sources--shared drives, customer-submitted content, crawled web pages, or multi-tenant corpora.

What Is Prompt Injection?

Knowledge Base Poisoning as a Sustained Campaign

Knowledge base poisoning generalizes indirect injection from a single document to a corpus-level campaign. Attackers insert documents containing false information or injection payloads, which the retrieval system then surfaces to users as authoritative answers ^[5]. Defenses at the ingestion layer include content moderation on all ingested documents, provenance tracking to establish where each document came from and who can modify it, and anomaly detection for unusual document content ^[5][2]. Training data poisoning--the RAG analogue at the fine-tuning layer--demands validation and review of all training data before use ^[5].

From Injection to Exfiltration: How Data Leaves the Building

Why Traditional DLP Fails

Once an attacker achieves injection, the next phase is data exfiltration--and this is where conventional security tooling breaks down. Traditional Data Loss Prevention (DLP) systems are fundamentally unequipped to police these threats because the exfiltration occurs through legitimate AI workflows rather than obviously unauthorized transfers. Attackers leverage the LLM's authorized plugins, API calls, and even rendering capabilities to bypass perimeter defenses ^[6]. The document was retrieved legitimately; the model summarized it legitimately; the summary was sent through an approved email tool. Every individual step looks normal. Only the composition is an attack.

Real-world incidents confirm this is not theoretical: side-channel exfiltration techniques have yielded critical CVEs against Microsoft 365 Copilot, GitHub Copilot, and Google Gemini throughout 2024-2025 ^[1].

Tool Hijacking and the Agentic Attack Surface

Tool-calling agents represent the highest-severity prompt injection targets. When a model can send emails, query databases, execute code, or call APIs, a successful injection translates directly into real-world action ^[3]. The MCPTox benchmark demonstrated greater than 72% tool-hijacking success against 20 LLM agents using the Model Context Protocol ^[1]. A representative production case study illustrates the pattern: an internal HR automation agent built to process employee benefits inquiries had access to a database query tool, an email sending tool, and a file read tool pointed at a shared HR directory--a seemingly benign combination that becomes a complete exfiltration chain in the hands of an injected instruction ^[3].

Security auditors consistently find the same failure modes in real deployments: retrieval systems that expose documents without enforcing user authorization, agents with tools that can act across tenant or privilege boundaries, and protections that rely only on prompt wording rather than architectural controls ^[4]. The industry term for the resulting dynamic is the "confused deputy": the AI system holds legitimate privileges and is tricked into abusing them on the attacker's behalf ^[6].

There is also a purely economic dimension: exposed or poorly secured APIs allow external systems to query the AI repeatedly and harvest information at scale, while unbounded agent loops create "denial of wallet" cost risks ^[5][7].

Threat Modeling for Production RAG Systems

Effective defense starts with structured threat modeling adapted to LLM-specific attack vectors. A practical methodology proceeds in four steps ^[2]:

1. Identify where untrusted input enters the system. Enumerate every channel: user prompts, retrieved documents, tool outputs, web content, emails, third-party API responses. Each is a potential injection vector.
2. Map data flows through LLM processing. Trace how retrieved content and user input combine in the context window, and where model outputs travel next--including rendered outputs in browsers (XSS), generated SQL, and shell commands ^[2].
3. Identify trust boundaries the LLM cannot reliably enforce. Assume the model cannot distinguish instructions from data. Any boundary that depends on the model "understanding" privilege levels is not a boundary ^[2].
4. Enumerate tools and privileges accessible to a compromised agent. For every tool, ask: if an attacker fully controlled the model's next action, what could this tool exfiltrate, modify, or destroy? Cross-tenant or cross-privilege tool access demands immediate remediation ^[4][8].

Systems exhibiting the lethal trifecta--privileged access, untrusted input processing, and exfiltration capability--should be flagged for maximum scrutiny and designed with the explicit assumption that prompt injection will eventually succeed ^[1][2]. In practice, this means treating data sensitivity segmentation as an architectural requirement: AI workflows should be segmented by data sensitivity level, with rulesets governing how sensitive data is handled in each environment ^[7].

Navigating The Threat Of Prompt Injection In AI Models

Defense-in-Depth: Layered Strategies for Production

Because no single mechanism fully solves prompt injection, effective defense requires multiple reinforcing layers. Research consistently shows that single controls provide only partial protection, while combinations of provenance verification, anomaly monitoring, and context isolation create genuine defense-in-depth ^[2].

Ingestion-Time Controls

The first defensive layer operates before any document reaches the vector store:

• Content moderation and scanning on all ingested documents to detect injection payloads, suspicious instruction patterns, and anomalous content ^[5].
• Provenance tracking so every document carries trusted metadata about its origin and permitted editors, enabling trust-weighted retrieval ^[5][2].
• Anomaly detection for unusual documents or sudden corpus changes that could indicate a poisoning campaign ^[5].

Retrieval-Time Authorization

The retrieval layer must enforce the same access controls as the underlying data source. If a user lacks permission to view a document, the RAG system must not retrieve it--even if it is the most semantically relevant result ^[5]. Document-level authorization, source trust scoring, and per-user filtered retrieval are non-negotiable for multi-user enterprise deployments ^[4][5]. Without this layer, an injected instruction that tricks the model into "searching for salary data" can only succeed within the requester's existing privileges.

Input and Output Guardrails

Commercial and open-source LLM security platforms now provide dedicated screening layers. Enterprise solutions such as Lakera Guard and HiddenLayer offer architectures for detecting jailbreaks and injection attempts while minimizing latency and blocking data exfiltration patterns in production RAG systems ^[9]. Guardrails should operate on both sides of the model:

• Input inspection of user prompts and retrieved content for known injection signatures.
• Output validation that treats every LLM response as untrusted user input, applying sanitization appropriate to the consumption context--because LLM outputs that flow into databases, browsers, or shells can themselves carry SQL injection, XSS, and command injection payloads ^[2].

Architectural Isolation and Least Privilege

The most robust defenses are structural rather than behavioral:

• Minimize agency. Every tool granted to an agent expands the impact of a successful injection. Remove tools that are not strictly required, and constrain each tool's scope to the minimum necessary privileges ^[1][8].
• Enforce authorization boundaries at the tool layer. Tool calls should be validated against user permissions server-side, with robust validation layers between the LLM and critical systems ^[8]. An agent should never hold credentials that span tenant or privilege boundaries ^[4].
• Break the exfiltration chain. Separate systems with access to private data from those with network egress. Rendered-content side channels, URL fetches, and outbound email from LLM contexts deserve specific scrutiny.
• Secure the periphery. System prompts should never contain credentials; logs and traces--which quietly retain sensitive user and business data--need the same retention and access controls as production databases ^[4][3].

Monitoring, Testing, and Continuous Validation

Defense does not end at deployment. Regular adversarial AI testing--red-teaming prompts, retrieval pipelines, and tool chains--keeps behavioral analytics tuned to detect the subtle interaction patterns indicative of exfiltration attempts ^[7]. Teams should run targeted prompt injection and data exfiltration test cases matched to their specific architecture, monitor for anomalous retrieval patterns and tool usage, and maintain clear escalation paths when guardrails fire ^[4]. Structured security audits covering prompt injection, tool misuse, RAG poisoning, and output validation against the OWASP LLM Top 10 provide an external check before and after production launch ^[4].

What Is Prompt Injection?

Conclusion

Prompt injection and data exfiltration are not edge cases in enterprise RAG systems--they are the predictable consequences of an architecture in which instructions and data share a single channel, privileged data sits one retrieval away from untrusted content, and connected tools provide ready-made exfiltration paths. The empirical evidence is stark: poisoning attacks succeed 90% of the time with a handful of planted documents, tool hijacking succeeds against most evaluated agents, and even first-party copilots from major vendors have shipped with exploitable exfiltration channels ^[1].

The strategic response is not to abandon RAG but to design with clear eyes. The lethal trifecta framework gives teams a triage heuristic: wherever private data, untrusted content, and egress capability converge, assume injection will succeed and engineer so that success is contained ^[1][2]. In practice, that means ingestion scanning and provenance tracking, document-level retrieval authorization, input and output guardrails from specialized security platforms ^[9], least-privilege tool boundaries validated server-side, and continuous adversarial testing ^[7]. No single layer is sufficient--defense-in-depth is the only strategy the current state of the art supports ^[2]. Organizations that treat their RAG pipeline with the same rigor as any other privileged, internet-facing system will capture the productivity benefits of enterprise AI without becoming the next exfiltration case study.

---CONTENT---