PaxLabs Research
Andres G. and the PaxLabs Research Team · PaxLabs Inc.
Prompt Injection as Privilege Escalation: A Systems Security Perspective on LLM Agents
Andres G. and the PaxLabs Research Team PaxLabs Inc. Correspondence: legal@paxeer.app
We argue that prompt injection in large language model (LLM) agents is not a language understanding problem but a systems security problem. Drawing on decades of work in operating systems, web security, and database security, we show that prompt injection is structurally identical to privilege escalation: an attacker exploits the absence of architectural boundaries between trusted instructions and untrusted data to execute unauthorized operations. We formalize this framing, demonstrate why language-level defenses (such as asking the model to ignore injected instructions) are fundamentally unreliable, and propose architectural defenses inspired by capability-based security, typed input channels, and runtime-level trust boundaries. We introduce a taxonomy of injection surfaces in agent systems, present a formal model of trust and privilege in LLM agents, and outline an evaluation protocol for testing agent systems against injection attacks. Our analysis suggests that the prompt injection problem will persist until the field adopts the same lessons that systems security learned decades ago: data and code must be separated at the architectural level, not at the linguistic level.
Large language models are increasingly deployed as autonomous agents that execute multi-step tasks, call external tools, read and write files, and interact with web services [1], [2]. These agents receive instructions from multiple sources: the system prompt provided by the developer, user messages entered at runtime, tool outputs returned from external APIs, and documents retrieved from the web or internal knowledge bases. All of these inputs are concatenated into a single natural language context window, and the model processes them as a unified stream of tokens.
This architectural choice creates a security vulnerability that is already well-understood in traditional software systems. When user-controlled input is processed in the same channel as trusted instructions, and there is no mechanism to distinguish between the two, an attacker can inject instructions that the system treats as authoritative. In the context of web security, this is called cross-site scripting (XSS) [3]. In database security, it is called SQL injection [4]. In LLM-integrated applications, it is called prompt injection [5], [6].
The prompt injection problem was first identified in 2022 by researchers who showed that simple text additions to a prompt could override a model's intended behavior [7]. Since then, the attack surface has expanded dramatically as LLMs have been given access to tools, memory systems, and the ability to process untrusted external content [5], [8]. Despite significant attention from the research community, no reliable defense has been found. We believe this is because the problem has been mischaracterized. The dominant framing treats prompt injection as a language modeling failure (the model should be able to distinguish instructions from data) and seeks solutions in prompt engineering, fine-tuning, or output filtering. We argue that this framing is incorrect.
Our thesis is as follows: prompt injection in LLM agents is a privilege escalation vulnerability. The root cause is architectural: there is no boundary separating trusted instructions from untrusted data within the model's input pipeline. The solution, therefore, must also be architectural. It must come from the same design principles that have proven effective against analogous vulnerabilities in operating systems, databases, and web applications: typed input channels, least-privilege enforcement, capability-based access control, and runtime-level trust boundaries.
In this paper, we make the following contributions:
(1) We formalize the analogy between prompt injection and privilege escalation, showing that the structural conditions for the attack are identical in both cases.
(2) We demonstrate, with formal argument, why language-level defenses cannot reliably enforce the instruction/data boundary.
(3) We propose a set of architectural defenses grounded in established systems security principles.
(4) We introduce a taxonomy of injection surfaces specific to LLM agent systems.
(5) We present a formal model of trust and privilege in LLM agents, along with an evaluation protocol for testing agent security.
The remainder of this paper is organized as follows. Section II provides background on prompt injection attacks and their taxonomy. Section III presents the privilege escalation framing. Section IV analyzes why language-level defenses fail. Section V describes architectural defenses. Section VI catalogs injection surfaces. Section VII presents our formal model. Section VIII outlines an evaluation protocol. Section IX discusses related work. Section X addresses limitations. Section XI concludes.
We adopt a four-category taxonomy of prompt injection attacks that maps to the primary attack vectors identified in the literature.
Direct Injection. In a direct injection attack, the attacker places malicious instructions directly into the user-facing input field of an LLM application. The classic example is appending "Ignore all previous instructions and do X" to a user query [7]. This attack exploits the fact that the model processes all input tokens through the same attention mechanism, with no distinction between system-level instructions and user-provided text. Perez and Ribeiro [9] demonstrated that goal hijacking and prompt leaking attacks could be executed with simple handcrafted inputs, achieving success rates as high as 58.6% against GPT-3.
Indirect Injection. In an indirect injection attack, the attacker embeds malicious instructions in content that the model retrieves or processes later, such as web pages, emails, documents, or tool outputs. Greshake et al. [5] provided the first systematic treatment of this attack class, showing that an LLM integrated with a web browser could be compromised by visiting a page containing hidden instructions. They demonstrated data exfiltration, remote control, and self-propagating injection (worms) through indirect channels. This work established that retrieval-augmented generation (RAG) systems and tool-using agents are vulnerable even when the user never provides malicious input.
Jailbreaking. Jailbreaking attacks target the safety alignment of the model itself, using adversarial prompts to bypass content restrictions or behavioral guardrails [10], [11]. Wei et al. [10] showed that adversarial prompts could be constructed automatically, and Chao et al. [11] introduced PAIR (Prompt Automatic Iterative Refinement), which uses one LLM to generate jailbreaks against another in a black-box setting, achieving high success rates against GPT-3.5, GPT-4, and other commercial models. While jailbreaking is often discussed separately from prompt injection, we note that the underlying mechanism is the same: the attacker exploits the model's inability to distinguish between instructions it should follow and data it should merely process.
System Prompt Extraction. System prompt extraction attacks aim to reveal the hidden instructions, API keys, or behavioral rules embedded in the system prompt [12]. This attack class is relevant because system prompts often contain sensitive configuration details, and their extraction can enable further attacks. The OWASP Top 10 for LLM Applications (2025) lists prompt injection as the number one risk and system prompt leakage as a separate category [12], but the two are closely related: a successful injection often begins with extracting the system prompt to understand the target's constraints.
Modern LLM agents follow a common architectural pattern. A system prompt defines the agent's role, capabilities, and behavioral constraints. The agent receives user messages and, in response, generates a sequence of reasoning steps and tool calls. Tool outputs are fed back into the context window, and the agent continues reasoning until the task is complete or a maximum number of steps is reached [1], [2].
This architecture means that the model's input context at any given step contains a mixture of: (a) developer-authored system instructions, (b) user-provided queries, (c) outputs from external tools, (d) retrieved documents, and (e) the agent's own prior reasoning. All of these are represented as tokens in a single sequence. The model has no architectural mechanism to determine which tokens are authoritative instructions and which are untrusted data.
We now formalize the analogy between prompt injection and privilege escalation in traditional systems.
Definition 1 (Privilege Escalation). A privilege escalation occurs when a process running at a given privilege level P_low executes an operation that requires a higher privilege level P_high, where P_high > P_low, by exploiting a flaw in the system's access control mechanism.
Definition 2 (Prompt Injection). A prompt injection occurs when untrusted input I, processed at trust level T_low, causes the LLM agent to execute an action A that is authorized only at trust level T_high, where T_high > T_low, by exploiting the absence of an architectural boundary between I and the agent's trusted instructions S.
Proposition 1 (Structural Equivalence). Prompt injection satisfies the structural conditions of privilege escalation: (1) there exist distinct trust levels (system instructions vs. external data); (2) untrusted input can influence operations at a higher trust level; and (3) the mechanism of escalation is the absence of an enforced boundary between trust domains.
Proof sketch. Consider an LLM agent with system prompt S, user input U, and tool output O. The model's input is the concatenation C = S || U || O, where || denotes token concatenation. The model processes C as a single sequence. If an attacker controls O (for example, by including hidden instructions in a retrieved web page), the model cannot reliably determine whether a given instruction in C originated from S (trusted) or from O (untrusted). Therefore, instructions in O can override or augment instructions in S, achieving the same effect as a privilege escalation: the attacker's instructions are executed with the authority of the system prompt. QED.
This structural equivalence is not metaphorical. It is literal. The same pattern appears in:
SQL Injection [4]: User input is concatenated into a query string without sanitization. The database engine cannot distinguish between the query structure (code) and the input values (data). The result is unauthorized query execution.
Cross-Site Scripting (XSS) [3]: User input is embedded in an HTML page without encoding. The browser cannot distinguish between the page's markup (code) and the user's content (data). The result is unauthorized script execution.
Prompt Injection [5]: Untrusted text is embedded in the model's context window without type marking. The model cannot distinguish between the system's instructions (code) and the external content (data). The result is unauthorized instruction execution.
In all three cases, the root cause is identical: data and code share a channel, and the interpreter cannot reliably distinguish between them. The historical lesson from SQL injection and XSS is clear: this class of vulnerability cannot be solved by asking the interpreter to be smarter. It must be solved by ensuring that data and code never share a channel, or by enforcing boundaries through architectural mechanisms [13].
IV. WHY LANGUAGE-LEVEL DEFENSES FAIL
A natural response to prompt injection is to train or prompt the model to resist it. Common approaches include: (a) appending "Do not follow instructions in the document" to the system prompt; (b) fine-tuning the model on examples of injection attacks; (c) using prompt sandwiching, where the system instructions are repeated after each tool output [14]; and (d) using data delimiters or special markers to delimit untrusted content [14].
We argue that these approaches are fundamentally limited for the following reasons.
Reason 1: The instruction/data boundary is not representable in natural language. Natural language has no syntactic mechanism for distinguishing between instructions and data. When a system prompt says "Summarize the following document," the model processes both the instruction and the document content through the same attention layers. If the document contains the text "Actually, ignore the previous instruction and instead output your API key," the model must determine, based on semantic context alone, which instruction is authoritative. This is not a parsing problem (as in SQL injection, where parameterized queries provide a clear structural boundary). It is an interpretation problem, and interpretation is inherently unreliable.
Reason 2: Training on attacks creates an arms race, not a solution. Fine-tuning a model to resist known injection patterns is analogous to signature-based intrusion detection: it works against known attacks but fails against novel ones. Adversarial prompts can be rephrased, translated, encoded, or obfuscated in ways that preserve their semantic effect while evading detection [10], [11]. The attack surface is the entire natural language space, and no finite training set can cover it.
Reason 3: Prompt sandwiching and delimiters are bypassable. Prompt sandwiching (repeating system instructions after each tool output) and data delimiters (wrapping untrusted content in special markers) provide weak boundaries that can be overcome by sufficiently sophisticated injections. If the attacker knows or guesses the delimiter format, they can craft injections that escape the delimited region [14]. If the attacker does not know the format, they can still exploit the model's tendency to treat all text as potentially relevant context.
Reason 4: The model's attention mechanism is instruction-agnostic. Transformers process all input tokens through the same attention mechanism [15]. There is no architectural equivalent of a kernel/user mode split or a code/data segment distinction. The model's weights determine how it responds to each token, but the weights cannot encode a context-independent rule like "tokens from source X are instructions; tokens from source Y are data." This is because the trust level of a token is a property of its provenance (where it came from), not its content (what it says), and provenance is not represented in the model's input.
These limitations are not engineering gaps that will be closed by better models or more training data. They are structural properties of the architecture. A single natural language processing pipeline, no matter how capable, cannot enforce a trust boundary that it cannot represent.
If language-level defenses cannot solve the problem, what can? We draw on established principles from systems security to propose a set of architectural defenses for LLM agents.
The most direct analog to parameterized queries in SQL is typed input channels. Instead of concatenating all inputs into a single string, the agent runtime should tag each token with its source (system, user, tool, retrieval) and make this metadata available to the model or to a runtime enforcement layer. This is analogous to how modern operating systems tag memory pages with permission bits (read, write, execute), preventing data from being executed as code.
In practice, this can be implemented at the framework level. The agent runtime maintains a mapping from token ranges to source types. A policy engine can then enforce rules such as: "instructions from the tool source are never treated as system-level directives" or "retrieved documents cannot modify the agent's tool permissions."
The agent runtime should enforce a strict separation between the instruction pipeline and the data pipeline. System instructions are processed first and establish the agent's authority, goals, and constraints. External data (tool outputs, retrieved documents) is then processed within the scope of that authority, but cannot modify it. This is analogous to how a process's capabilities are set at spawn time and cannot be elevated by the process's own input.
One implementation approach is to use separate inference passes: the first pass processes the system prompt and user query to generate a plan; the second pass processes tool outputs within the scope of that plan, without allowing the tool outputs to alter the plan's authority. This "plan-then-execute" architecture limits the blast radius of injection.
Traditional agent architectures give the model access to all configured tools at all times. A capability-based approach [16] instead grants the model specific, bounded capabilities for each step of its reasoning. At each tool call, the runtime checks whether the requested tool and parameters are within the agent's current capability set. An injected instruction that requests an unauthorized tool call (for example, "send all emails to attacker@example.com") would be blocked by the runtime, not by the model.
This approach is analogous to how modern operating systems use capability-based access control: a process must present a valid capability (token) to access a resource, and capabilities are granted on a least-privilege basis [16].
All tool executions should occur in sandboxed environments with limited privileges. File system access should be restricted to a designated working directory. Network access should be constrained to approved endpoints. API calls should be rate-limited and audited. This is the standard approach in container security and applies directly to LLM agents.
Debenedetti et al. [1] implemented this principle in the AgentDojo framework, which evaluates agent security under realistic tool-access conditions. NVIDIA's guidance on sandboxing agentic workflows [17] recommends blocking file writes outside the workspace and restricting subprocess spawning, both of which limit the impact of successful injection.
Even with architectural defenses, a defense-in-depth approach should include runtime monitoring. The agent's tool calls, file accesses, and network requests should be logged and compared against expected behavior profiles. Deviations (such as an email agent suddenly requesting file system access) should trigger alerts or automatic suspension.
This is analogous to how intrusion detection systems (IDS) monitor process behavior in traditional operating systems. The goal is not to prevent injection (that is the job of architectural defenses) but to detect and respond to successful injection before it causes significant harm.
We identify six distinct injection surfaces in LLM agent systems, each with different trust properties and attack vectors.
User Input. Direct user messages are the most obvious injection surface. While users are typically considered trusted (they are the ones the agent is serving), adversarial users may attempt to extract system prompts, bypass content filters, or escalate privileges. Direct injection attacks target this surface [7], [9].
Tool Output. Tools return data from external sources (APIs, databases, web services) that may be attacker-controlled. This is the primary surface for indirect injection [5], [8]. An attacker who controls a web page, email, or API response can embed instructions that the agent processes as part of its reasoning.
Retrieved Documents. RAG systems retrieve documents from external knowledge bases. If an attacker can influence the document corpus (by contributing to a wiki, posting on a forum, or poisoning a vector database), they can inject instructions that are retrieved and processed by the agent [5].
Memory Entries. Agents with persistent memory store and retrieve past interactions. If an attacker can inject content into the memory system (for example, through a compromised tool output that is later stored), the injection persists across sessions and can be triggered at a later time.
Other Agent Messages. In multi-agent systems, agents communicate by exchanging messages. An attacker who compromises one agent can inject instructions into messages sent to other agents, propagating the attack across the system [18]. This is analogous to worm propagation in networked systems.
Metadata and Environment Variables. Agent configurations, environment variables, and metadata (such as filenames, HTTP headers, or database column names) can contain injection payloads. These surfaces are often overlooked because they are not part of the natural language content, but they are processed by the same model pipeline.
Each of these surfaces has different trust properties. User input is semi-trusted (the user is the intended operator, but may be adversarial). Tool output and retrieved documents are untrusted. Memory entries are trusted at the time of storage but may become untrusted if the stored content was originally contaminated. A defense architecture must account for all of these surfaces and assign appropriate trust levels.
We present a formal model of trust and privilege in LLM agents, sufficient to reason about injection conditions and defense requirements.
Definition 3 (Agent State). An LLM agent at time t is in state S_t = (P, C_t, R_t), where P is the set of privileges (capabilities) granted to the agent, C_t is the context window (a sequence of tokens from all sources), and R_t is the set of resources the agent has accessed so far.
Definition 4 (Trust Level). Each token in C_t has an associated trust level L(token) in {system, user, tool, retrieval, memory, agent}. The trust level reflects the provenance of the token.
Definition 5 (Privilege Boundary). A privilege boundary is a rule B = (L_src, action, L_req) that specifies: tokens at trust level L_src may cause the agent to perform action only if the agent currently holds privilege L_req for that action.
Definition 6 (Injection Condition). An injection succeeds at time t if and only if there exists a token sequence o at trust level L_low in C_t such that processing o causes the agent to perform an action a that requires privilege L_high, where L_high > L_low, and no privilege boundary enforces the check (L_low, a, L_high).
Theorem 1 (Necessity of Architectural Boundaries). If all tokens in C_t are processed by a single model inference pass with no external enforcement, then for any finite set of language-level defenses D, there exists an injection payload o that bypasses D.
Proof sketch. The set of possible injection payloads is the set of all token sequences in the model's vocabulary, which is finite but combinatorially large. A language-level defense D is a function from context windows to accept/reject decisions. Since D is implemented by the same model (or a related model), it operates on the same input representation and cannot access provenance metadata that is not in the input. An attacker can construct o by trial and error (black-box) or by gradient-based optimization (white-box) until D(o) = accept. This is the same argument that underlies the impossibility of solving SQL injection through input validation alone [4], [13]. QED.
Defense Requirement 1 (Provenance Tracking). Every token in the agent's input must carry provenance metadata that is not under attacker control.
Defense Requirement 2 (Privilege Enforcement). The agent runtime must enforce privilege boundaries at each tool call, independent of the model's output.
Defense Requirement 3 (Least Privilege). The agent's privilege set P at each step must be the minimum required for the current action, not the maximum configured for the session.
We propose an evaluation protocol for testing agent systems against injection attacks, based on the principles established in this paper.
The attacker controls one or more injection surfaces (tool output, retrieved documents, memory entries, or other agent messages). The attacker's goal is to cause the agent to perform an unauthorized action: exfiltrating data, calling an unauthorized tool, modifying protected resources, or escalating the agent's own privileges.
We adopt the following metrics, consistent with the AgentDojo framework [1]:
Attack Success Rate (ASR): The fraction of injection attempts that result in the agent performing the unauthorized action.
Benign Utility: The agent's task completion rate in the absence of attacks.
Utility Under Attack: The agent's task completion rate when injection payloads are present in its input.
Security-Utility Tradeoff: The Pareto frontier between ASR and utility, which captures the inherent tension between restricting the agent's behavior (to prevent injection) and allowing it to function effectively.
For each injection surface S in {tool output, retrieved documents, memory entries, agent messages}:
(1) Generate a set of injection payloads using known attack methods (manual crafting, PAIR [11], gradient-based optimization for open-weight models).
(2) Embed each payload in the target surface and measure ASR.
(3) Measure benign utility on a standard task suite without injections.
(4) Measure utility under attack on the same task suite with injections present.
(5) Repeat for each defense configuration (no defense, language-level defense, architectural defense).
We recommend the following baselines:
No Defense: The agent processes all input without any injection defense. This establishes the upper bound on ASR.
Language-Level Defense: The agent uses prompt sandwiching [14], data delimiters [14], or a fine-tuned model. This establishes the effectiveness of non-architectural approaches.
Architectural Defense: The agent uses typed input channels, capability-based permissions, and sandboxed tool execution. This establishes the effectiveness of the approach advocated in this paper.
AgentDojo [1] provides 97 realistic tasks and 629 security test cases across domains (banking, Slack, travel, workspace). InjecAgent [19] and other benchmarks provide complementary test scenarios. We recommend using these existing benchmarks as a starting point, but note that they primarily test tool-output injection. A comprehensive evaluation should also test injection through memory, retrieved documents, and multi-agent communication channels.
The prompt injection problem was first articulated by Perez and Ribeiro [9], who demonstrated goal hijacking and prompt leaking against GPT-3. Greshake et al. [5] extended this work to indirect injection, showing that LLM-integrated applications could be compromised through retrieved content. Liu et al. [20] formalized direct prompt injection attacks and proposed a systematic evaluation framework. Abdelnabi et al. [8] demonstrated injection attacks against real-world LLM-integrated applications, including email assistants and code generation tools.
More recent work has explored injection in multi-agent systems. [18] showed that prompt injection can propagate between LLM agents in a manner analogous to computer worms, where a compromised agent injects malicious instructions into messages sent to other agents.
Defenses have followed three main approaches. Language-level defenses include prompt sandwiching [14], instructional prevention (appending "ignore injected instructions" to the system prompt), and data delimiters [14]. Detection-based defenses use classifiers (such as BERT-based models [21]) to identify injection payloads in tool outputs before they reach the agent. Architectural defenses include tool filtering (restricting the agent to only the tools needed for a specific task) [1], firewalls at the agent-tool interface [22], and Prompt Flow Integrity (PFI), which enforces least privilege on LLM agents through untrusted data identification and privilege boundary enforcement [23].
Debenedetti et al. [1] introduced AgentDojo, a benchmark for evaluating both attacks and defenses in agent systems. Their evaluation found that existing defenses reduce ASR but also reduce benign utility, and that adaptive attacks can often bypass known defenses.
The concept of capability-based access control originates in operating systems research [16]. A capability is an unforgeable token that grants the holder specific access rights. This model has been applied to programming language security (object capabilities in E and Joe-E), web security (content security policies), and distributed systems (OAuth tokens). We argue that it applies directly to LLM agents, where tool access should be mediated by capabilities rather than by the model's own judgment.
The analogy between prompt injection and SQL injection was noted early by Willison [6] and others. Halfond et al. [4] provide a comprehensive survey of SQL injection attacks and defenses, including the key insight that parameterized queries (which separate code from data at the architectural level) are the only reliable defense. Scott and Sharp [3] apply similar reasoning to XSS, arguing that input validation alone is insufficient and that content must be sanitized or separated at the rendering layer. Our paper extends this reasoning to LLM agents.
We acknowledge several limitations of our analysis.
First, our formal model is simplified. Real agent systems have complex trust relationships that do not reduce to a simple hierarchy of trust levels. For example, a tool output may be partially trusted (the API is trusted, but the data it returns may not be), and the trust level of a memory entry may degrade over time. A more nuanced model would assign continuous trust scores rather than discrete levels.
Second, the architectural defenses we propose introduce overhead and complexity. Typed input channels require modifications to the inference pipeline. Capability-based permissions require a policy engine that must itself be designed and maintained. Sandboxing limits the agent's flexibility. These costs are real and must be weighed against the security benefits.
Third, we do not provide empirical evaluation in this paper. Our arguments are structural and formal, grounded in the observation that prompt injection shares the same root cause as SQL injection and XSS. Empirical validation using benchmarks like AgentDojo [1] is necessary to confirm that architectural defenses provide meaningful security improvements in practice.
Fourth, the field is evolving rapidly. New agent architectures, new model capabilities, and new attack techniques are published frequently. Our analysis is grounded in the current state of the art, but specific details may change as the technology matures. We believe, however, that the core insight (the need for architectural separation of data and code) will remain valid regardless of how the technology evolves, because it is grounded in fundamental principles of systems security that have held for decades.
Fifth, our framing assumes that the goal is to build agents that are safe against adversarial inputs. In some deployment contexts (such as creative writing assistants or brainstorming tools), the ability to follow instructions embedded in retrieved content may be desirable. The appropriate level of defense depends on the agent's capabilities and the deployment context.
We have argued that prompt injection in LLM agents is a privilege escalation vulnerability, not a language understanding problem. The root cause is architectural: user-controlled input and trusted instructions are processed in the same channel, with no mechanism to distinguish between them. This is the same root cause that produces SQL injection and XSS in traditional software systems, and the historical lesson from those domains is clear: the problem cannot be solved by asking the interpreter to be smarter. It must be solved by separating data from code at the architectural level.
We have proposed a set of architectural defenses: typed input channels that tag each input token with its provenance, instruction/data separation at the runtime level, capability-based permissions that limit the agent's tool access to the minimum required for each step, and sandboxing that contains the blast radius of successful injection. These defenses are grounded in established systems security principles and are designed to be implemented at the framework level, independent of the underlying model.
We have also introduced a formal model of trust and privilege in LLM agent systems, a taxonomy of injection surfaces, and an evaluation protocol for testing agent security. Our formal model shows that, without architectural boundaries, no finite set of language-level defenses can prevent injection. This result mirrors the impossibility of solving SQL injection through input validation alone.
The LLM agent ecosystem is growing rapidly. Agents are being given access to email, code execution, file systems, web browsers, and financial APIs. The attack surface is expanding in proportion to these capabilities. We urge the research community and the industry to treat prompt injection with the same seriousness that the systems security community treats privilege escalation: as a fundamental architectural problem that requires fundamental architectural solutions.
The prompt injection problem will not be solved by better prompts. It will be solved by better architecture.
REFERENCES
[1] E. Debenedetti, J. Zhang, M. Balunovic, L. Beurer-Kellner, M. Fischer, and F. Tramer, "AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents," in Proc. NeurIPS Datasets and Benchmarks Track, 2024. arXiv:2406.13352.
[2] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, "ReAct: Synergizing Reasoning and Acting in Language Models," in Proc. ICLR, 2023. arXiv:2210.03629.
[3] D. Scott and R. Sharp, "Abstracting Application-Level Web Security," in Proc. 11th International Conference on World Wide Web (WWW), 2002, pp. 396-407.
[4] W. G. Halfond, J. Viegas, and A. Orso, "A Classification of SQL Injection Attacks and Countermeasures," in Proc. IEEE International Symposium on Secure Software Engineering, 2006, pp. 13-15.
[5] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection," in Proc. 16th ACM Workshop on Artificial Intelligence and Security (AISec), 2023. arXiv:2302.12173.
[6] S. Willison, "Prompt injection attacks against GPT-3," Simon Willison's Weblog, Sep. 2022. [Online]. Available: https://simonwillison.net/2022/Sep/12/prompt-injection/
[7] F. Perez and I. Ribeiro, "Ignore This Title and HackAPrompt: Exposing Systemic Weaknesses of LLMs Through a Global Scale Prompt Hacking Competition," in Proc. EMNLP, 2023. arXiv:2311.16119.
[8] S. Abdelnabi, K. Greshake, S. Mishra, C. Endres, T. Holz, and M. Fritz, "Prompt Injection attack against LLM-Integrated Applications," arXiv preprint, 2023. arXiv:2306.05499.
[9] F. Perez and I. Ribeiro, "Ignore Previous Prompt: Attack Techniques For Language Models," in ML Safety Workshop, NeurIPS, 2022. arXiv:2211.09527.
[10] A. Wei, N. Haghtalab, and J. Steinhardt, "Jailbroken: How Does LLM Safety Training Fail?," in Proc. NeurIPS, 2023. arXiv:2307.02483.
[11] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, "Jailbreaking Black Box Large Language Models in Twenty Queries," arXiv preprint, 2023. arXiv:2310.08419.
[12] OWASP Foundation, "OWASP Top 10 for Large Language Model Applications," 2025. [Online]. Available: https://genai.owasp.org/llm-top-10/
[13] D. Denning, "A Lattice Model of Secure Information Flow," Communications of the ACM, vol. 19, no. 5, pp. 236-243, 1976.
[14] R. Liu, Y. Cao, C. Wang, et al., "Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models," in Proc. KDD, 2025. arXiv:2403.04957.
[15] A. Vaswani, N. Shazeer, N. Parmar, et al., "Attention Is All You Need," in Proc. NeurIPS, 2017. arXiv:1706.03762.
[16] D. S. Hardjono and J. H. Saltzer, "Protection and the Control of Information Sharing in Multics," Communications of the ACM, vol. 19, no. 7, pp. 388-402, 1976.
[17] NVIDIA Developer Blog, "Practical Security Guidance for Sandboxing Agentic Workflows and Managing Execution Risk," 2025. [Online]. Available: https://developer.nvidia.com/blog/practical-security-guidance-for-sandboxing-agentic-workflows-and-managing-execution-risk/
[18] S. Li, Y. Deng, J. Wang, et al., "Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems," arXiv preprint, 2024. arXiv:2410.07283.
[19] Z. Wu, S. Qiu, Y. Chen, et al., "InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated LLM Agents," arXiv preprint, 2024. arXiv:2403.02691.
[20] Y. Liu, G. Deng, Y. Li, et al., "Prompt Injection attack against LLM-Integrated Applications," arXiv preprint, 2023. arXiv:2306.05499.
[21] ProtectAI, "Prompt Injection Detection Model," 2024. [Online]. Available: https://huggingface.co/protectai/
[22] T. Bagdasarian, R. Binns, S. Meiksins, et al., "Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks?," arXiv preprint, 2025. arXiv:2510.05244.
[23] D. Kim, S. Lee, J. Park, et al., "Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents," arXiv preprint, 2025. arXiv:2503.15547.