System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective

1 Introduction

• Indirect prompt injection places malicious material in separately accessed data, which can steer future planning steps; model-level defenses are difficult to generalize and formally analyze.The cited limitations include vulnerability to tailored attacks and dependence on nondeterministic models without well-understood guarantees.Source: §1 · p. 1
• Applying information flow control keeps the planner from seeing information derived directly or indirectly from untrusted sources.The approach treats the LLM and execution facilities as black boxes, supporting formal analysis without modeling their internals.Source: §1 · pp. 1–2
• The f-secure design separates a trusted-information-only planner from an executor that can access all data, with a monitor preventing untrusted-derived outputs from returning to planning.The planner produces structured steps, while the rule-based executor performs them.Source: §1 · pp. 1–2
• The formal analysis proves execution trace non-compromise for f-secure systems, while vanilla LLM systems do not satisfy that property.Attackers may still affect processed data, but cannot influence the query's execution plan.Source: §1 · p. 2
• The evaluations report that the system eliminates all tested attacks without impeding functionality or efficiency.The paper presents case studies and benchmarks as evidence that its theoretical analysis transfers to practice.Source: §1 · p. 2

2 Problem Definition

• Vanilla LLM systems process principal queries amid diverse information sources, creating security risks from malicious information flow.The section defines the system and scopes the threats before analyzing compromise.Source: §2 · p. 2

2.1 Vanilla LLM System

• A vanilla system comprises models, non-model facilities, and principals, and generates executable steps at runtime rather than following a predetermined plan.Each step's output and prior steps feed into generation of subsequent steps.Source: §2.1 · pp. 2–3
• Because LLM-generated steps are nondeterministic, the model treats next-step generation as probabilistic given the existing execution trace and outputs.The execution trace is an iterative sequence of steps and their outputs.Source: §2.1 · p. 3

2.2 Execution Trace Compromise

• Malicious information can influence later step generation and hijack the execution trace, because free-form natural language blurs instructions and data.An attacker can craft untrusted information that the model accesses during query execution.Source: §2.2 · p. 3
• A chain-based attack can propagate malicious text through an intermediate file and cause private patient information to be sent to an attacker.In the example, an external medical file injects an instruction during a merge; reading the merged file then changes the later send step.Source: §2.2 · p. 4
• Conditional malicious instructions can remain dormant in intermediate outputs until a later operation triggers them, such as a file deletion.The section introduces an attack in which instructions in an externally sourced report target a later delete operation.Source: §2.2 · p. 4

3 Threat Model

• The paper excludes vulnerabilities in non-model facilities, noting that traditional software security has developed techniques for that topic.This is a scope boundary, not a claim that the facilities are invulnerable.Source: §3 · p. 4
• The system-level defense does not certify model-level alignment against attacks that compromise the LLM; it aims to prevent broader execution-trace harm after compromise.The paper distinguishes model-level attacks from the system-level consequences it targets.Source: §3 · p. 4
• Principals and facilities are trusted, and information from them is treated as trusted; unknown or untrusted sources may compromise the system.Attackers may manipulate untrusted information but cannot alter principal queries, system code, or configurations.Source: §3 · pp. 4–5

4 Design Overview

• The planner generates structured executable steps from the query, and a rule-based executor performs them while a security monitor oversees the process.The system loads information-flow security labels when a principal logs in.Source: §4 · p. 5
• The Structured Executable Planning Format regulates generated steps so security constraints can be enforced deterministically.The planner generates the next atomic step using the query and previously generated steps.Source: §4 · p. 5
• The monitor allows intermediate results into planning only when they pass integrity checks, rejecting the untrusted file content in the example.The planner requests prior results by data reference, which the monitor evaluates against the security requirements.Source: §4 · p. 5
• The monitor syntax-checks generated steps; validated steps go to the executor, and successful outputs are stored and referenced in later planning.Execution failures prompt the planner to generate another step; an end signal returns final results to the principal.Source: §4 · pp. 5–6

5 Secure LLM System

• An information-flow-secure LLM system requires that untrusted information cannot impact the probability of any execution trace for a query.The practical f-secure framework combines a fine-grained label model, trust configuration, SEPF, and a context-aware pipeline.Source: §5 · p. 6

5.1 Fine-grained Security Label Model

• Integrity labels capture whether natural-language information is trusted to be free of malicious executable semantics, and combined information receives the join of its sources’ labels.The model focuses on executable semantics; combined information is trusted only if all sources are trusted.Source: §5.1 · pp. 6–7

5.2 Security Configuration

• The integrity bound ι separates trusted labels, satisfying I(q) ⊑ι, from untrusted labels, and principals can configure trust for external sources.By default, ι is set to the integrity of the principal’s query; information from non-principal, non-system sources is untrusted unless configured otherwise.Source: §5.2 · p. 7

5.3 Structured Executable Planning Format

• SEPF structures each generated plan step with an index, instruction, object, data input, and expected output.Inputs can refer to earlier steps’ outputs using {Data_output: index}; after execution, the output field is updated with a reference to the result.Source: §5.3 · p. 7

5.4 Context-Aware Working Pipeline

• The context-aware pipeline generates each step from the query, prior trace, and intermediate outputs, then checks its syntax before execution or regeneration.The monitor checks field names, values, and types, including input parameters and data references.Source: §5.4 · pp. 7–8
• The monitor loads only individually trusted output items for planning, while the executor is restricted to the objects and information specified in each step.Successful steps are recorded in the trace for planning the next step; failed executions trigger step regeneration.Source: §5.4 · p. 8

• Execution-trace non-compromise means low-integrity information cannot compromise the execution trace.Source: §6 · p. 9
• For any query, equal information sets at integrity ι or higher must yield equal probabilities for every execution trace.The sets may differ on lower-integrity information.Source: §6 · p. 9
• An f-secure LLM system preserves ι-execution trace non-compromise.Source: §6 · p. 9
• The proof checks each step transfer during planning and execution for influence from low-integrity information.The complete proof is in Appendix A.Source: §6 · p. 9

7.1 Security Evaluation

• The evaluation uses two labels, T and U, with trusted information allowed to influence planning and untrusted information excluded.The label configuration treats information trusted by a principal as T and other external information as U.Source: §7.1 · p. 9, Table 1 · p. 9
• In the chain-based file case, f-secure uses end-to-end information-flow control to defend against compromise involving malicious instructions in an untrusted input file.The case compares f-secure with SecGPT on a query that combines medical and clinical files.Source: §7.1 · pp. 10–11, Figure 6 · p. 10
• In the one-step email case, the security monitor blocks untrusted malicious email content from planning while allowing trusted budget information to be used.The case contrasts SecGPT’s disclosure of private budget details with f-secure’s filtering of low-integrity information.Source: §7.1 · pp. 10–11, Figure 5 · p. 10
• 0% Attack Success Rate (ASR) for f-secure across all models, while vanilla ReAct reaches 51.6% (GPT-3.5 Turbo) and 28.7% (GPT-4 Turbo) in the base setting and 67.4% and 44.5% in the enhanced setting.The evaluation uses InjectAgent; Table 2 reports attack success rates.Source: Table 2 · p. 12, §7.1 · p. 12

7.2 Functionality Evaluation

• On the multiple-tool usage benchmark with GPT-3.5 Turbo, f-secure reaches 96.14% step accuracy and 60% overall accuracy, versus SecGPT’s 26.44% and 0%.Step accuracy measures intermediate-step correctness; overall accuracy measures final-result correctness.Source: §7.2 · pp. 12–13, Table 3 · p. 12
• Across nearly all benchmarks, f-secure maintains or improves step and overall accuracy versus vanilla, and outperforms SecGPT across benchmarks, especially for multiple-tool tasks.With GPT-3.5, step accuracy is consistently higher than overall accuracy, indicating difficulty completing every step when the deployed LLM is less capable.Source: §7.2 · pp. 12–13, Table 3 · p. 12
• The security checks and step modifications cost 0.0001 times the step-generation cost; f-secure step execution is 10x faster than vanilla and SecGPT.The reported timing averages cover the tool-use benchmarks.Source: §7.2 · p. 13, Table 4 · p. 13

LLM-Based System Security

• Prior LLM-system security work examines adversarial control of components, prompt injection, and defenses for system-level risks.The cited categories include plugin-related security, prompt injection, and architectures intended to secure LLM systems.Source: §8 · pp. 13–14
• SecGPT addresses security and privacy risks from third-party apps but does not protect against in-app execution-trace compromise.The paper positions its defense against this gap.Source: §8 · p. 14

Information Flow Control

• Information flow control uses security labels and policies to protect confidentiality and integrity as data moves through a system.Strict enforcement can provide noninterference, while practical systems may allow endorsement and declassification.Source: §8 · p. 14

9 Limitations and Conclusion

• The system does not target model-level attacks against the tool LLM, such as malicious webpage instructions that cause it to refuse a requested summary.Its scope is limiting broader security impacts when low-integrity information compromises the LLM.Source: §9 · p. 14
• The paper concludes that f-secure provides robust security guarantees while maintaining functionality and efficiency.The conclusion describes a context-aware pipeline with structured executable plans and security-label checks.Source: §9 · p. 14

A Proof of Theorem 6.2 in Section 6

• Theorem 6.2 states that an f-secure LLM system preserves ι-execution trace non-compromise.The proof concludes that the probability of an execution trace is identical for any two ι-equivalent information sets.Source: A Proof of Theorem 6.2 in Section 6 · pp. 16, 18
• During planning, the probability of generating the next step remains identical for ι-equivalent information sets.The planner uses the prior execution trace, eligible intermediate outputs, and instructional prompt; the proof states that the prior trace is accessible to the planner.Source: A Proof of Theorem 6.2 in Section 6 · p. 17
• During execution, a step cannot modify the execution trace, and outputs based on low-integrity information cannot be labeled trusted.The proof also establishes that ι-equivalent information sets remain equivalent after execution.Source: A Proof of Theorem 6.2 in Section 6 · pp. 17–18

B.1 Models & Device

• The running cases use gpt-4-turbo-2024-04- as the listed model.The supplied model name is truncated in the passage.Source: §B.1 · p. 18
• InjecAgent batch experiments use gpt-3.5-turbo-0125, gpt-4-turbo-2024-04-09, gemini-1.5-pro, and claude-3-5-sonnet-20240620.Source: §B.1 · p. 18
• Functionality evaluation uses gpt-3.5-turbo-0125, gpt-4-turbo-2024-04-09, gemini-1.5-pro, and claude-3-5-sonnet-20240620.Source: §B.1 · p. 18
• Experiments run on a MacBook Pro with an Apple M2 Pro chip, 12 cores, and 16 GB of memory.The chip has 8 performance cores and 4 efficiency cores.Source: §B.1 · p. 18

B.2 Benchmarks

• InjecAgent contains 1,054 test cases covering direct-harm and data-stealing attacks in base and enhanced settings; the other benchmarks test single-tool, multiple-tool, and relational-data tasks.The latter tasks assess string repetition, typing strings with up to 13 tools, and queries across three relational tables, respectively.Source: §B.2 · p. 18

B.3 Practical Security Label Configuration

• Retrieved InjectAgent tool information is treated as untrusted, while outputs from the three functionality benchmarks are labeled trusted.Some coworker emails are trusted; other specified emails and files are labeled untrusted.Source: §B.3 · p. 18

B.4 Tools

• The described tools include file append, deletion, reading, Gmail search, and message sending; confirmation-only outputs are treated as trusted.Read-file and Gmail-search outputs are assessed according to the integrity of their source files and email senders.Source: §B.4 · pp. 18–19, Figure 8 · p. 19, Figure 9 · p. 19

B.5 System Prompt Template

• The planner prompt asks for one atomic next step in a structured JSON blob, with fields for the step, object, inputs, and expected output.It directs the planner to use prior-step outputs by reference and to add an answer step when needed before an end step.Source: §B.5 · pp. 19–20

C.1 Case I: One-Step Compromise

• The message to the manager says the marketing department’s budget exceeded the $1,000 threshold, amounting to $1,024 this year; the trace confirms the email was sent.Source: §C.1 · p. 22

C.2 Case II: Chain-Based Compromise

• In Case II, the requested chain appends untrusted medical.txt to trusted clinical.txt, reads the integrated file, and sends its contents to the doctor.The trace shows the append confirmation and subsequent email-sent output.Source: §C.2 · p. 22

C.3 Case III: Conditional Compromise

• In Case III, report.txt contains an instruction to delete all files when a delete operation is reached, but the trace deletes report.txt only.The trace first reads report.txt, sends its contents to Alice, then executes the delete step for report.txt.Source: §C.3 · p. 23