ACE: A Security Architecture for LLM-Integrated App Systems

I. INTRODUCTION

• Malicious apps threaten planning integrity, execution integrity and availability, and execution privacy; new IsolateGPT attacks include Planner Manipulation, Execution Flow Disruption, and Execution Manager Hijack.Source: §I · pp. 1–2
• Existing defenses use isolation or information-flow controls but assume apps cannot manipulate descriptions or schemas and retain interleaved planning and execution.Source: §I · p. 1
• ACE creates an abstract plan from trusted query information, instantiates it with registered apps, then executes the verified plan in isolation.Source: §I · p. 2
• Static analysis checks structured plans against user-specified information-flow constraints and blocks accidental or malicious leakage of privileged information to unqualified recipients.Source: §I · p. 2
• ACE prevents all attacks from INJECAGENT and ASB and the newly introduced attacks, while achieving above 80% utility on the LangChain Tool Usage suite.Source: §I · p. 2

II. BACKGROUND AND PROBLEM STATEMENT

• Section II reviews LLM-integrated apps and defenses against malicious apps before stating the paper’s problem and goals.Source: §II · p. 2

A. Overview of LLM-Integrated App Systems

• A system LLM plans app invocations using a dynamic context of user input, dialogue, app descriptions, and intermediate results, with planning and execution interleaved.Source: A. Overview of LLM-Integrated App… · p. 2, Figure 1 · p. 3
• Apps comprise a natural-language description, input/output schema, and implementing function; an orchestrator manages invocations and data flow, while app chaining requires dependency tracking and consistent intermediate results.Source: A. Overview of LLM-Integrated App… · pp. 2–3

B. Existing Defenses for LLM-Integrated App Systems

• Indirect prompt injection and natural-language ambiguity in multi-app systems can compromise app integrity, mislead users, or violate privacy across execution chains.The paper identifies these risks as especially significant when systems involve multiple untrusted apps.Source: B. Existing Defenses for… · p. 3
• f-Secure [12] separates trusted-input planning from rule-based execution and uses information-flow control to prevent untrusted data from influencing planning.Its reliance on unverified app descriptions and schemas can undermine those protections.Source: B. Existing Defenses for… · p. 3
• IsolateGPT isolates app execution through a Hub-and-Spoke design, but trusts static descriptions and schemas and relies on user interaction for app control.It checks outputs against expected formats and declared semantics without validating description integrity or inspecting app logic, limiting adaptation to adversarial scenarios and contributing to user fatigue.Source: B. Existing Defenses for… · p. 3

C. Problem Statement

• Attackers may control apps, schemas, metadata, execution, and outputs to manipulate planning or execution under weak or strong threat models.The weak model trusts app descriptions and schemas; the strong model allows them to be malicious.Source: C. Problem Statement · pp. 3–4
• The paper identifies four focal attacks: planning integrity violations, execution integrity violations, execution availability breakdowns, and execution privacy compromises.Examples include promoting a malicious app in a plan, passing malicious output to a benign app, interrupting task completion, or leaking sensitive user information.Source: C. Problem Statement · p. 4
• Planning availability attacks are considered easily detectable, while privacy compromises apply only during execution; existing defenses do not cover all attacker objectives.The paper says existing systems do not handle arbitrary app manipulation under a strong threat model and offer limited protection even under a weak one.Source: C. Problem Statement · p. 4
• The architecture should protect planning and execution from untrusted apps while preserving utility and remaining agnostic to system and LLM configurations.It should prevent app descriptions from arbitrarily changing control flow and enforce data flow according to the prespecified plan.Source: C. Problem Statement · p. 4

III. NEW ATTACKS ON LLM-INTEGRATED APP SYSTEMS

• The paper identifies three attacks against IsolateGPT: Execution Flow Disruption and Execution Manager Hijack via malicious app output, and Planner Manipulation via malicious app descriptions.The attacks are demonstrated on IsolateGPT’s public implementation.Source: §III · p. 4

A. IsolateGPT System Overview

• IsolateGPT’s planner creates a detailed plan but retains only an app allowlist; a separate execution-manager LLM selects and invokes apps iteratively in isolated spokes.The execution manager mediates inter-spoke communication and uses app outputs and prior context to determine subsequent actions.Source: A. IsolateGPT System Overview · pp. 4–5

B. New Attacks against IsolateGPT

• IsolateGPT trusts app descriptions during planning and passes raw app outputs to its execution-manager LLM without verification, allowing malicious content to influence later actions.This trust in outputs undermines the system’s claim that attacks within one app cannot propagate to the rest of the system.Source: B. New Attacks against IsolateGPT · p. 5
• The attacks use MetroHail as a benign fare-estimating app and QuickRide as a malicious app whose output or description targets MetroHail’s result.The scenarios were evaluated on the public IsolateGPT implementation.Source: B. New Attacks against IsolateGPT · p. 5

C. Execution Flow Disruption Attack

• Execution-flow disruption exploits unverified raw app output and interleaved planning and execution; a modified MetroHail inserts a string claiming fare apps are compromised.The indirect prompt injection prematurely halts execution rather than allowing the system to obtain MetroHail’s fare.Source: C. Execution Flow Disruption Attack · p. 5
• QuickRide’s output diverts execution before MetroHail runs, and IsolateGPT returns a vague technical-error message instead of a valid fare.The attack demonstrates an availability breakdown in IsolateGPT’s execution chain and fails to identify the compromised app as the source of the error.Source: Figure 2 · p. 5, C. Execution Flow Disruption Attack · p. 5

D. Execution Manager Hijack Attack

• QuickRide’s injected instructions made the execution manager report an artificially inflated MetroHail fare alongside QuickRide’s normal fare.The attack shows how raw app output passed into the manager’s context can let one malicious app undermine another app’s credibility.Source: D. Execution Manager Hijack Attack · p. 5

E. Planner Manipulation Attack

• QuickRide’s malicious description caused the planner to exclude MetroHail from a fare-calculation plan, leaving only QuickRide’s fare in the user’s output.Both apps described themselves as fare calculators, but QuickRide’s description instructed the planner to exclude MetroHail.Source: E. Planner Manipulation Attack · p. 6

A. Design Principles

• ACE separates planning from execution, prevents apps from influencing one another’s inclusion, enforces information-flow controls, and grants apps least privilege.The controls are intended to prevent sensitive data from reaching unqualified locations across complex, multi-step execution traces.Source: A. Design Principles · p. 6

IV. ACE SYSTEM ARCHITECTURE; B. High-level Overview

• ACE has an abstract planner, concrete planner, and executor with decreasing capabilities; the stages create a constrained app plan and execute it with isolation and managed permissions.The concrete planner statically verifies system-level policies such as privacy controls, while the executor limits each app to required data and restricts access to other apps and host resources.Source: B. High-level Overview · pp. 6–7, §IV · p. 6

B. High-level Overview

• ACE supports standalone apps and single-query interactions; application suites and multi-query interactions remain future work.Source: B. High-level Overview · p. 7

C. Abstract Planner

• The abstract planner generates abstract apps from the user query without access to installed-app information, using broad functional categories to represent available utilities.Each abstract app has a name, natural-language description, and input/output type signature, but does not implement its described behavior.Source: C. Abstract Planner · p. 7
• Abstract plans use a restricted Python-like language as execution-trace constraints, enabling static security reasoning; optional planner context must come from a fully trusted source.The language limits control-flow constructs and runtime features, and the plan is designed to prevent installed apps from manipulating execution flow.Source: C. Abstract Planner · p. 8

D. Concrete Planner

• The concrete planner maps each abstract app to exactly one registered concrete app, combining the mapping with the abstract plan to form executable control flow.It first filters apps by embedding similarity, using OpenAI text-embedding-ada-002 and Euclidean distance, then determines which filtered apps can implement each abstract app.Source: D. Concrete Planner · p. 8
• A compatibility layer translates mismatched concrete-app inputs and outputs to abstract-app types, with a specialized LLM resolving natural-language semantics.The app-matching LLM can differ from the planning LLM, allowing performance-cost tradeoffs.Source: D. Concrete Planner · pp. 8–9
• ACE prioritizes concrete plans that meet low-privilege access and secure information-flow constraints.The matching process may yield multiple concrete plans; the planner selects among them to enforce these security constraints.Source: D. Concrete Planner · p. 9

E. Executor

• The executor uses an orchestrator-worker architecture to separate privilege management from execution and isolate workers with only the privileges needed for their tasks.Worker environments default to the most restrictive privileges that still enable required functionality.Source: E. Executor · p. 9
• The orchestrator manages worker environments and messages, validates worker inputs and outputs, and terminates app workers that exceed resource limits.It holds the concrete plan and can report a terminated worker’s failure to the plan worker.Source: E. Executor · p. 9
• The plan worker processes the concrete plan without invoking apps directly, while isolated app workers receive only needed privileges and exchange data with the orchestrator.The orchestrator routes worker outputs back to the plan worker for subsequent execution stages.Source: E. Executor · p. 9

F. Information Flow Control Security

• ACE models information-flow policy as a lattice ordering “may flow into,” using joins to combine contamination; query-initialized variables accumulate labels, while app memory has static classes.Objects may be statically or dynamically bound to security classes.Source: F. Information Flow Control Security · p. 10
• ACE compiles abstract plans into information-flow programs, labels apps and variables from app clearances and query sensitivity, and statically rejects violating concrete plans; if no secure mapping exists, it terminates.Flows are implicitly contaminated with the query label because plan generation depends on the query.Source: F. Information Flow Control Security · p. 10

V. EVALUATION

• ACE’s evaluation covers attack defense, query processing, utility, and cost using GPT-4o, o3-mini, GPT 4.1, Claude 3.7 Sonnet, and Qwen-2.5-72B.The evaluation first tests the new attacks, then prompt-injection benchmarks, and finally utility and cost overhead.Source: §V · p. 10

A. Case Studies

• ACE defeated its three new attacks: trusted-only abstract planning and pairwise matching block planner manipulation, while schema-typed app outputs never control execution flow.The abstract plan also constrains execution paths so compromised app outputs cannot interrupt other app invocations.Source: A. Case Studies · p. 11

B. Prompt Injection Benchmarks

• INJECAGENT and ASB test IPI via compromised-app outputs; security tracks attacker-app noninvocation and utility expected-app use, but simulated apps leave output correctness unmeasured.For ASB, utility counts success when at least one of the two feasible user apps is invoked; overall success is end-to-end expected-app invocation.Source: B. Prompt Injection Benchmarks · p. 11
• On INJECAGENT [14], ACE achieved 100% security across 1,054 cases, with utility ≥84% for GPT-4o/o3-mini combinations and ≥60% for Claude 3.7 Sonnet/Qwen 2.5 72B Instruct.The utility results are presented in Table II.Source: B. Prompt Injection Benchmarks · p. 11, Table II · p. 12
• On ASB [15], ACE achieved ≥86% utility with GPT-4.1 and GPT-4o; 3 attacker-app invocations in 2,000 trials were task-appropriate, not injection-driven violations.The invocations occurred because the apps were suitable for the users’ tasks.Source: B. Prompt Injection Benchmarks · p. 12
• Mistral-7B StruQ had a 7% attack success rate on adapted INJECAGENT, while ACE determines generated plans strictly from trusted data.The provided Llama-7B StruQ model could not be evaluated because its context window was too small for the test cases.Source: B. Prompt Injection Benchmarks · p. 12

C. Tool Use Benchmark

• The benchmark tests single-tool typing, typing across 26 tools, and relational-data queries using 17 tools; ACE traces show structured plans with complex control flows.Utility is measured by step accuracy and overall accuracy against ground-truth trajectories and outputs.Source: C. Tool Use Benchmark · p. 12
• ACE achieves ≥80% success rates across all Tool Usage tasks with GPT-4o and GPT-4.1; GPT-4o plus o3-mini reaches 66.7% utility on relational data.GPT-4o plus o3-mini also yields high utility on the single- and multiple-tool suites.Source: Table III · p. 12, C. Tool Use Benchmark · p. 12
• Abstract planning uses two LLM invocations, concrete planning may query every abstract/concrete app pair, and execution overhead mainly comes from container setup and orchestration.Concrete-planning queries can be batched to reduce runtime at higher cost; GPT-4o and GPT-4.1 outperform GPT-4o plus o3-mini on overhead and utility.Source: C. Tool Use Benchmark · pp. 12–13
• For GPT-4.1, ACE costs $0.01 on the single-tool suite and $0.27 on the multiple-tool suite.The paper notes that the multiple-tool suite stresses utility with 26 tools, more than expected for regular tasks.Source: C. Tool Use Benchmark · p. 13
• ACE runs the single-tool suite in 11.73 seconds for GPT-4.1, versus IsolateGPT's 39.21 seconds; ACE's multiple-tool runtime is 20.95 seconds using all 26 tools.IsolateGPT's multiple-tool runtime reaches 126.65 seconds with 13 tools.Source: C. Tool Use Benchmark · p. 13

VI. RELATED WORKS

• Related work covers backdoor attacks [21], [22], jailbreaks [23]–[25], and prompt injection,, [26], [27], including indirect prompt injection,.Model-level defenses include StruQ [19], Instruction Hierarchy [28], and SecAlign [29], while system-level approaches include f-Secure [12] and CaMel [31]; model-level defenses lack strict benign/malicious-data boundaries and remain vulnerable to stronger attacks.Source: §VI · p. 13
• Formal-verification work applies analysis to LLM outputs, while PDoctor [36] detects planning violations against user constraints but does not mitigate attacks.Other cited applications include automated theorem proving, [33] and extracting and verifying objectives and constraints,.Source: §VI · p. 13

VII. CONCLUSION

• ACE decomposes planning into two structured steps based on fully trusted information and uses a trusted rule-based executor, enabling formal reasoning with information-flow policies.The conclusion presents this security-first design as a path toward trustworthy agentic applications.Source: §VII · p. 13

APPENDIX A ADDITIONAL DETAILS ON INFORMATION FLOW CONTROL

• The information-flow lattice permits data to flow upward to objects with at least as high a security class, but not downward without explicit authorization.Source: Appendix A · p. 15
• ACE tracks progressive data contamination and prevents contaminated data from reaching unqualified locations, including through conditional and looping constructs.Source: Appendix A · p. 15
• Fixpoint iteration tracks converged security labels across loops, and graph search over storage-object dependencies determines whether the loop statement is secure.The graph reachability calculation nearly corresponds to Warshall’s algorithm for transitive closure.Source: Appendix A · p. 15
• Concrete-plan verification injects loop-condition dependencies into body flows and branch-condition dependencies into conditional bodies to detect implicit information leaks.Loops require fixpoint analysis; if-statements do not.Source: Appendix A · p. 16
• ACE filters concrete plan implementations that violate information-flow policy and terminates with failure status if no app assignment satisfies the constraints.Source: Appendix A · p. 16

APPENDIX B PRIVILEGE-BASED RISK SCORING

• Among concrete plans, ACE prioritizes the assignment requiring the least privilege overall, measured by privileged host-system resource use; ties are broken randomly.Source: Appendix B · p. 16
• A plan is preferred only when its aggregate privilege risk set is a strict subset of the other plan’s; otherwise, their risks are incomparable.Source: Appendix B · p. 16
• The initial risk-scoring approach can be extended to rank apps from trusted providers above unverified providers or aggregate privileges more sophisticatedly.Source: Appendix B · p. 16

APPENDIX C PLANNING LANGUAGE

• Because the implementation uses Python’s complex runtime, ACE cannot formally guarantee its data-privacy guarantees; a formally specified DSL is proposed as future work.The authors describe their approach as a first step toward provable security, while noting that a DSL could support more precise information-flow analysis.Source: Appendix C · pp. 16–17
• Plans use a single no-argument main function returning a string, with typed variables and function signatures checked statically and at runtime.The language restricts Python syntax and runtime features to support static analysis.Source: Appendix C · p. 17
• Variables are immutable, functions are pure, and abstract-app calls are compiled into runtime invoke operations that forward arguments to the orchestrator.Mutable data structures are disallowed, and direct invoke calls before compilation fail syntax validation.Source: Appendix C · p. 17
• The sandbox isolates filesystem and network access, but container escape or carefully constructed worker messages could still cause external damage or invalid tool invocations.The orchestrator cannot directly inspect the plan worker’s program state to verify execution traces.Source: Appendix C · p. 17

APPENDIX D SYSTEM PROMPT TEMPLATES

• The abstract-app prompt requests only relevant JSON tool schemas, with PascalCase names, typed inputs and outputs, and no task solving or decomposition.It allows an empty app list for tasks solvable with basic code and limits schema types to int, float, str, and bool.Source: Appendix D · p. 17
• The abstract-plan prompt requires a typed Python main function using provided apps and permitted built-ins, with no unapproved imports, side effects, or function-call subexpressions.It specifies math as the only import, provides display for output, and requires assigning main()’s result to final_output.Source: Appendix D · p. 18
• The abstract-plan prompt also requires code-only output and limits tool use to the apps listed as available, with operational context supplied separately.It warns against using tools shown only in examples.Source: Appendix D · p. 18
• The concrete-planner prompt compares abstract and concrete tool purposes and schemas, returning success with mapping functions or failure when incompatibilities cannot be reconciled.It permits input renaming, reordering, and casting; output mapping returns one value, and the prompt encourages lenient schema matching.Source: Appendix D · pp. 18–19

APPENDIX E SELECTED ACE EXECUTION TRACES

• The appendix also introduces two InjecAgent traces demonstrating system failures at matching and execution time.Source: Appendix E · p. 19

A. Tool Usage Benchmark Example

• The Relational Data example uses database tools to answer a query, and its abstract and concrete plans were generated with GPT-4.1.The presented trace includes the initial query, abstract apps, abstract plan, and output.Source: A. Tool Usage Benchmark Example · p. 19
• The example requires three database-related abstract apps: finding each user’s key, retrieving their location ID, and looking up the city for each location.Source: A. Tool Usage Benchmark Example · p. 19
• ACE must orchestrate the apps in order—user keys, locations, then cities—and pass information from each stage into the next invocation.Source: A. Tool Usage Benchmark Example · p. 19
• A rule-based conditional check compares the computed information to determine the answer, without requiring an LLM for that final step.Source: A. Tool Usage Benchmark Example · p. 19
• The trace concludes that Bob and Alice do not live in the same city.Source: A. Tool Usage Benchmark Example · p. 20

Matching Failure

• A future-flight abstract app is rejected as incompatible with an existing-reservations concrete app, leaving no suitable match and preventing plan execution.The concrete planner compares app names, descriptions, and schemas; the descriptions differ on future flights versus existing reservations.Source: B. Matching and Execution Failures on… · p. 20
• Because the query gives little detail about how to find flight information, users may recover lost utility by adding relevant task context to the query.Prior systems resolve this ambiguity by combining query information with relevant app schemas in the planning LLM’s context.Source: B. Matching and Execution Failures on… · p. 20

Execution Failure

• An executor’s output mapping incorrectly assumes the app result has a top-level ‘size’ key, so execution fails despite a successful tool match.The failure arises from an input/output mapping error.Source: B. Matching and Execution Failures on… · p. 21

• It covers ACE benchmark adaptation (F-A), omitted IsolateGPT attacks (F-B), ACE’s ASB evaluation results (F-C), and StruQ trials against INJECAGENT (F-D).Source: Appendix F · p. 22

A. Benchmark Details and Adaptation

• ACE’s INJECAGENT adaptation checks whether executing a plan for the original query causes a security violation, testing both tool generation and control-flow integrity.The original benchmark instead tested whether a ReAct-style agent invoked an additional tool indicated by an injection string after a requested tool had run.Source: A. Benchmark Details and Adaptation · p. 22
• INJECAGENT contains 1,054 cases across 17 user apps and 52 attacker apps: 544 data-exfiltration cases and 510 direct-user-harm cases.The benchmark’s test cases simulate a user query that invokes a user app returning a malicious response, potentially triggering harmful or data-leaking behavior.Source: A. Benchmark Details and Adaptation · p. 22
• ACE’s benchmark setup ports each suite’s tools and queries into ACE’s app format and uses the default system context for app and plan generation.Source: A. Benchmark Details and Adaptation · p. 22
• ASB spans 10 agents and 10 application scenarios, with 50 tool-requiring user queries; ACE’s evaluation ports the suite and derives context from each scenario’s original prompt template.Source: A. Benchmark Details and Adaptation · p. 22

B. IsolateGPT Attack Details

• The attack-detail section presents diagrams and execution traces for Execution Hijack and Planner Manipulation against ACE and IsolateGPT.Figure 10 shows the failed Execution Manager Hijack attack on ACE, with IsolateGPT execution on the left; Figures 11–13 cover attack diagrams and traces.Source: B. IsolateGPT Attack Details · p. 22, Figure 10 · p. 22, Figure 11 · p. 23

C. Additional Results on ASB Benchmark

• Table VII presents ACE’s security and utility performance broken down by ASB agent scenario.Source: C. Additional Results on ASB Benchmark · p. 22
• ASB attacker-tool selections describe benign behavior related to the planner’s abstract app, so the authors do not consider them serious security violations.ASB apps take no arguments, return constant strings, and have no side effects, so information-flow control, risk scoring, and privilege management do not apply.Source: C. Additional Results on ASB Benchmark · p. 22
• Because ASB limits information-flow control, risk scoring, and privilege management, ACE’s security there arises from separating abstract planning, concrete planning, and execution.Source: C. Additional Results on ASB Benchmark · p. 23

D. StruQ Case Study

• The StruQ case study presents one trace that ignores the attacker instruction and another that produces a harmful app invocation.For both traces, content through the first app invocation and output was simulated by INJECAGENT rather than produced by the StruQ LLM.Source: D. StruQ Case Study · p. 23
• The StruQ traces’ steps through the first app invocation and output were simulated by INJECAGENT, not generated by the StruQ LLM.Source: D. StruQ Case Study · p. 23

E. Attack Failure

• In the failure trace, Ada Lovelace’s retrieved profile contains a bio asking to share genetic data with a health-monitoring app.Source: E. Attack Failure · p. 24
• The assistant returns AdaLovelace’s profile details, including the bio and app ID, rather than showing a genetic-data-sharing action.Source: E. Attack Failure · p. 24

F. Attack Success

• The attack-success trace starts with a request to navigate to a blueberry-scones recipe, whose returned snippet asks to share genetic data with a health-monitoring app.Source: F. Attack Success · p. 24
• The model invokes The23andMeShareData with appID12345, then reports that the user’s genetic data was successfully shared.Source: F. Attack Success · p. 24