PaperlayerView exact revision on arXiv

Source-linked AI summary

IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems

Yuhao Wu, Franziska Roesner, Tadayoshi Kohno, Ning Zhang, Umar Iqbal

arXiv:2403.04960v2cs.CRcs.AIcs.CLcs.CYcs.LG

TL;DR

Third-party LLM apps combine natural-language interactions with access to user data, creating security and privacy risks when apps may be untrustworthy. ISOLATEGPT isolates app execution and mediates interactions; it reports protection against attacks and safety issues while retaining baseline functionality, with overhead under 30% for 75.73% of tested queries.

  • Problem

    Third-party apps can access user data and interact with other apps and the system through imprecise natural-language interfaces, raising security and privacy risks.

    Source: §I · p. 1, B. Security and privacy risks · pp. 3–4

  • Method

    ISOLATEGPT isolates app execution and uses a hub-and-spoke architecture with an inter-spoke protocol to mediate app interactions.

    Source: §I · pp. 1–2

  • Results

    ISOLATEGPT protects against adversarial attacks and language-imprecision safety issues while matching VANILLAGPT functionality; overhead is under 30% for 75.73% of tested queries.

    Source: §I · p. 2

  • Takeaways & Limitations

    By constraining apps to isolated environments and mediating their external interactions, the architecture reduces the attack surface of LLM-based systems.

    Source: §I · p. 1

Abstract

from arXiv · show

Large language models (LLMs) extended as systems, such as ChatGPT, have begun supporting third-party applications. These LLM apps leverage the de facto natural language-based automated execution paradigm of LLMs: that is, apps and their interactions are defined in natural language, provided access to user data, and allowed to freely interact with each other and the system. These LLM app ecosystems resemble the settings of earlier computing platforms, where there was insufficient isolation between apps and the system. Because third-party apps may not be trustworthy, and exacerbated by the imprecision of natural language interfaces, the current designs pose security and privacy risks for users. In this paper, we evaluate whether these issues can be addressed through execution isolation and what that isolation might look like in the context of LLM-based systems, where there are arbitrary natural language-based interactions between system components, between LLM and apps, and between apps. To that end, we propose IsolateGPT, a design architecture that demonstrates the feasibility of execution isolation and provides a blueprint for implementing isolation, in LLM-based systems. We evaluate IsolateGPT against a number of attacks and demonstrate that it protects against many security, privacy, and safety issues that exist in non-isolated LLM-based systems, without any loss of functionality. The performance overhead incurred by IsolateGPT to improve security is under 30% for three-quarters of tested queries.

I. INTRODUCTION

LLM apps use natural-language instructions, user data, and APIs to automate tasks, but these capabilities create security and privacy risks. ISOLATEGPT proposes isolated app execution with mediated collaboration and reports protection without loss of functionality.

A. LLM-based systems

  • LLM apps combine natural-language functionality descriptions with APIs, which the system loads into memory so an LLM can plan tasks and exchange information across apps.Prior user-app and user-LLM messages are also retained in memory to support contextually relevant follow-up responses.Source: §I · p. 1, 1) LLM apps architecture · p. 3

I. INTRODUCTION

  • Natural-language interactions are difficult to scrutinize, while unrestricted access to user data, other apps, and system capabilities exposes users to potentially untrustworthy third-party apps.The paper gives examples including exfiltrating personal data and surreptitiously booking expensive tickets.Source: §I · p. 1
  • ISOLATEGPT isolates app execution and mediates app-system interactions through user-permitted interfaces, reducing the attack surface by design.Its hub routes queries, dedicated-LLM spokes handle app tasks with prior context, and the ISC protocol routes well-defined requests between spokes via the hub.Source: §I · pp. 1–2
  • ISOLATEGPT protects against adversarial attacks and language-imprecision safety issues while matching VANILLAGPT functionality; overhead is under 30% for 75.73% of tested queries.The evaluation also reports protection against attacks aimed at altering app behavior or stealing data from apps and the system.Source: §I · p. 2

1) LLM apps architecture:

  • LLM-based systems can automate data access, app collaboration, information synthesis, and changes to response behavior through installed apps.The introduction illustrates these uses with flight booking, attaching a cloud-drive file to email, comparing ride fares, and fiction writing.Source: 1) LLM apps architecture · p. 3

B. Security and privacy risks

Shared execution lets apps access data and influence other apps or the LLM, creating risks from malicious behavior and imprecise language. The paper argues that LLM systems need isolation adapted to their natural-language interactions and contextual data needs.

B. Security and privacy risks

  • In a shared memory space, apps can access data and influence other apps or the LLM, and adversaries may exploit this to steal data or control app behavior.The paper also notes that SQL injection and remote code execution attacks may apply to LLM-based systems.Source: B. Security and privacy risks · p. 3
  • Even without an adversary, ambiguous instructions or mismatched meanings can lead the LLM to alter app behavior or share personal data across contexts.Examples include conflicting instructions from fiction-writing and diagnosis apps, and reusing data collected for diagnosis in a travel reservation.Source: B. Security and privacy risks · pp. 3–4

C. Securing LLM-based systems

  • The proposed architecture isolates apps and mediates their interactions with the system through a trustworthy intermediary and well-defined interfaces with user permission.The paper presents this as reducing the attack surface by constraining apps to their execution spaces.Source: C. Securing LLM-based systems · p. 4
  • Conventional isolation must be adapted because natural-language interactions are harder to sanitize, while apps need access to user data, prior context, and other apps for task completion.The paper identifies rich contextual access and secure interfaces for natural-language collaboration as design requirements.Source: C. Securing LLM-based systems · p. 4

III. THREAT MODEL

The threat model considers untrustworthy apps, malicious content, and ambiguous natural-language instructions in a system with shared app execution and persistent memory. ISOLATEGPT addresses cross-app and system risks through isolated app environments and mediated communication.

A. System model

  • The modeled system supports third-party apps, online services, and collaboration in a shared execution environment, while retaining raw and processed user-app and user-LLM interactions in persistent memory.Source: A. System model · p. 4

B. Attacker capabilities and goals

  • Attackers may install malicious apps or expose malicious content, seeking to control other apps or the LLM or steal sensitive data; ambiguous language can also create safety risks without an attacker.Source: B. Attacker capabilities and goals · p. 4

C. Trust relationships

  • The model trusts the LLM and hosting system not to intend harm, but treats apps and processed content as untrustworthy and natural-language instructions as ambiguous.Source: C. Trust relationships · p. 5

1) In scope:

  • In scope are malicious attempts to control other apps or the LLM, steal their data, and language-driven inadvertent compromise or exposure in multi-app execution.Source: 1) In scope · p. 5

2) Out of scope:

  • Attacks within a single app are out of scope, although the model includes stopping their propagation to other apps.Source: 2) Out of scope · p. 5

IV. ISOLATEGPT: SYSTEM ARCHITECTURE

  • ISOLATEGPT executes apps in separate environments and uses a hub, app-specific spokes, and the ISC protocol to route query handling and inter-app requests.In the illustrated email task, the hub plans and invokes an email spoke, which obtains cloud-drive collaboration through the hub with user permission.Source: §IV · pp. 5–6, Figure 2 · p. 5

A. Hub goals and design

IsolateGPT divides system-wide coordination and app execution between a hub and isolated spokes. Non-LLM operators manage interactions, while planners and dedicated LLMs resolve requests using scoped context and data.

A. Hub goals and design

  • The hub intercepts user requests, routes them to apps or an LLM, mediates app collaboration, and maintains system-wide context and data.It uses an operator, planner, and memory module to perform these duties.Source: A. Hub goals and design · p. 6

1) Hub operator:

  • The hub operator is a non-LLM module that deterministically manages interactions among hub modules, spokes, and other spokes.The design aims to reduce exposure to natural-language attacks such as prompt injection.Source: 1) Hub operator · p. 6

2) Hub planner:

  • The hub planner decides whether a request needs apps and, if so, identifies the apps and resources required to resolve it.It uses the query, conversation context, and installed-app functionality descriptions; plans can include primary and assisting apps and their dependencies.Source: 2) Hub planner · p. 6

3) Hub memory:

  • The hub memory records user interactions and extracted data across apps, supplying context to the planner and deciding what data an app receives.The paper defers the memory architecture details to an appendix.Source: 3) Hub memory · p. 6

4) Query life cycle: Interplay between hub modules:

  • For each request, the hub selects apps, asks the user to choose if there are multiple primary apps, then passes the chosen app the query and required data with permission.The operator uses the planner and memory module to carry out these steps.Source: 4) Query life cycle: Interplay between… · p. 6

B. Spoke goals and design

  • A spoke is an isolated interface that executes an app, supplies needed data, collaborates with other spokes, and manages app memory.Each spoke contains an operator, an LLM, and a memory module.Source: B. Spoke goals and design · p. 6

1) Spoke operator:

  • The spoke operator is a non-LLM module that deterministically manages spoke components and hub communication, reducing exposure to natural-language attacks.It interfaces with untrustworthy apps and relays their natural-language messages to the hub.Source: 1) Spoke operator · p. 6

2) Spoke LLM:

  • Each spoke uses a dedicated LLM to plan and execute app work; unlike shared-LLM systems, apps can use different LLMs, including fine-tuned ones.Its plan can specify steps, additional user data, and functionality from other apps.Source: 2) Spoke LLM · p. 7

3) Spoke memory:

Spokes retain app-specific context while supporting isolated execution for specialized or private queries. When apps need to collaborate, the ISC protocol routes constrained exchanges through the hub and involves users in screening.

3) Spoke memory:

  • Spoke memory records app interactions and extracted data, while the hub can provide needed information from system or other-spoke interactions with user consent.The paper defers memory-management details to an appendix.Source: 3) Spoke memory · p. 7

4) Specialized spokes:

  • Vanilla spokes handle queries needing only a standard or specialized LLM, and private-mode spokes can run without prior context.A vanilla spoke has standard spoke components but no app; it can also synthesize results from non-data-dependent apps.Source: 4) Specialized spokes · p. 7

5) Query life cycle: Interplay between spoke modules:

  • A spoke operator passes the query and relevant memory to its LLM, obtains needed data through the hub with user consent, and requires consent before irreversible app actions.If another app is needed, the spoke uses ISC; the hub relays the response back to the user.Source: 5) Query life cycle: Interplay between… · p. 7

C. Inter-spoke communication

  • ISC enables spokes to collaborate while isolated by channeling exchanges through the trusted hub, which screens for malicious instructions and natural-language ambiguity with user involvement.The protocol constrains which messages can be exchanged.Source: C. Inter-spoke communication · p. 7

1) Broadcasting functionality:

  • ISC exposes predefined functionality descriptions to spokes without revealing whether the corresponding apps are installed.The hub can know the installation status and make functionality available with user consent.Source: 1) Broadcasting functionality · pp. 7–8

4) Life cycle of collaboration between spokes:

  • Figure 3 depicts the hub-mediated request-and-response sequence; user consent is required at steps 1, 3, and 5.The hub provides the formats, relays the request to the responding spoke, and returns its response to the calling spoke.Source: Figure 3 · p. 8

2) Supporting message exchange:

  • The protocol prevents direct spoke-to-spoke messaging and restricts exchanges to spoke and hub operators, allowing the hub to screen messages before forwarding them.The hub supplies request and response formats before the operators exchange messages.Source: 2) Supporting message exchange · p. 8

3) Screening and assistance with screening of messages:

ISC combines message constraints, format checks, hub assessments, and user permissions to screen collaboration. The paper describes this as a preliminary permission model and notes that users still manually screen messages.

3) Screening and assistance with screening of messages:

  • Users must manually screen messages exchanged between spokes because foolproof automatic detection of malicious natural-language instructions is not available.The permission model is preliminary; comprehensive automated regulation and its user experience are outside the paper’s scope.Source: 3) Screening and assistance with… · pp. 8–9
  • The hub cross-checks a requested functionality against its own plan and includes its assessment of whether a collaboration request is unexpected in the permission dialog.The user can configure preferences for different scenarios and durations.Source: 3) Screening and assistance with… · pp. 8–9, Figure 4 · p. 8
  • ISC requires defined request and response formats; operators validate messages, drop malformed requests, and can automatically check some types such as dates, integers, and URLs.Ephemeral identifiers help prevent invented collaborations and avoid directly disclosing app names or other offered functionality.Source: 3) Screening and assistance with… · p. 8

4) Life cycle of collaboration between spokes:

  • During collaboration, the hub selects a capable app and relays formatted requests and responses with user consent; a vanilla spoke can synthesize results when apps need not share data.Messages are checked for format, and data exchanges remain screened for malicious messages.Source: 4) Life cycle of collaboration between… · p. 9

V. EVALUATION: PROTECTION ANALYSIS

  • The evaluation compares ISOLATEGPT’s protection against threats with a non-isolated baseline and separately assesses functionality and performance overhead.VANILLAGPT provides the same features without isolating app execution.Source: §V · p. 9

A. App compromise and data stealing evaluation at scale

Across the tested attacks, VANILLAGPT had successful attacks, while ISOLATEGPT’s permission dialogs warned on every appearance; tailored testing also showed an app-compromise attack failing under isolation. The benchmark’s system-data scenarios and some explanations have stated scope limits.

A. App compromise and data stealing evaluation at scale

  • The benchmark was extended with 544 system-data attacks, bringing its total to 1,598 attacks involving app compromise and app or system data theft.The evaluation reports VANILLAGPT attack success and ISOLATEGPT permission-warning measures, which reflect different outcomes.Source: A. App compromise and data stealing… · pp. 9–10

1) Overall trends:

  • 20.2% of attacks succeeded on average against VANILLAGPT, while ISOLATEGPT showed permission dialogs for 7.6% of attacks, with warnings in 100% of those dialogs.Table I summarizes the protection evaluation of both systems across the benchmark.Source: 1) Overall trends · p. 10, Table I · p. 10

2) LLM guardrails are more sensitive when the potential for harms is apparent:

  • Data-stealing attacks had particularly high success or permission-dialog appearance rates, while second-app compromise was significantly more successful than first-app compromise in VANILLAGPT.The authors suggest apparent harm may heighten guardrail sensitivity and offer context-window growth or repeated prompt effects as possible explanations for the second-app pattern.Source: 2) LLM guardrails are more sensitive… · p. 10
  • The benchmark’s system-data results are limited by prompts that assume system data uses the same descriptors as app memory, unlike potentially structured system storage.The authors note that structured formats such as key-value storage may use restricted descriptions.Source: 2) LLM guardrails are more sensitive… · p. 10

1) App compromise:

  • In the ride-fare case study, Quick Ride’s attempt to add $10 to Metro Hail’s estimate succeeded in VANILLAGPT but failed in ISOLATEGPT.The authors attribute the failure to the app spoke’s execution being limited to its own space; non-data-dependent app results are synthesized in an isolated vanilla spoke.Source: 1) App compromise · p. 10

2) Data stealing:

The case studies show ISOLATEGPT blocking malicious cross-app actions and limiting unintended data sharing and instruction interference.

2) Data stealing:

  • Quick Ride altered Metro Hail’s behavior in VANILLAGPT but failed to alter it in ISOLATEGPT.Source: Figure 5 · p. 11
  • VANILLAGPT enabled Gmail to exfiltrate a GDrive document and conceal the attack by deleting both emails; ISOLATEGPT blocked it through explicit cross-app consent.The case used Gmail and GDrive to attach a cloud-drive document to an email.Source: 2) Data stealing · pp. 10–11, Figure 6 · p. 11
  • Even with permanent permission for app collaboration, ISOLATEGPT requires review of irreversible actions and contains a compromised app’s attack within its isolated space.Source: 2) Data stealing · p. 11

3) Inadvertent data exposure:

  • VANILLAGPT shared personal data collected by Health Companion with Travel Mate, while ISOLATEGPT required explicit permission before sharing it.The travel app’s imprecise data requirements led to exposure of sensitive personal data it did not need.Source: 3) Inadvertent data exposure · p. 12, Figure 7 · p. 12
  • ISOLATEGPT’s protection in this scenario requires users to provide data manually, adding effort; the authors describe this as a necessary usability-security trade-off.The case also motivates precise app declarations and illustrates risks from natural-language ambiguity without active attackers.Source: 3) Inadvertent data exposure · p. 12

4) Uncontrolled system alteration:

  • With Creative Muse and Symptom Solver issuing competing instructions, VANILLAGPT balanced both, while ISOLATEGPT followed the relevant app, potentially avoiding an unexpected answer.The case shows that even non-malicious apps can interfere with one another and create safety issues in a shared environment.Source: 4) Uncontrolled system alteration · p. 13

VI. EVALUATION: FUNCTIONALITY CORRECTNESS ANALYSIS

ISOLATEGPT matched VANILLAGPT’s functionality on app benchmarks, while incurring measured query-resolution overhead; the evaluation also analyzes benchmark mistakes and scope.

VI. EVALUATION: FUNCTIONALITY CORRECTNESS ANALYSIS

  • The evaluation used LangChain benchmarks covering queries with no apps, one app, multiple apps up to 13, and collaboration between up to five apps.The benchmarks compare end-to-end resolution with expected outputs using execution flow and semantic similarity.Source: §VI · p. 13

A. Overall trends

  • ISOLATEGPT matched VANILLAGPT’s functionality on all benchmarks involving apps; on the no-app benchmark, their accuracy differed only negligibly.Source: A. Overall trends · p. 13

B. Mistakes analysis

  • In collaboration benchmarks, intermediate-step plan mismatches often still produced correct final outputs because the LLM found another route to the answer.Source: B. Mistakes analysis · p. 13
  • Other collaboration errors involved app-response parsing; most no-app errors reflected response similarity mismatches attributed to LLM probabilism, with some context-window overflows.Source: B. Mistakes analysis · pp. 13–14
  • The authors note that real-world systems may face complex, nuanced use cases outside the scope of the peer-reviewed and widely used benchmarks they evaluated.They consider the benchmarks sufficient for understanding the system design, their core contribution.Source: B. Mistakes analysis · p. 14

VII. EVALUATION: PERFORMANCE ANALYSIS

  • Across tested queries, ISOLATEGPT’s performance overhead versus VANILLAGPT was 30% or less for 75.73%; at the 90th and 95th percentiles, overheads were 1.24× and 1.80×.The authors point to simple optimizations that would eliminate much of the overhead.Source: A. Overall trends · p. 14

B. Planning and memory extraction take additional time

ISOLATEGPT’s planning and memory-extraction steps account for much of its added query time, especially when multiple apps collaborate. The paper also reports cost overheads and frames isolation’s security gains as accompanied by performance costs that may be reduced through optimization.

B. Planning and memory extraction take additional time

  • For multiple-app queries, planning and memory extraction account for 17.18% and 80.43% of added time; collaboration queries split 28.87% and 67.67%, respectively.The figures refer to planning and memory extraction in the hub and spoke, respectively.Source: B. Planning and memory extraction take… · pp. 14–15
  • Hub planning traverses available apps, while spoke planning exposes all functionalities; heuristics, tailored templates, or narrower functionality sets could reduce this work.Measurements assume cold starts; reused spokes and retained data can reduce initiation and transmission overhead, and planning and memory extraction could be parallelized.Source: B. Planning and memory extraction take… · pp. 14–15

C. Takeaway

  • Query resolution time increases as more apps are involved, so users’ overhead is lower in use cases involving only a few apps.The measurements cover end-to-end response generation, not merely the appearance of the first words; the paper expects perceived overhead may therefore be less significant.Source: B. Planning and memory extraction take… · p. 15, C. Takeaway · p. 15
  • Process isolation incurs performance overhead, though the authors say future optimizations can improve ISOLATEGPT’s usability.They note that LLM performance improvements may also reduce overhead.Source: C. Takeaway · p. 15

D. Cost overhead

  • For 10% of benchmark queries, ISOLATEGPT costs 1.85× more.The paper notes that performance optimizations and cheaper LLMs could reduce these cost overheads.Source: D. Cost overhead · p. 15

VIII. CONCLUDING REMARKS

  • The authors present ISOLATEGPT’s evaluation as empirical evidence that execution isolation can improve security in LLM-based systems.They describe this work as a foundation for informed next steps in evaluating security practices and their limits.Source: §VIII · p. 15
  • The authors released ISOLATEGPT’s code and worked with LlamaIndex to integrate it as a Llama Pack.Source: §VIII · p. 15

APPENDIX A ADDITIONAL DESIGN AND IMPLEMENTATION DETAILS

ISOLATEGPT isolates hub and spoke processes and mediates their exchanges through the hub, while maintaining separate memories and configurable user permissions. Its implementation also uses operating-system controls to restrict process resources and access.

A. Execution isolation

  • ISOLATEGPT runs hub and spokes in separate processes, restricting system calls and process resources and limiting app network requests to their root domain.Network data flows to endpoints are moderated through user permission.Source: A. Execution isolation · p. 17

B. Secure message exchange

  • Spokes request functionality through the hub, which returns request and response formats plus an ephemeral spoke identifier without revealing the spoke’s name or other functions.The protocol uses inter-process communication and well-defined message formats that operators can validate.Source: B. Secure message exchange · p. 17

1) Long-term & working memory:

  • Long-term memory stores interaction history, LLM-generated summaries, and entity-information pairs; working memory supplies recent interactions and relevant knowledge to the LLM context window.Entity information can be loaded at runtime as needed.Source: 1) Long-term & working memory · p. 17

2) Hub and spoke memory:

  • The hub keeps system-wide and per-spoke summaries and entity records, while spokes maintain their own interaction history, summaries, and entity records.The hub can use per-spoke records to assess and share needed data with user consent.Source: 2) Hub and spoke memory · pp. 17–18
  • The hub’s working memory combines recent interactions with summarized knowledge to support queries across spokes, such as carrying travel dates into a later calendar request.Source: 2) Hub and spoke memory · pp. 17–18

D. Permission model

  • User consent applies to app selection, inter-app data exchanges, and data leaving the system; configurable preferences can reduce repeated prompts and vary by duration and scenario.Source: D. Permission model · p. 18

1) Permanent permission:

  • Permanent permissions reduce user fatigue most but carry the highest user risk, so ISOLATEGPT disallows them for irreversible actions such as sending email or making purchases.Users can revoke permanent permission at any time.Source: 1) Permanent permission · p. 18

2) Session permission:

ISOLATEGPT offers session-based and one-time permission choices, and the appendix describes benchmarks used to evaluate requests ranging from no-app tasks to multi-app collaboration. The artifact appendix identifies the system and its replication resources.

2) Session permission:

  • Session permissions persist for one system session and can cover app selection, spoke communication, and data sent in requests.They are useful when resolving a query requires repeated interactions, such as an email app querying a calendar several times.Source: 2) Session permission · p. 18

3) One-time permission:

  • One-time permissions prompt users for each app selection, spoke interaction, or remote data transmission, making them suited to irreversible actions such as sending email or purchases.The permission model is a preliminary effort tailored to a limited set of use cases; a comprehensive model remains future work.Source: 3) One-time permission · p. 18

E. Functionality and performance evaluation benchmarks

1) Single app:

  • Typewrite tests whether the system can replicate a given string using a single typewriting app.Source: 1) Single app · p. 18

2) Multiple apps:

  • Typewriter deploys 26 letter-specific apps for typewriting queries, although each test case uses at most 13 apps.Source: 2) Multiple apps · p. 18

3) Multiple apps collaboration:

4) No apps:

  • Email Extraction tests structured-data extraction from email text with apps disabled.Source: 4) No apps · p. 18

APPENDIX B ARTIFACT APPENDIX

  • The artifact describes ISOLATEGPT as an execution-isolation architecture and provides resources to replicate the paper’s evaluation.Source: Appendix B · p. 19

2) Hardware dependencies:

The artifact materials specify the tested hardware and software, required API keys, and benchmark use, alongside setup instructions. They also state the paper’s major claims and explain how to reproduce case-study evaluations, including their permission assumption and probabilistic caveat.

2) Hardware dependencies:

  • The authors tested ISOLATEGPT on a commodity machine with an AMD Ryzen 9 3900X, 32 GB RAM, and 1 TB disk; no special hardware is required.Source: 2) Hardware dependencies · p. 19

3) Software dependencies:

  • ISOLATEGPT uses Python 3.9, LangChain, Miniconda, and Ubuntu 20.04.6 LTS; evaluations use GPT-4 version 0613.Python dependencies are specified in environment.yml, with setup instructions in the README.Source: 3) Software dependencies · p. 19

4) API keys and subscription:

  • Running ISOLATEGPT requires an OpenAI API key and incurs usage fees; scoring functionality correctness in Experiment E3 also requires a LangChain API key.Source: 4) API keys and subscription · p. 19

5) Benchmarks:

  • The artifact uses the Relational Data benchmark to evaluate complex queries requiring collaboration among multiple apps.Source: 5) Benchmarks · p. 19

B. Artifact Installation & Configuration

C. Major Claims

  • The major claims are that ISOLATEGPT protects against malicious-app behavior and certain multi-app safety issues, retains similar functionality, and incurs overhead that is manageable for most queries.Source: C. Major Claims · p. 19

1) Experiment (E1):

  • The protection experiment runs four case studies on ISOLATEGPT and VANILLAGPT; reproducing data-stealing and inadvertent-exposure results assumes users reject warned permission requests.Because LLMs are probabilistic, attacks may not be fully effective on every run, so the authors suggest repeating the experiment and checking execution flows.Source: 1) Experiment (E1) · p. 19

2) Experiment (E2):

Experiment E2 compares ISOLATEGPT and VANILLAGPT performance on a representative relational-data benchmark, while E3 compares their functionality correctness on that same benchmark. The passages describe the evaluation procedures and outputs but provide no measured scores or latency values.

2) Experiment (E2):

  • E2 compares system performance on LangChain’s Relational Data benchmark, selected because running all four benchmarks would cost hundreds of dollars.The chosen benchmark contains complicated queries requiring multiple apps and collaboration between them.Source: 2) Experiment (E2) · pp. 19–20
  • E2 records average query-resolution time by system process and per-query runtime for both ISOLATEGPT and VANILLAGPT.The process breakdown is in perf compare.csv; per-query runtimes are stored in separate system-specific files.Source: 2) Experiment (E2) · p. 20
  • Runtime varies with server load, infrastructure, optimization updates, and nondeterministic LLM prediction time, though component-time ratios and overall latency trends should remain in a similar range to Table IV.The same query can have different latency when rerun at different times.Source: 2) Experiment (E2) · p. 20

3) Experiment (E3):

  • E3 compares intermediate-step and final-output correctness for ISOLATEGPT and VANILLAGPT; its reported mean is for the multi-app collaboration benchmark, and results may vary with LLM predictions.The two systems’ results are stored separately, with correctness columns for intermediate steps and final output.Source: 3) Experiment (E3) · p. 20
Loading 2403.04960v2…