Source-linked AI summary
Secure and Efficient Access Control for Computer-Use Agents via Context Space
Haochen Gong, Chenxiao Li, Rui Chang, Wenbo Shen
TL;DR
Computer-use agents can take unintended actions because LLMs are uncertain, while user confirmations and runtime LLM-based validation have usability, security, or performance limitations. CSAgent uses development-time intent- and context-aware policies, generated and refined with an automated toolchain and enforced by an OS service. Across three benchmarks, it blocked 100% of attacks with policy evolution, with 1.99% average additional latency and 5.42% utility decrease.
Problem
LLM uncertainty and runtime safeguards leave computer-use agents at risk of unintended actions, while confirmation and dynamic validation approaches have usability, security, and performance limitations.
Source: §1 · p. 1Method
CSAgent uses development-time static policies indexed by intent and context, an automated toolchain to generate and refine them, and an OS service to enforce them at runtime.
Source: §1 · pp. 1–2Results
100% of attacks were blocked with policy evolution, with 1.99% average additional latency and 5.42% utility decrease across evaluations on AgentBench, AgentDojo, and AndroidWorld.
Source: §1 · p. 2Takeaways & Limitations
CSAgent provides contextual and intent-aware validation for computer-use agent actions, with evaluation showing strong security guarantees and acceptable overhead.
Source: §10 · p. 15CSAgent currently focuses on single-agent scenarios; multi-agent collaboration challenges such as cross-agent context transfer remain future work.
Source: §8 · p. 14
Abstract
from arXiv · showhide
Large language model (LLM)-based computer-use agents represent a convergence of AI and OS capabilities, enabling natural language to control system- and application-level functions. However, due to LLMs' inherent uncertainty issues, granting agents control over computers poses significant security risks. When agent actions deviate from user intentions, they can cause irreversible consequences. Existing mitigation approaches, such as user confirmation and LLM-based dynamic action validation, still suffer from limitations in usability, security, and performance. To address these challenges, we propose CSAgent, a system-level, static policy-based access control framework for computer-use agents. To bridge the gap between static policy and dynamic context and user intent, CSAgent introduces intent- and context-aware policies, and provides an automated toolchain to assist developers in constructing and refining them. CSAgent enforces these policies through an optimized OS service, ensuring that agent actions can only be executed under specific user intents and contexts. CSAgent supports protecting agents that control computers through diverse interfaces, including API, CLI, and GUI. We implement and evaluate CSAgent, which successfully defends against all attacks in the benchmarks while introducing only 1.99% performance overhead and 5.42% utility decrease.
1 Introduction
Computer-use agents can control devices through APIs, CLIs, and GUIs, but LLM uncertainty and broad permissions can lead to unintended, irreversible actions. CSAgent uses development-time intent- and context-aware policies, an automated policy toolchain, and an optimized OS service to protect agents across these modalities.
1 Introduction
- LLM attacks, hallucinations, and user-level permissions can lead agents to perform unintended irreversible actions with serious consequences.Examples include deleting data, transferring funds, or unlocking devices.Source: §1 · p. 1
- CSAgent applies system-level static access-control policies to agent functions, with execution permitted only under specified contexts such as user intent and system state.The OS service enforces the policies at runtime; developers and administrators can author them during development.Source: §1 · p. 1
- Intent-aware context space indexes policies by function and intent, while an LLM predicts potential user intents during policy generation.This addresses variation in security constraints that depends on the user's request.Source: §1 · p. 2
- The context analyzer generates context spaces across modalities; for GUI apps, it links identified elements to event handlers and builds call graphs, while policy evolution supports refinement.The framework supports iterative improvement based on app updates or runtime feedback.Source: §1 · p. 2
- Parallel processing and systematic context management target runtime intent-extraction and context-loading bottlenecks.The manager also categorizes contexts by update frequency and caches context spaces during app switching.Source: §1 · p. 2
- 100% of benchmark attacks are blocked with policy evolution, with 1.99% average additional latency and 5.42% utility decrease.The evaluation spans AgentBench, AgentDojo, and AndroidWorld.Source: §1 · p. 2
2 Background
Computer-use agents control computers through API, CLI, or GUI interfaces. These modalities differ in how they expose actions and in their scope and constraints.
2 Background
- API-based control provides reliable, efficient high-level actions through defined interfaces, but agents are limited to available APIs and require new tool development for added functionality.Examples include document processing, ticket booking, and route planning.Source: §2 · p. 3
- CLI-based control offers broad access to system functions through command-line tools, but command injection can enable code execution, privilege escalation, or system compromise.Tool providers typically use sandboxing and command filtering to mitigate these risks.Source: §2 · p. 3
- GUI-based control lets agents interact with any GUI app by perceiving screens or GUI trees and manipulating elements, but comprehension challenges and per-action LLM inference can hinder completion and performance.This broad compatibility includes legacy apps without API or CLI support.Source: §2 · p. 3
3 Motivation and Threat Model
Computer-use agent safeguards face a tension between retaining useful actions and preventing harmful ones. Context-sensitive pre-execution rules offer a more flexible approach, but runtime policy generation and the stated threat assumptions constrain the setting.
3.1 Security-usability Trade-offs in CUAs
- Unconditional filtering and sandboxing cannot resolve the security-functionality tension: allowing risky actions preserves capability, while prohibiting them limits it.Frequent confirmations also burden usability and add latency through repeated user interaction and LLM inference.Source: §3.1 · p. 3
3.2 Rule-based Context Validation for CUAs
- Context validation can distinguish safe from risky uses of the same action, but runtime LLM policy generation remains costly and inconsistent.Prior generated policies had overall similarity around 0.76, even at temperature 0.Source: §3.2 · p. 4
3.3 Threat Model and Assumptions
- The threat model targets excessive agency and sensitive-information disclosure, assumes benign instructions and trusted software components, and excludes attacks on LLM training or deployment infrastructure and content-safety issues.The agent's LLM may still produce untrustworthy outputs when handling untrusted external inputs or making security-critical decisions.Source: §3.3 · p. 4
4 CSAgent Design
CSAgent organizes access control around application-specific policies that account for user intent and runtime context. Its design combines automated policy construction and refinement with runtime validation and context-management optimizations.
4.1 Design Goals and Overview
- CSAgent abstracts API, CLI, and GUI operations as functions, using a development toolchain to construct policies and an OS service to validate actions at runtime.The design also supports developer-written policies and offers user-decision or contextual-guidance fallbacks when validation fails.Source: §4.1 · pp. 4–5
4.2 Intent-aware Context Space
- Intent-aware context spaces organize application policies by function and user intent, allowing the same function to have different constraints for different intents.Validation requires all rules in the matching policy to pass; unmatched sensitive actions are denied unless explicitly authorized.Source: §4.2 · pp. 5–6
4.3 LLM-based Context Analyzer
- The context analyzer generates policies through staged LLM reasoning, while GUI policy generation combines event-handler discovery with static call-graph analysis.For conditional functions, it predicts possible intents, then identifies dependent contexts, constraints, and metadata; GUI event handlers are identified with LLM code analysis and linked to their execution flows.Source: §4.3 · p. 7
4.4 Optimized Context Manager
- The context manager extracts intents concurrently with agent processing, updates contexts according to their temperature levels, and caches recently used context spaces.Cold, warm, and hot contexts are updated respectively on loading or switching, new instructions, and before each policy validation; the cache uses LRU replacement.Source: §4.4 · p. 8
4.5 Policy Evolution Framework
- The Policy Evolution Framework validates generated policies, synchronizes them with app updates, and uses runtime feedback to suggest refinements for developer review.It can also adapt user-customizable parameters after user approval, while strict defaults are used initially.Source: §4.5 · p. 9
5 Implementation
The implementation combines a policy-generation and evolution toolchain with an OS service for runtime protection, and maps agent actions to policies for validation.
5 Implementation
- CSAgent is implemented in Python as a context-space generation and evolution toolchain plus an OS service for runtime protection.Context spaces use JSON, and the interface is designed to integrate with existing agent frameworks with minimal modification.Source: §5 · p. 9
- The implementation maps API and CLI calls directly to function entries, while GUI actions are mapped through the targeted element’s package, class, and resource ID.Before each action, the service retrieves the policy using function and intent indexing.Source: §5 · p. 10
- Context values are acquired from user instructions, systems, and agent operations, with agent parameters captured during interception and history updated after actions complete.System-state contexts can be obtained through CLI or APIs, or through dedicated services requiring platform-vendor participation.Source: §5 · p. 10
6 Security Analysis
The security analysis formalizes system and agent behavior, states three security properties, and proves safety and utility preservation while examining threats and policy reliability.
6.1 Formal Model
- The formal model represents the system as a tuple, its state as context and environment, and agent decisions as potentially erroneous function executions.It also partitions functions by security risk.Source: §6.1 · p. 10
6.2 Security Properties
- The stated properties cover context-dependent safety for conditional functions, complete policies for security-sensitive functions, and prevention of unauthorized conditional or dangerous functions.Source: §6.2 · p. 10
6.3 Main Security Theorems
- CSAgent's theorems preserve safety by requiring validation for executions and preserve legitimate safe operations by permitting normal functions and policy-satisfied conditional functions.For conditional functions, execution requires intent matching and policy satisfaction; safe conditional operations pass validation.Source: §6.3 · pp. 10–11
6.4 Threat Analysis
- Prompt-injection validation relies on trusted user instructions and system context, while hallucinated or incorrectly classified actions remain subject to policy checks.Incorrect intent selection typically causes validation failure rather than unsafe execution; policy incompleteness remains a reliability concern, with user intervention or replanning as fallback.Source: §6.4 · p. 11
7 Evaluation
CSAgent was evaluated across CLI, API, and GUI benchmarks for integration, GUI analysis, attack defense, overhead, and policy evolution. Policy refinement improved the security–utility–performance results.
7.1 Experimental Setup and Benchmarks (RQ1)
- The evaluation used AgentBench for CLI, AgentDojo for API, and AndroidWorld for GUI agents, with policies generated without access to security test sets.Vanilla agents and PVAgent provided comparison configurations; integration used minimal RPC hooks.Source: §7.1 · pp. 11–12
7.2 Capability of GUI Analysis (RQ2)
- CSAgent extracted 0.93× more GUI element information than AutoDroid and 3.12× more than UI-CTX.The analyzer ignored non-functional elements that did not produce actual control effects.Source: Figure 6 · p. 12, §7.2 · p. 12
7.3 Effectiveness of CSAgent Access Control (RQ3)
- 0.44% average ASR meant CSAgent defended against approximately 99.56% of attacks; after policy evolution, it blocked all tested attacks on AgentBench and AndroidWorld.CSAgent had 100% successful defense across AgentDojo injection scenarios; PVAgent defended against 96.59% of attacks.Source: Table 3 · p. 12, Table 4 · p. 12, §7.3 · p. 12
7.5 Efficacy of Policy Evolution (RQ5)
- Across all benchmarks, CSAgent-RF reduced overall utility decrease to 5.42% and average latency overhead to 1.99%, with lower overhead primarily stemming from fewer completion steps on some complex tasks.Source: §7.5 · p. 14
7.4 Performance and Cost Overhead (RQ4)
- Less than 5% latency overhead across all benchmarks accompanied a 15.58% average utility decrease before refinement.AgentDojo had a more pronounced utility decrease; reflection reduced its utility drop to 9.93% but raised latency overhead to 21.04%.Source: Figure 7 · p. 13, §7.4 · p. 13
- 96.00% of context spaces loaded within one second, and cached retrieval latency was negligible.Average policy counts were 120 for AgentBench, 13.25 for AgentDojo, and 150.4 for AndroidWorld.Source: Figure 8 · p. 13, §7.4 · p. 13
- 35.30% was the geometric mean token-consumption overhead, with over 96.41% consisting of prompt tokens.CSAgent used one additional LLM inference per task regardless of complexity; its development-phase policies applied across devices and users without regeneration.Source: Table 5 · p. 13, §7.4 · pp. 13–14
7.5 Efficacy of Policy Evolution (RQ5)
- Policy evolution converged in an average of 7.25 iterations per context space.Refinement produced more explicit intents, more precise constraints and validation logic, and relaxed overly conservative constraints.Source: §7.5 · p. 14
8 Discussion
The discussion identifies policy testing coverage and deployment scope as remaining boundaries for CSAgent.
8 Discussion
- Broader automated testing is still needed to cover more agent actions and attacks and assess policy completeness at scale.Current protection focuses on single-agent scenarios; multi-agent context transfer remains future work.Source: §8 · p. 14
- Web protection is bounded by fragmented environments and content-security issues, while GUI policy generation through source-code analysis remains an adoption hurdle.The evaluation included TypeScript apps, but the authors note these web-environment challenges.Source: §8 · p. 14
9 Related Work
Related work spans OS access-control models, agent policy validation, and other system-level protections.
Access control and contextual integrity
- Many agent access-control approaches use runtime LLM security decisions, while AgentSpec uses static policies but lacks user-intent differentiation.The authors characterize runtime LLM decision-making as having security or performance limitations.Source: §9 · p. 14
Other system-level LLM agent protection
- Other system-level protections address input separation, information flow, sandboxing, isolation, and agent planning.The paper presents these approaches and CSAgent as potentially complementary parts of defense in depth.Source: §9 · p. 15
10 Conclusion
The conclusion presents CSAgent as a context-space framework for intent- and context-aware access control over computer-use agents.
10 Conclusion
- Evaluation shows strong security guarantees with acceptable overhead for CSAgent’s contextual and intent-aware policy validation.Source: §10 · p. 15