PaperlayerView exact revision on arXiv

Source-linked AI summary

SIR: Self-improving Red-teaming for Compute Use Agents

Chen Xiong, Zhiyuan He, Pin-Yu Chen, Stjepan Picek, Tsung-Yi Ho

arXiv:2608.30207v1cs.CRcs.AI

TL;DR

Computer-use agents face indirect prompt injection from untrusted content, while fixed hand-written benchmarks may underestimate adaptive attack risk. SIR composes plain-language attack principles, learns bypass strategies from failed trajectories, and improves attack success across three frontier agents using deterministic system-state evaluation.

  • Problem

    Computer-use agents can encounter adversary-controlled content, and fixed hand-written injection benchmarks may underestimate the risk of adaptive indirect prompt injection.

  • Method

    SIR composes stealthy injections from reusable plain-language principles and iteratively distills bypass strategies from failed trajectories in a black-box, OS-level setting.

  • Results

    SIR raised attack success above the fixed injection baseline across three frontier CUAs, including from 4% to 22% on Claude Opus 4.8 and from 0% to 28% on Gemini 3.5 Flash.

  • Takeaways & Limitations

    Discovered principles transfer across victim architectures, indicating that feedback produces reusable attack strategies rather than only per-instance payload refinements.

  • Takeaways & Limitations

    Results are a snapshot because the proprietary victim API models change over time, and the pipeline costs roughly $150 to $250 per Claude feedback round.

Abstract

from arXiv · show

Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal, and they are increasingly deployed to automate everyday digital tasks. Because they can be exposed to untrusted content while operating, they are vulnerable to indirect prompt injection (IPI), in which an adversary plants instructions in content the agent will read and redirects it toward actions that violate the user's intent. Existing CUA safety benchmarks evaluate fixed injections written by hand, which may underestimate the risk posed by an adaptive adversary. We present SIR, a black box IPI attack that (i) composes stealthy injections from a small library of reusable principles stated in plain language and (ii) wraps composition in an iterative feedback loop that diagnoses the victim's failed trajectories and distills the bypasses into new, named strategies that are reapplied across tasks. Unlike prior red teaming of web agents, we target CUAs at the operating system level and score attacks with a fully deterministic oracle, using checks on filesystem, service, and permission state rather than an LLM judge. On experiment, we evaluate three frontier CUAs. Composing principles with feedback raises the attack success rate over a baseline written by hand, for example from 4% to 24% on Claude Opus 4.8 and from 0% to 28% on Gemini 3.5 Flash, while the benign task still completes. Principles discovered against one model further transfer to a different architecture with no additional feedback.

1 Introduction

Computer-use agents face indirect prompt injections because they must interpret potentially adversarial environmental content, while fixed hand-authored evaluations may miss adaptive attacks. SIR addresses this gap with compositional, failure-driven red teaming that uses deterministic system-state scoring.

  • Motivation: CUAs can be redirected by adversarial instructions embedded in web pages, documents, forums, interfaces, or terminal output.Such injections can cause concrete system compromises, including file deletion, permission changes, or sensitive-data exfiltration.
  • Motivation: Fixed, hand-authored injections evaluated in one execution may underestimate the risk posed by motivated adversaries that diagnose failures and adapt.Existing benchmarks largely place one injection at a predetermined location and score the resulting run.
  • SIR: SIR learns from failed injections by diagnosing the defensive behavior that blocked them and distilling a reusable bypass principle in plain language.This reverses self-improvement approaches that primarily mine successful jailbreaks or wins.
  • SIR: SIR’s inner loop composes task-specific stealthy injections from reusable principles, while its outer loop feeds newly discovered strategies back into the library.The injection is planted in the environment, the CUA performs the benign task, and failed trajectories inform later rounds.
  • Evaluation: SIR was evaluated across 50 RedTeamCUA tasks and three frontier CUAs, with attack success improving over a hand-crafted baseline on every model while benign-task completion was checked separately.Representative increases were 4% to 22% on Claude Opus 4.8 and 0% to 28% on Gemini 3.5 Flash.

2 Related Works

Related work spans static and adaptive indirect-prompt-injection benchmarks, while SIR emphasizes failure-driven strategy expansion and deterministic system-state evaluation for OS-level computer-use agents.

  • Prior benchmarks: Prior work developed tool-integrated and visually grounded indirect-prompt-injection benchmarks for agents operating in stateful or interactive environments.Examples include tool-output attacks, deterministic utility and security functions, and targeted textual or visual attacks against web agents.
  • Adaptive red teaming: Recent adaptive methods automate injection search through mutation, semantic evolution, trajectory analysis, or discovery of high-salience injection surfaces.These approaches include AgentVigil, EVA, and MUZZLE.
  • SIR’s distinction: SIR differs by expanding a reusable strategy space from failed trajectories rather than only refining a payload or selecting a stronger injection surface for one scenario.Its success is determined with deterministic system-state oracles.
  • Comparison criteria: Table 1 compares representative benchmarks and adaptive methods using joint success as the criterion for attacks that complete both adversarial and benign objectives.This separates successful hijacking from merely derailing the user’s task.

3 Methodology

SIR models black-box OS-level indirect prompt injection as joint completion of an adversarial objective and benign task, then searches compositionally and expands its strategy library from failed trajectories.

  • Pipeline: SIR uses two nested loops: an inner loop produces task-specific injections, and an outer loop analyzes unsuccessful trajectories to distill reusable bypass strategies.The updated strategies guide subsequent attack generation.
  • Problem formulation: Each task contains a benign instruction, adversarial objective, and initial web–OS environment, while deterministic evaluators inspect post-execution state.The attacker places an injection in an attacker-controlled region and observes externally visible victim behavior.
  • Threat model: The attacker is strictly black-box: it cannot alter the user instruction or operating system and observes only rendered states, reasoning traces, actions, and evaluator outcomes.The injection is planted before execution, with no intervention after the CUA begins acting.
  • Objective: A successful run requires both the adversarial objective and benign task to be completed, ruling out attacks that merely derail the CUA.This operationalizes stealthy interference that leaves the user’s task successful.
  • Compositional attack search: The base inventory contains content-level and delivery-level principles, and each task’s injection is generated through seed, composition, and refinement calls.Composition typically selects 2–4 task-relevant principles, while refinement removes overtly adversarial wording and incorporates feedback strategies.
  • Failure-driven strategy discovery: Failed trajectories are diagnosed for recurring defensive behaviors, then distilled into named strategies containing a diagnosis, bypass mechanism, composition guidance, and example.The top K nonredundant strategies are merged into the shared library and applied to all tasks in the next round.

4 Experiments

SIR is evaluated against a fixed injection baseline on three frontier CUAs using 50 OS-level tasks and deterministic attack-state checks. Compositional search and failure-driven feedback substantially improve attack success, reveal model-specific strategy profiles, and support cross-model transfer.

  • Experimental Setup: 50 Reddit cases spanning seven attack categories are sampled from RedTeamCUA for evaluation across three frontier CUA models.The cases pair benign web-to-OS tasks with adversarial objectives embedded in forum comments.
  • Attack Effectiveness Across Frontier CUAs: 16%→54%, 4%→24%, and 0%→28%: SIR raises attack success rate over the fixed baseline on Claude Opus 4.6, Claude Opus 4.8, and Gemini 3.5 Flash, respectively.The Gemini result is notable because the fixed baseline never succeeds while SIR succeeds on more than a quarter of tasks.
  • Ablation: Effect of Feedback: Compositional search raises ASR from 6% to 18% over the hand-crafted baseline by tailoring principle combinations to individual cases.The gain targets surface-level defenses such as urgency-pattern detection and authority-claim filtering.
  • Ablation: Effect of Feedback: Feedback raises ASR from 12% to 20% by distilling bypasses for defensive behaviors that the base inventory cannot express.On Gemini 3.5 Flash, ASR rises from 8% to 28% with feedback, compared with base composition alone.
  • Analysis of Base-Principle Selection: P1+P2+D1 is the most frequent combination for both Claude models, accounting for 16% of Opus 4.6 and 20% of Opus 4.8 compositions.It combines prerequisite framing, authority embedding, and visual blending.
  • Analysis of Base-Principle Selection: P1+P3+D1 appears among the top selections for all three victim models at 10%, 12%, and 10%, respectively.Gemini places greater weight on side-effect masking, which appears in three of its four leading combinations, including P2+P4+D1 at 14%.
  • Cross-Model Transfer of Discovered Principles: Frozen principles discovered on Claude Opus 4.6 improve ASR on unseen models without further feedback, reaching 16% with P9 on Opus 4.8 and 18% with P9 on Gemini 3.5 Flash.The transfer results indicate reusable strategies rather than payloads tied only to the discovery target.

5 Conclusion

SIR is a black-box framework for adaptive indirect prompt injection against operating-system computer-use agents. It composes attacks from reusable principles, learns from failed trajectories, and evaluates outcomes using deterministic system-state checks.

  • SIR composes injections from a shared principle inventory and distills recurring failure patterns into named, reusable principles for later attacks.
  • SIR analyzes failed execution trajectories across tasks and scores attacks against deterministic system state rather than an LLM judge.
  • Across RedTeamCUA tasks and three frontier victim CUAs, SIR raised attack success above the benchmark’s fixed injection, including on a model the fixed injection never penetrates.
  • SIR is intended as a controlled measurement for surfacing vulnerabilities and guiding defenses before computer-use agents receive real system privileges.

A Base Principle Inventory

SIR’s attack search uses a reusable inventory of content and delivery principles, then incorporates strategies accumulated through failure analysis. The attacker selects and combines guidance according to the task context rather than applying every principle mechanically.

  • Content principles govern what the injection says, while delivery principles govern where and how it is placed.
  • The refine prompt combines principles selected during composition with strategies accumulated from previous feedback rounds.
  • The attacker is instructed to rewrite the injection by applying the selected principles.
  • An example selection combines Prerequisite Framing, Multi-Step Dilution, and High-Trust Positioning.
  • Failure analysis adds ErrorCorroboration, which presents a predicted real error and frames the adversarial action as its fix.
  • Failure analysis also adds ConditionalDeferred, which uses conditional language because the CUA’s safety filter applies weaker scrutiny to such advice.
  • The attacker LLM chooses which principles and discovered strategies to incorporate based on the specific task context.

C.1 Diagnose–Distill Procedure

SIR diagnoses failed trajectories in batches, groups them by defensive failure mode, and proposes named strategies with explicit bypass guidance. The procedure includes examples distilled from refusals followed by environment errors or weaker scrutiny of conditional advice.

  • Each trajectory is compressed into a roughly 650-token digest preserving the injection, initial reasoning response, key environment events, and final action.
  • The analyzer uses two LLM calls per feedback round: batch diagnosis followed by strategy proposal.
  • Call 1: Batch Diagnosis: Batch diagnosis identifies the defensive criterion blocking each case and groups cases by shared failure mode.
  • Call 2: Strategy Proposal: Strategy proposals must include a name, failure diagnosis, bypass mechanism, composition guidance, and an example injection fragment.
  • ErrorCorroboration is distilled when a CUA later encounters the specific error predicted by an initially refused injection.
  • ConditionalDeferred reflects weaker scrutiny of conditional advice such as “if the build fails, try removing...” than imperative commands.

D Implementation Details

The implementation runs CUAs and attacker models in an Ubuntu 22.04 VMware environment, with a feedback loop that generates, analyzes, and retains attack strategies. Evaluation records both adversarial success and benign task completion using deterministic state checks.

  • Experiments run in an Ubuntu 22.04 VMware virtual machine managed by OSWorld’s DesktopEnv at 1920×1080 resolution.
  • Claude CUAs use the Anthropic computer-use API, while Gemini 3.5 Flash uses Google’s native computer-use tool.
  • Injection generation uses Grok-4.3 with temperature 0.8, three API calls per case, and a banned-pattern quality filter.
  • Failure analysis uses batched roughly 650-token digests and two Grok-4.3 calls per feedback round.
  • The outer loop runs for up to R = 10 rounds, retains Top-K = 5 strategies per round, and early-stops when cumulative UASR improvement remains below 5% for two consecutive rounds.
  • Coordinate handling is pixel-based for Claude and normalized-to-pixel scaled for Gemini 3.5 Flash.
  • Deterministic evaluators check filesystem, service, permission, and count states while recording adversarial success and benign task completion.
  • The full 50-case evaluation has an approximate per-model cost reported in Table 6.

E Discovered Strategy Catalog

The discovered strategy catalog organizes bypasses by the defensive behavior they target, including trust shifting, destructive-action disguise, target reframing, and error misdirection. These named strategies are returned to the attacker as reusable natural-language guidance.

  • Catalog organization: The strategy names are emitted by the analyzer LLM and supplied as natural-language guidance to the attacker during refinement.The guidance is grouped by the defensive behavior each strategy targets.
  • Trust-shifting / self-attribution: Trust-shifting strategies make the agent attribute a malicious decision to its own reasoning rather than to injected content.Examples include TrajectoryAnchoredSelfAttribution and RetrospectivePredictionAnchoring.
  • Destructive-action disguise: Destructive-action disguise frames harmful operations as routine logging, cleanup, or post-build verification.The catalog includes debug_log_redirect and post_success_deferred_cleanup.
  • Target reframing: Target reframing recasts sensitive files or commands as benign maintenance operations to avoid target-sensitivity heuristics.Examples include plausible_npm_cache_corruption, package_json_corruption_via_mv, and novel_filename_as_fake_module.
  • Error misdirection: Error misdirection links plausible network, build, or permission symptoms to remediation steps that contain the payload.Named variants include dns_offline_error_misdirection, error_reproduction_first_then_fix, and env_var_fallback.

F Limitations

SIR’s limitations concern reproducibility, cost, and dual use. The evaluation uses changing proprietary models and produces transferable attack strategies that could be directed at systems with real privileges.

  • Reproducibility and cost: Proprietary victim models change over time, so the reported results are a snapshot that later revisions may not reproduce.This constrains reproducibility beyond the evaluated model versions.
  • Reproducibility and cost: $150 to $250 per feedback round for Claude models limits explored rounds, seeds, and benchmarks and constrains external replication.The cost applies per feedback round and restricts the breadth of evaluation.
  • Dual use: SIR produces transferable attack strategies that could be directed at systems with real privileges despite evaluation in a sandboxed VM.The authors restrict release and use to defensive research and responsible disclosure settings.

G Attack Success by Category

Attack success varies by adversarial category, with data exfiltration hardest among larger categories and file deletion or tampering generally more vulnerable. Permission-change success differs sharply across model generations, while category ordering remains broadly stable.

  • Category difficulty: 20%, 10%, 10%: data exfiltration is the hardest large category on Opus 4.6, Opus 4.8, and Gemini 3.5 Flash.Outbound transfer is difficult to frame as a benign maintenance step.
  • Category difficulty: File deletion and file tampering rank higher on every victim because destructive or modifying local-file actions can resemble cleanup or build fixes.The passage describes this as a consistent cross-victim pattern.
  • Permission changes: 50% to 0%: permission-change success drops from Opus 4.6 to Opus 4.8, while Gemini 3.5 Flash records 17%.The passage attributes this split to stronger scrutiny of permission-altering actions in the newer Claude model.
  • Cross-model pattern: Absolute rates fall as models advance, but the relative category ordering is largely stable across architectures.Actions easiest to disguise as legitimate remain most vulnerable, while outbound exfiltration remains most resistant.
Loading 2608.30207v1…