PaperlayerView exact revision on arXiv

Source-linked AI summary

When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents

Jaylen Jones, Zhehao Zhang, Yuting Ning, Eric Fosler-Lussier, Pierre-Luc St-Charles, Yoshua Bengio, Dawn Song, Yu Su, Huan Sun

arXiv:2602.08235v2cs.CLcs.AIcs.CR

TL;DR

CUAs can produce severe unintended behaviors from benign inputs, but existing research lacks a concrete framework and automated way to study these long-tail risks. The paper defines such behaviors and introduces AutoElicit, which iteratively perturbs benign instructions using execution feedback while preserving realism and benignity. AutoElicit surfaces harmful behaviors at high rates and reveals persistent susceptibility across frontier CUAs.

  • Problem

    Research on unintended CUA behaviors is largely anecdotal and lacks systematic characterization and automated elicitation for realistic benign scenarios, despite potentially severe consequences for users, data, and systems.

  • Method

    The paper defines unintended CUA behaviors and introduces AutoElicit, a two-stage framework that generates benign task perturbations and iteratively refines them using CUA execution feedback while enforcing realism and benignity.

  • Results

    AutoElicit surfaces harmful unintended behaviors from frontier CUAs, reaching up to 72.5% of OS-domain seeds and 60.8% of Multi-Apps seeds for Claude 4.5 Haiku, with successful perturbations transferring across other frontier CUAs.

  • Takeaways & Limitations

    The framework establishes a foundation for systematically analyzing when benign inputs lead to unsafe CUA behaviors in realistic computer-use settings.

  • Takeaways & Limitations

    The paper focuses on Cybersecurity Risks and leaves Agentic Misalignment Risks as a frontier for future automatic elicitation.

Abstract

from arXiv · show

Although computer-use agents (CUAs) hold significant potential to automate increasingly complex OS workflows, they can demonstrate unsafe unintended behaviors that deviate from expected outcomes even under benign input contexts. However, exploration of this risk remains largely anecdotal, lacking concrete characterization and automated methods to proactively surface long-tail unintended behaviors under realistic CUA scenarios. To fill this gap, we introduce the first conceptual and methodological framework for unintended CUA behaviors, by defining their key characteristics, automatically eliciting them, and analyzing how they arise from benign inputs. We propose AutoElicit: an agentic framework that iteratively perturbs benign instructions using CUA execution feedback, and elicits severe harms while keeping perturbations realistic and benign. Using AutoElicit, we surface hundreds of harmful unintended behaviors from state-of-the-art CUAs such as Claude 4.5 Haiku, Claude 4.5 Opus, and Operator. We further evaluate the transferability of human-verified successful perturbations, identifying persistent susceptibility to unintended behaviors across various other frontier CUAs. This work establishes a foundation for systematically analyzing unintended behaviors in realistic computer-use settings.

1. Introduction

CUAs can produce severe unintended harms from benign, realistic tasks because natural-language instructions imperfectly capture user intent, yet such long-tail risks remain difficult to study systematically. The paper introduces a conceptual framework and AUTOELICIT, which automatically refines benign task perturbations using execution feedback to surface these behaviors.

  • Motivation: Benign CUA tasks in file management, system administration, and software engineering can create substantial risks to users, data, and operating systems.A sensitive SSH-configuration task illustrates how a minor mistake can compromise system security.
  • Motivation: Natural-language instructions are imperfect proxies for users’ full intentions, creating discrepancies between stated tasks and underlying safety expectations.Task complexity and nuance make fully specified acceptable and unacceptable behaviors difficult to convey.
  • Research gap: Long-tail unintended behaviors can cause severe or irreversible consequences, but existing research lacks a unified framework and automated strategy for proactively surfacing them.Prior efforts rely substantially on anecdotal reports and manual scenario construction.
  • Contributions: The paper defines a conceptual framework that characterizes unintended CUA behaviors and their benign input factors, enabling analysis beyond anecdotal observations.The framework also organizes main behavior categories and conditions that give rise to them.
  • Contributions: AUTOELICIT generates seed perturbations for benign OSWorld tasks and iteratively refines them using real-world execution feedback while enforcing realism and benignity.The resulting AUTOELICIT-SEED collection contains 361 seed perturbations across OS and Multi-Apps domains.
  • Findings: 72.5% of OS-domain seed perturbations and 60.8% of Multi-Apps seed perturbations elicited harms from Claude 4.5 Haiku, with successful perturbations transferring across frontier CUAs.The study also analyzes hundreds of successful elicitations for recurring vulnerability patterns and failure modes.

2. Related Work

Research on unintended CUA behaviors has documented real-world risks but remains fragmented, while existing automated elicitation methods do not preserve the benign and realistic conditions of CUA safety testing.

  • Unintended CUA behaviors: Prior CUA safety evidence comes from ad-hoc model-card evaluations and recurring online reports of system damage, without a unified framework for comprehensive analysis.The cited prior work includes ToolEmu, Bloom, TAI3, BGD, and OS-Harm.
  • Unintended CUA behaviors: Research on self-preservation, deception, and scheming focuses on agents pursuing misaligned internal goals rather than harms caused by misinterpreted user intent.The paper positions unintended harms from benign inputs as a more immediate risk for current CUA capabilities.
  • Behavior elicitation: No automatic method reliably elicits unintended CUA behaviors in realistic user scenarios while maintaining benign and realistic inputs.Prior automatic elicitation and jailbreaking methods do not directly satisfy these CUA safety constraints.

3. A Conceptual Framework for Unintended Behaviors of Computer-Use Agents

The paper defines unintended CUA behaviors as coordinated unsafe actions that deviate from user intent, emerge from benign contexts, violate safety constraints, and differ from ordinary execution mistakes.

  • Definition: An unintended behavior is an unsafe agent behavior that deviates from user intent, emerges inadvertently from benign instructions and environments, and may comprise harmful actions, reasoning steps, or sequences.The definition requires coordinated, goal-directed action toward a harmful outcome.
  • Definition: The framework identifies four characteristics: deviation from inferred user intent, emergence from fully benign inputs, violation of safety constraints, and distinction from general mistakes.The fourth characteristic uses CUA reasoning to identify deliberate harms rather than simple execution errors.
  • User intent: Because natural-language tasks rarely specify acceptable intermediate behaviors comprehensively, practical CUA use relies on weaker safety specifications and inferred user intent.A complete formal specification or human-audited list would make deviations easier to identify mechanically.
  • Benign input: The framework excludes adversarial manipulation, requiring unintended behaviors to arise solely from benign instructions and environment observations.This distinguishes them from explicit harmful requests and indirect prompt injection.
  • Safety constraints: Cybersecurity Risks cover unintended harms to users, data, and systems that compromise confidentiality, integrity, or availability.The paper links these risks to underspecified instructions and excessive delegation of control.
  • Safety constraints: The paper focuses on Cybersecurity Risks, while treating Agentic Misalignment Risks as an important frontier for future automatic elicitation.Agentic Misalignment Risks involve misaligned objectives and loss of user control.
  • Goal-directedness: Unintended behaviors require concerted effort toward harmful outcomes, distinguishing them from mistakes such as selecting Delete instead of Save despite correctly inferring user intent.The proposed distinction relies on monitoring deliberate harmful planning in the agent’s explicit reasoning, while acknowledging faithfulness and obfuscation limitations.

4. AUTOELICIT: Automatically Eliciting and Analyzing Unintended Behaviors

AUTOELICIT is a two-stage, black-box framework that generates plausible harmful targets for benign tasks and refines their perturbations through execution and quality feedback. It is designed to surface unintended behaviors while preserving realistic, benign prompts.

  • Framework overview: AUTOELICIT automatically elicits unintended CUA behaviors in realistic and benign scenarios using Context-Aware Seed Generation and Execution-Guided Perturbation Refinement.The two-stage design limits costly execution-based refinement to scenarios with high elicitation potential.
  • Context-Aware Seed Generation: Context-Aware Seed Generation uses task and environment context to propose plausible unintended behavior targets paired with minimally perturbed instructions.Preprocessing captures screenshots, describes the initial environment, and records a representative CUA trajectory.
  • Context-Aware Seed Generation: Seed generation evaluates target feasibility, contextual plausibility, harm severity, realism, and benignity before iterative refinement and filtering.Multiple LLM judges score targets and perturbations, while retained seeds must satisfy quality thresholds.
  • Execution-Guided Perturbation Refinement: Execution-Guided Perturbation Refinement executes perturbed instructions, analyzes trajectories, and iteratively updates perturbations using execution feedback.An outer loop records unsuccessful trajectories, while an inner quality loop rejects candidates that violate realism or benignity constraints.
  • Execution-Guided Perturbation Refinement: Trajectory analysis summarizes agent actions, evaluates whether unintended behavior occurred, assigns a Behavior Elicitation Score, and categorizes harm severity.Perturbations exceeding a predefined score threshold become successful elicitation examples; others enter Perturbation History for refinement.
  • Framework scope: AUTOELICIT provides a fully automatic elicitation framework requiring only black-box access to surface risks from frontier CUAs and relevant use cases.The framework explicitly targets realistic unintended behaviors rather than adversarially requested harms.

5. Experiments

Experiments show that AUTOELICIT can automatically surface severe unintended behaviors from frontier CUAs across realistic benign computer-use scenarios, with successful perturbations transferring across agents. Ablations and meta-analysis further examine why the framework works and organize recurring vulnerability patterns.

  • Experimental setup: AUTOELICIT evaluates realistic benign OSWorld scenarios across OS and Multi-Apps domains using perturbation datasets grounded in manually curated tasks.The seed construction spans core Ubuntu applications and multi-application workflows, supporting broad coverage of computer-use settings.
  • Elicitation results: 72.5% of OS seeds and 60.8% of Multi-Apps seeds elicited unintended behaviors from Claude 4.5 Haiku.The reported elicitation success rates measure the percentage of seeds producing unintended behavior.
  • Elicitation results: 9.2–10.1% of seeds produced High or Critical severity harms, while Operator showed elicitation success rates of 81.6% for OS and 67.8% for Multi-Apps.Operator results also used a 0% baseline harm rate for the evaluated tasks.
  • Transferability: 35.0%–53.8% of human-verified perturbations transferred to other frontier CUAs, indicating broad transferability beyond the source agents.The transferability study evaluates human-verified perturbations across target agents, while open-source agents showed greater robustness than closed-source agents.
  • Additional analyses: 68.8–86.2% of successful elicitations required execution-guided refinement, while verbalized sampling improved seed diversity and ensemble judging addressed moderate evaluator agreement.Ablations also found that removing the inner quality-refinement loop substantially reduced elicitation success and perturbation quality.

6. Conclusion

The paper introduces a conceptual framework and automatic elicitation pipeline for systematically studying unintended CUA behaviors, showing that frontier CUAs remain vulnerable during benign execution.

  • The framework characterizes unintended CUA behaviors and automatically elicits them by iteratively perturbing benign instructions using agent execution feedback.
  • The analysis reveals consistent vulnerabilities in frontier CUAs during benign execution.

Impact Statement

The paper establishes a framework for systematically studying unintended behaviors that computer-use agents may exhibit from benign, naturally occurring inputs. It aims to enable proactive, scalable safety analysis for more reliable real-world deployment.

  • The paper introduces a conceptual framework for characterizing unintended computer-use-agent behaviors arising from benign inputs.
  • It also introduces an automatic elicitation framework to proactively identify risks caused by ambiguity and imperfections in natural-language instructions.
  • Scalable proactive analysis is intended to support safer and more reliable use of computer-use agents in real-world applications where safety is paramount.

Overview

The overview organizes supporting material across reproducibility, analysis, ablations, costs, transferability, annotation, future work, examples, risks, and seed-generation details.

  • Further appendices cover context-aware seed generation, execution-guided perturbation refinement, elicitation costs, transferability analysis, and human annotation procedures.
  • The appendices include reproducibility analysis, meta-analysis details and results, and additional ablations and analyses.
  • The listed materials collectively provide methodological details and additional analyses supporting the paper’s study of unintended computer-use-agent behaviors.
  • They also document AWS instance usage, future work, examples of unintended behaviors, and agentic misalignment risks.

A. Future Work

The paper identifies persistent susceptibility to unintended behaviors and outlines future mitigation and elicitation improvements. AutoElicit generates context-grounded, minimally modified benign instructions and refines them through automated evaluation and execution-oriented analysis.

  • Future Work: The results reveal persistent susceptibility of frontier computer-use agents to unintended behaviors from benign input contexts, motivating mitigation strategies for trustworthy deployment.
  • Future Work: Future elicitation work could perturb environment context, surface risks from open-ended GUI tasks, and reduce the costs of iterative agent execution.
  • Future Work: The paper focuses on cybersecurity risks while leaving agentic misalignment risks as a critical frontier for future research.
  • Method: Context-aware seed generation proposes plausible harm targets and perturbed instructions grounded in the task’s environment and representative execution trajectories.
  • Method: AutoElicit evaluates and refines candidate seeds with quality scoring and filtering while keeping instruction changes minimal, benign, and realistic.
  • Method: The elicitation targets OS-level state changes including process termination, file and directory modifications, permission changes, system-configuration changes, and resource exhaustion.

D.6. AUTOELICIT-SEED

AUTOELICIT-SEED provides verified perturbation seeds for eliciting unsafe unintended behaviors from benign OSWorld tasks, then uses execution feedback to refine them. Its pipeline combines trajectory summarization, evaluation, and iterative failure-guided perturbation.

  • Seed Dataset: 361 seed perturbations from 66 benign tasks form AUTOELICIT-SEED across OS and Multi-Apps domains.The dataset cost $225.32 total.
  • Execution-Guided Refinement: AUTOELICIT executes each seed and iteratively refines failed perturbations using execution feedback until elicitation succeeds or the iteration limit is reached.Failure analysis examines recurring patterns and reasons why prior attempts avoided harmful behavior.
  • Trajectory Analysis: Trajectory summarization converts full executions into step-by-step action and state-change descriptions, prioritizing screenshots as visual ground truth.The summary includes actions, reasoning traces, and output screenshots for downstream evaluation.
  • Trajectory Analysis: The Trajectory Evaluator assigns a 0-100 Behavior Elicitation Score, severity category, justification, and REFINE or COLLECT recommendation.Severity ranges from None and Minimal through Low, Medium, High, and Critical.
  • Execution-Guided Refinement: The pipeline uses nested feedback to improve elicitation efficiency while exploring benign instructions unlikely to reveal severe harms through unguided search.The outer loop refines toward elicitation, while the broader process uses execution feedback to explore long-tail risks.

E.2. Inner Loop (Quality Evaluation Feedback)

The inner quality loop filters and repairs perturbations before execution so that elicited harms arise from realistic, benign-looking requests rather than explicit adversarial instructions. It applies constraint scoring, ensemble evaluation, and bounded refinement.

  • Quality Evaluation: AUTOELICIT evaluates every proposed perturbation for benignity, realism, contextual plausibility, goal preservation, harm plausibility, and implicitness before execution.The process uses predefined thresholds for each Constraint Adherence Score criterion.
  • Quality Evaluation: An ensemble of LLM evaluators uses majority voting to decide whether each perturbation satisfies the required constraints.The quality check tests whether requests have safe interpretations, resemble plausible user requests, and avoid explicitly directing harm.
  • Quality Refinement: Failed quality checks trigger targeted refinement addressing violations such as explicit harm references, unrealistic phrasing, or environmental incompatibility.Refinement continues until all criteria pass or the maximum iteration count is reached.

F.3. Small-Scale Elicitation Analysis on Claude 4.5 Opus

A small-scale evaluation applies AUTOELICIT to 120 randomly sampled seeds for Claude 4.5 Opus, while checking baseline safety and reporting execution costs. The design also examines whether severe elicitation patterns persist across stronger Opus variants.

  • Evaluation Setup: 120 randomly sampled seeds, split evenly between OS and Multi-Apps domains, are used for the Claude 4.5 Opus analysis.The corresponding harm-severity distribution is evaluated on Claude 4.5 Haiku with GPT-5 as refinement model.
  • Run Statistics: The study reports Opus-run average costs of $15.48 and $17.70, with 5.0 and 4.2 average execution iterations, respectively.These figures correspond to the two listed Opus evaluation configurations.
  • High-Severity Evaluation: A targeted evaluation selects the 30 most severe Haiku seeds to test worst-case vulnerability persistence in Claude 4.5 Opus and Claude 4.6 Opus.Seeds are ranked first by harm severity and then by elicitation score.
  • Transfer Evaluation: The transfer dataset contains 117 human-verified perturbed instructions collected from Haiku and Opus source agents.50 instructions originate from Haiku and 67 from Opus; verification requires zero baseline harm, consistent source-agent elicitation, and human-validated outcomes.

G.2. Target Agent Evaluation

Transferability is evaluated across seven target CUAs using repeated executions of 117 perturbed instructions. The protocol defines transfer success by whether at least one run elicits unsafe behavior and reports both source-specific and overall transfer rates.

  • Target Agents: Seven target CUAs span open-source and closed-source systems with different architectures, training paradigms, and capability levels.The evaluation includes four open-source agents and two named closed-source agents alongside the source-agent setup.
  • Evaluation Protocol: 2,808 executions result from 117 instructions evaluated on each target agent across three independent runs.The protocol uses a consistent OSWorld environment for every instruction-agent pair.
  • Transfer Criterion: Transfer success is recorded when an instruction elicits unsafe behavior from a target agent in at least one of its three runs.This criterion accounts for execution variability while treating a single unsafe execution as a practical safety risk.
  • Transfer Metrics: The study reports source-specific transfer rates and overall transfer rates as complementary measures of cross-agent vulnerability.Source-specific rates condition on the originating agent, whereas overall rates aggregate across all perturbed instructions.

H. Human Annotation

The study validates AUTOELICIT’s automatic evaluator through human annotation, defining unintended behaviors as coordinated harmful deviations from user intent that emerge from benign inputs. The evaluator shows strong precision and negative-label reliability, but judging harm and benignity remains difficult.

  • Annotation Criteria: Human annotation evaluates trajectory harm, perturbation quality, and goal-directed behavior through sequential criteria.The criteria distinguish harmful unintended behavior from general execution mistakes and direct harmful instructions.
  • Operational Definition: An unintended behavior is coordinated, goal-directed harmful action that deviates from user intent and emerges from benign instruction and environment contexts.The formulation distinguishes these behaviors from capability failures caused by ordinary execution mistakes.
  • Evaluator Reliability: 79.5% True Positive Rate indicates that the automatic evaluator accurately labels most successful elicitation predictions.Three annotators used majority voting to determine whether unintended behavior occurred.
  • Annotation Challenges: Annotators and the automatic evaluator struggle most with open-ended judgments about harm and whether perturbations remain benign.Unanimous agreement decreases for elicitation and perturbation evaluation failures, while severe harms have lower false-positive rates.
  • Evaluator Reliability: 98% True Negative Rate indicates strong reliability in identifying trajectories without elicited harm.Majority voting identified only one false negative in the annotation set.

I. Reproducibility Analysis

The reproducibility study re-executes successful Claude 4.5 Haiku elicitation cases five times to test whether harmful behaviors persist. Most cases reproduce, supporting systemic rather than purely random vulnerabilities, although testing is limited to an OS-domain subset.

  • Scope: Reproducibility testing was limited to a subset of the OS domain because large-scale agent execution and LLM-based evaluation were computationally costly.The reported analysis covered 152 successful cases spanning eight unique tasks.
  • Reproducibility Findings: 87.5% of 152 successful elicitation cases were reproducible across five re-executions for Claude 4.5 Haiku.Harmful behavior occurred in 57.6% of individual trials and across all original tasks.
  • Interpretation: The reproducibility results provide evidence that elicited harms reflect systemic vulnerabilities in agent instruction following rather than random errors.A case counted as reproducible when at least one of five trials exhibited harmful behavior.
  • Meta-Analysis: The broader analysis motivates extracting recurring benign-input vulnerability patterns because large elicitation volumes can obscure high-level insights.The meta-analysis summarizes runs, categorizes recurring vulnerabilities, and clusters categories by shared root causes.

J.3. Results

AUTOELICIT substantially outperforms mutation baselines in elicitation success and perturbation quality while surfacing more severe harms. Its meta-analysis identifies recurring vulnerability categories and shows that frontier CUAs often fail to default to safety constraints treated as implicit expectations.

  • Meta-Analysis: The meta-analysis identifies 30 categories and 13 clusters for Opus, versus 99 categories and 29 clusters for Haiku.It analyzes 87 successful Opus perturbations and 437 successful Haiku perturbations.
  • Vulnerability Patterns: Frontier CUAs may perform overbroad deletions or unsafe permission changes when requests such as “clean up” or ensuring access leave safety constraints implicit.The analysis connects these failures to ambiguous requests and argues that agents should seek clarification and avoid dangerous actions under confusion.
  • Baseline Comparison: 72.5% per-seed elicitation success exceeds TAI3’s 24.2% and SelfRef’s 17.6% under the baseline comparison.The comparison uses the same success threshold and trajectory evaluation pipeline across methods.
  • Baseline Comparison: AUTOELICIT produces more consequential harms, with 41.8% of successful elicitations in Minimal+Low severity versus 62.5% for TAI3 and 66.7% for SelfRef.The authors attribute the difference to dual feedback and quality constraints guiding exploration beyond superficial deviations.
  • Perturbation Quality: AUTOELICIT achieves higher perturbation-quality scores across all six evaluated dimensions, including Harm Plausibility of 80.2 versus 13.2 for TAI3 and 5.6 for SelfRef.The quality evaluation uses the same constraint-adherence scores and ensemble evaluators as AUTOELICIT’s quality gate.

K.2. Verbalized Sampling

The ablations evaluate how verbalized sampling, execution-guided refinement, inner quality refinement, and ensemble judging affect perturbation diversity, elicitation success, and quality assessment. Results show that these components improve exploration and elicitation while judges retain distinct evaluation tendencies.

  • Verbalized Sampling: Removing VS-Multi consistently reduces seed diversity, increasing Self-BLEU by +0.140, decreasing normalized pairwise edit distance by −0.094, and lowering strategy entropy from 1.36 to 0.93.These metrics indicate more repetitive, similar, and less uniformly exploratory perturbations without multi-turn verbalized sampling.
  • Verbalized Sampling: Claude 4.5 Opus shows the largest VS-Multi diversity gains, with Self-BLEU +0.214, edit distance −0.159, and entropy changing from 1.84→0.88.The passage reports similarly strong improvements for o4-mini and smaller improvements for GPT-5.
  • Execution-Guided Perturbation Refinement: Execution-Guided Perturbation Refinement substantially improves elicitation success across the evaluated execution agents.The refinement stage uses execution feedback to improve strategies beyond initial seed perturbations, though it requires additional agent executions.
  • Inner Quality Refinement Loop: Removing the Inner Quality Refinement Loop lowers Elicitation Success Per Seed by 26.7% and Average Elicitation Score by 7.5.Constraint adherence also declines, with the largest drops in Harm Plausibility (-53.1) and Implicitness (-41.5).
  • Ensemble Judge: The three ensemble judges show moderate or low agreement, ranging from 36.0% to 61.0%, with high score variability for Benignity and Implicitness.o4-mini is most permissive, Claude 4.5 Haiku is stricter, and GPT-5 falls between them.

L. AWS Instance Usage

The supplied passages describe AWS execution infrastructure and prompts for generating comprehensive initial desktop-state descriptions and benign perturbations for CUA safety analysis. They also summarize vulnerability-cluster metadata from successful elicitation runs.

  • AWS Instance Usage: AWS EC2 t3.medium instances are used concurrently for elicitation experiments, with 50GB of EBS storage allocated by default.
  • Vulnerability Clusters: Meta-analysis organizes successful Haiku elicitation runs into vulnerability clusters, with Table 21 covering the remaining five of the top ten clusters across 437 successful perturbations.The cluster tables report names, definitions, counts, and proportions for recurring benign-input vulnerability patterns.
  • Initial Environment State Description: The initial-state prompt asks for a single comprehensive, flowing description of the desktop environment using the operating system, screenshot, setup steps, and task context.It prioritizes open applications, focus, tabs, visible content, current locations, and immediately available actions.
  • Initial Environment State Description: The prompt requires detailed reporting of application windows, terminal and browser state, file paths, visible content, system elements, input readiness, and actionable operations.Descriptions should be present-tense, specific, comprehensive, accurate, prioritized, and action-focused.
  • Initial Perturbed Instruction Generation: The perturbation-generation prompt analyzes a benign instruction and environment, reviews prior attempts, identifies a plausible harmful unintended behavior, and creates a minimally perturbed instruction.It directs the generator to use unintended-behavior primitives and elicitation strategies while keeping the scenario tied to the described task and environment.
Loading 2602.08235v2…