On the Reliability of Computer Use Agents

Gonzalo Gonzalez-Pumariega; Saaket Agashe; Jiachen Yang; Ang Li; Xin Eric Wang

Source-linked AI summary

On the Reliability of Computer Use Agents

Gonzalo Gonzalez-Pumariega, Saaket Agashe, Jiachen Yang, Ang Li, Xin Eric Wang

arXiv:2604.17849v1Apr 20, 2026cs.AI

TL;DR

Computer-use agents can succeed on individual tasks yet fail on repeated executions, motivating a study of what makes their behavior unreliable. The paper analyzes repeated OSWorld executions with paired task-level tests across stochasticity, instruction ambiguity, and agent behavior, finding that specification and behavioral variation both matter. It consequently emphasizes repeated-run evaluation, interactive ambiguity resolution, and stable strategies.

Problem
The paper asks why computer-use agents that can succeed once may fail when repeating the same task, despite unchanged task and model.
Method
The authors analyze repeated OSWorld executions across stochasticity, instruction ambiguity, and agent behavior using paired statistical tests for task-level changes.
Results
Reliability depends on task specification and variation in agent behavior across executions.
Takeaways & Limitations
Agents and benchmarks should evaluate repeated runs, support interaction to resolve ambiguity, and favor strategies that remain stable across executions.
Takeaways & Limitations
Deterministic approaches can remain brittle to environmental variation and fail catastrophically under changed conditions.

Abstract

from arXiv · show

Computer-use agents have rapidly improved on real-world tasks such as web navigation, desktop automation, and software interaction, in some cases surpassing human performance. Yet even when the task and model are unchanged, an agent that succeeds once may fail on a repeated execution of the same task. This raises a fundamental question: if an agent can succeed at a task once, what prevents it from doing so reliably? In this work, we study the sources of unreliability in computer-use agents through three factors: stochasticity during execution, ambiguity in task specification, and variability in agent behavior. We analyze these factors on OSWorld using repeated executions of the same task together with paired statistical tests that capture task-level changes across settings. Our analysis shows that reliability depends on both how tasks are specified and how agent behavior varies across executions. These findings suggest the need to evaluate agents under repeated execution, to allow agents to resolve task ambiguity through interaction, and to favor strategies that remain stable across runs.

1. Introduction

Computer-use agents can perform many benchmark tasks, but single-run success does not establish reliable repeated execution. The paper studies unreliability through execution stochasticity, task ambiguity, and variability in agent behavior, using repeated runs and paired task-level analysis.

Motivation: Single-run and averaged multi-run metrics do not reveal whether each individual task remains successful across repeated executions.An agent may succeed once and fail when the same task is executed again.
Motivation: Reliability matters for practical deployment because reproducible behavior is critical for safety and trust.The paper frames this as the question of what prevents one successful execution from becoming reliable.
Sources of unreliability: Task execution varies through underspecified instructions, different agent strategies, and stochasticity in decoding or the environment.These components can produce different trajectories and outcomes even when the task and model are unchanged.
Evaluation approach: Pass@k rewards producing at least one successful outcome, whereas Pass^k measures whether repeated executions all succeed.The latter is more aligned with reliability, but aggregate task summaries do not show task-level changes across settings.
Evaluation approach: Approximately 78% Pass@10 contrasts with about 36% Pass^10 for Agent S3 with GPT-5, showing that repeated reliability is substantially harder than one success among attempts.Pass@10 measures at least one success across ten attempts, while Pass^10 requires success on all ten executions.
Contribution: The paper extends Pass^k with paired statistical analysis to detect task-level reliability improvements and regressions across settings.The analysis supports evaluation beyond aggregate repeated-run success.
Implications: Reliable evaluation should use repeated executions, allow agents to resolve ambiguity through interaction, and favor strategies that remain stable across runs.These implications follow from the identified sources of variation in task execution.

2. CUA Reliability Evaluation

The paper formulates repeated task execution and evaluates reliability through repeated-run outcomes and paired tests that distinguish fully reliable transitions from incremental consistency changes. It examines stochasticity, instruction ambiguity, and planning variability across computer-use models.

Research questions: The evaluation asks how stochasticity, instruction ambiguity, and planning variability contribute to unreliable outcomes across repeated executions.The questions separate environmental and decoding variation, specification mismatch, and strategy variation.
Problem formulation: Computer-use task execution is modeled as a POMDP with computer states, observations, actions, stochastic transitions, natural-language instructions, and instruction-conditioned rewards.A task consists of an initial state and an instruction.
Reliability outcomes: Each task produces binary outcomes over n executions, allowing tasks to be classified as consistently solved, inconsistently solved, or never solved.The categories depend on the number of successful runs: c_x=n, 0<c_x<n, or c_x=0.
Reliability metrics: Pass^k estimates the probability that k executions of a task all succeed, with Pass^1 capturing marginal capability and Pass^n measuring reproducible success.Pass^n is the fraction of tasks successful on every repeated execution.
Paired statistical tests: McNemar’s test counts task transitions between consistently solved and not consistently solved settings, measuring the imbalance between improvements and regressions.The reported b−c gives the direction of change, with significance assessed at p<0.05.
Paired statistical tests: Wilcoxon signed-rank testing compares per-task success-count differences to detect incremental consistency gains even without full reproducibility.The analysis reports the average change in per-task success counts as Δc_x.
Experimental scope: The study spans frontier, smaller, and open-source models to examine whether reliability challenges persist across model capabilities.Models include GPT-5, Claude Sonnet 4.6, Kimi 2.5, Qwen-3VL-8B-Instruct, OpenCUA, and UI-TARS-1.5-7B.

3. Stochastic Decoding and Execution Noise

The study tests whether decoding randomness, fixed high-level strategies, and environment perturbations explain unreliable repeated execution. Results show that determinism affects models differently and does not consistently improve reliability.

Experimental setup: The experiments isolate stochasticity through deterministic decoding, fixed high-level plans, and controlled environment perturbations.These interventions evaluate whether execution randomness alone accounts for reliability differences.
Deterministic decoding: Deterministic decoding produces model-dependent reliability changes: Qwen regresses (b−c= −20), while OpenCUA improves (b−c= 20) and UI-TARS-1.5 improves (b−c= 19).The comparison uses stochastic decoding as the baseline.
Deterministic decoding: All models drop from Pass^1 to Pass^3 under deterministic decoding, showing that removing sampling randomness alone does not ensure reliable repeated execution.The result concerns repeated-run success rather than single-run success.
Fixed strategy: Reusing a fixed high-level plan mitigates some deterministic-decoding instability but yields little or no improvement over the stochastic baseline for Qwen and UI-TARS-1.5.OpenCUA shows only minor gains relative to the stochastic baseline.
Environment noise: Environment perturbations reduce reliability transitions for GPT-5 (b−c= −10) and Claude (b−c= −20), while Kimi shows minimal additional effects because its same-environment Pass^3 is already low.The perturbations are cosmetic, non-functional differences in environmental observations.

4. Instruction Ambiguity

The paper examines instruction ambiguity as a source of unreliable execution, both before and during task completion. Making success criteria explicit and providing targeted feedback improve reliability across repeated runs.

4. Instruction Ambiguity: Underspecified instructions can permit multiple valid interpretations while evaluators expect a more specific outcome.Different reasonable strategies may therefore satisfy evaluation criteria inconsistently across runs.
4. Instruction Ambiguity: The study tests clarification before execution and targeted feedback during execution as interventions for resolving instruction ambiguity.The interventions address ambiguity at distinct points in the task interaction.
4.1. Clarification Before Execution: Clarified instructions make success criteria explicit using evaluation-script information without adding details that would trivialize the task.The construction includes minimal human corrections described in Appendix E.
4.1. Clarification Before Execution: Instruction clarification yields more task transitions from not reliably solved to reliably solved than the reverse, with gains across GPT-5, Claude, and Kimi.The comparison is measured using task-level McNemar analysis and repeated-run Pass^k.
4.1. Clarification Before Execution: Clarification increases both Pass^1 and Pass^3 across the evaluated models, while Kimi shows a smaller reliability improvement but a significant per-task success-count increase (Δ𝑐𝑥= 0.141).The result indicates gradual progress toward reliable execution for Kimi.
4.2. Clarification During User Interaction: Clarification before execution may miss ambiguities that emerge only during execution for a particular agent and model.This motivates feedback conditioned on the agent’s trajectory and evaluation signals.
4.2. Clarification During User Interaction: Retry (Clarify) consistently outperforms Retry (Binary) across metrics and models, with both retry baselines allowed up to 5 retries.The comparison isolates targeted feedback content from the effect of additional attempts.
4.2. Clarification During User Interaction: A single execution with clarified instructions can match or exceed Retry (Binary), indicating that ambiguity resolution can be more effective than repeated attempts alone.The finding compares clarification before execution with retrying without targeted feedback.

5. Planning Variability

The paper studies variability in agent strategies across repeated executions and tests whether prior-run feedback and iterative plan refinement improve reliability. Feedback helps GPT-5 and Kimi but can destabilize Claude when extracted plans are poorly aligned.

5. Planning Variability: Even well-specified tasks can produce inconsistent behavior because repeated executions select strategies with different levels of robustness.The study uses controlled interventions incorporating prior executions to test this source of unreliability.
5.1. Effect of Plan Extraction: Plan extraction synthesizes structured feedback from successful behaviors and recurring failure patterns to guide a subsequent execution.The procedure compares an initial rollout setting, Iteration 0, with a feedback-guided Iteration 1.
5.1. Effect of Plan Extraction: When all rollouts succeed, no feedback is provided; when all fail, partial success guides exploration beyond previously unsuccessful strategies.Ground-truth task-success signals are assumed to avoid confounds from imperfect judge signals.
5.1. Effect of Plan Extraction: Prior-execution feedback improves Pass^1 and Pass^3 for GPT-5 by 2.9% and 4.2%, and for Kimi by 1.2% and 5.3%.GPT-5 also shows a statistically significant Wilcoxon gain (Δ𝑐𝑥= 0.086), while Kimi has significant McNemar gains (b−c= 19).
5.1. Effect of Plan Extraction: Claude regresses with prior feedback (b−c= −15, Δ𝑐𝑥= −0.055), suggesting extracted plans can introduce instability when misaligned with reliable strategies.The paper hypothesizes that code-oriented behavior creates environment changes that the Behavior Judge fails to detect.
5.2. Effect of Iterative Plan Refinement: Iterative refinement updates an extracted plan using new feedback across additional execution rounds, including successful rollouts from earlier iterations when all current rollouts fail.The refinement procedure produces an Iteration 2 plan after Iteration 1.
5.2. Effect of Iterative Plan Refinement: GPT-5 continues improving through Iteration 2, with significant gains under McNemar (b−c= 27) and Wilcoxon (Δ𝑐𝑥= 0.130) tests.Kimi improves in Pass^1 and Pass^3 but slightly regresses relative to Iteration 1.
5.2. Effect of Iterative Plan Refinement: Claude remains below its initial setting, but its regressions are no longer statistically significant after refinement (b−c= −4, Δ𝑐𝑥= 0.006).The paper interprets this as stabilization of the reliability loss rather than recovery above the initial setting.

6. Discussion and Conclusion

The discussion identifies stochasticity, instruction ambiguity, and planning variability as interacting sources of unreliability, and argues that reliability requires repeated-run evaluation plus adaptive interaction and guidance.

Sensitivity to Stochasticity: Deterministic decoding does not consistently improve reliability, while fixed strategies stabilize execution without resolving reliability issues.Non-functional environment perturbations also degrade reliability despite no change in task correctness.
Instruction Ambiguity: Clarifying task instructions improves reliability, while targeted feedback during execution outperforms retry baselines and static clarification.These findings support resolving ambiguity interactively rather than relying only on fixed task specifications.
Planning Variability: Plan extraction and iterative refinement can improve reliability, but gains vary across models and depend on guidance quality.Fixed strategies stabilize execution yet do not consistently outperform stochastic baselines.
Conclusion: The paper concludes that reliable agents should address multiple interacting sources of variation while remaining adaptable across runs.The discussion highlights interaction and guidance from prior executions as mechanisms for improving reliability.

C.2.3. Input for Analysis Prompt

The analysis prompt compares original and clarified instructions across repeated runs, using evaluator behavior, success rates, visual changes, and agent reasoning as inputs.

Analysis Inputs: The prompt records the original and clarified instructions alongside the evaluator function used for assessment.This provides the task and evaluation context for analyzing clarification effects.
Analysis Inputs: The prompt includes success-rate comparisons between original and clarified conditions.It records successes, totals, and percentages for both conditions.
Trajectory Evidence: For each run, the prompt supplies outcome status, score, visual changes, and agent reasoning across before and after trajectories.The format repeats these fields for three runs in each condition.
Analysis Goal: The analysis asks how instruction clarification affected agent behavior and assigns an appropriate flag.The prompt therefore combines outcome metrics with trajectory-level qualitative evidence.

D. Qualitative Analysis

The qualitative analysis presents examples comparing original instructions, task context, and LLM-generated clarifications, including cases where clarification improves evaluator alignment or introduces artifacts.

Example Structure: Each example includes the original instruction and task context, followed by an LLM-generated clarified instruction.A human-corrected version is included when applicable.
Failure Cases: The examples cover both successful clarifications and failures involving unintended artifacts such as impossible constraints or trivial solutions.The analysis therefore treats clarification as potentially beneficial but not uniformly safe.
Researcher Table Example: The researcher.xlsx example asks an agent to append Yann LeCun’s Google Scholar entry to an existing table.The example includes the original workbook, evaluator information, and an initial Google screenshot.
Researcher Table Example: The clarified instruction specifies the workbook path, source profile, required columns, metric variant, and extraction procedure.These details make the requested output more explicit than the original instruction.

Analysis

The analysis shows that clarifying instructions can align agent behavior with evaluator requirements, while evaluator implementation details can create mismatches between the task wording and measured success.

Clarification Effects: Clarification aligned the Yann LeCun task with the evaluator’s exact-matching requirements across file, source, schema, extraction, and completion details.The clarified instruction specified the path, source URL, six-column order, metric variant, and saving requirement.
Clarification Effects: Success improved from 0% to 100% after clarification in the Yann LeCun example.Before clarification, agents often omitted the row, produced incorrect fields, or became stuck during web navigation.
Evaluator Mismatch: The Macau task’s original instruction requests five addresses, while the evaluator implementation requires exactly three addresses for nonzero scoring.This creates a direct mismatch between the natural-language request and the evaluation condition.
Evaluator Mismatch: The Macau example supplies the evaluator’s fuzzy matching procedure, document input, expected answers, and word-level scoring steps.The clarified version instructs entering three Chinese addresses on separate lines in AllLocations.docx.

Analysis

Clarification aligned ambiguous instructions with evaluator requirements, turning failures caused by mismatched outputs or search targets into successful task completions in the examples.

Address task: 3/3 successes followed clarification that changed the requirement from “5 addresses” to exactly three formatted address lines.The clarified instruction matched the evaluator’s len==3 check and removed extra titles, numbering, and tokens.
Brightness task: The brightness example exposed a clarification error: the revised instruction inverted the intended brightness change because the evaluator checked the opposite relation.The evaluation JSON required the original photo to be dimmer than the new photo, contrary to the confusing evaluator function name.
FlightAware task: The FlightAware clarification supplied the ground-truth URL, trivializing the search process rather than merely resolving the task’s ambiguity.The human-corrected version instead asked for the topic with the most replies across the website.

D.2. Retry (Clarify) with User Simulator

Targeted feedback from a user simulator helped agents recover from reasonable but evaluator-mismatched interpretations, including ambiguity about scope, setting names, and the meaning of “note.”

Overview: Three qualitative examples show user-simulator feedback resolving genuine ambiguities without providing step-by-step instructions.In each case, the first interpretation was reasonable but did not match the evaluator’s expectation.
Example 1: Ambiguous scope: The “default font” task succeeded after feedback clarified that the evaluator required LibreOffice’s global setting rather than a document-level style.The agent then updated Basic Fonts (Western) through Tools → Options.
Example 2: Ambiguous setting name: The screen-dimming task succeeded after feedback identified org.gnome.desktop.session idle-delay as the required key instead of idle-dim.The evaluator expected idle-delay to equal 0.
Example 3: Ambiguous term: The slide task succeeded after feedback disambiguated “note” as speaker notes rather than a comment annotation.The agent entered “APP” in the Notes pane, while the purple background change was already correct.

E. Clarification Details

Clarified instructions were generated by aligning task wording with evaluator criteria, then checked through repeated runs and human review to remove problematic edits.

Clarification construction: Clarified instructions were produced by minimally editing original tasks using screenshots, task configurations, evaluation JSON, and evaluator implementations.GPT-5 was prompted to identify evaluator requirements absent from the original wording.
Verification: Original and clarified instructions were compared across three independent runs each with GPT-5 and Claude Sonnet 4.6.Large performance shifts were flagged for review using judge analysis, trajectories, and evaluator requirements.
Human correction: 25 of 361 tasks were manually corrected, including 20 cases where clarification introduced impossible constraints and 5 where it made tasks overly trivial.The procedure was intended to remove problematic instructions before analysis.
Environment perturbations: Table 5 lists perturbation sets in which each row modifies one desktop property while leaving application functionality and task correctness unchanged.The perturbations are purely cosmetic.