PaperlayerView exact revision on arXiv

Source-linked AI summary

VeriCUA: Verification as Agentic Exploration for Computer-Using Agents

Qiushi Sun, Da Yu, Alexander Knop, Amer Sinha, Lingpeng Kong, Ben Kao, Ravi Kumar, Pritish Kamath

arXiv:2610.40578v1cs.AIcs.CLcs.CV

TL;DR

Computer-using-agent verification must judge whether requested effects occurred, despite partial observations and varying access to application state. VeriCUA uses a shared agentic process to inspect context, build task rubrics, and gather evidence; it improves agreement with reference labels across evaluated settings, with its largest offline gains on hard cases and stronger rejection of false completion claims online.

  • Problem

    VLM judges and generated state-checking programs rely on partial observations or accessible state, while a shared verification procedure must work across access settings and platforms.

    Source: §1 · p. 1

  • Method

    VeriCUA inspects the available context, constructs a rubric, selects GUI or code checks for unresolved requirements, and combines findings under explicit evidence rules.

    Source: §1 · p. 2

  • Results

    VeriCUA improves agreement with reference labels over compared baselines across evaluated settings, with the largest offline gains on hard trajectories and fewer false positives online.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The results support agentic verification for CUA evaluation and motivate its use in data curation and reinforcement learning.

    Source: §6 · p. 11

    Offline verification is limited by retained recording observations; unresolved tasks use an initial-judgment fallback, which occurs on 23% of WebArena trajectories.

    Source: §4.3 · pp. 8–9

Abstract

from arXiv · show

Building and scaling computer-using agents (CUAs) across desktop, mobile, and web environments requires reliable task-completion verification for data curation, evaluation, and reinforcement learning. Typical automated approaches prompt vision-language models (VLMs) to judge trajectories or generate programs to inspect application state. However, both rely on partial views of complex digital environments. VLM judges are restricted to supplied observations, while generated programs depend on accessible state. These views can omit decisive evidence or leave completion judgments unsupported. Tailoring separate verifiers to different platforms and access settings adds further engineering cost. We present VeriCUA, a platform-agnostic framework for verification through agentic exploration. It treats verification as dynamic evidence gathering and assessment within a shared procedure for online and offline settings across platforms and observation modalities. After inspecting the available context to construct a task rubric, the verifier selects GUI or code checks for unresolved requirements and combines their findings under explicit evidence rules. Experiments show improved agreement with reference labels over the compared baselines across these settings, with the largest offline gains on hard cases and stronger rejection of false completion claims online. Further analyses connect these gains to complementary evidence and test-time exploration.

1. Introduction

Computer-using agents need reliable verification because trajectories and completion reports can fail to establish whether requested effects occurred. VeriCUA addresses this with a shared, platform-agnostic process that gathers evidence for task requirements and improves agreement with reference labels across online and offline settings.

1. Introduction

  • Trajectories and completion reports can mistake attempted actions for actual effects, while reliable judgments are needed for evaluation, data curation, and reinforcement learning.Edits may remain unsaved, operations may fail without a visible warning, or an agent may complete only part of an instruction.Source: §1 · p. 1
  • VeriCUA first inspects the available context to construct a rubric, then selects GUI or code checks for unresolved requirements in a shared online and offline workflow.Findings guide further inspection of relevant frames, artifacts, or other available sources.Source: §1 · p. 2, Figure 1 · p. 2
  • VeriCUA links findings to supporting observations and source properties, then applies explicit evidence rules to classify requirements as supported, refuted, or unresolved.Offline checks use retained records, while online checks can inspect accessible machine state such as files and databases.Source: §1 · p. 2
  • VeriCUA achieves higher agreement with reference labels than compared baselines in both settings and rejects more false completion claims than the online VLM judge.Adding code-based state reads enables more decisions without increasing false positives; adding visual evidence improves agreement and decision coverage at matched verification passes.Source: §1 · p. 2
  • The evaluation examines how evidence access and complementary exploration affect decision coverage and reliability across platforms.The paper frames agentic verification as a shared approach for online and offline tasks.Source: §1 · p. 3

2. Related Work

Prior work evaluates computer-using agents with task-specific state checkers, trajectory judgments, structured critics, and agentic evidence gathering. VeriCUA builds on these lines by inspecting context before rubric construction and selecting checks within a shared workflow.

2. Related Work

  • Computer-using-agent evaluation spans task-specific programmatic evaluators, human-labeled trajectory datasets, evaluator audits, and benchmarks of evidence acquisition.Completion judgments also support trajectory curation, reinforcement-learning feedback, reward modeling, and safety-oriented workflow evaluation.Source: §2 · p. 3
  • Trajectory verification includes VLM judges, structured critics, adaptive rubrics, generated state-inspection programs, and agentic approaches that gather additional evidence.Evidence-first verification checks proposed claims against their state or trace sources before accepting rubric verdicts.Source: §2 · p. 3
  • VeriCUA inspects the available environment or recording before constructing its rubric, then chooses GUI or code checks and combines findings under source-aware rules.This places rubric construction and evidence gathering in one shared offline and online workflow.Source: §2 · p. 3

3. Method

VeriCUA treats verification as active evidence gathering after an agent’s attempt, using the instruction and available context to define checkable requirements. It selects inspections based on unresolved findings and applies source-aware rules, with online and offline procedures adapted to their available evidence.

3.1. Verification as Active Evidence Gathering

  • After an execution, VeriCUA uses the instruction and available sources to assess completion, allowing success, failure, or abstention when evidence is insufficient.Its workflow inspects context, synthesizes requirements, explores with checks, and arbitrates findings under setting-specific evidence rules.Source: §3.1 · p. 4

3.2. Environment-Aware Rubric Synthesis

  • Context inspection grounds a rubric in the application or recording, separating requested task requirements from locations that merely provide evidence.Requirements are checkable and marked hard or soft; soft checks add findings without relaxing any hard requirement.Source: §3.2 · pp. 4–5

3.3. Adaptive Hybrid Action Space

  • The action space spans retained trajectories, the current interface, and persisted state, with GUI or code checks selected to investigate unresolved requirements.Reads retain source metadata, and an online inspection can combine permitted reads with a predicate that evaluates a requirement.Source: §3.3 · pp. 5–6

3.4. Evidence Discipline

  • Findings must cite executed checks, and source attributes constrain what observations can establish; requirements are met only with attested support and unmet only with corroborating refutations.If hard requirements remain unknown and none is unmet, the task is unresolved and follows the setting’s abstention or fallback policy.Source: §3.4 · p. 6

3.5. Online and Offline Instantiations

  • Online verification can combine GUI inspection with code reads of stored artifacts, while offline verification audits fixed, time-stamped recordings through forward and backward inspection.Online unresolved tasks may be abstained on; the described offline configuration uses an initial single-pass VLM judgment as a fallback when exploration cannot resolve a task.Source: §3.5 · pp. 6–7

4. Experiments

Across online environments and offline benchmarks, VeriCUA reports stronger verification performance than compared baselines, with distinct trade-offs around abstention and evidence availability. The experiments evaluate multiple platforms, verifiers, and metrics, including accuracy, agreement, and false-positive behavior.

4.1. Experimental Setup

  • The online evaluation uses AndroidWorld, OSWorld, and ScienceBoard after benchmark-agent episodes, while offline evaluation covers WebArena and OSReward trajectories.Verifiers run on gemini-3-flash-preview, and the backbone study also reruns them on gemini-3.1-pro-preview; reported metrics include agreement, precision, recall, F1, and false-positive rate.Source: §4.1 · p. 7

4.2. Online Verification; 4.3. Offline Verification

  • Across online and offline evaluations, VeriCUA leads the compared baselines on reported accuracy measures, with the online study also finding the lowest false-positive rate.The online comparison covers three environments; the offline comparison reports the highest accuracy and balanced accuracy across benchmarks.Source: §4.2 · p. 7, Table 1 · p. 8, §4.3 · p. 8

4.2. Online Verification

  • On ScienceBoard, VeriCUA’s false-positive rate is roughly a quarter of the auto-generated verifier’s, while the generated verifier has higher recall on OSWorld and ScienceBoard.VeriCUA has higher precision and accuracy than the generated verifier in those comparisons.Source: §4.2 · p. 8
  • VeriCUA abstains when findings cannot settle a task or decided verification passes disagree, yielding higher accuracy on decided episodes across all three online environments.Counting abstentions as failures can lower recall without adding false positives, a policy the paper describes as favoring conservative selection for data curation or reward assignment.Source: §4.2 · p. 8

4.3. Offline Verification

  • On OSReward, VeriCUA’s accuracy advantage over the strongest compared judge is larger on the hard subset than on the full set.Offline verification is limited by retained observations; when they do not resolve a task, the reported configuration uses an initial-judgment fallback, occurring on 23% of WebArena trajectories.Source: §4.3 · pp. 8–9

5. Analysis

The analyses test how evidence sources, observation inputs, verification passes, and backbone strength affect VeriCUA’s decisions. They also examine false completion reports, confident failure judgments, and suspected reference-label errors.

5.1. GUI Versus Hybrid Action Space

  • On AndroidWorld, code-based state checks increase decision rate and recall without increasing false positives.Every additional correct decision supported by the code channel is backed by a file or application-store read.Source: Figure 3 · p. 9, §5.1 · p. 9

5.2. The Effect Of Observation

  • On WebArena, adding a11ytrees to screenshots raises VeriCUA’s decision rate without changing accuracy; trees alone trade accuracy for higher decision rate and lower cost.More final frames improve OSReward accuracy on hard cases, but it remains below VeriCUA, while full-set accuracy changes only modestly.Source: §5.2 · p. 9, Figure 4 · p. 10

5.3. Test-Time Exploration

  • VeriCUA rejects about 80% of false completion reports and outperforms both baselines in each online environment.The difference is largest on ScienceBoard, where the generated verifier rarely rejects incorrect completion claims.Source: §5.3 · p. 10
  • 87% agreement and decision rate are reached after adding the visual pass to default AndroidWorld verification.Visual evidence also improves both metrics at matched pass counts; combined verdicts produce fewer incorrect fail decisions than majority voting, and VeriCUA retains higher matched-budget accuracy on OSReward and OSReward-Hard.Source: §5.3 · pp. 9–10, Figure 5 · p. 10

5.4. Gains From A Stronger Backbone

  • Replacing the flash backbone with the pro backbone improves VeriCUA’s accuracy and balanced accuracy on OSReward-Hard.All frameworks improve in the online AndroidWorld comparison, with VeriCUA retaining the highest accuracy on both backbones.Source: Table 4 · p. 11, §5.4 · p. 11

5.5. Confident Failures and Gold-Label Audit

  • Exploration raises failure precision on each offline benchmark while retaining fewer initial failure judgments.A blinded adjudicator supports VeriCUA on 63% of flagged trajectories, with the clearest support on OSWorld; the authors suggest this can help prioritize suspected label errors for inspection.Source: §5.5 · p. 11

6. Conclusion

VeriCUA frames computer-using-agent verification as platform-agnostic agentic exploration, using a shared workflow across offline recordings and online environments. Across evaluated platforms it improves agreement with reference labels over compared baselines, with the largest offline gains on hard trajectories.

6. Conclusion

  • VeriCUA uses context inspection, rubric construction, GUI or code checks, and source-aware arbitration in a shared online and offline workflow.The framework is presented as platform-agnostic.Source: §6 · p. 11
  • Across evaluated platforms, VeriCUA improves agreement with reference labels, with fewer false positives online and the largest offline gains on hard trajectories.Analyses connect decision-coverage gains to complementary evidence sources and verification passes, and report further accuracy improvement from stronger backbones.Source: §6 · p. 11
  • The results support agentic verification for CUA evaluation and motivate its use in data curation and reinforcement learning.Source: §6 · p. 11

A. Implementation Details

The implementation adapts a shared verification procedure to different evidence sources, settings, and modalities. It uses evidence aggregation rules and read-only controls while processing both live machine state and recorded trajectories.

A.1. Verification Settings

A.2. Action Space and Evidence Types

  • The action space assigns evidence types to verbs by source and modality; the online getter inventory has 17 verbs, with image reading reserved for vision fallback and clipboard reads run last.A miss from find counts only as weak evidence.Source: §A.2 · p. 16, Table A.2 · p. 16, Table A.3 · p. 16

A.3. Evidence Aggregation

  • Offline aggregation uses up to three audit paths, requiring attested support across completed paths for met and corroborating refutations for unmet; online unmet decisions require two corroborating refutations.Online unresolved tasks produce abstentions, counted as fail in binary evaluation; across passes, disagreement between decided verdicts also yields abstention.Source: §A.3 · pp. 16–17

A.4. Read-Only Enforcement

  • Recorded-trajectory reads are pure functions of disk files, while machine reads use an allowlist, reject shell-control arguments, and compare persistent-target digests before and after execution.Source: §A.4 · p. 17

A.5. Online Environment Adapters

  • Online adapters verify AndroidWorld snapshots with adb-readable stores and pre-state anchors, and use read-only machine getters for OSWorld and ScienceBoard.AndroidWorld snapshots preserve an identical end state for replay; ScienceBoard uses the benchmark-shipped virtual-machine image.Source: §A.5 · p. 17

A.6. Recorded Observation Processing

  • Recorded inputs include actions, reasoning, screen frames, and available accessibility trees, with dataset-specific frame semantics determining whether evidence is pre-action or post-effect.OSReward uses pure GUI observations with pre-action frame semantics; WebArena includes accessibility trees, which are condensed before reading.Source: §A.6 · pp. 17–18

B.1. Evaluation Sets

The evaluation sets are summarized in Table B.1, and the multi-pass suite focuses on cases where verifier judgments are uncertain or conflict with gold labels.

B.1. Evaluation Sets

  • Table B.1 summarizes the evaluation sets.Source: §B.1 · p. 18, Table B.1 · p. 18
  • The multi-pass suite retains snapshots with a completion claim contradicted by gold, disagreement among configurations, or abstention by any pass.Source: §B.1 · p. 18

B.2. Baseline Implementations

This section specifies baseline implementations, verifier settings, evaluation metrics, and supplementary comparisons across evidence inputs, pass combinations, backbones, and computational cost.

B.2. Baseline Implementations

  • Offline baselines use single-call or staged trajectory judgments, while online baselines judge the final screenshot or generate a task-specific evaluator script.The offline judges differ in score, key-point, and ordered-milestone formats; the auto-generated online verifier runs once with full device access.Source: §B.2 · p. 18

B.3. Verifier Configuration

  • The verifier runs three offline paths by default, uses candidate rubrics and checker synthesis online, and abstentions count as fail in binary evaluation metrics.The binary metrics include agreement, precision, recall, F1, and false positive rate; decision rate and accuracy on decided episodes are also reported for abstaining verifiers.Source: §B.3 · p. 19, §B.4 · p. 19

C.1. Action Space and Evidence Rules

C.2. Observation Inputs and Rollout Agents

  • WebArena analyses report VeriCUA by observation input, results by rollout agent and overall, and judge comparisons using accessibility trees.The tree-fed judge comparison uses a11ytrees alone, while VeriCUA retains GUI and a11ytree input.Source: §C.2 · p. 20, Table C.2 · p. 20, Table C.3 · p. 20

C.3. Test-Time Scaling and Budget

  • Adding visual inspection improves agreement and decision rate over repeated default runs, including at matched pass counts, while the combination rule yields fewer incorrect fail verdicts than majority voting.Source: §C.3 · p. 21, Table C.5 · p. 21

C.4. Backbone Comparisons

  • The OS-Themis budget ablation reports mean balanced accuracy, and the online backbone comparison reports accuracy and balanced accuracy across flash and pro tiers.The OSReward table compares single-call judge backbones alongside VeriCUA on gemini-3-flash-preview.Source: §C.3 · p. 21, Table C.6 · p. 21, §C.4 · p. 22

C.5. Computational Cost

  • Token usage is reported per trajectory or episode, and an AndroidWorld verifier pass takes about 100 seconds per episode.Checker synthesis takes about half the time and predicate evaluation about a third; getter execution and context inspection together take under ten seconds.Source: §C.5 · p. 23, Table C.9 · p. 23

C.6. Confident Failures and Gold-Label Audit

The audit adjudicates verifier disagreements with benchmark labels, while case studies illustrate checks that rely on machine state or earlier trajectory evidence.

C.6. Confident Failures and Gold-Label Audit

  • The audit flags decided VeriCUA verdicts that contradict benchmark gold labels among 814 recorded trajectories, then uses a blinded adjudicator to re-judge flagged cases from trajectory evidence.The adjudicator sees the instruction, action and thought log, and up to ten evenly spaced frames, but not the verifier verdict, gold label, or reasoning.Source: §C.6 · pp. 23–24
  • Of 81 flagged trajectories, adjudication confirmed 51 against gold, refuted 20, and left 10 unsure; on OSWorld, 37 of 57 flags were confirmed.A second adjudicator confirmed 34 of the 51 flags it decided, and about 18 of 28 online OSWorld false positives were episodes the official evaluator marked fail in error.Source: §C.6 · p. 24

D.1. Online Environments

  • Online cases show a media-store check rejecting three clips submitted for one requested video and a playback-database read confirming a queue in the requested order.Both baselines certified the first run and failed the second.Source: §D.1 · p. 24, Figure D.1 · p. 24, Figure D.2 · p. 25

D.2. Recorded Trajectories

  • Offline cases use earlier or final recorded evidence to catch misplaced text, a wrong search page with over-limit prices, and a nearer park missed by the agent.Two of three judges passed the mobile example despite the closer park appearing 18 steps before the end.Source: §D.2 · p. 26, Figure D.3 · p. 26, Figure D.4 · p. 26

E. Limitations and Broader Impacts

VeriCUA’s verification has substantial inference and tool-call costs, and offline judgments are bounded by evidence preserved in the recording. Its prompts frame rubric and checker generation around task end states, with requirements intended to avoid assumptions about unstated details.

E. Limitations and Broader Impacts

  • VeriCUA issues tens of tool and model calls per trajectory—an order of magnitude more than a single-call judge—and suits filtering and reward assignment better than interactive use.Recorded trajectories prompt abstention when evidence for a requirement is absent; online access lifts that coverage limit only in instrumentable environments. The verifier does not write to the environment it verifies.Source: E. Limitations and Broader Impacts · p. 27

F. Prompts

  • Prompt inputs such as the task instruction and device context are appended at run time.Source: F. Prompts · p. 28

F.1. Recorded-Trajectory Verification

  • Recorded-trajectory rubric synthesis targets checkable final-state requirements, honoring exact details, separating hard from soft requirements, and not requiring a particular route.Source: §F.1 · p. 28

F.2. Online Verification

  • Online checker synthesis uses read-only getter plans and predicates, favors independent surfaces and broad reads, and leaves unstated encodings or storage locations to the checkers.Source: §F.2 · p. 29

Related papers

Loading 2610.40578v1…