Video-Based Reward Modeling for Computer-Use Agents

Linxin Song; Jieyu Zhang; Huanxin Sheng; Taiwei Shi; Gupta Rahul; Yang Liu; Ranjay Krishna; Jian Kang; Jieyu Zhao

Source-linked AI summary

Video-Based Reward Modeling for Computer-Use Agents

Linxin Song, Jieyu Zhang, Huanxin Sheng, Taiwei Shi, Gupta Rahul, Yang Liu, Ranjay Krishna, Jian Kang, Jieyu Zhao

arXiv:2603.10178v1Mar 10, 2026cs.CVcs.CL

TL;DR

CUA evaluation remains difficult to scale with methods that generalize across tasks and agent implementations, especially because execution videos combine redundant layouts with subtle success cues. The paper builds ExeVR-53k, synthesizes hard negatives, and prunes spatiotemporal tokens for video-based reward modeling. ExeVRM 8B achieves 84.7% accuracy and 87.7% recall, outperforming proprietary and open-weight baselines across multiple environments.

Problem
CUA evaluation remains difficult to scale because existing checks are task- and environment-specific, while video-based assessment must preserve subtle cues amid redundant layouts and limited negative supervision.
Method
The paper constructs ExeVR-53k, generates hard mismatched instruction–trajectory pairs, and fine-tunes ExeVRM with spatiotemporal token pruning on execution-video sequences.
Results
84.7% accuracy and 87.7% recall are achieved by ExeVRM 8B, outperforming proprietary and open-weight baselines across Ubuntu, Mac/Win, and Android settings.
Takeaways & Limitations
Video-execution reward modeling provides a scalable, model-agnostic evaluator for computer-use agents within the evaluated settings.

Abstract

from arXiv · show

Computer-using agents (CUAs) are becoming increasingly capable; however, it remains difficult to scale evaluation of whether a trajectory truly fulfills a user instruction. In this work, we study reward modeling from execution video: a sequence of keyframes from an agent trajectory that is independent of the agent's internal reasoning or actions. Although video-execution modeling is method-agnostic, it presents key challenges, including highly redundant layouts and subtle, localized cues that determine success. We introduce Execution Video Reward 53k (ExeVR-53k), a dataset of 53k high-quality video--task--reward triplets. We further propose adversarial instruction translation to synthesize negative samples with step-level annotations. To enable learning from long, high-resolution execution videos, we design spatiotemporal token pruning, which removes homogeneous regions and persistent tokens while preserving decisive UI changes. Building on these components, we fine-tune an Execution Video Reward Model (ExeVRM) that takes only a user instruction and a video-execution sequence to predict task success. Our ExeVRM 8B achieves 84.7% accuracy and 87.7% recall on video-execution assessment, outperforming strong proprietary models such as GPT-5.2 and Gemini-3 Pro across Ubuntu, macOS, Windows, and Android, while providing more precise temporal attribution. These results show that video-execution reward modeling can serve as a scalable, model-agnostic evaluator for CUAs.

1 Introduction

Evaluating computer-use agents remains difficult because rule-based checks do not scale across tasks, while successful evaluation requires method-agnostic evidence from execution videos. The paper addresses redundant visual content and scarce negative supervision with a dataset, adversarial instruction translation, and spatiotemporal pruning, achieving strong task-success assessment.

Motivation: Rule-based CUA evaluation limits scalability and transfer, motivating a learned reward model based on observable execution videos rather than internal reasoning or action formats.Execution video provides a representation that is comparable across different agent designs.
Challenges: Execution videos contain redundant static layouts alongside subtle local changes, so reward models must suppress homogeneous content without losing decisive UI cues.Examples include cursor focus shifts, small text edits, and transient dialogs.
Approach: ExeVR-53k standardizes trajectories into step-level keyframe sequences and uses adversarial instruction translation to create hard mismatched instruction–trajectory pairs with step-level mismatch signals.The resulting video summary preserves temporal progression while remaining computationally manageable.
Approach: Spatiotemporal token pruning removes visually homogeneous regions and persistent tokens while retaining localized UI elements and meaningful state transitions.Spatial pruning targets static backgrounds, while temporal pruning suppresses tokens that remain nearly unchanged across consecutive frames.
Results: 84.7% accuracy and 87.7% recall are achieved by ExeVRM 8B on video-execution assessment, outperforming proprietary baselines and improving temporal localization of decisive error spans.The model takes only a user instruction and its corresponding execution video sequence as input.

2 Related Work

Prior CUA reward evaluation relies heavily on handcrafted checks, sparse observations, and supervision dominated by successful trajectories. The paper positions unified video representations, scalable hard negatives, and GUI-specific pruning as responses to these limitations.

Reward evaluation: Existing CUA evaluation protocols commonly depend on handcrafted rules and environment-specific parsers, limiting generality across interfaces and tasks.Final-state checks are scalable but coarse, while full-screenshot evaluation can be memory-intensive.
Negative supervision: Negative supervision is scarce because public computer-using datasets are dominated by successful trajectories.Prior negative construction includes passive failures, expert or model annotation, and rule-based corruption.
Negative supervision: The paper generates scalable hard negatives by pairing successful trajectories with semantically mismatched instructions and standardizes heterogeneous data sources into a unified representation.This contrasts with passive failure collection and annotation-based approaches.
Token pruning: GUI execution videos require pruning methods that preserve subtle, transient UI evidence because existing visual token-pruning methods were mainly designed for natural or egocentric video.Large static regions are common, but removing semantically important cues can harm reward assessment.

3 Execution Video Reward Modeling

ExeVR-53k combines diverse computer-use trajectories into a unified video representation and adds adversarial negatives for reward modeling. Spatiotemporal token pruning reduces redundant visual content by identifying large spatially homogeneous regions and repeated tokens across frames.

ExeVR-53k: ExeVR-53k combines AgentNet, ScaleCUA, and OSWorld trajectories into a training corpus spanning human demonstrations and solutions from 30 diverse computer-using agents.The corpus targets the data bottleneck caused by limited scale and behavioral diversity in existing resources.
Video representation: Each trajectory is converted into a unified step-level video by extracting one representative post-action key frame per step and concatenating frames temporally at 1 FPS.This creates a compact progression of interface states while keeping input length manageable.
Adversarial instruction translation: Adversarial translation generates plausible but unpaired task instructions from valid trajectory segments to provide informative synthetic counterexamples for reward modeling.The procedure is motivated by the predominantly positive supervision in AgentNet and ScaleCUA.
Spatiotemporal token pruning: Spatiotemporal token pruning combines spatial pruning of large homogeneous regions with temporal pruning of repeated tokens to retain transient UI evidence while reducing video context length.The training pipeline applies both masks, repacks the retained tokens, and trains the language-model parameters using reward labels.
Spatial Token Pruning: Spatial pruning builds per-frame feature graphs by connecting neighboring patches whose feature distance is below τs, then marks sufficiently large connected components for removal.Patch features are reshaped into H′ × W′ grids, and Union-Find is used to obtain connected components before constructing the spatial mask.
Temporal Token Pruning: Temporal pruning compares each token with its most recent distinct reference using cosine similarity, removes sufficiently similar tokens, and updates the reference when a meaningful change occurs.The first-frame token is always retained, allowing the mask to capture cursor moves, menu expansion, scrolling, and window switching while ignoring static backgrounds.

4 Experiments

ExeVR-Bench evaluates reward models on binary task judgment and temporal failure attribution across diverse computer-use trajectories. ExeVRM 8B achieves the strongest reported overall performance, with gains over proprietary and open-weight baselines and benefits from higher-resolution inputs.

Benchmark and Metrics: ExeVR-Bench evaluates binary correctness and attribution of the first error across Ubuntu, Mac/Win, and Android trajectories.The benchmark contains 789 instances after removing unsolvable tasks, with 49.94% positive and 50.06% negative examples.
Benchmark and Metrics: Temporal IoU measures overlap between predicted and ground-truth failure intervals, rewarding accurate localization and penalizing overly broad predictions.The metric compares predicted interval ˆI with ground-truth interval I.
Main Results: 84.7 accuracy, 82.9 precision, and 87.7 recall make ExeVRM 8B the top overall model, surpassing Seed-2.0 Pro and GPT-5.2.Seed-2.0 Pro reports 80.3/83.9/74.7 and GPT-5.2 reports 75.0/82.7/66.5 for accuracy/precision/recall.
Main Results: +17.1 accuracy points over Qwen3-VL 8B and +28.2 over InternVL-3.5 8B show ExeVRM 8B’s advantage over open-weight baselines.Recall also reaches 87.7 versus 49.9 and 55.9 for those models.
Main Results: +4.6 overall accuracy and +5.2 recall result from scaling ExeVRM from 4B to 8B, with the largest gains on Android and Ubuntu Agent settings.Overall accuracy rises from 80.1 to 84.7, while recall rises from 82.5 to 87.7.
Ablations: Increasing input resolution from 360p to 720p improves accuracy and recall for both Qwen3-VL 4B and 8B with spatiotemporal token pruning.For Qwen3-VL 8B, accuracy rises from 81.5 to 84.7 and recall from 80.5 to 87.7.

5 Discussion & Ablation Studies

Video-based assessment benefits from dense, high-resolution execution context, while spatiotemporal pruning makes that context tractable. Temporal pruning contributes more than spatial pruning alone, and joint pruning offers favorable efficiency with high recall.

Dense video context outperforms sparse snapshots: Omitting causal transitions makes completion judgment difficult, while processing every key frame at full resolution can cause OOM on an 80GB A100.At 360p, performance can fall below final-state assessment, motivating both video-based assessment and token pruning.
Higher resolution brings more benefit for reward modeling: For Qwen3-VL 8B, increasing resolution from 360p to 720p raises accuracy from 81.5 to 84.7 and recall from 80.5 to 87.7 (+7.2).Precision also increases from 82.5 to 82.9.
Asymmetric effects of spatial and temporal pruning: TTP alone provides a stronger overall balance than STP alone, with accuracy/recall of 80.3/79.3 versus 77.9/72.6.The analysis attributes this pattern to reward prediction relying primarily on inter-frame state transitions.
Asymmetric effects of spatial and temporal pruning: Adding STP to TTP changes accuracy only marginally from 80.3 to 80.1 while increasing recall to 82.5.The full configuration mainly introduces a precision–recall trade-off rather than a substantial performance drop.
Spatiotemporal pruning improves training efficiency: At 50 frames, joint STP+TTP uses about 49.5 GiB and 5.8 s/step, compared with 56 GiB and 6.2 s/step for STP-only.Joint pruning reduces redundancy across spatial and temporal dimensions, enabling 720p long-horizon training under the reported hardware setup.

6 Conclusion

The paper presents video-execution reward modeling for computer-using agents through ExeVR-53k, adversarial negatives, and spatiotemporal token pruning. ExeVRM 8B achieves strong benchmark performance across environments, while joint pruning improves efficiency as videos lengthen.

Conclusion: ExeVRM 8B achieves 84.7% accuracy and 87.7% recall, outperforming proprietary and open-weight baselines across Ubuntu, Mac/Win, and Android settings.The system uses ExeVR-53k, adversarial instruction translation, and STP+TTP to preserve decisive UI evidence while reducing redundancy.

A Implications for Future Work

The discussion identifies limits in long-horizon outcome judgment and in the computational cost of extended multimodal reasoning. It proposes process-aware supervision as a future direction, but notes that suitable fine-grained annotations are unavailable.

Limitations: Very long-horizon Ubuntu trajectories with substantial exploration are less effective because binary outcome supervision can mistake locally unsuccessful attempts for failure-like behavior.This exposes a gap between outcome-level reward modeling and process-level credit assignment.
Limitations: Long-horizon video reasoning remains expensive for think-in-the-image multimodal models because frame count, resolution, and reasoning depth can exhaust token budgets.The study therefore focuses on supervised fine-tuning rather than large-scale reinforcement learning with long-form reasoning rollouts.
Future work: Process-aware supervision could decompose long trajectories into intent-consistent sub-trajectories, but constructing such a benchmark requires fine-grained human annotations that are not yet available.The proposed direction would evaluate intermediate progress and recovery behaviors rather than only final outcomes.

B Method list for OSWorld trajectory rollout

The OSWorld rollout pool spans proprietary frontier and open-weight CUA systems, creating heterogeneous trajectories for training video-only reward modeling. This diversity supplies successful, near-miss, and behaviorally varied examples.

Method list for OSWorld trajectory rollout: The dataset intentionally rolls out diverse proprietary and open-weight CUA systems on OSWorld to avoid overfitting to agent-specific interaction habits.ExeVRM is trained from execution video without relying on internal thoughts or tool traces.
Method list for OSWorld trajectory rollout: The evaluated pool includes systems such as UI-TARS, Agent S2, Claude, Gemini, OpenCUA, Qwen2.5-VL, and MobileAgent V3.The list spans multiple model families, backends, and parameter scales.
Method list for OSWorld trajectory rollout: Across this pool, agents differ in action efficiency, recovery behavior, and error modes, producing successful and near-miss trajectories under shared tasks.This heterogeneity provides harder negatives and richer temporal divergence patterns for learning fine-grained success/failure signals.

C Visualization of STP and TTP

Figure 5 illustrates how spatiotemporal token pruning reduces redundancy in execution videos while retaining localized UI evidence relevant to task success.

Tokenization and masking: Each frame produces 880 tokens after spatial patch merging, followed by temporal merging of adjacent frames.STP masks are computed on the original spatial patch grid before temporally adjacent masks are merged.
Temporal merging: A token merged from two frames remains masked if either constituent frame pruned the corresponding region.Because tokens retain neighboring-frame information, content appearing early may remain visible in later visualizations.
Spatiotemporal pruning: STP removes large homogeneous regions but is relatively conservative in UI-dense scenes containing many small elements.This limitation motivates complementing spatial pruning with temporal pruning.
Spatiotemporal pruning: TTP removes temporally invariant tokens while preserving regions associated with subtle local evidence such as text edits, focus changes, and transient dialogs.It also removes repeated sidebars, window chrome, and static wallpaper or background regions.

D Prompts

The section presents prompts for several GUI-agent evaluation strategies, contrasting screenshot-and-history inputs with sequential execution-frame analysis.

AER: AER evaluates success using the user’s intent, action history, final application or webpage state, and the agent’s response.Its adapted evaluation uses only the final screenshot and a compact action-history list.
Simplified Judge: Simplified Judge receives the task goal, interface state, action sequence, thought process, and both the first and last screenshots.Its detailed action history covers each action and longer reasoning by the GUI agent.
Sequential keyframe evaluation: The sequential-keyframe prompt represents the full execution trajectory and asks the evaluator to assess task completion from the instruction and screenshots.It requires describing the successful target state, analyzing each screenshot, and identifying UI changes or mistakes.
Prompt materials: SE-WSM, ZeroGUI, AER, and Simplified Judge are represented through separate prompt figures in the evaluation prompt materials.The supplied figures identify prompts for SE-WSM, ZeroGUI, AER, and Simplified Judge.

Prompt of ZeroGUI [73]

The ZeroGUI prompt example specifies an email-backup task, requests structured screenshot analysis and reasoning, and concludes that the demonstrated execution was unsuccessful because saving was not observable.

Evaluation procedure: The prompt requires evaluators to describe the successful target state, analyze each screenshot, and identify UI changes or mistakes.Its required response format contains task requirement, screenshot analysis, reasoning, final answer, and score fields.
Task requirement: The example task is to back up Thunderbird emails from the “Bills” folder as subject-named .eml files in the Google Drive folder “emails.”Successful completion requires the emails to be visible there or at least a saving action to be performed.
Screenshot analysis: The initial screenshots show Thunderbird, the Bills folder, two emails, and their selection without an observable export or save attempt.The listed messages are “Amazon Web Services Invoice Available” and “Your receipt from X (formerly Twitter).”
Screenshot analysis: Later screenshots show the Save As option and file-picker navigation, but no confirmed Google Drive destination or completed save.The file picker remains open while a software-update notification appears, leaving the save location unclear.
Final judgment: The final assessment marks the task unsuccessful because the screenshots provide no observable saving action or completion evidence.The reasoning specifically notes the absence of confirmation that the files were saved with subject names in the required folder.