PaperlayerView exact revision on arXiv

Source-linked AI summary

ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reduction

Amirhossein Abaskohi, Yuhang He, Peter West, Giuseppe Carenini, Pranit Chawla, Vibhav Vineet

arXiv:2605.11212v3cs.CL

TL;DR

Computer-use agents struggle to incorporate long screenshot histories because repeated visual content rapidly consumes context. ReVision trains multimodal models on trajectories with redundant patches removed by a learned temporal selector, achieving lower token usage and higher success rates across three benchmarks. The results indicate that removing redundancy allows performance to continue improving with longer histories, although effectiveness is bounded by predictor errors, lossy token removal, and GUI-focused evaluation.

  • Problem

    Repeated screenshot content rapidly increases visual-token costs, limiting the history that computer-use agents can incorporate under fixed context and compute budgets.

  • Method

    ReVision trains multimodal language models on trajectories where a learned selector compares consecutive screenshot patches, removes redundant regions, and preserves spatial structure.

  • Results

    Across OSWorld, WebTailBench, and AgentNetBench, ReVision reduces token usage by approximately 46% on average and improves success rate by +3% over the no-drop baseline with 5 history screenshots.

  • Takeaways & Limitations

    ReVision achieves higher success rates with fewer tokens, and performance continues improving with longer histories when redundancy is removed.

  • Takeaways & Limitations

    Effectiveness depends on the redundancy predictor, token removal can affect pixel-level precision, and the experiments focus on GUI agents and vision-language models.

Abstract

from arXiv · show

Computer-use agents (CUAs) rely on visual observations of graphical user interfaces, where each screenshot is encoded into a large number of visual tokens. As interaction trajectories grow, the token cost increases rapidly, limiting the amount of history that can be incorporated under fixed context and compute budgets. This has resulted in no or very limited improvement in the performance when using history unlike other domains. We address this inefficiency by introducing ReVision, which is used to train multimodal language models on trajectories where redundant visual patches are removed using a learned patch selector that compares patch representations across consecutive screenshots while preserving spatial structure required by the model. Across three benchmarks, OSWorld, WebTailBench, and AgentNetBench, when processing trajectories with 5 history screenshots using Qwen2.5-VL-7B, ReVision reduces token usage by 46% on average while improving success rate by 3% over the no drop baseline. This establishes a clear efficiency gain, enabling agents to process longer trajectories with fewer tokens. With this improved efficiency, we revisit the role of history in CUAs and find that performance continues to improve as more past observations are incorporated when redundancy is removed.

1 Introduction

Computer-use agents need longer visual histories for multi-step GUI tasks, but repeated screenshot content rapidly consumes context. ReVision trains models on redundancy-filtered trajectories, reducing visual tokens while improving success rates across benchmarks.

  • Motivation: CUAs often use limited screenshot history because extended visual trajectories strain constrained context budgets.Past states and actions can matter for complex workflows, but each screenshot adds hundreds or thousands of visual tokens.
  • ReVision: ReVision removes redundant patches across consecutive screenshots with a learned selector while preserving spatial structure.Unlike inference-only pruning, it trains the multimodal model on filtered trajectories without architecture changes.
  • Results: With longer histories, ReVision consistently outperforms most same-size baselines by at least 2% on average when using 5 or more images.The framework is intended to enable longer visual histories under similar compute budgets.
  • Contributions: The paper identifies temporal redundancy, introduces redundancy-aware training, and evaluates reduced-token histories across long-horizon computer-use benchmarks.Its stated contributions include delaying visual-history saturation while reducing token usage and improving success rates.

2 Related Work

Related work spans vision-first computer-use agents and visual-token compression, while ReVision targets patch-level filtering across consecutive screenshots to preserve temporal evidence.

  • Computer-use agents: Vision-first computer-use agents reason directly over pixels rather than relying on structured DOM or accessibility representations.Examples include CogAgent, AGUVIS, OpenCUA, FARA, WebSTAR, and UI-TARS.
  • Temporal redundancy: 36%–56% of visual tokens are redundant across steps in the reported computer-use benchmarks.Table 1 reports average steps, patches per image, and redundant patches across environments.
  • Visual token pruning: Prior visual-token methods prune redundant regions within one image, compress tokens by saliency, or reduce the number of past trajectory steps.These approaches include ShowUI, FocusUI, Focus-Scan-Refine, adaptive compression, and FocusAgent.
  • Positioning: ReVision instead filters visual history at patch granularity across consecutive screenshots while preserving temporal evidence for long-horizon decisions.Its setting differs from methods that prune within a single screenshot or remove entire past steps.

3 Temporal Visual Redundancy

Consecutive GUI screenshots contain substantial overlap, causing standard multimodal models to repeatedly encode unchanged content and limiting useful history.

  • Trajectory representation: CUAs encode each screenshot into many visual tokens alongside accumulated textual context to predict the next action.As trajectories grow, the visual-token count increases linearly.
  • Redundancy: Consecutive screenshots often differ only in a small interface region while most of the screen remains unchanged.Examples include a button click or text update against an otherwise stable interface.
  • Measurement: 45.4% of patches are unchanged on average across steps, exceeding 56% in some settings and corresponding to over 1,000 redundant patches per step on average.The analysis compares corresponding patches in consecutive screenshot pairs across multiple benchmarks.
  • Implication: Repeated visual content consumes computation and context budget, limiting the model’s ability to incorporate useful history.The measured redundancy motivates removing unchanged visual tokens from trajectory histories.

4 REVISION

ReVision constructs compact visual histories by comparing corresponding patches across consecutive screenshots and training the multimodal model to reason with filtered inputs while retaining spatial positions.

  • Overview: ReVision uses a learned selector to compare corresponding tokens or patches across consecutive screenshots and retain informative regions.The approach preserves spatial alignment before passing the reduced visual input to the language model.
  • Problem formulation: CUA trajectories combine screenshots, accumulated textual context, and predicted actions at each step.The model conditions on visual and textual histories to generate the next reasoning and action.
  • Token filtering: For each step, ReVision computes a binary mask from corresponding patches in consecutive images and forms a reduced token set from retained patches.The mask marks which current-image patches remain, while the filtered tokens are combined with the textual context.
  • RTS classifier: RTS is a lightweight three-layer MLP that predicts whether a current-image patch is redundant given its corresponding previous-image patch.Training labels use semantic-region matching and intersection-over-union rather than raw pixel or embedding similarity.
  • Training: Filtered trajectories keep the first image in each history window unchanged and remove redundant patches from later images.The model is fine-tuned on the same filtered setting used at inference, learning to recover omitted information from earlier observations.
  • Implementation: The implementation retains original position identifiers for kept tokens and applies the selector as a plug-in mechanism on multimodal language models.The pipeline encodes images, selects tokens, builds multimodal inputs, and sends them to the language-model decoder.

5 Experiments and Results

Across three computer-use benchmarks, ReVision improves the performance–efficiency trade-off by reducing visual tokens and often shortening trajectories while preserving or improving success. Its temporal token selection and history scaling analyses show that removing redundant patches supports longer, more useful visual histories.

  • Experimental Settings: ReVision is evaluated on OSWorld, AgentNetBench, and WebTailBench using success rate under interactive and offline benchmark settings.The setup includes desktop, web-based, and fixed-trajectory evaluations, with baseline comparisons across multimodal and specialized UI agents.
  • Performance–Efficiency Trade-offs: ReVision shifts the success-rate versus token-usage frontier, achieving high success at comparable or lower token budgets across the evaluated benchmarks.Figure 3 compares success rate with average tokens per step for OSWorld, AgentNetBench, and WebTailBench.
  • Performance–Efficiency Trade-offs: 34% fewer tokens per image accompanies success-rate gains of up to 7 points on OSWorld and AgentNetBench and up to 14 points on WebTailBench.The resulting efficiency supports 9-image histories under a token budget comparable to 5-image baselines; WebTailBench reaches nearly 50% success rate versus below 30% for strong baselines.
  • Trajectory Efficiency: ReVision reduces average trajectory length by up to 4 steps while improving success rate, with WebTailBench reaching nearly 50% success rate in approximately 25–30 steps.Strong baselines require 33–37 steps and remain below 40% success rate on WebTailBench.
  • Token Selection: RTS provides the best reported performance–efficiency trade-off, improving OSWorld success rate from 32.3 to 34.0 while reducing tokens per step by 48% on average.Random and pixel-based strategies degrade performance, while OmniParserV2 has latency above 550 ms compared with approximately 22 ms for ReVision.
  • Visual History Scaling and Saturation: ReVision continues improving with longer histories to around 11 images, whereas No Drop peaks around 7 images and declines as saturation occurs near 23k tokens.Forward removal performs best for history sizes H = 5 and H = 9, while backward removal degrades performance, especially for longer histories.
  • Ablations: Increasing history from 3 to 5 images consistently improves performance across OSWorld and AgentNetBench while maintaining predictable token scaling.The relative gains remain consistent across benchmarks and architectures, although improvements are slightly smaller for Qwen3-VL-8B.

6 Conclusion

ReVision improves long-horizon computer-use by representing screenshot history around temporal redundancy. Across multiple benchmarks, it achieves higher success with fewer tokens and shorter trajectories, while motivating future work on spatial redundancy and long-context saturation.

  • Conclusion: ReVision models temporal redundancy across consecutive screenshots to reduce unnecessary visual tokens in computer-use history.The conclusion frames history representation, rather than simply the number of past images, as central to preserving useful information within limited context budgets.
  • Conclusion: Across multiple benchmarks, ReVision achieves higher success rates while using fewer tokens and shorter trajectories.The authors connect this result to improved decision-making in long-horizon computer-use tasks.
  • Conclusion: Future work should extend redundancy modeling from temporal overlap to spatial redundancy and clarify performance saturation in long-context multimodal reasoning.These directions are stated as open questions in the conclusion.

Limitations

ReVision’s effectiveness is bounded by patch-prediction errors, visual dynamism, task precision requirements, and the evaluated GUI-agent setting. Token removal can discard information needed for subtle state transitions or pixel-level reasoning, and transfer to other multimodal domains is unverified.

  • Scope and Limitations: Patch-predictor errors can remove visually small but important changes, especially in interfaces with dense text, small icons, or dynamic elements.The limitation concerns failures to identify patches that later become critical for GUI reasoning.
  • Scope and Limitations: ReVision offers its largest gains when consecutive screenshots substantially overlap, while highly dynamic scenes or frame-by-frame comparison reduce achievable compression.The method is designed around temporal redundancy in screenshot histories.
  • Scope and Limitations: Token removal is lossy and may affect tasks requiring pixel-level precision despite preserving positional indices for spatial reasoning.This caveat follows directly from the method’s compression operation.
  • Scope and Limitations: Experiments focus on GUI-based computer-use agents and vision-language models, so direct transfer to video understanding or robotic perception is not established.The authors state that applying the core idea to other multimodal settings would require additional adaptation.

B Detailed Results Tables

Across benchmarks, ReVision improves the efficiency–performance trade-off by removing redundant visual tokens while preserving or improving success and action efficiency. Its gains are especially clear on long-horizon settings, where longer histories become useful rather than increasingly redundant.

  • OSWorld: ReVision matches corresponding no-drop baselines before token dropping, indicating that its training procedure does not degrade performance.
  • OSWorld: More than 2× fewer tokens per step accompanies a 34.5 to 35.9 improvement in 50-step success rate for Qwen2.5-VL-7B with five-image history on OSWorld.Average steps also decrease from 22.7 to 19.8.
  • AgentNetBench: 72.5 to 73.8 average success rate is achieved on AgentNetBench with Qwen2.5-VL-7B and five-image history, with gains across coordinate, content, and functional metrics.Improvements are also observed with stronger backbones, including Qwen-3-VL-8B.
  • WebTailBench: Standard baselines on WebTailBench gain little beyond three history images, with OpenCUA reaching 29.5 from 25.8 at 50-step success rate before slightly dropping to 29.1 at five images.Token cost continues to grow as history expands.
  • Longer histories: With Qwen2.5-VL-7B, increasing ReVision history from three to nine images raises 50-step success rate from 28.4 to 40.8 and 100-step success rate from 35.2 to 48.9.Average steps decrease from 27.9 to 23.6 at five images while token usage remains significantly lower.

C Error Analysis

Repeated-run evaluations show that the reported ReVision trends are stable across benchmarks and history lengths, including longer histories with substantially larger visual contexts.

  • Stability analysis: Small success-rate standard deviations across benchmarks and history lengths indicate robust trends across three runs.The stability analysis covers all evaluated settings.
  • Long histories: ReVision maintains low variance at H = 7 and H = 9 despite substantially increased visual tokens and reasoning context.

E Training on a Fixed Context Window Generalizes to Other Window Sizes.

The supplied passages describe fixed-window training evaluations alongside qualitative evidence that ReVision preserves updated, task-relevant visual content while removing unchanged regions. The resulting representation supports a favorable efficiency–performance trade-off.

  • Fixed-window generalization: Models trained with ReVision using fixed history windows w ∈ 3, 5 are evaluated under matched and mismatched inference windows on OSWorld and AgentNetBench.The evaluation reports success rates and average tokens per step, with best performance when training and inference windows are matched.
  • Qualitative comparison: Random and Spiral dropping often discard important UI elements while retaining redundant background content, producing fragmented visual context.
  • Qualitative comparison: Pixel-based similarity can retain redundant regions or remove semantically important details because it is sensitive to small visual variations.Although it achieves stronger token reduction, it often harms downstream reasoning.
  • Qualitative comparison: Embedding-based DINO and Qwen methods better preserve semantically meaningful regions but can retain large redundant areas or miss fine-grained interface updates.
  • Qualitative comparison: ReVision removes unchanged regions while preserving newly updated and task-relevant content across consecutive screenshots.The model can rely on previous images for redundant information and attend to necessary current-step updates.
  • Efficiency–performance trade-off: ReVision achieves higher success rates with substantially fewer tokens per step in efficiency–performance trade-off views across benchmarks and step budgets.

H Region-Level Grouping and Learned Filtering

Learned filtering improves redundancy detection over fixed cosine similarity, while region-level grouping can improve results at substantial inference cost. ReVision instead uses structure during training to retain low-latency inference.

  • Learned filtering: Classifier-based filtering improves success rate from 72.3 to 72.9 with Qwen embeddings and from 71.7 to 72.1 with DINOv2 embeddings.It also slightly reduces token usage by learning an adaptive decision boundary instead of using a fixed cosine-similarity threshold.
  • Region-level grouping: OmniParser region grouping reaches 74.6 average success rate on AgentNetBench and 35.2 SR@100 on OSWorld with lower visual token ratios.The improvement comes with approximately 558–572 ms latency.
  • ReVision trade-off: ReVision maintains approximately 22–23 ms latency and achieves 73.8 success rate without region parsing at inference time.Its approach leverages region structure during training.
  • Patch selection comparison: Qualitative comparisons show ReVision retaining more focused visual information than random, spiral, pixel-based, DINO, and Qwen selection strategies.Figure 6 compares patch retention across two consecutive steps using lower-resolution visualizations.
  • Efficiency–performance analysis: Figure 7 compares success rate with average tokens per step across OSWorld and WebTailBench settings.Detailed numerical results are provided in Appendix B tables.

I Training and Implementation Details

ReVision trains and evaluates computer-use agents with history windows whose redundant visual patches are filtered while spatial structure and relevant information are preserved. The implementation uses matched filtered training and inference settings, deterministic evaluation, and benchmark analyses of success, token usage, trajectory length, generalization, and filtering choices.

  • Data construction: Training samples use a sliding window of up to k recent images with all preceding reasoning and actions, producing step-level supervision from AgentNet trajectories.Separate models are trained for each history-window size k to match the inference distribution.
  • Token filtering: ReVision compares corresponding patches in consecutive screenshots, removes redundant tokens from later images, and keeps the first image in each history window intact.The same filtering pipeline is applied during training and inference so the model operates on partially observed visual inputs.
  • Evaluation: Evaluation uses temperature T=0.0, averages results over three runs, and applies GPT-4O LLM-as-a-judge scoring for WebTailBench step-level correctness and final success.Success rate is the primary metric; OSWorld and WebTailBench also use different step budgets and efficiency analyses relate success to token usage and interaction steps.
  • Case study: The OSWorld case study shows substantial token savings while the agent preserves correct actions and coherent reasoning across browser privacy-settings steps.Unchanged interface regions are omitted, while the agent continues navigating settings and disabling ad-personalization options on reduced visual context.
  • Analysis and benchmarks: The study reports benchmark results, run variability, generalization across model families and history sizes, cross-window robustness, and filtering ablations with latency trade-offs.Matched training and inference windows are emphasized; classifier-based filtering improves over cosine similarity, while OmniParser further reduces redundancy but adds substantial latency.
Loading 2605.11212v3…