PaperlayerView exact revision on arXiv

Source-linked AI summary

Reflection with Action-Induced Visual Differences for Desktop GUI Agents

Yijie Ma, Chaoyue Niu, Fan Wu, Guihai Chen

arXiv:2608.24015v1cs.AI

TL;DR

Desktop GUI agents must detect subtle or spatially distributed changes across pre- and post-action screenshots, yet conventional reflectors combine change detection with success verification in one step. EFR separates these operations by extracting and filtering SoM-grounded visual evidence before judging success, improving end-to-end task success on both evaluated benchmarks.

  • Problem

    Desktop interfaces make reflection difficult because subtle or distributed visual changes must be identified across large, dense screenshots, while conventional reflectors couple change detection and outcome verification.

  • Method

    EFR is a two-stage reflector that uses SoM to annotate action locations and candidate changes, filters action-relevant evidence, and verifies success from the cleaned evidence.

  • Results

    EFR improves average end-to-end task success by 5.94% on OSWorld-Verified and 4.95% on WindowsAgentArena.

  • Takeaways & Limitations

    EFR makes reflection better grounded in screen transitions while reducing visual search complexity and reasoning burden.

  • Takeaways & Limitations

    EFR adds per-task cost: latency increases by 15.8% on OSWorld-Verified and 13.3% on WindowsAgentArena.

Abstract

from arXiv · show

The Planner-Operator-Reflector (POR) framework is widely used in GUI agents to maintain objective alignment in complex tasks through modular collaboration. However, desktop GUIs introduce a key challenge: large, dense interfaces often exhibit subtle or scattered state changes, placing most of the burden on the reflector, which must compare pre- and post-action screens, while the planner and operator reason over a single state. Existing reflectors collapse change detection and outcome verification into one step, leaving evidence implicit and yielding weakly grounded decisions. To address this limitation, we propose Evidence-First Reflection (EFR), a two-stage reflector that explicitly decouples action-induced visual differences extraction from outcome verification. EFR identifies the action location and candidate changed regions with Set-of-Marks annotations, describes and filters action-relevant changes, and makes the final judgment from the cleaned evidence. This evidence-reasoning decoupled design makes reflection better grounded in screen transitions, while reducing both visual search complexity and reasoning burden. Experiments on OSWorld-Verified and WindowsAgentArena demonstrate that EFR improves reflector accuracy by 7.11%, yielding average end-to-end task success gains of 5.94% and 4.95% on the two benchmarks, respectively.

1 Introduction

Desktop GUI reflection is difficult because large interfaces contain subtle or distributed changes that the reflector must compare across screenshots. EFR addresses this by separating visual evidence extraction from action-success verification and improves task success across two benchmarks.

  • Challenge: Desktop screens make comprehensive change detection difficult because they are large, complex, and information-dense.The reflector must analyze both pre-action and post-action screenshots, unlike the planner and operator, which primarily reason over the current state.
  • Challenge: 27.35% of failed OSWorld-Verified trajectories were caused by incorrect reflector judgments in conventional POR.Among these reflection failures, 70.49% came from page-difference identification errors and 29.51% from success-judgment errors despite correctly identified differences.
  • EFR: EFR explicitly decouples visual evidence extraction from action-outcome verification in a two-stage reflector.It uses Set-of-Marks annotations to organize action-induced changes before judging success from the extracted evidence.
  • Results: 5.94% and 4.95% are EFR’s average end-to-end task-success gains on OSWorld-Verified and WindowsAgentArena, respectively.The gains are averaged across model-benchmark settings over three runs.

2 Related Work

Prior GUI-agent research builds on modular planning, execution, reflection, and visual grounding components. POR-related work refines reflection for error correction, while SoM traditionally supports target-element localization during action execution.

  • POR Framework: POR explicitly separates planning, execution, and feedback verification for complex tasks.Mobile-Agent-v3 extends this paradigm with a Planner-Operator-Reflector-Notetaker framework for multi-step navigation and error correction.
  • Reflector: Reflectors support reusable memory and hierarchical closed-loop correction in GUI-agent pipelines.Related systems apply reflection at different execution stages or granularities and use it to identify and correct errors.
  • SoM in GUI Agents: SoM overlays explicit markers on interface content to identify candidate elements before actions are taken.Prior GUI agents use markers such as numbers or bounding boxes so models can operate through indexed elements rather than raw coordinates.

3 Problem Formulation

The conventional POR reflector compares pre- and post-action screenshots to assess success, but this combines visual change identification and goal verification in one brittle inference step. In dense desktop interfaces, subtle or intricate changes can be overlooked or fragmented.

  • Conventional POR: The conventional POR loop plans from the current state, executes a grounded action, and reflects by comparing pre- and post-action screenshots.The planner derives a sub-goal, the operator grounds it into an executable action, and the reflector produces feedback for the loop.
  • Problem Formulation: Conventional reflection simultaneously identifies visual transformations and determines whether they satisfy the intended goal.Both objectives are typically processed in a single VLM pass using the action description and two screenshots.
  • Problem Formulation: High-resolution desktop screenshots make monolithic reflection brittle because subtle changes are overlooked and intricate changes produce fragmented analysis.The resulting conclusions can be hasty when the reflector cannot comprehensively organize page differences.

4 Method

EFR replaces conventional reflection with staged visual evidence extraction and outcome judgment. It annotates screenshot changes, describes and filters region-level evidence, then evaluates success from the cleaned evidence.

  • 4 Method: EFR replaces the conventional reflector within the POR loop with a two-stage evidence-based reflection framework.The framework is designed to extract salient features under complex visual transformations and ground judgments in explicit evidence.
  • 4.1 Page Changes Annotated With SoM: SoM marks changed regions with indexed red boxes and the executed action location with a blue circle across pre- and post-action screenshots.The annotations support region-level comparison and provide spatial context for the executed action.
  • 4.1 Page Changes Annotated With SoM: EFR identifies candidate changes through pixel-wise comparison, merges nearby regions, maps them to bounding boxes, and overlays the indexed boxes on both screenshots.The resulting marked representations show action coordinates on the pre-action screenshot and change areas on the post-action screenshot.
  • 4.2 Action-Induced Page Difference Analysis and Filtering: The reflector compares marked screenshots region by region and generates change descriptions while distinguishing meaningful state transitions from system noise.The grounded action contains the executed action type and coordinates rather than the original intended action, reducing intention bias in intermediate evidence extraction.
  • 4.2 Action-Induced Page Difference Analysis and Filtering: EFR retains meaningful regional changes, filters noise-related descriptions and markers, and redraws the original screenshots into a clean evidence pair.The filtered screenshots and cleaned descriptions are used in the subsequent success judgment phase.
  • 4.3 Action Success Judgment: The final reflector receives the cleaned post-action screenshot, filtered change descriptions, and intended action, then outputs a success label, key evidence, and status summary.This judgment is based on filtered visual evidence and is provided to downstream modules.

5 Evaluation

EFR is evaluated against conventional POR across two desktop-GUI benchmarks, four backbone models, task categories, reflector accuracy, ablations, and per-task cost. It consistently improves mean task success and reflector judgments, while adding moderate reflection overhead.

  • Evaluation Setup: EFR is compared with conventional POR on OSWorld-Verified and WindowsAgentArena using four GUI-specialized and general-purpose backbone models.The evaluation uses three-run means and benchmark-provided environments.
  • Main Results: 5.94% average gain on OSWorld-Verified and 4.95% on WindowsAgentArena in mean overall success rate.Across all four backbones, POR-EFR consistently improves the mean overall success rate; Kimi-K2.5 reaches 59.74% and 52.88%, respectively.
  • Task Categories: EFR yields its clearest category-level gains on dense, frequently changing tasks, including Office and Workflow on OSWorld-Verified and Web Browsing and Windows Utilities on WindowsAgentArena.Reported gains include 10.23%, 9.16%, 12.33%, and 15.80%, respectively; scattered category-level decreases also occur.
  • Reflector Accuracy: 7.11% improvement in reflector judgment accuracy accompanies a strong positive correlation with end-to-end task success.The correlation across GUI-Owl-32B trajectories on OSWorld-Verified is Pearson r = 0.86.
  • Ablation Study: Ablating SoM, page-difference identification, grounded-action conditioning, or screenshots reduces success or accuracy by 2.72%, 6.70%, 1.38%, and 10.58%, respectively.These ablations test localized visual focus, explicit difference analysis, observed-action conditioning, and screenshot access during verification.
  • Cost Analysis: EFR increases per-task latency and token usage but reduces executed steps, with the extra VLM call only partially offset by fewer executions.Overheads are 15.8% and 14.9% on OSWorld-Verified and 13.3% and 17.9% on WindowsAgentArena; executed steps fall by 11.5% and 3.7%.

6 Conclusion

The paper presents EFR as a two-stage replacement for conventional reflection, explicitly extracting visual evidence before judging action success. Its evaluations report improved performance and emphasize that desktop GUI agents require more than directly applying existing methods.

  • 6 Conclusion: EFR decouples explicit visual evidence extraction from subsequent success judgment within the POR framework.The conclusion describes EFR as a two-stage reflection framework.
  • 6 Conclusion: The evaluations demonstrate superior performance for EFR while highlighting the broader complexity of building desktop GUI agents.The conclusion states that directly applying existing methods is insufficient.

A Reflector Accuracy

EFR improves reflector accuracy by making visual evidence explicit before judging action outcomes. Its two-stage process annotates changed regions, anchors the action location, and verifies success from filtered evidence.

  • 87.50% reflector accuracy with EFR versus 80.39% for the conventional reflector on OSWorld-Verified.Average task success also rises from 44.91% to 51.94%.
  • 13.78 percentage points fewer reflector-attributed failures occur with EFR: 13.57% versus 27.35%.Page-difference identification errors fall by 12.25 percentage points, while success-judgment errors fall by 1.54 percentage points.
  • The Chrome History example shows EFR producing intermediate evidence before the final reflection judgment, while both reflectors reach the correct outcome.The evidence retains action-relevant areas and removes an unrelated taskbar-clock update.
  • EFR first identifies and organizes visual differences, then verifies action success from the extracted evidence.Set-of-Marks annotations and merged difference boxes make changed regions consistently referenceable across screenshots.
  • Action-location annotation projects the executed action’s geometric parameters onto the pre-action screenshot without additional visual inference.Point actions use execution coordinates, while drag actions use start and end points; coordinate-free actions receive no location marker.

D Benchmark Task Grouping Details

The evaluation groups benchmark tasks using mappings between app-level and domain-level categories, while EFR’s prompts define separate analysis and verification stages.

  • D Benchmark Task Grouping Details: The left category-mapping table reports OSWorld-Verified and the right table reports WindowsAgentArena.Mappings follow the official benchmark definitions.
  • D Benchmark Task Grouping Details: Stage 1 identifies and describes page changes directly related to the specified action.It filters every marked region as relevant or noise, then summarizes only the relevant areas.
  • D Benchmark Task Grouping Details: Stage 2 determines whether the previous action was successful by comparing screenshots and the visual-change description against the latest action.The prompt instructs the verifier to prioritize the visual-change description when it conflicts with the screenshots.
  • D Benchmark Task Grouping Details: The operator action space is adopted from Mobile-Agent-v3, with grounding requirements differing across point, drag, and coordinate-free actions.Clicks and scrolling require one screen point, drags require start and end points, and hotkey, wait, and done do not require visual grounding.

G Per-Step Cost Analysis

EFR increases per-step computational cost but changes task outcomes mainly by recovering trajectories that otherwise remain failures.

  • 30.8% higher latency and 29.8% higher token usage per step occur on OSWorld-Verified with POR-EFR.On WindowsAgentArena, latency rises 17.7% and token usage rises 22.4%.
  • Figure 10 compares average latency and token usage per executed step across the two benchmarks.The first row represents OSWorld-Verified and the second represents WindowsAgentArena.
  • 12.68% of OSWorld-Verified trajectories and 10.19% of WindowsAgentArena trajectories transition from failure to success after applying EFR.Success-to-failure transitions are 3.27% and 4.30%, respectively.
  • Figure 11 shows trajectory proportions that remain failed, recover, or regress after applying EFR.The charts provide benchmark-specific outcome transition distributions.

H.1.1 Case 1: Change the 2 in “H2O” to a subscript in LibreOffice Writer

In the H2O subscript task, EFR recognizes subtle cursor, selection, and formatting changes, whereas the conventional reflector repeatedly misses the initial cursor evidence and retries.

  • EFR converts a nearly invisible cursor change into usable evidence for the H2O subscript task.This prevents repeated false failure judgments in the small-change regime.
  • Figure 10’s case comparison places the failed conventional-reflector trajectory above the successful EFR trajectory.The figure is identified as the small-change case involving subtle cursor and formatting evidence.
  • The conventional reflector judges the first click as failed despite a small cursor appearing next to the character “2”.It similarly misses the cursor after the second and third clicks.
  • After four repeated clicks, the conventional-reflector trajectory remains stuck and ultimately fails.The reflector continues reporting no visible change.
  • EFR recognizes the text cursor, selection highlight, and final subscript rendering as successive successful states.The three actions are clicking “2”, dragging over it, and pressing Ctrl+Shift+B.

H.1.2 Case 2: Add Wednesday 12:00 course slot in Calc schedule

The Calc trajectory illustrates how EFR rejects a superficially successful edit when unrelated spreadsheet regions are also damaged, then supports rollback and successful retry.

  • Case 2: Add Wednesday 12:00 course slot in Calc schedule: The case demonstrates EFR’s large-change handling: it organizes dispersed visual changes rather than checking only whether the target region changed.
  • Case 2: Add Wednesday 12:00 course slot in Calc schedule: EFR judges the edit as failed because unrelated page changes and missing text accompany the correctly filled target cell.The conventional reflector focuses only on the target cell and accepts the intermediate state.
  • Case 2: Add Wednesday 12:00 course slot in Calc schedule: The harmful edit is rolled back successfully, returning the spreadsheet to its earlier state.
  • Case 2: Add Wednesday 12:00 course slot in Calc schedule: The retry changes only the intended cell, so EFR judges the action successful and the task is completed.

H.4 EFR Failure Case Analysis

EFR’s remaining failures arise from broader agent-pipeline limitations: correct local failure detection may not produce recovery, and explicit visual evidence may still be semantically misinterpreted.

  • H.4 EFR Failure Case Analysis: Fine-grained grounding errors can prevent the planner and operator from converting EFR’s detected execution failure into rollback or a revised plan.
  • H.4 EFR Failure Case Analysis: Input errors and semantic misunderstanding can lead EFR to accept export.jpg.jpg as harmless despite the incorrect target filename.
  • H.4 EFR Failure Case Analysis: The negative results are attributed to broader pipeline limitations rather than a systematic weakness on a particular task category.
Loading 2608.24015v1…