PaperlayerView exact revision on arXiv

Source-linked AI summary

PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher, Maximilian Schilling

arXiv:2610.31274v1cs.AIcs.HC

TL;DR

Existing evaluations emphasize long-horizon task completion or conventional GUI controls, leaving fine-grained grounding less directly tested. POINTERBENCH introduces three diagnostic suites and evaluates general-purpose and specialist models; results identify text grounding as a major bottleneck, particularly at finer target granularities.

  • Problem

    Interactive evaluations conflate grounding with reasoning, execution, and environment control, while GUI grounding benchmarks lack fine-grained targets such as spreadsheet boundaries and positions between characters.

    Source: §1 · p. 1

  • Method

    POINTERBENCH provides three 500-example suites for spreadsheet geometry, fine-grained textual targets, and professional interfaces, and evaluates general-purpose models and specialist grounding configurations.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    No specialist exceeds 34.0% on POINTERBENCH-Text, making fine-grained text grounding the primary bottleneck across specialist configurations.

    Source: §3.3 · p. 5

  • Takeaways & Limitations

    By exposing failures at individual grounding primitives, POINTERBENCH provides a targeted test bed for measuring and improving fine-grained GUI grounding.

    Source: §4 · p. 5

    POINTERBENCH isolates one-step grounding rather than temporal planning, recovery, or environment dynamics, and does not directly measure robustness to arbitrary display resolutions or coordinate representations.

    Source: H Additional Limitations · p. 16

Abstract

from arXiv · show

Multimodal Large Language Models have shown promising performance in computer-use tasks, from planning user workflows to grounding natural-language actions to precise UI targets. Existing end-to-end benchmarks primarily emphasize long-horizon task completion, while GUI grounding benchmarks remain focused on conventional controls and a limited set of professional interfaces. We introduce POINTERBENCH, a diagnostic suite of 1,500 screenshot-instruction pairs probing three complementary capabilities: thin sub-element identification and geometric grounding in spreadsheets (POINTERBENCH-Sheets), character-level grounding in long-text documents (POINTERBENCH-Text), and mixed GUI grounding across 100 professional applications (POINTERBENCH-Pro). We carefully curate each task with pixel-exact target geometry and evaluate both general-purpose multimodal models and 39 specialist GUI grounding configurations. We find text grounding to be the most challenging setting, with Claude Fable 5, GPT-5.6 Sol, and the leading native specialist UI-Venus-2-9B achieving 69.6%, 53.4%, and 25.8% success on POINTERBENCH-Text despite respective 83.2%, 77.7%, and 54.5% average success across POINTERBENCH. These results reveal substantial headroom in fine-grained GUI grounding across frontier and specialist models. We publicly release the benchmark and evaluation code at https://github.com/warmwindOS/pointerbench.

1 Introduction

Computer-use agents must ground actions to precise screen regions, but existing evaluations either conflate grounding with other agent capabilities or omit fine-grained targets. POINTERBENCH addresses this gap with three complementary diagnostic tasks and an evaluation spanning general-purpose and specialist models.

1 Introduction

  • Precise screen grounding remains challenging in dense, visually ambiguous GUIs and high-resolution professional applications.Computer-use agents must ground each action to screen coordinates or regions as well as select an appropriate action sequence.Source: §1 · p. 1
  • End-to-end interactive benchmarks conflate failures in reasoning, grounding, execution, and environment control.They evaluate long-horizon task completion in websites or executable desktop environments.Source: §1 · p. 1
  • ScreenSpot and ScreenSpot-Pro lack fine-grained actions such as targeting spreadsheet boundaries, positions between characters, or narrow text spans.These benchmarks isolate localization, but the introduction calls for complementary tests that vary target geometry to diagnose precise control.Source: §1 · p. 2
  • POINTERBENCH evaluates three complementary grounding tasks, isolating spreadsheet structure, fine-grained text, and professional interfaces.The introduction frames its contributions as a diagnostic suite with precisely defined target geometry and a comprehensive evaluation of general-purpose multimodal models and specialist configurations.Source: §1 · p. 2
  • The paper's contributions combine a diagnostic benchmark suite with evaluation showing rank reversals across subsets and a persistent textual-grounding bottleneck.The stated bottleneck concerns character-level and region-level textual grounding.Source: §1 · p. 2

2 Related Work

Prior work evaluates complete interactive agents or screenshot-based GUI localization, while specialized models and methods improve conventional-control localization. POINTERBENCH complements these efforts by testing specific sub-element geometries independently of temporal planning and environment execution.

2 Related Work

  • Interactive benchmarks evaluate long-horizon task execution, whereas POINTERBENCH removes temporal planning to test precise grounding of an already specified action.The cited examples include open-ended web tasks, reproducible websites, desktop applications, and mobile applications.Source: §2 · p. 2
  • GUI grounding benchmarks and models cover screenshot localization, state-conditioned grounding, and conventional controls, while POINTERBENCH targets thin and tightly bounded sub-elements.Its focus includes spreadsheet boundaries, character- and caret-level text locations, and narrowly bounded regions.Source: §2 · pp. 2–3
  • POINTERBENCH provides a reproducible diagnostic of grounding errors independent of temporal planning and environment execution.It broadens professional-interface coverage to 100 interfaces while systematically isolating pixel-exact sub-element geometries.Source: §2 · p. 3

3 PointerBench

POINTERBENCH isolates spreadsheet, textual, and professional-interface grounding through complementary suites. Results vary substantially by task: text grounding is a bottleneck for specialists, while frontier models perform strongly on spreadsheet and professional-interface targets.

3 PointerBench

  • POINTERBENCH defines one-step grounding targets across three 500-example suites, using 1024×768 screenshots and point or bounding-box outputs.The benchmark separates spreadsheet structure, fine-grained text, and heterogeneous professional interfaces; bounding boxes require at least 0.90 target coverage and 0.70 prediction precision.Source: §3 · p. 3

3.1 Frontier Model Performance

  • 83.2% average success makes Claude Fable 5 the POINTERBENCH leader, ahead of Claude Opus 5 at 81.1% and GPT-5.6 Sol at 77.7%.Performance varies by suite: Opus leads Sheets at 98.8%, while Fable leads Text at 69.6%.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Specialist GUI Grounding Models

  • MVP-32B leads specialist configurations on all three suites, scoring 68.4% on Sheets, 34.0% on Text, 75.4% on Pro, and 59.3% average.Trailing models reverse rank across suites: UI-Venus-2-9B is second on Pro, GTA1-32B on Text and overall, and OpenCUA-32B on Sheets.Source: §3.2 · p. 4, Table 2 · p. 4

3.3 Per-Suite Analysis

  • 98.8% Sheets success for Opus contrasts with Fable’s 78.1% on row-resize handles, showing that aggregate performance can conceal errors on thin structural targets.Conventional cell and header targets are localized almost perfectly.Source: §3.3 · p. 4
  • No specialist exceeds 34.0% on POINTERBENCH-Text; word localization is easiest, while character, caret, and tightly bounded region targets remain difficult.Frontier character accuracy reaches 51.9% for Opus and 48.1% for Sol; Fable reaches 56.0% on Region.Source: Table 3 · p. 4, §3.3 · p. 5, Figure 2 · p. 5
  • 82.4% Pro success for Fable, 81.2% for Opus, and 82.0% for Sol accompany disproportionate icon failures: icons are 38.0% of examples but 57.3% of pooled failures.Text controls are easiest, followed by other interface controls; icons are hardest.Source: §3.3 · p. 5
  • Converting parseable Gemini 3.7 Flash and Qwen3.8 Max predictions to the requested coordinate frame raises their scores from 3.1% to 78.1% and from 2.8% to 55.4%.Both models predominantly return coordinates in a 0–1000 frame despite instructions to use absolute 1024×768 coordinates.Source: §3.3 · p. 5

4 Conclusion

POINTERBENCH exposes distinct fine-grained grounding failures across spreadsheets, text, and professional interfaces. Its three suites use controlled or generated interfaces with pixel-exact targets and reproducible benchmark data.

4 Conclusion

  • Text grounding degrades from words to characters, carets, and precisely bounded regions, while professional-interface failures concentrate disproportionately on icons.The conclusion also notes that leading models reliably localize conventional spreadsheet and application controls but remain challenged by thin structural elements.Source: §4 · p. 5

A Benchmark Construction and Reproducibility

  • Sheets examples vary grid layout, style, offsets, and optional application chrome; targets include cells, headers, colors, relative references, handles, edges, and corners.Generation is deterministic, producing a distinct screenshot and metadata record for each example.Source: A Benchmark Construction and… · p. 8
  • Text examples use exact glyph geometry across words, characters, punctuation, carets, spans, and structured fields, with 366 point targets and 134 bounding-box targets.The language mix is 50% English and 10% each German, French, Spanish, Italian, and Dutch; examples are retained only when the target is unique.Source: A Benchmark Construction and… · p. 9
  • Pro contains generated professional interfaces, with five examples from each of 100 applications and targets spanning icons, text, and other controls.Its 190 icon, 154 text, and 156 other-control examples retain application metadata for tracing; selection is deterministic and reproducible.Source: A Benchmark Construction and… · p. 9

B Benchmark Composition and Geometry

The benchmark spans varied professional-application domains and target geometries. Text targets are substantially smaller than those in Sheets or Pro, and target size relates to accuracy without fully explaining errors.

B Benchmark Composition and Geometry

  • Table 4 summarizes the task composition of the 500-example Sheets suite.Source: B Benchmark Composition and Geometry · p. 9, Table 4 · p. 9
  • Median target area is 0.215% for Sheets, 0.0286% for Text, and 0.129% for Pro; the Text median is approximately 7.5× smaller than Sheets and 4.5× smaller than Pro.Figure 4 presents target-center distributions and target areas; Sheets follows spreadsheet geometry, Text spans document surfaces, and Pro clusters near upper-interface controls.Source: B Benchmark Composition and Geometry · p. 9
  • Development and creative software are the largest Pro domain groups, while examples also span scientific, engineering, productivity, communication, enterprise, office, operating-system, and consumer interfaces.Figure 3 summarizes the 100 represented applications by domain.Source: B Benchmark Composition and Geometry · p. 9
  • The largest-area target quartile is 10.7 percentage points more accurate than the smallest on average, though the size effect varies by suite and does not fully explain errors.Figure 5 reports target width and height separately because area can obscure highly anisotropic targets such as thin spreadsheet boundaries.Source: B Benchmark Composition and Geometry · p. 9

C Evaluation Protocol and Inference Settings

The evaluation uses shared screenshot–instruction inputs and point or bounding-box outputs in the benchmark coordinate frame. It scores target localization with suite-level success rates and specifies common frontier-model prompting and inference settings.

C Evaluation Protocol and Inference Settings

  • Bounding-box predictions count as correct when coverage is at least 0.90 and precision is at least 0.70; the asymmetric criterion penalizes clipping while allowing a modest margin.Coverage and precision are computed from overlap between predicted and annotated boxes.Source: C Evaluation Protocol and Inference… · p. 9
  • Table 5 distinguishes Text bounding-box region targets from the remaining groups, which require point predictions.Source: Table 5 · p. 10
  • Figure 3 summarizes Pro application-domain coverage; each of the 100 applications contributes five examples, and parenthetical numbers count applications by domain.Source: Figure 3 · p. 10
  • Malformed or unusable outputs count as failures, and the benchmark reports binary success rate for each suite and their unweighted mean.Official scores use unmodified outputs in the requested coordinate frame; alternate coordinate conventions are analyzed separately.Source: C Evaluation Protocol and Inference… · p. 10
  • Frontier models receive a shared system prompt specifying absolute pixel coordinates in a 1024×768 frame and JSON point or bounding-box output.The prompt prohibits normalized coordinates and cropping or resizing the coordinate frame.Source: C Evaluation Protocol and Inference… · p. 10
  • Inference settings use original 1024×768 images and enforced JSON schemas; response caps differ by reasoning setting, and filtered or budget-exhausted responses are not retried.Exact endpoint identifiers, request manifests, hashes, and per-example outputs are included with the released evaluation artifacts.Source: C Evaluation Protocol and Inference… · pp. 10–11
  • Figure 4 shows normalized target-center distributions, their marginals, and target areas as a percentage of the 1024×768 screen, with dashed lines marking suite medians.Source: Figure 4 · p. 11
  • Figure 5 shows normalized target width and height by suite as percentages of the corresponding screen dimension on a logarithmic scale.Source: Figure 5 · p. 11

D Reasoning-Budget Sensitivity

Reasoning-budget effects vary across models rather than uniformly changing the ranking. Paired low-versus-disabled comparisons are statistically significant for Sonnet, Luna, and Sol, including Sonnet’s small decline.

D Reasoning-Budget Sensitivity

  • The study evaluates six models across their available reasoning settings on the same 1,500 examples.Source: D Reasoning-Budget Sensitivity · p. 11, Table 6 · p. 12
  • 72.7% at medium reasoning is Luna’s highest reported result, while Sol reaches 77.3% at low and 77.7% at medium after 67.5% with reasoning disabled.Sonnet is non-monotonic, and medium reasoning reaches 72.2%; medium improves Fable by only 1.4 percentage points over low, raises Grok from 1.8% to 8.7%, and does not improve Qwen.Source: D Reasoning-Budget Sensitivity · pp. 11–12, Table 7 · p. 12
  • Low-versus-disabled changes are statistically significant for Sonnet, Luna, and Sol, including Sonnet’s small decline.Table 8 counts examples corrected and regressed by the low setting and reports two-sided exact McNemar tests.Source: D Reasoning-Budget Sensitivity · p. 12, Table 8 · p. 12

E Additional Diagnostic Breakdowns

The additional breakdowns show that aggregate accuracy can hide errors on thin spreadsheet geometry, while text-region and coordinate-frame failures vary by model. Converted coordinate scores are diagnostic only, not leaderboard results.

Spreadsheet target types

  • Edges range from 81.0% to 98.3% and corners from 89.7% to 97.4%; Opus reaches 98.3% on edges and 94.9% on corners.Fable reaches 91.4% on edges despite perfect cell, header, and color grounding, illustrating that aggregate Sheets accuracy can conceal fine-geometric errors.Source: E Additional Diagnostic Breakdowns · pp. 12–13, Table 9 · p. 13

Coordinate-frame compliance

  • 78.1% for Gemini 3.7 Flash and 55.4% for Qwen3.8 Max are converted diagnostic accuracies, up from 3.1% and 2.8%, respectively.Uniform conversion from 0–1000 coordinates to the requested frame produces diagnostic scores only; converted scores are not used for ranking.Source: E Additional Diagnostic Breakdowns · pp. 13–14, Table 13 · p. 14

Bounding-box failures

  • 26.1% of bounding-box examples are oversized for both Fable and Opus, while clipping is the most common failure for Sol (39.6%) and Luna (41.8%).The Region group contains 134 bounding-box examples; the strongest models do not share one failure mode.Source: E Additional Diagnostic Breakdowns · p. 13, Table 12 · p. 14

F Full Model Evaluation

The full-model evaluation presents success rates across POINTERBENCH-Sheets, Text, and Pro, including all evaluated systems and a center-click baseline.

Full evaluation

  • Table 14 reports success rates across the three POINTERBENCH suites for the full evaluation.New Gemini and Qwen entries use unmodified official outputs, without coordinate-frame post-processing.Source: Table 14 · p. 14

Reported metrics

Evaluation scope

  • The evaluation includes all 37 evaluated systems together with the center-click baseline.The table continues across pages.Source: F Full Model Evaluation · p. 15

G Complete Specialist GUI Grounding Evaluation

The specialist evaluation covers 39 open-weight GUI-grounding configurations under their native evaluation protocols, across all 1,500 benchmark examples.

Evaluation scope

Evaluation protocol

  • Specialist models use their official native prompts, coordinate formats, or inference-time scaling wrappers rather than the unified absolute-pixel JSON prompt.The passage gives examples including normalized coordinates, scaled coordinates, model-specific grounding formats, and wrappers.Source: G Complete Specialist GUI Grounding… · p. 15

Scoring

H Additional Limitations

POINTERBENCH tests one-step grounding rather than temporal planning, recovery, or environment dynamics, and its fixed coordinate frame limits the settings it directly measures.

Benchmark scope

  • POINTERBENCH isolates one-step GUI grounding and does not measure temporal planning, recovery, or environment dynamics.Its scores are not a complete measure of computer-use-agent capability; interactive-benchmark success likewise does not guarantee robustness to these fine-grained targets.Source: H Additional Limitations · p. 16

Coordinate-frame scope

  • The benchmark uses a fixed 1024×768 frame and does not directly test arbitrary resolutions, resizing policies, or alternative coordinate representations.This bounds conclusions about robustness to other display and coordinate settings.Source: H Additional Limitations · p. 16

Pretraining exposure

  • Proprietary pretraining exposure cannot be independently verified for closed models, although the generated benchmark examples reduce direct overlap with established GUI-grounding datasets.Generated interfaces and tasks do not eliminate possible exposure to similar interfaces or tasks.Source: H Additional Limitations · p. 16
Loading 2610.31274v1…