PaperlayerView exact revision on arXiv

Source-linked AI summary

GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding

Md Abrar Jahin, Md Rizwan Parvez

arXiv:2608.21832v1cs.CL

TL;DR

Existing GUI benchmarks do not isolate whether models understand spatial relations or merely localize salient interface elements. GUI-PRIMITIVES addresses this with 994 contrastive pairs across seven relations and evaluates 19 vision-language models. Models achieve low strict grounding accuracy, with most errors arising from predictions outside candidate regions, while candidate marking improves selection but remains an oracle diagnostic.

  • Problem

    Existing GUI benchmarks do not isolate whether models bind relational language to the correct interface element.

  • Method

    GUI-PRIMITIVES uses 994 contrastive instruction pairs over seven spatial relations, fixing each screenshot and anchor while switching the relation so the target moves between candidates.

  • Results

    Most models’ failures reflect candidate localization: predictions fall outside both candidate regions on 60–92% of items, while within-region selection follows relation words except for containment and occlusion.

  • Takeaways & Limitations

    GUI-PRIMITIVES diagnoses GUI grounding failures more specifically than overall click accuracy, and its accuracy correlates with ScreenSpot-Pro performance at Spearman ρ = +0.74.

  • Takeaways & Limitations

    Containment and occlusion appear only in synthetic screenshots, limiting their external validity; the benchmark covers static pre-action English click targets.

Abstract

from arXiv · show

Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element. We introduce GUI-Primitives, a 994-item benchmark of contrastive instruction pairs over seven spatial relations in graphical user interfaces (left/right, above/below, containment, alignment, proximity, list ordinal, occlusion). Each pair holds the screenshot and anchor fixed while changing the relation expression, so the correct target moves between two designated candidates. Five annotators validate a 196-item subset ($κ= 0.94$ well-formedness; $κ= 0.79$ target selection). Nineteen vision-language models reach at most $32\%$ strict point-in-box accuracy. Because models emit unconstrained coordinates, we classify each prediction by the candidate region it falls within. Predictions fall outside both candidates on $60-92\%$ of items. Conditional on falling within a candidate region, target selection reaches 0.82-0.90 for horizontal position, vertical position, proximity, and list ordinal, but does not differ significantly from 0.50 for containment and occlusion: most failures reflect candidate localization rather than relation understanding. Across ten models, benchmark accuracy correlates with ScreenSpot-Pro accuracy (Spearman $ρ= +0.74$), an exploratory association at this sample size. Marking the two designated candidates raises selection accuracy by 35--57 percentage points, an oracle diagnostic that supplies the candidate set rather than a deployable method. We release the benchmark, predictions, and code.

1 Introduction

GUI-PRIMITIVES isolates whether vision-language models bind spatial language to the correct GUI element, a capability existing end-to-end and coarse grounding benchmarks do not separate. Across 19 models, performance is low, while candidate-level analysis distinguishes localization failures from relation-selection failures.

  • Motivation: Existing GUI benchmarks combine grounding with planning and action or measure clicks within target boxes without separating spatial primitives.Consequently, a wrong click does not identify which elementary skill failed.
  • Benchmark: GUI-PRIMITIVES contains 994 contrastive-pair questions over seven GUI spatial relations, fixing the screenshot and anchor while changing the relation so the target switches candidates.The construction controls screenshot-specific salience and fixed answer preferences.
  • Evaluation: 19 vision-language models are evaluated, with five annotators validating a 196-item subset at Fleiss κ = 0.94 for well-formedness.The annotators also selected the instruction-consistent candidate on 96.9% of retained items.
  • Findings: 31% strict point-in-box accuracy is reached by Claude Opus 4.7, the strongest model, while every model falls below 0.50 on containment, occlusion, alignment, and proximity.On real desktop screenshots, 96.2% of predictions fall outside both candidate regions.
  • Findings: Spearman ρ = +0.74 links GUI-PRIMITIVES accuracy with ScreenSpot-Pro accuracy across ten models.The paper reports this as an exploratory association at the evaluated sample size.
  • Findings: 35–57 percentage points is the selection-accuracy improvement from marking the two candidates, an oracle diagnostic rather than a deployable method.The intervention supplies the candidate set that baseline models must otherwise locate.

2 Related Work

Prior work evaluates GUI grounding, spatial reasoning, minimal-pair contrasts, and inference-time interventions, but GUI-PRIMITIVES combines these ideas for controlled spatial diagnosis in GUI screenshots.

  • GUI grounding and computer-use agents: OSWorld evaluates real software tasks, while ScreenSpot and ScreenSpot-Pro isolate pixel grounding; other benchmarks add hierarchical GUI evaluation or trained grounding agents.These approaches ask whether agents land clicks or complete tasks, rather than isolating each GUI spatial primitive.
  • Spatial reasoning in vision-language models: Natural-image benchmarks show that vision-language models struggle with spatial relations and perceptual primitives despite stronger captioning or general capabilities.The cited benchmarks cover controlled minimal pairs, broad spatial relations, and 2D/3D perceptual primitives.
  • Spatial reasoning in vision-language models: RocketScience separates object localization from spatial reasoning with contrastive pairs, and GUI-PRIMITIVES extends controlled evaluation to dense GUI layouts with text, similar icons, and visual clutter.The GUI setting differs substantially from the visual prior of natural-image benchmarks.
  • Minimal-pair evaluation: Minimal-pair evaluation uses a single lexical or syntactic edit to test understanding rather than surface statistics, and GUI-PRIMITIVES adds pair-consistency as a relational-failure diagnostic.This design follows earlier psycholinguistic and vision-language applications.
  • Inference-time interventions: Set-of-Mark prompting replaces coordinate regression with numbered visual-region selection, while activation and attention steering modify model internals without weight updates.Related work also explores zooming, modality-aware prompting, and test-time scaling for GUI agents.

3 The GUI-PRIMITIVES Benchmark

GUI-PRIMITIVES is a controlled benchmark designed to test seven elementary spatial skills required for GUI grounding. Its contrastive pairs, mixed real and synthetic sources, and human verification make relation understanding separable from candidate localization.

  • Design goals: GUI-PRIMITIVES tests seven elementary spatial primitives using minimal pairs and a human verification protocol.The full construction pipeline is summarized in Figure 1.
  • Primitive set: The seven primitives cover horizontal and vertical position, containment, alignment, proximity, list ordinal, and occlusion, with a 0.50 two-candidate reference level.All primitives are binary minimal-pair contrasts.
  • Minimal-pair construction: Each pair shares one screenshot and anchor while changing the relation expression so the correct target becomes the contrastive candidate.A model using a fixed salience or centrality shortcut should pass one twin and fail the other.
  • Sources: The benchmark combines real desktop screenshots with procedurally rendered synthetic stimuli containing known positions and controlled distractors.Synthetic data supplies containment and occlusion because real annotations lack parent–child and overlay–target metadata.
  • Human verification: Five annotators judged a 196-item stratified subset, yielding Fleiss κQ1 = 0.942 for well-formedness and κQ2 = 0.787 for target selection.After 11 invalid items were dropped, the human-clean core contained 185 items with 96.9% human accuracy.

4 Models and Protocol

The protocol evaluates a broad set of vision-language models with standardized coordinate handling and point-in-box metrics. Because predictions are unconstrained coordinates, the analysis explicitly separates candidate selection from predictions outside both regions.

  • Models: 19 vision-language models are evaluated, including proprietary systems, Qwen-family models, InternVL3, Llama Vision, Gemma, PaliGemma, and MiniCPM-V.The set spans open-weight and commercial API models.
  • Prompting and decoding: All models use greedy decoding at T = 0 with a fixed seed, while a robust parser handles pixel and normalized coordinate conventions.Screenshots are resized to account for provider-side downsampling and predictions are rescaled to original pixels.
  • Metrics: Point-in-box accuracy counts a prediction as correct when both predicted coordinates lie within the target box in the original pixel frame.Loose accuracy instead uses a radius based on twice the target diagonal.
  • Metrics: 0.50 is a forced-choice reference for the binary contrasts, not the chance level of unconstrained clicking.A prediction may fall outside both candidate regions, so each baseline prediction is classified as target, distractor, neither, or invalid.

5 Diagnostic Results

GUI-PRIMITIVES exposes a large human–model gap and uneven primitive competence. Candidate-level and pair-consistency analyses indicate that failures often arise from localization and relational binding rather than simple chance selection.

  • Shortcut controls: Blank-canvas controls reduce accuracy from 22.0% to 6.5%, while shuffled screenshots reduce it to 14.8%.Both highly significant drops support dependence on the visual input and matching instruction.
  • Per-primitive structure: 0.80–0.83 is the top-five accuracy range for list-ordinal, while containment, occlusion, alignment, and proximity remain below the 0.50 reference for every model.Rel-pos-horizontal reaches 0.51 for Claude Opus 4.7, whereas rel-pos-vertical peaks at 0.32 for GPT-5.
  • Candidate-level analysis: 60–92% of baseline predictions fall outside both candidate regions, and within-region target selection reaches 0.89–0.90 for horizontal position, vertical position, and proximity.This pattern indicates that much of the strict-accuracy deficit reflects candidate localization rather than relational interpretation.
  • Overall accuracy and human gap: 32% is the strongest model’s accuracy on the human-clean core, versus 96.9% for annotators.The 65-point gap persists across the full benchmark and cleaned core, with stable model rankings across evaluation subsets.
  • Pair consistency: Pair-consistency lies below the independent-guessing curve y = x^2, indicating negatively correlated errors across contrastive twins.The relation word, rather than the picture, is identified as the source of confusion.

6 Primitive Competence Predicts Grounding

GUI-PRIMITIVES accuracy is positively associated with ScreenSpot-Pro grounding accuracy across ten models. The association also holds separately within the real-screenshot and synthetic subsets.

  • Cross-benchmark association: Spearman ρ = +0.736 (p = 0.015, n = 10) between GUI-PRIMITIVES and ScreenSpot-Pro accuracy.The correlation is +0.705 in the real-screenshot subset and +0.760 in the synthetic subset.

7 Training-Free Interventions

Training-free interventions distinguish candidate search from the spatial skills tested by GUI-PRIMITIVES. Oracle marking produces large gains, whereas reasoning prompts and activation steering do not significantly improve accuracy.

  • Oracle two-candidate marking: 35–57 percentage points is the gain from marking the two designated candidates across three of four tested models.The intervention replaces coordinate search with symbolic selection and therefore functions as an oracle diagnostic.
  • Oracle two-candidate marking: +57.1 points lifts GPT-5 from 30% to 87%, the highest absolute accuracy reached under any tested condition.OS-Atlas-Base-7B rises from 10% to 52%, while Claude Opus 4.7’s +9.2-point gain is marginal and nonsignificant.
  • Per-primitive intervention effects: Six of seven primitives gain at least 40 points on GPT-5 with SoM.List-ordinal regresses by −32.4 points on Qwen2.5-VL after converting an already-saturated task into mark selection.
  • Negative results: +0.5 points for primitive-aware CoT and +1.5 points for activation steering are both nonsignificant.The model often names the relation correctly in intermediate reasoning but still emits the wrong coordinate.

8 Discussion

GUI-PRIMITIVES reveals that most baseline errors arise from candidate localization, while oracle candidate marking substantially improves discrimination without repairing the underlying skill gap.

  • Oracle two-candidate marking yields the largest training-free improvement, but supplies candidate regions rather than solving the grounding deficit.SoM shifts part of relational grounding into symbolic mark selection and depends on upstream detection quality.
  • 60–92% of predictions fall outside both candidate regions, indicating candidate localization dominates baseline errors.Within-region selection follows the relation word on all primitives except containment and occlusion, where it is indistinguishable from 0.50.

9 Limitations

The benchmark’s scope and interventions impose important boundaries on how its findings and gains should be interpreted.

  • Containment and occlusion appear only in synthetic items, limiting their external validity because real GUI datasets lack scalable relationship metadata.List-ordinal has only a 24-item real slice, and these results are reported separately.
  • GUI-PRIMITIVES v1 covers static pre-action grounding for English click targets, not drag targets, bidirectional or CJK layouts, or end-to-end OSWorld success.The paper identifies these as planned extensions or future work.
  • SoM gains may not generalize to realistic multi-candidate scenes because the benchmark overlays at most two marks and deployment inherits detector failures.Point-in-box accuracy is also binary and does not capture structured spatial errors.

Ethical Considerations

The paper documents evaluation choices, benchmark composition, and implementation boundaries relevant to interpreting its evidence and supporting safer GUI-agent development.

  • The work is intended to expose elementary spatial-grounding failures before deployment, supporting safer GUI-agent development.Screenshots remain under their original license; only derived annotations are redistributed.
  • Closed-vision APIs may silently misalign predictions through server-side downsampling, so wrappers should report coordinate-frame handling.The paper recommends documenting provider-side preprocessing in high-resolution evaluations.
  • The benchmark includes 994 items across full, human-verified, and cleaned splits, with per-primitive bootstrap intervals released alongside the artifact.Table 3 covers all 19 evaluated models, while closed models were omitted from the full split because of API cost.
  • Five annotators judged well-formedness and target selection on 196 items, yielding Fleiss κ = 0.942 and 0.787 respectively.Eleven items were removed after majority judgments to form the 185-item human-clean core.
  • Real targets are much smaller than synthetic targets, with median areas of 0.05% versus 1.3% of image area.Both distributions span approximately 3.5 orders of magnitude, providing varied target granularity.
  • The primary metric is strict point-in-box accuracy in the original pixel frame, complemented by a loose 2×-diagonal metric and bootstrap 95% confidence intervals.The 0.50 reference is a forced-choice benchmark reference, not the prediction task’s chance level because coordinates are unconstrained.

M Details of CoT and Activation Steering: Negative Results

CoT and activation steering did not significantly improve performance, while candidate marking raised selection accuracy by bypassing the spatial grounding skill. The broader results indicate that failures are concentrated in candidate localization and relation binding rather than simple instruction parsing.

  • Negative intervention results: Δ = +0.5 points for primitive-aware CoT on Qwen2.5-VL-7B was not significant.The model could name the relation during reasoning but still emitted the wrong coordinate.
  • Negative intervention results: Δ = +1.5 points for activation steering on Qwen2.5-VL-7B was not significant.No tested layer and α configuration crossed the Holm-adjusted significance threshold.
  • Overall benchmark performance: All models remained far below the 96.9% annotator target-selection accuracy, with the strongest model reaching 31–32% strict accuracy.The strongest system, Claude Opus 4.7, reached 31% on the human-clean subset and 31–32% overall.
  • Per-primitive failure structure: Within-region selection favored the target at 0.89–0.90 for horizontal position, vertical position, and proximity, but containment and occlusion were indistinguishable from 0.50.Most below-reference point-in-box accuracy came from predictions outside both candidate regions, occurring on 60–92% of predictions.
  • Downstream association: GUI-Primitives accuracy correlated with ScreenSpot-Pro accuracy at Spearman ρ = +0.736 across ten models.The association held in both real-screenshot and synthetic subsets, but the authors claim it conservatively as a model-level association.

Q Model-Specific Sensitivity Analysis

Sensitivity analyses identify directional, target-size, infrastructure, and candidate-localization effects. Left-side bias is significant, while real-screenshot failures are dominated by predictions outside both candidate regions rather than incorrect relational selection.

  • Directional asymmetry: 15/19 models scored higher on “left of” than “right of” items, with a mean gap of +0.169 accuracy points.The asymmetry was significant (t = 5.6, p < .001) and appeared across proprietary and strongest open models.
  • Directional asymmetry: The above–below preference was weaker and not significant, with mean Δ = +0.047 and p = .10.OS-Atlas was an outlier with a strong above preference of Δ = −0.28.
  • Target-size sensitivity: Large targets reached mean accuracy 0.092 versus 0.022 for tiny targets when pooled across models and primitives.The effect was largest for containment and list-ordinal and smallest for alignment and proximity.
  • Real-screenshot localization: 96.2% of real-screenshot predictions fell outside both candidate regions, while pooled within-region selection was 0.747.For GPT-5 and Claude Opus 4.7, 87.8% fell outside both regions and within-region selection was 0.914.
  • Human comparison: Within-region selection for the strongest models was 0.95–0.97, comparable to human accuracy of 0.969.Their lower overall accuracy primarily reflected that only approximately one third of predictions entered either candidate region.
  • Shortcut controls: Removing the screenshot reduced accuracy by 15.5 points, shuffling it reduced accuracy by 7.2 points, and heavy blur reduced accuracy by 5.3 points.The blur result suggests reliance on global layout, but blur also degrades icons and boundaries, so it does not isolate OCR from cross-modal binding.

U Error Analysis

The error analysis separates relational composition failures from fallback and infrastructure failures. Anchor collapse, twin-identical predictions, and off-candidate clicks recur across models, while some interventions can worsen already-solved primitives.

  • Composition failures: Anchor-collapse occurs when a model locates the anchor but clicks it instead of the relational neighbor.It affected 28% of failed binary-primitive items on Opus 4.7 and 21% on GPT-5.
  • Composition failures: Twin-identical predictions show that models emit essentially the same click for both members of a contrastive pair.This relation-ignored signature pushes pair-consistency below accuracy and affected 37% of paired failures on Qwen2.5-VL, 25% on Opus 4.7, and 24% on GPT-5 after rescaling.
  • Fallback failures: Constant-default errors are model-side fallback behavior, including Llama-3.2-11B-Vision emitting (0, 0) on 46% of diagnostic items.The raw output is a well-formed click command, so this category is distinct from parser failure.
  • Intervention-design failures: SoM on Qwen2.5-VL’s list-ordinal regressed by 32.4 points because it converted an already-solved counting task into mark selection.The result supports applying training-free interventions selectively rather than uniformly.
  • Candidate localization: Candidate-level analysis found that below-reference point-in-box accuracy primarily reflected predictions outside both candidates, not systematic distractor selection.The target was selected at least as often as the distractor on every primitive.
  • Error taxonomy: Coordinate-frame errors came from downsampled API coordinates, while composition-side errors involved detected elements not being combined with the relation word.The paper treats coordinate-frame and fallback errors as infrastructure failures rather than spatial-reasoning failures.

V Limitations

The benchmark’s conclusions are bounded by synthetic-only primitives, limited downstream validation, intervention scope, coarse metrics, and changing proprietary APIs. These constraints restrict generalization and causal interpretation.

  • Data scope: Containment and occlusion are synthetic-only because real GUI datasets lack scalable parent–child and overlay–target metadata.Their failure rates and absent relation-word signal may not transfer directly to uncurated real-world GUIs.
  • Inference scope: Per-primitive regression coefficients are small after model fixed effects, so the paper claims only a joint model-level association.The authors explicitly do not interpret individual primitive coefficients causally.
  • Downstream scope: The downstream validation stops at ScreenSpot-Pro rather than testing end-to-end OSWorld task success.An OSWorld grounding slice is left for future work.
  • Task scope: GUI-Primitives v1 covers English instructions and click targets, leaving drag prediction, bidirectional and CJK layouts, and OSWorld pre-action grounding for future versions.The stated action and language scope bounds direct application beyond these settings.
  • Intervention scope: SoM gains reflect symbolic mark-selection scaffolding rather than repair of the underlying perception-and-binding problem.The setup uses at most two candidate marks, and deployment would inherit proposal errors from an upstream component.
  • Metric scope: Point-in-box accuracy treats a near miss and a click on the wrong element identically, leaving structured spatial-error measurement for future work.The loose metric partially separates imprecision from misidentification but does not capture drift, side-of-screen errors, or anchor collapse.
  • Reproducibility scope: Proprietary API behavior may change through downsampling, decoding, or model-identity updates, so future runs may differ.The authors freeze dated snapshots and pin model identifiers in an appendix.
Loading 2608.21832v1…