PaperlayerView exact revision on arXiv

Source-linked AI summary

PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control

Jingxuan Wei, Xi Bai, Shan Liu, Caijun Jia, Zheng Sun, Xinglong Xu, Siyuan Li, Linzhuang Sun, Bihui Yu, Conghui He, Cheng Tan

arXiv:2605.15963v1cs.AI

TL;DR

Region-tolerant GUI agents leave unresolved whether they can reliably construct geometry when actions must hit precise points in continuous visual space. This paper introduces PAGE Bench and PAGER, whose topology-aware approach achieves 4.1× higher task success than the strongest general baseline.

  • Problem

    Existing GUI agents mainly target region-tolerant components, leaving point-precise operation in continuous visual space insufficiently addressed.

  • Method

    The paper introduces PAGE Bench and PAGER, combining dependency-structured planning with pixel-level execution, pixel-grounded tuning, and precision-aligned reinforcement learning.

  • Results

    PAGER achieves 4.1× higher task success than Gemini-3.1-Pro, indicating stronger trajectory-level stability and final geometric quality.

  • Takeaways & Limitations

    Precision-sensitive GUI control requires converting correct operation semantics into state-conditioned parameter control across dependent construction steps.

Abstract

from arXiv · show

Large vision-language models have significantly advanced GUI agents, enabling executable interaction across web, mobile, and desktop interfaces. Yet these gains largely rely on a forgiving region-tolerant paradigm, where many nearby pixels inside the same component remain valid. Precise geometric construction breaks this assumption: actions must land on points in continuous canvas space rather than tolerant regions. Because geometric primitives carry ontological dependencies, a local coordinate error can induce cascading topological failures that distort downstream objects and invalidate the final construction. We identify this regime as precision-sensitive GUI tasks, requiring point-level accuracy, geometry-aware verification, and robustness to dependency-driven error propagation. To benchmark it, we introduce PAGE Bench, with 4,906 problems and over 224K process-supervised, pixel-level GUI actions. We further propose PAGER, a topology-aware agent that decomposes construction into dependency-structured planning and pixel-level execution. Pixel-grounded supervised tuning establishes executable action grammar, while precision-aligned reinforcement learning mitigates rollout-induced exposure bias through state-conditioned geometric feedback. Experiments reveal a pronounced Semantic-Execution Gap: general multimodal models can exceed 88% action type accuracy yet remain below 6% task success. PAGER closes this gap, delivering 4.1x higher task success than the strongest evaluated general baseline and raising step success rate from below 9% for GUI-specialized agents to over 62%, establishing a new state of the art for point-precise GUI control.

1 Introduction

Precision-sensitive GUI tasks require point-level accuracy on continuous canvases, where small coordinate errors can propagate through dependency-coupled geometric constructions. PAGER addresses this regime with dependency-structured planning, pixel-level execution, and training that aligns action generation with geometric precision.

  • Task setting: Precision-sensitive GUI tasks replace region-tolerant component selection with point-level manipulation on a continuous canvas.Success depends on landing within a small pixel tolerance rather than merely selecting the correct interface region.
  • Task setting: Small coordinate errors propagate through dependent lines, circles, intersections, angles, and polygons, potentially invalidating the construction.Geometric operations are coupled through dependencies, so local mistakes affect downstream objects.
  • PAGER: PAGER factorizes construction into dependency-structured planning and pixel-level execution conditioned on the current canvas state.Its planner induces a construction graph and topologically valid sub-task order, while its executor grounds each sub-task in concrete GUI actions.
  • PAGER: Pixel-grounded supervised tuning establishes executable action grammar, while precision-aligned training targets accurate continuous-space control.Ablations identify pixel-grounded SFT as the execution prior and parameter-accuracy rewards as drivers of continuous-space control.
  • Benchmark: PAGE Bench evaluates precise geometric construction with process-supervised trajectories, pixel-level annotations, and process-level and final-result metrics.The benchmark is introduced as a dedicated evaluation of precision-sensitive GUI tasks.

2 Related Work

Related work spans GUI agents that translate multimodal instructions into executable actions across computing environments and geometric reasoning research that evaluates diagram understanding and mathematical problem solving. GUI systems increasingly combine perception, decomposition, reasoning, and native end-to-end execution, while geometric benchmarks broaden across tasks and dimensional settings.

  • GUI Agents: GUI agents map multimodal instructions to executable actions across web, mobile, desktop, and broader computer-use environments.This research line studies executable interaction across multiple interface settings.
  • GUI Agents: Perceptual-action abstractions include CogAgent’s high-resolution interface understanding and CoCo-Agent’s environment perception with conditional decomposition for mobile action prediction.These systems connect interface perception with structured action prediction.
  • GUI Agents: Recent GUI systems such as UI-TARS and GUI-Libra move toward native end-to-end execution with reasoning-aware action generation.The passage identifies these systems as part of the field’s shift toward integrated execution.
  • Geometric Reasoning: Geometric reasoning research examines diagram interpretation, principle identification and application, and mathematically valid solutions from multimodal inputs.Diagnostic benchmarks analyze failures in principle identification, application, perception, planning, theorem use, and reflection.
  • Geometric Reasoning: Evaluations broaden geometric reasoning beyond plane geometry to 3D settings, larger diagram-based problem spaces, and visually aided mathematical reasoning.The passage describes this as a subsequent expansion of benchmark scope.

3 Methodology

PAGER addresses precision-sensitive geometric GUI drawing by modeling point-level success and dependency-coupled error propagation, then factorizing construction into dependency-consistent planning and pixel-grounded execution. Pixel-precise supervision and geometric-reward reinforcement learning align action generation with executable behavior and rendered construction validity.

  • Problem Formulation: Precision-sensitive drawing requires executed pixels to fall within an ε-distance of reference points rather than merely inside valid target regions.The framework contrasts point-level success with region-tolerant interaction using Succpt(a) = I[∥p(a) − p∗∥2 ≤ ϵ].
  • Problem Formulation: Small coordinate deviations can perturb downstream objects because geometric construction dependencies couple parameter errors to canvas changes.The dependency structure is represented through Jℓ and Bℓ, which capture construction dependencies and map parameter errors to canvas perturbations.
  • Planning and Execution: PAGER’s Planning Module builds a construction graph and dependency-consistent sub-task list, while its Task Execution Module grounds each sub-task into pixel-level GUI actions.Graph nodes represent primitives or relations, graph edges encode dependencies, and execution instantiates step specifications with coordinates, geometric parameters, style, and label position.
  • Pixel-Precise Supervision: Pixel-Precise Data Construction supplies trajectories with sub-tasks, screenshots, histories, next actions, feedback, and spatial annotations for supervised training.Geometric coordinates are projected into visible-window pixels, binding anchors for points, lines, circles, arcs, polygons, and labels to pixel targets.
  • Pixel-Precise Supervision: SFT learns executable action grammar and state-conditioned prediction, while precision-aware rollout training addresses the mismatch between reference screenshots during training and self-generated screenshots at inference.This mismatch motivates training on rollouts rather than relying only on teacher-forced reference states.
  • Precision Optimization: RL scores actions by operation-type correctness, typed-parameter accuracy, and rendered geometric validity, while a KL anchor preserves executable behavior learned by SFT.The geometric terms assess anchors, relations, layout, region validity, text consistency, and pixel deviation; the admissible action set is used only during training.

4 Dataset

PAGE Bench is built as a closed, verified GeoGebra execution loop that converts constructible geometry problems into structured, replayable GUI trajectories. It contains 4,906 problems and 224,497 pixel-level GUI actions spanning substantial dependency chains and continuous-canvas operations.

  • Problem collection and executable screening: PAGE Bench screens public K–12 geometry problems for GeoGebra-executable constructions, then removes under-specified, non-constructive, or non-operational instances through manual verification.The pipeline retains only instances that remain valid after execution and verification.
  • Structured task generation and standardization: Each retained problem becomes an ordered function+args task sequence, which standardization parses, repairs, and normalizes with aligned metadata.This structured intermediate representation preserves intended geometric dependencies.
  • Execution mapping and environment-grounded reconstruction: Abstract construction steps are mapped in live GeoGebra to tool selections and parameterized canvas manipulations, yielding executable click, paint, and type actions.The resulting interactions form replayable interface trajectories.
  • Benchmark composition: 4,906 problems are split into 4,443 training and 463 test instances, comprising 2,049 multiple-choice and 2,857 open-ended problems.Open-ended instances account for 58.23% of the benchmark.
  • Process scale: 224,497 GUI actions span 53,277 high-level tasks, averaging 10.86 tasks and 45.76 actions per problem; click and paint comprise 88.03% of actions.The trajectory lengths create dependency chains, while paint requires continuous-canvas control.

5 Experiment

PAGER achieves the strongest overall performance on precise geometric GUI tasks, substantially outperforming general VLMs and GUI-specialized agents. Ablations and qualitative analyses attribute these gains to pixel-grounded supervision, parameter-aligned feedback, and robust point-level execution.

  • Main results: 29.52 Overall is PAGER’s best score, exceeding Gemini-3.1-Pro by 5.15 points, or 21.1%.PAGER also leads Task, Middle, and Final scores despite Gemini-3.1-Pro leading Param and Step.
  • Semantic-execution gap: 23.78 Task Success is PAGER’s result versus 5.82 for Gemini-3.1-Pro, approximately 4.1× higher despite weaker action semantics in general VLMs.Claude-Sonnet-4.6 reaches 95.85 Action Accuracy, while GPT-5.4 and Gemini-3.1-Pro reach 88.04 and 89.18, but their Task Success remains 1.11, 0.56, and 5.82, respectively.
  • GUI-specialized baselines: 62.20 Step Success is PAGER’s result, compared with below 9% for UI-TARS and OS-ATLAS and 16.18 for the strongest GUI-agent baseline.The comparison indicates that region-tolerant, component-level grounding is too coarse when exact points determine geometric validity.
  • Ablation study: 20.47 Overall is achieved by PAGER-SFT, alongside 48.47 Parameter Accuracy and 47.91 Step Success, demonstrating the value of pixel-grounded process supervision.Removing the parameter-accuracy reward reduces Overall from 20.47 to 20.07, identifying parameter alignment as the key bottleneck.
  • Qualitative and human evaluation: r = 0.9397 shows near-perfect correlation between automated evaluation and human judgment, with PAGER achieving high automated success and superior human preference.Qualitatively, PAGER preserves rectangular structure, diagonal intersection, and coordinate consistency, while competing models exhibit distortions, omissions, or propagated vertex errors.

6 Conclusion and Limitations

The paper frames precision-sensitive GUI tasks as point-precise control in continuous visual space, where dependency-coupled geometric errors can invalidate downstream structures. PAGE Bench measures this failure mode, while PAGER addresses it through dependency-structured planning.

  • Precision-sensitive GUI tasks shift success from region-tolerant component selection to point-precise control in continuous visual space.This regime is exposed by precise geometric construction.
  • Small coordinate errors can cascade through dependency-coupled primitives, producing invalid downstream structures.
  • PAGE Bench makes this failure mode measurable through process-supervised, pixel-level geometric interactions.
  • PAGER addresses precision-sensitive GUI failures through dependency-structured planning.

A Performance Results with Radar Chart Analysis

PAGER is evaluated against fourteen baselines on PAGE Bench, revealing a pronounced Semantic-Execution Gap: strong semantic planning does not translate into comparable final results.

  • Evaluation Setup: PAGER is compared with fourteen baselines across three categories on precision-sensitive GUI tasks, as summarized in Figure 7.The evaluation uses PAGE Bench.
  • Semantic-Execution Gap: 32.4 Middle Process score is achieved by Gemini-3.1-Pro, while GPT-5.4 reaches 27.4.Both are state-of-the-art closed models evaluated for high-level semantic planning.
  • Semantic-Execution Gap: 11.8 Final Result score is recorded for GPT-5.4, sharply below its 27.4 Middle Process score.The passage identifies this drop as evidence of the Semantic-Execution Gap.

B Fine-Grained Classification Results Analysis · C GeoGebra Screening Prompt · D GeoGebra Automation Prompt

The section analyzes ten geometric capabilities, showing that PAGER is strongest on basic semantic-to-tool grounding while dependency-heavy and complex scenarios expose severe weaknesses in existing models. It also specifies GeoGebra prompts for screening suitable K12 problems and generating structured, executable construction tasks.

  • B Fine-Grained Classification Results Analysis: Ten geometric capabilities reveal severe performance variance across architectures in continuous spatial reasoning and ontological dependencies.The evaluation decomposes precision-sensitive GUI performance into ten distinct geometric dimensions.
  • B Fine-Grained Classification Results Analysis: 61.12 and 59.52 are PAGER’s scores for natural language to geometric tool mapping and basic geometric object construction, respectively.These capabilities are described as fundamental prerequisites, while semantic grounding to specific drawing tools remains relatively manageable.
  • B Fine-Grained Classification Results Analysis: 34.68 is GPT-5.4’s sequential-planning score, while Qwen3-VL-8B falls below 23 as tasks require multi-step dependencies, geometric relations, and structural constraints.Small early coordinate deviations can accumulate into cascading topological failures.
  • B Fine-Grained Classification Results Analysis: Scores near 10 mark UI-TARS and OS-ATLAS failures on auxiliary elements, implicit object introduction, composite management, and real-world geometric modeling, whereas PAGER adapts robustly.These scenarios require abstract spatial reasoning and strict geometry-aware verification.
  • C GeoGebra Screening Prompt: The K12-GeoGebra screening prompt identifies geometry questions suitable for GeoGebra visualization and excludes pure algebraic calculations or logical reasoning.It accepts batches containing question stems, options, and solution steps, separated by blank lines, and returns structured JSON decisions.
  • D GeoGebra Automation Prompt: The automation prompt converts mathematical construction descriptions, answer hints, and image context into specific Python calls from a strictly defined GeoGebra API library.It also categorizes mathematical skills, grade level, and drawing difficulty.

E Dataset Quality Assurance Prompt

The dataset quality-assurance prompt evaluates GeoGebra entries by comparing their problem content, image, generated canvas, constructions, and executed tasks. It produces three 1–5 scores and a comprehensive filtering result, with checks including exact label consistency and positional accuracy.

  • Review Inputs: The reviewer inspects each entry’s question, answer, image, and sequence of GeoGebra function calls.The image may contain geometry, function graphs, or coordinate systems relevant to the problem.
  • Task Checks: Text labels must match task specifications exactly, including case, mathematical symbols, brackets, spaces, and units.For add_text_label tasks, the evaluator also checks whether the anchor point is near the specified coordinates and associated with the intended location.
  • Scoring Tasks: Three scores, each from 1 to 5, plus one comprehensive result summarize image, canvas, construction, and execution quality.The prompt evaluates the original image, generated canvas, auxiliary constructions, and task execution before filtering dataset entries.

F Metrics

The evaluation uses a four-stage protocol to assess geometric generative reasoning across action execution, final geometric outcomes, stratified analysis, and an overall score.

  • F Metrics: The four-stage protocol evaluates action execution, final geometric results, stratified performance, and a unified overall score.It is designed to comprehensively measure a model’s geometric generative reasoning capability.

F.1 Middle Process Metrics (Rule-Based)

The section defines four rule-based metrics for evaluating predicted action sequences against ground-truth annotations, from action-type and parameter correctness to step- and task-level success. These metrics are macro-averaged across evaluation IDs, with Middle Process Score weighting task success most heavily.

  • Metric definitions: Four rule-based metrics evaluate predicted action logs against ground-truth action sequences: Action Type Accuracy, Parameter Accuracy, Step Success Rate, and Task Success Rate.They assess action categories, parameters, simultaneous per-step correctness, and all-step task completion, respectively.
  • Metric definitions: Parameter Accuracy requires case-insensitive string equality for type actions, bounding-box inclusion for clicks, and 5-pixel Euclidean tolerance for paint actions.Paint correctness is evaluated on the 1280 × 720 screen.
  • Metric definitions: Task Success Rate is binary per task and requires every step to be simultaneously successful, making it stricter than Step Success Rate.Step Success Rate requires both action type and parameter correctness at each step.
  • Aggregation: All four metrics are computed per task, averaged within each evaluation ID, and macro-averaged across IDs for the model-level score.This aggregation produces evaluation-ID-balanced model scores.
  • Middle Process Score: MPS = 0.6 · TSR + 0.2 · SSR + 0.1 · PA + 0.1 · AA, weighting task-level success most heavily and action-type accuracy as a lower-level sanity check.The weighting reflects the hierarchy of difficulty across task, step, parameter, and action-type criteria.

F.2 Final Result Metrics · F.3 Overall Score

The paper evaluates final geometric outputs using symbolic completion metrics and holistic VLM-based scores, then combines final-result quality with procedural performance into a unified Overall Score. This framework balances geometric correctness, visual fidelity, and action-sequence correctness across models.

  • F.2 Final Result Metrics: Final-output quality is evaluated from two complementary perspectives: symbolic geometric completion and holistic visual assessment.These perspectives target both structured construction correctness and properties that resist purely symbolic evaluation.
  • F.2 Final Result Metrics: OTC computes geometric similarity by matching each model point to its closest ground-truth point with an exponential-kernel contribution.The ground-truth and model constructions contain labeled points with coordinates and construction-command lists.
  • F.2 Final Result Metrics: The point-matching tolerance is τ = 0.5, while command correctness is checked through mapped inputs and signature matches against ground-truth commands.Command signatures use the command name and sorted inputs after applying point correspondence.
  • F.2 Final Result Metrics: OTC combines point placement and geometric structure, assigning greater weight to geometric structure than to point placement.This makes the completion metric sensitive to construction organization as well as coordinate proximity.
  • F.2 Final Result Metrics: Gemini-2.5-Pro scores outputs in [0, 1] using task data, JSON constructions, and rendered images under data-logic and visual-presentation views.The evaluator captures rendering fidelity, label legibility, and cross-modal consistency.
  • F.2 Final Result Metrics: The holistic metrics are TC for element presence, VS for pixel-level rendering fidelity, and GL for mathematical geometric correctness.GL also detects visual hallucinations, while VS covers line thickness, color, label positioning, and coordinate scaling.
  • F.2 Final Result Metrics: FRS = 0.3 · OTC + 0.3 · TC + 0.2 · VS + 0.2 · GL combines symbolic completion with holistic task, visual, and geometric scores.The weighting gives OTC and TC larger contributions than VS and GL.
  • F.3 Overall Score: OS is the equally weighted average of MPS and FRS, balancing procedural action-sequence correctness against final geometric-output quality for unified model ranking.This scalar supports comparison across all evaluated models.
Loading 2605.15963v1…