Source-linked AI summary
FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection
Mingyu Ouyang, Kevin Qinghong Lin, Mike Zheng Shou, Hwee Tou Ng
TL;DR
High-resolution UI screenshots create severe token redundancy and make naïve visual-token pruning especially damaging to positional information. FocusUI combines instruction-aware saliency selection with PosPad, and across four benchmarks it reports strong accuracy-efficiency trade-offs. Its stated scope is spatial visual-token reduction, leaving temporal multi-round UI interaction for future work.
Problem
High-resolution UI screenshots contain many redundant visual tokens, while naïve pruning breaks positional continuity and causes severe accuracy drops in precise grounding.
Method
FocusUI fuses instruction-conditioned bounding-box overlap with a UI-graph prior for saliency selection and uses PosPad markers to preserve positional continuity.
Results
Across four UI grounding benchmarks, FocusUI models outperform GUI-specific baselines at the same size while maintaining strong accuracy-efficiency trade-offs.
Takeaways & Limitations
FocusUI provides an efficient UI grounding framework that reduces visual tokens while preserving positional continuity for precise localization.
Takeaways & Limitations
FocusUI primarily gains efficiency from spatial visual token reduction; temporal multi-round and sequential UI actions remain future work.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) have shown remarkable performance in User Interface (UI) grounding tasks, driven by their ability to process increasingly high-resolution screenshots. However, screenshots are tokenized into thousands of visual tokens (e.g., about 4700 for 2K resolution), incurring significant computational overhead and diluting attention. In contrast, humans typically focus on regions of interest when interacting with UI. In this work, we pioneer the task of efficient UI grounding. Guided by practical analysis of the task's characteristics and challenges, we propose FocusUI, an efficient UI grounding framework that selects patches most relevant to the instruction while preserving positional continuity for precise grounding. FocusUI addresses two key challenges: (1) Eliminating redundant tokens in visual encoding. We construct patch-level supervision by fusing an instruction-conditioned score with a rule-based UI-graph score that down-weights large homogeneous regions to select distinct and instruction-relevant visual tokens. (2) Preserving positional continuity during visual token selection. We find that general visual token pruning methods suffer from severe accuracy degradation on UI grounding tasks due to broken positional information. We introduce a novel PosPad strategy, which compresses each contiguous sequence of dropped visual tokens into a single special marker placed at the sequence's last index to preserve positional continuity. Comprehensive experiments on four grounding benchmarks demonstrate that FocusUI surpasses GUI-specific baselines. On the ScreenSpot-Pro benchmark, FocusUI-7B achieves a performance improvement of 3.7% over GUI-Actor-7B. Even with only 30% visual token retention, FocusUI-7B drops by only 3.2% while achieving up to 1.44x faster inference and 17% lower peak GPU memory.
1. Introduction
FOCUSUI targets efficient UI grounding by selecting instruction-relevant visual tokens while preserving positional continuity for precise localization. It combines dense saliency supervision with position-preserving token selection and reports strong accuracy-efficiency trade-offs across four benchmarks.
- Motivation: High-resolution UI screenshots create substantial computational overhead because thousands of visual tokens dominate the sequence budget.Naïve visual token pruning also breaks positional continuity and causes severe accuracy drops on precise UI grounding tasks.
- Framework: FOCUSUI selects instruction-relevant visual tokens while preserving positional continuity needed for precise localization.The framework is presented as an efficient UI grounding approach addressing visual redundancy and positional disruption.
- Token selection: Its Query-Guided Saliency Scorer uses fused supervision from instruction-conditioned bounding-box overlap and a rule-based UI-graph prior.The prior down-weights large homogeneous regions to help filter irrelevant tokens.
- Position preservation: POSPAD preserves sequence continuity by compacting each dropped contiguous token sequence into a learnable marker at the sequence’s last index.This transformation addresses the positional-information failure of general pruning methods on precise UI grounding tasks.
- Results: Across four UI grounding benchmarks, FOCUSUI models outperform previous state-of-the-art models and show favorable accuracy-efficiency trade-offs.The framework is implemented with Qwen2.5-VL and Qwen3-VL backbones of multiple sizes.
2. Efficient UI Grounding: Task Characteristics and Challenges
Efficient UI grounding is challenging because high-resolution, structured screenshots contain severe visual-token skew and because naïve pruning disrupts spatial position information. FOCUSUI addresses these challenges by selecting instruction-relevant patches and preserving continuity across dropped-token sequences.
- Task characteristics: UI screenshots are high-resolution, compositionally structured, and dominated by large homogeneous panes interspersed with small widgets.These characteristics make UI grounding different from natural visual understanding.
- Task characteristics: ≥85.4% of tokens are visual screenshot tokens across two benchmarks and two grounding models, confirming severe visual-token imbalance.Study 1 links this imbalance to significant computational overhead.
- Redundancy: Instruction-aware selection prioritizes patches relevant to the instruction and de-emphasizes visually repetitive regions.The Instruction-to-Patch saliency score fuses ground-truth bounding-box overlap with a UI-graph prior that down-weights large connected components.
- Position sensitivity: Direct pruning creates positional jumps in M-RoPE’s height and width dimensions, producing pronounced localization offsets on fine-grained targets.Precise UI grounding is therefore sensitive to the positional information of visual embeddings.
- Position sensitivity: POSPAD replaces each contiguous dropped-token sequence with one learnable marker at its last index, inheriting that index’s spatial position.The strategy preserves positional continuity and mitigates disruption to the model’s spatial understanding.
3. FOCUSUI
FOCUSUI selects instruction-relevant visual tokens through fused patch-level saliency supervision and preserves positional continuity with POSPAD during compression.
- Instruction-to-Patch Saliency Score: FOCUSUI fuses bounding-box overlap with a UI-graph prior to identify instruction-relevant patches while down-weighting large homogeneous regions.The bounding-box score measures patch overlap with the ground-truth box, while the UI-graph score assigns lower weights to large connected components.
- Instruction-to-Patch Saliency Score: The UI-graph score groups RGB-similar 4-neighborhood patches with union-find, then assigns each component an inverse logarithmic size weight.This suppresses visually repetitive backgrounds and enhances distinctive elements.
- Query-Guided Saliency Scorer: A Query-Guided Saliency Scorer predicts per-patch relevance from similarities between patch embeddings and instruction text embeddings, then selects the top-K patches.The retention ratio r determines K = floor(rM), and the remaining patches are dropped.
- POSPAD: Positional Continuity Preservation: POSPAD replaces each contiguous sequence of dropped tokens with one learnable marker at that sequence’s last index, preserving positional continuity while shortening the visual sequence.Each dropped sequence reduces length by |Ru| − 1, and the resulting sequence remains compatible with common M-RoPE implementations without downstream architecture changes.
- Efficient UI Grounding Framework: FOCUSUI integrates token selection before the language-model decoder, producing a compact sequence while retaining the original decoder architecture.The framework is applied to Qwen2.5-VL and Qwen3-VL models, with deep visual embeddings gathered only for kept image tokens in Qwen3-VL.
4. Experiments
FOCUSUI is evaluated across four UI grounding benchmarks, retention ratios, model backbones, pruning baselines, efficiency settings, qualitative examples, and ablations. Results show strong accuracy-efficiency trade-offs, with position-preserving selection retaining performance while reducing tokens.
- Performance: FOCUSUI achieves state-of-the-art grounding performance across ScreenSpot-V2, ScreenSpot-Pro, OS-World-G, and UI-Vision at 30–50% visual-token retention.The experiments compare multiple retention ratios against dense baselines and GUI-specific methods of the same model size.
- Comparison to General Pruning Methods: At 30% retention, FOCUSUI remains within 0.5, 3.2, and 2.5 points of baseline accuracy on ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G, respectively.General visual-token pruning methods show severe degradation under the same setting.
- Efficiency Analysis: Reducing retention from 100% to 30% yields up to 1.44× faster inference and about 17–18% lower peak memory with only 3.2-point accuracy loss.The efficiency analysis covers Qwen2.5-VL and Qwen3-VL backbones on ScreenSpot-Pro.
- Qualitative Results: Qualitative heatmaps show that FOCUSUI selects instruction-relevant visual tokens while suppressing background regions at a 30% retention ratio.Examples span web, desktop, and mobile interfaces from ScreenSpot-V2 and ScreenSpot-Pro.
- Ablation Study: Removing either the UI-graph prior or bounding-box overlap supervision degrades accuracy, while lower retention produces a smooth accuracy-retention trade-off.Full retention matches the dense baseline, 50% retains most performance, and more aggressive settings incur larger drops.
5. Related Work
Related work covers VLM-powered GUI agents, UI visual grounding, and visual token reduction. These lines of research establish the surrounding progress in interface interaction, grounding, and efficiency-oriented token compression.
- VLM-Powered GUI Agents: VLM-powered GUI agents have progressed from text-dependent systems toward visual solutions for planning, element grounding, and interface control.The transition increasingly exploits VLM capabilities for graphical interfaces.
- UI Visual Grounding: UI visual grounding locates an interaction target from a screenshot and natural-language instruction, with performance improving through model and data scaling.Recent work includes more advanced model designs and larger datasets.
- Visual Token Reduction: Visual token reduction methods exploit redundancy and importance variation through training-free pruning or encoder-side compression to lower computation cost.These methods address redundancy in visual tokens, especially relative to information-dense text.
6. Conclusion
The paper introduces FOCUSUI as a query-guided framework that selects instruction-relevant visual tokens while preserving positional continuity. Across four UI grounding benchmarks, it reports strong accuracy-efficiency trade-offs, while identifying temporal interaction as future work.
- Conclusion: FOCUSUI combines instruction-relevant visual-token selection with positional-continuity preservation for efficient UI grounding.The framework is integrated with state-of-the-art VLMs.
- Limitations and Future Work: FOCUSUI primarily gains efficiency through spatial visual-token reduction, leaving temporal multi-round and sequential UI interactions for future work.The stated future direction extends the method beyond spatial reduction.
A.1. Training Data
FOCUSUI is trained on filtered public GUI datasets assembled according to GUI-Actor. Overlap with downstream test sets is excluded, and low-quality annotations are removed before training.
- Training Data: The training set compiles several public high-quality GUI datasets while excluding Wave-UI samples overlapping downstream test sets.This construction follows GUI-Actor and is intended to support fair evaluation.
- Training Data: Filtering samples with ground-truth-to-OmniParser box IoU below 0.3 reduces the number of elements by 22.9%.OmniParser V2 is used to refine annotation quality.
- Training Data: The training-dataset statistics are reported in Table 8.The passage identifies the table containing the final training statistics.
A.2. Training Details
FOCUSUI is trained in two stages and evaluated on four public UI-grounding benchmarks using reproducible training configurations.
- Training setup: FOCUSUI training uses 8 NVIDIA H200 GPUs, bfloat16 precision, DeepSpeed ZeRO-2, FlashAttention-2, and an effective batch size of 32 per GPU.The maximum image size is set to 5,720,064 pixels to match GUI-Actor.
- Training setup: Training begins with one epoch of Query-Guided Saliency Scorer pre-training at a 1e-4 learning rate, taking about 12 hours for both 3B and 7B models.
- Reproducibility: The final checkpoint is used for all FOCUSUI models, with full Weights & Biases logs provided for reproducibility.FOCUSUI-7B training loss and evaluation curves are shown in Figure 6.
- Evaluation benchmarks: Evaluation covers ScreenSpot-V2, ScreenSpot-Pro, OS-World-G, and UI-Vision, with benchmark overviews reported in Table 9.The benchmarks pair screenshots with instructions for UI grounding evaluation.
B.1. Visual Redundancy Analysis
Visual tokens dominate UI-grounding sequences, motivating analyses of instruction-relevant patch selection and coverage under reduced token budgets.
- Visual token skew: Visual tokens occupy at least 84.3% of the sequence across the studied benchmarks under default ScreenSpot-Pro evaluation settings.This token distribution confirms substantial visual redundancy in UI grounding tasks.
- Pruning comparison: Study 2 compares FOCUSUI with UI-grounding models using advanced visual token pruning methods.The detailed comparison appears in Table 11.
- Patch selection: Patch Recall@K% measures the fraction of ground-truth regions captured among the top K% saliency-ranked patches.The evaluation uses K values of 5%, 10%, 25%, and 50%.
- Patch selection: Full Coverage Budget reports the percentage of visual tokens needed to fully cover the ground-truth elements.Patch Recall and Full Coverage Budget results are reported on ScreenSpot-Pro in Table 12.
C.2. Analysis of POSPAD
The POSPAD analysis compares marker placement within dropped-token sequences and examines grounding accuracy, saliency behavior, and prompt formats across evaluation settings.
- POSPAD placement: POSPAD placement is compared at the beginning, middle, and end of each contiguous dropped-token sequence.The proposed sequence-end placement is evaluated against two alternatives.
- POSPAD placement: Across retention ratios, placing POSPAD at the end of each dropped sequence achieves the best performance, especially at low retention ratios.The authors relate this result to raster-scan ordering used by the vision encoder and M-RoPE.
- Accuracy under reduction: FOCUSUI maintains high UI-grounding accuracy across varying visual token reduction ratios on ScreenSpot-V2 and ScreenSpot-Pro.The corresponding performance curves are shown in Figure 7.
- Saliency behavior: The Query-Guided Saliency Scorer highlights instruction-relevant regions and suppresses irrelevant background in qualitative examples.For straightforward tasks, saliency peaks at ground-truth locations; for complex cases, it distinguishes potential targets from irrelevant background.
- Prompt templates: FOCUSUI prompt templates instruct GUI agents to locate screen elements corresponding to human instructions and output coordinate-based actions.Different model prompts specify either coordinate output or tool-mediated interaction formats.
- Prompt templates: The Qwen2.5-VL and Qwen3-VL evaluation prompts require the agent to output one point in the format (x, y) without external tools.The supplied prompt variants identify the evaluated model families.