PaperlayerView exact revision on arXiv

Source-linked AI summary

GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction

Hongxin Li, Yuntao Chen, Zhaoxiang Zhang

arXiv:2604.23941v1cs.CV

TL;DR

GUI element grounding is important for low-latency agents, but large VLMs are difficult to deploy on resource-constrained devices. GoClick combines a lightweight encoder-decoder model with Progressive Data Refinement, achieving strong grounding and device-cloud agent results while exposing deployment and data-curation limitations.

  • Problem

    Large GUI grounding VLMs, often exceeding 2.5B parameters, are difficult to run on resource-constrained mobile devices despite the need for low-latency grounding.

  • Method

    GoClick uses an encoder-decoder architecture and Progressive Data Refinement to train a 230M-parameter grounding model from a 3.8M-sample core set.

  • Results

    GoClick outperforms equal-sized models, compares favorably with larger models across grounding benchmarks, and improves step success rates in device-cloud GUI agents.

  • Takeaways & Limitations

    Lightweight grounding models can support practical GUI-agent deployment on resource-constrained devices within the evaluated device-cloud framework.

  • Takeaways & Limitations

    Evaluation used L20 GPUs rather than real embedded devices, so real-world memory, energy, and system-optimization challenges remain incompletely captured.

Abstract

from arXiv · show

Graphical User Interface (GUI) element grounding (precisely locating elements on screenshots based on natural language instructions) is fundamental for agents interacting with GUIs. Deploying this capability directly on resource-constrained devices like mobile phones is increasingly critical for GUI agents requiring low latency. However, this goal faces a significant challenge, as current visual grounding methods typically employ large vision-language model (VLM) (more than 2.5B parameters), making them impractical for on-device execution due to memory and computational constraints. To address this, this paper introduces GoClick, a lightweight GUI element grounding VLM with only 230M parameters that achieves excellent visual grounding accuracy, even on par with significantly larger models. Simply downsizing existing decoder-only VLMs is a straightforward way to design a lightweight model, but our experiments reveal that this approach yields suboptimal results. Instead, we select an encoder-decoder architecture, which outperforms decoder-only alternatives at small parameter scales for GUI grounding tasks. Additionally, the limited capacity of small VLMs encourages us to develop a Progressive Data Refinement pipeline that utilizes task type filtering and data ratio adjustment to extract a high-quality 3.8M-sample core set from a 10.8M raw dataset. Training GoClick using this core set brings notable grounding accuracy gains. Our experiments show that GoClick excels on multiple GUI element grounding benchmarks while maintaining a small size and high inference speed. GoClick also enhances GUI agent performance when integrated into a device-cloud collaboration framework, where GoClick helps cloud-based task planners perform precise element localization and achieve higher success rates. We hope our method serves as a meaningful exploration within the GUI agent community.

1 Introduction

GUI element grounding lets agents locate interface elements from natural-language instructions, but existing large VLMs are difficult to deploy on resource-constrained devices. GoClick addresses this with a 230M-parameter model, an encoder-decoder architecture, and refined training data.

  • Motivation: GUI element grounding enables agents to locate screenshot elements for actions such as clicking and mobile-app navigation.It is important for low-latency GUI agents, including assistance for visually impaired users and information collection.
  • Motivation: Models often exceeding 2.5B parameters create deployment challenges on mobile devices because of memory and computational constraints.This creates tension between grounding performance requirements and on-device deployment constraints.
  • GoClick: GoClick uses only 230M parameters while targeting grounding performance comparable to larger models and low latency.The model is introduced as a lightweight GUI element grounding expert for resource-constrained environments.
  • GoClick: An encoder-decoder architecture outperforms decoder-only alternatives at small parameter scales for GUI element grounding.Downsized 1B and 2B decoder-only models produced only moderate grounding performance, motivating the Florence-2 base architecture.
  • Training data: Progressive Data Refinement extracts a high-quality core set from a 10.8M raw dataset by filtering task types and adjusting task ratios.The pipeline removes outdated GUI patterns and harmful task samples before retaining useful training data.
  • Results: 3.8M refined samples train GoClick to outperform equal-sized grounding models and compare favorably with larger models across multiple benchmarks.The paper also reports a better speed-accuracy tradeoff and higher step success rates in a device-cloud collaboration agent.

2 Related Works

Vision-language research has progressed from encoder-decoder multimodal architectures to LLM-centered models and visually grounded GUI agents. However, many GUI-agent systems remain too large for mobile deployment, motivating lightweight grounding models.

  • Vision-Language Models: Vision-language models combine visual and textual inputs to support multimodal tasks such as visual question answering and image captioning.Early architectures include LXMERT, VL-BART, and Flamingo, while Florence-2 uses encoder and decoder blocks for textual outputs.
  • Vision-Language Models: LLM-centered VLMs inject visual information through projectors, prompts, or related feature-fusion mechanisms.LLaVA, LLaMA-Adapter, VisionLLM, Ferret, and Qwen-VL develop increasingly strong multimodal understanding and grounding abilities.
  • GUI Agents: These multimodal capabilities provide a route toward autonomous agents and motivate training element-grounding models from open-source VLMs.The paper targets improved GUI interaction through open-source visual-language grounding.
  • GUI Agents: Early GUI agents used textual GUI representations and proprietary LLM prompting, but were generally limited to narrow scenarios.Visual-observation approaches instead require pixel-level control in complex action spaces.
  • GUI Agents: Prompt-based systems such as MM-Navigator, AppAgent, SeeAct, and Mobile-Agent use GPT-4V reasoning or Set-of-Marks prompting, but simple prompt engineering is difficult to improve further.Other work trains open-source VLMs for GUI-agent tasks and combines proprietary task decomposition with open-source action execution.
  • Deployment gap: Many open-source GUI-agent models are at least 7B parameters and are not easily deployable on mobile devices.GoClick instead targets lightweight GUI element grounding, whereas OmniParser-v2 focuses on detection and captioning rather than grounding.

3.1 Preliminary

GUI element grounding maps a natural-language referring expression and screenshot to the target element’s coordinates. GUI agents then use such localization within complex, multi-step visual interactions under fine-grained and computational constraints.

  • GUI Element Grounding: GUI element grounding predicts a target element’s box or center coordinates from a screenshot and a natural-language referring expression.The task bridges user intentions and interface elements by mapping language to precise screen locations.
  • GUI Element Grounding: Referring expressions can describe spatial position, appearance, function, or combinations of these properties.Examples include a top-right button, a blue submit button, a login field, and a red close button in settings.
  • GUI Element Grounding: GUI grounding is harder than natural-scene grounding because interfaces contain high-resolution inputs and many small, visually similar elements.These conditions demand efficient processing and fine-grained visual comprehension.
  • GUI Agent Task: GUI agent tasks require multi-step operations such as navigation, form completion, and information retrieval from high-level user instructions.Examples include enabling dark mode, submitting forms, booking travel, and checking unread messages.
  • GUI Agent Task: Robust grounding is crucial for GUI-agent success because agents must localize elements across varied referring expressions and complex layouts.On mobile devices, this requirement is combined with computational constraints, motivating lightweight yet accurate models.
  • GoClick: GoClick is a lightweight GUI element grounding model based on a small VLM and deployed in GUI-agent tasks.Its design addresses the need for accurate element localization under resource constraints.

3.2 GoClick Model Architecture

GoClick adopts Florence-2’s encoder-decoder architecture, combining visual and textual representations before decoding grounding outputs. The model is fine-tuned on GUI grounding tasks to locate elements from diverse referring expressions.

  • Architecture: GoClick uses Florence-2’s visual encoder and multimodal encoder-decoder module as its architectural foundation.The visual encoder maps screenshot patches to projected visual embeddings that are combined with textual embeddings.
  • Capabilities: Fine-tuning produces a compact grounding expert that locates target elements from text, action intents, functionality descriptions, and brief visual descriptions.These referring expressions support varied GUI grounding tasks.
  • Training: For each sample, GoClick combines an image I and text prompt t to generate grounding output o, optimizing −log p(o|I, t).The vision-language projector and language-model embedding module prepare visual and textual inputs for decoding.
  • Architecture choice: The encoder-decoder design is chosen over decoder-only alternatives because precise spatial localization is central to GUI element grounding.The encoder handles visual-linguistic feature extraction while the decoder generates the grounding output.

3.3 Data Compilation Engine

The data compilation engine builds diverse GUI grounding samples, removes noisy and redundant elements, and progressively refines 10.8M raw samples into a 3.8M high-quality core set. The pipeline targets GUI-specific imbalance, outdated patterns, and task-type differences while improving data efficiency.

  • Data collection: The dataset represents each grounding task as a screenshot, referring expression, and coordinates triplet drawn from diverse GUI sources.Sources include webpages, mobile GUIs, and GUI-agent benchmark training splits.
  • Data collection: Four grounding task types—Text, Brief Description, Intent, and Functionality Grounding—are generated from multiple referring-expression forms.Reverse referring-expression generation tasks and widget-listing samples are also added, except Intent REG.
  • Data cleaning: 10.8M samples form the GoClick-Raw dataset after denoising and deduplication of GUI grounding examples.Denoising removes blank, invisible, invalid-box, and OCR-mismatched text elements; deduplication groups normalized expressions and discretized boxes.
  • Progressive Data Refinement: The coarse PDR stage removes non-element grounding tasks and outdated GUI patterns, reducing the dataset to 6.8M samples.Android 4.0 screenshots are given as an example of outdated data relative to post-2017 evaluation benchmarks.
  • Progressive Data Refinement: The fine PDR stage adjusts task-type volumes across major sources and produces a 3.8M-sample high-quality core set.The six major sources cover 90.8% of the coarse-stage dataset, while minor sources retain all samples.
  • Discussion: PDR identifies outdated GUI patterns and unequal task contributions as GUI-specific factors affecting grounding quality.SeeClick-Web exhibits a catastrophic performance decline at full inclusion, while reducing the dataset from 10.8M to 3.8M samples improves average precision.

4 Experiments

The experiments evaluate GoClick’s speed-accuracy trade-off, architecture choice, core-set curation, and contribution to GUI-agent tasks.

  • Experimental scope: The experiments cover speed-accuracy trade-offs, decoder-only architecture comparisons, core-set curation, and GUI-agent assistance.These four evaluations correspond to Sections 4.1 through 4.4.

4.1 Comparisons with State-of-The-Art Methods

GoClick delivers a strong speed–accuracy tradeoff across diverse GUI grounding benchmarks, using substantially fewer parameters and training samples than several larger competitors.

  • Benchmark Coverage: The benchmarks span functionality, brief-description, intent, and text grounding across mobile, web, and desktop GUI domains.FuncPred focuses on functionality descriptions, while ScreenSpot, ScreenSpot-v2, MOTIF, and RefExp use other referring-expression types.
  • Experimental Results: GoClick achieves a significantly better tradeoff between inference speed and grounding accuracy across multiple GUI element grounding benchmarks.Evaluation covers diverse GUI domains and referring-expression types, with Grounding Accuracy, TTFT, and TPOT used as metrics.
  • Experimental Results: 230M-parameter GoClick-B uses approximately 1/30 the parameters, 1/3 the TTFT, and 1/5 the TPOT of several competitive 7B models.It outperforms several 7B models on FuncPred, VWB EG, and obtains comparable accuracy on MOTIF and RefExp.
  • Data Efficiency: GoClick is fine-tuned with 3.8M samples, fewer than the 13.6M used by OS-ATLAS and 10M used by UGround.The reported results attribute the parameter and data efficiency to the encoder-decoder architecture and core-set curation pipeline.

4.2 Experiment on Model Architecture

The experiments find that an encoder-decoder base model is more effective than decoder-only alternatives for lightweight GUI grounding. GoClick also maintains robust performance across referring-expression types and GUI domains.

  • Experimental Results: GoClick based on Florence-2 outperforms InternVL-2.5-1B-SFT, Qwen2VL-2B-SFT, and SLiME-Gemma-2B-SFT across all evaluated benchmarks.All models were fine-tuned with the same 3.8M core set for two epochs under matched experimental conditions.
  • Experimental Results: 69.5% accuracy on FuncPred is achieved by GoClick, significantly surpassing the compared decoder-only models.FuncPred evaluates functionality grounding, where referring expressions do not mention visual or positional characteristics.
  • Parameter Efficiency: With one-tenth the parameters, GoClick (0.8B) achieves comparable or superior results to Qwen2VL-7B-SFT (8.3B).GoClick exceeds Qwen2VL-7B-SFT by +13.4 on FuncPred and +1.9 on VWB AG, while achieving similar performance on ScreenSpot-v2, MOTIF, and RefExp.
  • Cross-Domain Robustness: GoClick exhibits robust performance across Web, Desktop, and Mobile domains and across brief-description, intent, and functionality grounding.Its ScreenSpot results also show a smaller gap between text grounding and harder icon grounding than the Qwen2-VL-2B variant.
  • Conclusion: The results indicate an advantage for building GoClick on Florence-2’s encoder-decoder architecture.The comparison uses small-scale decoder-only alternatives including InternVL-2-1B, Qwen2-VL-2B, and SLiME-Gemma-2B.

4.3 Experiment on Fine-Tuning Data Refinement

The Progressive Data Refinement experiments show that selectively removing harmful or redundant samples improves GUI grounding while reducing training data. Integrating GoClick into device-cloud agents also raises task success across benchmarks.

  • Coarse-Grained Refinement: Old GUI patterns and REG samples are harmful to GUI element grounding performance.Removing outdated patterns reduces fine-tuning costs and improves benchmark results; functionality REG is especially harmful for Florence-2.
  • Fine-Grained Refinement: Fine-grained source and task-ratio adjustment shows that not all aggregated samples improve accuracy.Four sources show positive correlations across task types, while AndroidControl exhibits a negative correlation for Text Grounding.
  • Final Performance: Coarse refinement improves performance, and fine-grained refinement adds further gains with significantly fewer training samples.These results validate the proposed core-set curation pipeline for aggregating data from diverse sources.
  • Device-Cloud Collaboration: GoClick integration substantially improves GUI agent performance across AITW, AndroidControl, and GUIAct benchmarks.The gains occur when proprietary VLM planners delegate visual element grounding to GoClick.
  • Device-Cloud Collaboration: On AITW, Gemini-2-Flash-Exp reaches 47.2 overall Step SR with GoClick versus 34.8 using SoM prompting.GoClick also surpasses SoM on AndroidControl and both GUIAct benchmarks.
  • Device-Cloud Collaboration: Intent Grounding may be marginally more effective than Functionality Grounding for GoClick.The paper attributes this difference to indirect functional descriptions being harder to ground than direct appearance and positional descriptions.

5 Conclusion And Limitations

GoClick combines a lightweight grounding model with refined training data and improves device-cloud GUI agents. The authors identify limits in task generalization, heuristic refinement, and real-device evaluation.

  • Conclusion: GoClick uses an encoder-decoder architecture and systematic data curation to achieve strong GUI grounding at lightweight scale.The paper reports that the architecture outperforms decoder-only alternatives at small scales and that refinement removes harmful and redundant samples.
  • Conclusion: Integrating GoClick into device-cloud collaboration enhances GUI agent performance on resource-constrained devices.The framework combines lightweight on-device grounding with cloud-based planning.
  • Limitations: The encoder-decoder advantage may not generalize beyond GUI element grounding to other GUI-related tasks.The authors specifically identify planning, action prediction, and more complex reasoning as future test cases.
  • Limitations: The progressive data refinement pipeline remains partially heuristic and may vary slightly across random seeds.Limited computational resources make repeated experiments difficult, and outdated GUI patterns may require periodic retraining.
  • Limitations: More theoretically grounded sample-selection methods remain computationally expensive at the full dataset scale.Computing influence for 10.8M samples on a 0.8B model would require substantial resources.
  • Limitations: Evaluation uses L20 GPUs rather than real embedded devices.This controlled setting may not capture embedded deployment constraints such as memory, energy consumption, and system-specific optimization.
  • Conclusion: The findings offer implications for deploying multimodal GUI agents in resource-constrained environments.

Declarations

The authors report dataset, benchmark, and evaluation-resource availability for GoClick and its experiments.

  • Data Availability: Datasets used for data refinement and GoClick training are available through the project repository.
  • Data Availability: GUI agent task testing sets are also available through the project repository.
  • Benchmarks: The GUI element grounding benchmarks used in the experiments are identified by name.They include FuncPred, ScreenSpot, ScreenSpot-v2, MOTIF, RefExp, VWB AG, and VWB EG.
Loading 2604.23941v1…