PaperlayerView exact revision on arXiv

Source-linked AI summary

GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction

Hongxin Li, Yuntao Chen, Zhaoxiang Zhang

arXiv:2604.23941v1cs.CV

TL;DR

GUI element grounding is important for low-latency agents, but large VLMs are difficult to deploy on resource-constrained devices. GoClick combines a lightweight encoder-decoder model with Progressive Data Refinement, achieving strong grounding and device-cloud agent results while exposing deployment and data-curation limitations.

  • Problem

    Large GUI grounding VLMs, often exceeding 2.5B parameters, are difficult to run on resource-constrained mobile devices despite the need for low-latency grounding.

    Source: §1 · p. 1, §2.2 · p. 4

  • Method

    GoClick uses an encoder-decoder architecture and Progressive Data Refinement to train a 230M-parameter grounding model from a 3.8M-sample core set.

    Source: §1 · p. 2, §3.3.4 · p. 7

  • Results

    GoClick outperforms equal-sized models, compares favorably with larger models across grounding benchmarks, and improves step success rates in device-cloud GUI agents.

    Source: §1 · p. 3, §5 · p. 16

  • Takeaways & Limitations

    Lightweight grounding models can support practical GUI-agent deployment on resource-constrained devices within the evaluated device-cloud framework.

    Source: §5 · pp. 16–17

    Evaluation used L20 GPUs rather than real embedded devices, so real-world memory, energy, and system-optimization challenges remain incompletely captured.

    Source: §5 · p. 17

Abstract

from arXiv · show

Graphical User Interface (GUI) element grounding (precisely locating elements on screenshots based on natural language instructions) is fundamental for agents interacting with GUIs. Deploying this capability directly on resource-constrained devices like mobile phones is increasingly critical for GUI agents requiring low latency. However, this goal faces a significant challenge, as current visual grounding methods typically employ large vision-language model (VLM) (more than 2.5B parameters), making them impractical for on-device execution due to memory and computational constraints. To address this, this paper introduces GoClick, a lightweight GUI element grounding VLM with only 230M parameters that achieves excellent visual grounding accuracy, even on par with significantly larger models. Simply downsizing existing decoder-only VLMs is a straightforward way to design a lightweight model, but our experiments reveal that this approach yields suboptimal results. Instead, we select an encoder-decoder architecture, which outperforms decoder-only alternatives at small parameter scales for GUI grounding tasks. Additionally, the limited capacity of small VLMs encourages us to develop a Progressive Data Refinement pipeline that utilizes task type filtering and data ratio adjustment to extract a high-quality 3.8M-sample core set from a 10.8M raw dataset. Training GoClick using this core set brings notable grounding accuracy gains. Our experiments show that GoClick excels on multiple GUI element grounding benchmarks while maintaining a small size and high inference speed. GoClick also enhances GUI agent performance when integrated into a device-cloud collaboration framework, where GoClick helps cloud-based task planners perform precise element localization and achieve higher success rates. We hope our method serves as a meaningful exploration within the GUI agent community.

1 Introduction

GUI element grounding lets agents locate interface elements from natural-language instructions, but existing large VLMs are difficult to deploy on resource-constrained devices. GoClick addresses this with a 230M-parameter model, an encoder-decoder architecture, and refined training data.

Motivation

  • GUI element grounding enables agents to locate screenshot elements for actions such as clicking and mobile-app navigation.It is important for low-latency GUI agents, including assistance for visually impaired users and information collection.Source: §1 · p. 1
  • Models often exceeding 2.5B parameters create deployment challenges on mobile devices because of memory and computational constraints.This creates tension between grounding performance requirements and on-device deployment constraints.Source: §1 · p. 1

GoClick

  • GoClick uses only 230M parameters while targeting grounding performance comparable to larger models and low latency.The model is introduced as a lightweight GUI element grounding expert for resource-constrained environments.Source: §1 · pp. 1–2
  • An encoder-decoder architecture outperforms decoder-only alternatives at small parameter scales for GUI element grounding.Downsized 1B and 2B decoder-only models produced only moderate grounding performance, motivating the Florence-2 base architecture.Source: §1 · p. 2

Training data

  • Progressive Data Refinement extracts a high-quality core set from a 10.8M raw dataset by filtering task types and adjusting task ratios.The pipeline removes outdated GUI patterns and harmful task samples before retaining useful training data.Source: §1 · p. 2

Results

  • 3.8M refined samples train GoClick to outperform equal-sized grounding models and compare favorably with larger models across multiple benchmarks.The paper also reports a better speed-accuracy tradeoff and higher step success rates in a device-cloud collaboration agent.Source: §1 · pp. 2–3, Figure 2 · p. 2

2 Related Works

Vision-language research has progressed from encoder-decoder multimodal architectures to LLM-centered models and visually grounded GUI agents. However, many GUI-agent systems remain too large for mobile deployment, motivating lightweight grounding models.

Vision-Language Models

  • Vision-language models combine visual and textual inputs to support multimodal tasks such as visual question answering and image captioning.Early architectures include LXMERT, VL-BART, and Flamingo, while Florence-2 uses encoder and decoder blocks for textual outputs.Source: §2.1 · p. 3
  • LLM-centered VLMs inject visual information through projectors, prompts, or related feature-fusion mechanisms.LLaVA, LLaMA-Adapter, VisionLLM, Ferret, and Qwen-VL develop increasingly strong multimodal understanding and grounding abilities.Source: §2.1 · p. 3

GUI Agents

  • These multimodal capabilities provide a route toward autonomous agents and motivate training element-grounding models from open-source VLMs.The paper targets improved GUI interaction through open-source visual-language grounding.Source: §2.1 · p. 3
  • Early GUI agents used textual GUI representations and proprietary LLM prompting, but were generally limited to narrow scenarios.Visual-observation approaches instead require pixel-level control in complex action spaces.Source: §2.2 · p. 3
  • Prompt-based systems such as MM-Navigator, AppAgent, SeeAct, and Mobile-Agent use GPT-4V reasoning or Set-of-Marks prompting, but simple prompt engineering is difficult to improve further.Other work trains open-source VLMs for GUI-agent tasks and combines proprietary task decomposition with open-source action execution.Source: §2.2 · p. 4

Deployment gap

  • Many open-source GUI-agent models are at least 7B parameters and are not easily deployable on mobile devices.GoClick instead targets lightweight GUI element grounding, whereas OmniParser-v2 focuses on detection and captioning rather than grounding.Source: §2.2 · p. 4

3.1 Preliminary

GUI element grounding maps a natural-language referring expression and screenshot to the target element’s coordinates. GUI agents then use such localization within complex, multi-step visual interactions under fine-grained and computational constraints.

GUI Element Grounding

  • GUI element grounding predicts a target element’s box or center coordinates from a screenshot and a natural-language referring expression.The task bridges user intentions and interface elements by mapping language to precise screen locations.Source: §3.1.1 · p. 4
  • Referring expressions can describe spatial position, appearance, function, or combinations of these properties.Examples include a top-right button, a blue submit button, a login field, and a red close button in settings.Source: §3.1.1 · p. 4
  • GUI grounding is harder than natural-scene grounding because interfaces contain high-resolution inputs and many small, visually similar elements.These conditions demand efficient processing and fine-grained visual comprehension.Source: §3.1.1 · p. 4

GUI Agent Task

  • GUI agent tasks require multi-step operations such as navigation, form completion, and information retrieval from high-level user instructions.Examples include enabling dark mode, submitting forms, booking travel, and checking unread messages.Source: §3.1.2 · p. 4
  • Robust grounding is crucial for GUI-agent success because agents must localize elements across varied referring expressions and complex layouts.On mobile devices, this requirement is combined with computational constraints, motivating lightweight yet accurate models.Source: §3.1.2 · p. 4

GoClick

  • GoClick is a lightweight GUI element grounding model based on a small VLM and deployed in GUI-agent tasks.Its design addresses the need for accurate element localization under resource constraints.Source: §3.1.2 · p. 4

3.2 GoClick Model Architecture

GoClick adopts Florence-2’s encoder-decoder architecture, combining visual and textual representations before decoding grounding outputs. The model is fine-tuned on GUI grounding tasks to locate elements from diverse referring expressions.

Architecture

  • GoClick uses Florence-2’s visual encoder and multimodal encoder-decoder module as its architectural foundation.The visual encoder maps screenshot patches to projected visual embeddings that are combined with textual embeddings.Source: §3.2 · p. 4, Figure 3 · p. 5

Capabilities

  • Fine-tuning produces a compact grounding expert that locates target elements from text, action intents, functionality descriptions, and brief visual descriptions.These referring expressions support varied GUI grounding tasks.Source: Figure 3 · p. 5

Training

  • For each sample, GoClick combines an image I and text prompt t to generate grounding output o, optimizing −log p(o|I, t).The vision-language projector and language-model embedding module prepare visual and textual inputs for decoding.Source: §3.2 · pp. 4–5

Architecture choice

  • The encoder-decoder design is chosen over decoder-only alternatives because precise spatial localization is central to GUI element grounding.The encoder handles visual-linguistic feature extraction while the decoder generates the grounding output.Source: §3.2 · p. 5

3.3 Data Compilation Engine

The data compilation engine builds diverse GUI grounding samples, removes noisy and redundant elements, and progressively refines 10.8M raw samples into a 3.8M high-quality core set. The pipeline targets GUI-specific imbalance, outdated patterns, and task-type differences while improving data efficiency.

Data collection

  • The dataset represents each grounding task as a screenshot, referring expression, and coordinates triplet drawn from diverse GUI sources.Sources include webpages, mobile GUIs, and GUI-agent benchmark training splits.Source: §3.3.1 · pp. 5–6
  • Four grounding task types—Text, Brief Description, Intent, and Functionality Grounding—are generated from multiple referring-expression forms.Reverse referring-expression generation tasks and widget-listing samples are also added, except Intent REG.Source: §3.3.1 · p. 6

Data cleaning

  • 10.8M samples form the GoClick-Raw dataset after denoising and deduplication of GUI grounding examples.Denoising removes blank, invisible, invalid-box, and OCR-mismatched text elements; deduplication groups normalized expressions and discretized boxes.Source: §3.3.2 · pp. 6–7

Progressive Data Refinement

  • The coarse PDR stage removes non-element grounding tasks and outdated GUI patterns, reducing the dataset to 6.8M samples.Android 4.0 screenshots are given as an example of outdated data relative to post-2017 evaluation benchmarks.Source: §3.3.3 · p. 7
  • The fine PDR stage adjusts task-type volumes across major sources and produces a 3.8M-sample high-quality core set.The six major sources cover 90.8% of the coarse-stage dataset, while minor sources retain all samples.Source: §3.3.3 · p. 7

Discussion

  • PDR identifies outdated GUI patterns and unequal task contributions as GUI-specific factors affecting grounding quality.SeeClick-Web exhibits a catastrophic performance decline at full inclusion, while reducing the dataset from 10.8M to 3.8M samples improves average precision.Source: §3.3.4 · p. 7

4 Experiments

The experiments evaluate GoClick’s speed-accuracy trade-off, architecture choice, core-set curation, and contribution to GUI-agent tasks.

Experimental scope

  • The experiments cover speed-accuracy trade-offs, decoder-only architecture comparisons, core-set curation, and GUI-agent assistance.These four evaluations correspond to Sections 4.1 through 4.4.Source: §4 · p. 7

4.1 Comparisons with State-of-The-Art Methods

GoClick delivers a strong speed–accuracy tradeoff across diverse GUI grounding benchmarks, using substantially fewer parameters and training samples than several larger competitors.

Benchmark Coverage

  • The benchmarks span functionality, brief-description, intent, and text grounding across mobile, web, and desktop GUI domains.FuncPred focuses on functionality descriptions, while ScreenSpot, ScreenSpot-v2, MOTIF, and RefExp use other referring-expression types.Source: §4.1.1 · pp. 8–9

Experimental Results

  • GoClick achieves a significantly better tradeoff between inference speed and grounding accuracy across multiple GUI element grounding benchmarks.Evaluation covers diverse GUI domains and referring-expression types, with Grounding Accuracy, TTFT, and TPOT used as metrics.Source: §4.1.1 · pp. 8–9, §4.1.3 · p. 9
  • 230M-parameter GoClick-B uses approximately 1/30 the parameters, 1/3 the TTFT, and 1/5 the TPOT of several competitive 7B models.It outperforms several 7B models on FuncPred, VWB EG, and obtains comparable accuracy on MOTIF and RefExp.Source: §4.1.3 · p. 9

Data Efficiency

  • GoClick is fine-tuned with 3.8M samples, fewer than the 13.6M used by OS-ATLAS and 10M used by UGround.The reported results attribute the parameter and data efficiency to the encoder-decoder architecture and core-set curation pipeline.Source: §4.1.3 · p. 9

4.2 Experiment on Model Architecture

The experiments find that an encoder-decoder base model is more effective than decoder-only alternatives for lightweight GUI grounding. GoClick also maintains robust performance across referring-expression types and GUI domains.

Experimental Results

  • GoClick based on Florence-2 outperforms InternVL-2.5-1B-SFT, Qwen2VL-2B-SFT, and SLiME-Gemma-2B-SFT across all evaluated benchmarks.All models were fine-tuned with the same 3.8M core set for two epochs under matched experimental conditions.Source: §4.2.2 · p. 10, §4.2.1 · p. 10
  • 69.5% accuracy on FuncPred is achieved by GoClick, significantly surpassing the compared decoder-only models.FuncPred evaluates functionality grounding, where referring expressions do not mention visual or positional characteristics.Source: §4.2.2 · p. 10

Parameter Efficiency

  • With one-tenth the parameters, GoClick (0.8B) achieves comparable or superior results to Qwen2VL-7B-SFT (8.3B).GoClick exceeds Qwen2VL-7B-SFT by +13.4 on FuncPred and +1.9 on VWB AG, while achieving similar performance on ScreenSpot-v2, MOTIF, and RefExp.Source: §4.2.2 · p. 10

Cross-Domain Robustness

  • GoClick exhibits robust performance across Web, Desktop, and Mobile domains and across brief-description, intent, and functionality grounding.Its ScreenSpot results also show a smaller gap between text grounding and harder icon grounding than the Qwen2-VL-2B variant.Source: §4.2.2 · p. 11, Table 5 · p. 11

Conclusion

  • The results indicate an advantage for building GoClick on Florence-2’s encoder-decoder architecture.The comparison uses small-scale decoder-only alternatives including InternVL-2-1B, Qwen2-VL-2B, and SLiME-Gemma-2B.Source: §4.2 · p. 9, §4.2.1 · p. 10, §4.2.2 · p. 11

4.3 Experiment on Fine-Tuning Data Refinement

The Progressive Data Refinement experiments show that selectively removing harmful or redundant samples improves GUI grounding while reducing training data. Integrating GoClick into device-cloud agents also raises task success across benchmarks.

Coarse-Grained Refinement

  • Old GUI patterns and REG samples are harmful to GUI element grounding performance.Removing outdated patterns reduces fine-tuning costs and improves benchmark results; functionality REG is especially harmful for Florence-2.Source: §4.3.2 · pp. 11–12

Fine-Grained Refinement

  • Fine-grained source and task-ratio adjustment shows that not all aggregated samples improve accuracy.Four sources show positive correlations across task types, while AndroidControl exhibits a negative correlation for Text Grounding.Source: §4.3.2 · pp. 12–13

Final Performance

  • Coarse refinement improves performance, and fine-grained refinement adds further gains with significantly fewer training samples.These results validate the proposed core-set curation pipeline for aggregating data from diverse sources.Source: §4.3.2 · p. 13

Device-Cloud Collaboration

  • GoClick integration substantially improves GUI agent performance across AITW, AndroidControl, and GUIAct benchmarks.The gains occur when proprietary VLM planners delegate visual element grounding to GoClick.Source: §4.4.4 · p. 16
  • On AITW, Gemini-2-Flash-Exp reaches 47.2 overall Step SR with GoClick versus 34.8 using SoM prompting.GoClick also surpasses SoM on AndroidControl and both GUIAct benchmarks.Source: §4.4.4 · p. 16
  • Intent Grounding may be marginally more effective than Functionality Grounding for GoClick.The paper attributes this difference to indirect functional descriptions being harder to ground than direct appearance and positional descriptions.Source: §4.4.4 · p. 16

5 Conclusion And Limitations

GoClick combines a lightweight grounding model with refined training data and improves device-cloud GUI agents. The authors identify limits in task generalization, heuristic refinement, and real-device evaluation.

Conclusion

  • GoClick uses an encoder-decoder architecture and systematic data curation to achieve strong GUI grounding at lightweight scale.The paper reports that the architecture outperforms decoder-only alternatives at small scales and that refinement removes harmful and redundant samples.Source: §5 · p. 16
  • Integrating GoClick into device-cloud collaboration enhances GUI agent performance on resource-constrained devices.The framework combines lightweight on-device grounding with cloud-based planning.Source: §5 · p. 17

Limitations

  • The encoder-decoder advantage may not generalize beyond GUI element grounding to other GUI-related tasks.The authors specifically identify planning, action prediction, and more complex reasoning as future test cases.Source: §5 · p. 17
  • The progressive data refinement pipeline remains partially heuristic and may vary slightly across random seeds.Limited computational resources make repeated experiments difficult, and outdated GUI patterns may require periodic retraining.Source: §5 · p. 17
  • More theoretically grounded sample-selection methods remain computationally expensive at the full dataset scale.Computing influence for 10.8M samples on a 0.8B model would require substantial resources.Source: §5 · p. 17
  • Evaluation uses L20 GPUs rather than real embedded devices.This controlled setting may not capture embedded deployment constraints such as memory, energy consumption, and system-specific optimization.Source: §5 · p. 17

Conclusion

  • The findings offer implications for deploying multimodal GUI agents in resource-constrained environments.Source: §5 · p. 17

Declarations

The authors report dataset, benchmark, and evaluation-resource availability for GoClick and its experiments.

Data Availability

  • Datasets used for data refinement and GoClick training are available through the project repository.Source: Declarations · p. 17
  • GUI agent task testing sets are also available through the project repository.Source: Declarations · p. 17

Benchmarks

  • The GUI element grounding benchmarks used in the experiments are identified by name.They include FuncPred, ScreenSpot, ScreenSpot-v2, MOTIF, RefExp, VWB AG, and VWB EG.Source: Declarations · p. 17

Related papers

Loading 2604.23941v1…