Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents

1 Introduction

• Low latency, strong privacy guarantees, and resilience under limited connectivity motivate small, on-device GUI agents.The paper cites these needs for scenarios such as hands-free reminders and recipe access.Source: §1 · p. 1
• 53.3% accuracy on ScreenSpot-Pro lets Ferret-UI Lite (3B) surpass UI-TARS-1.5 (7B) by over 15%, while OSWorld navigation is only on par with that larger model.The figure contrasts strong grounding with limited multi-step navigation.Source: Figure 1 · p. 1, §1 · p. 2
• End-to-end agents streamline GUI interaction, but combining grounding, screen understanding, multi-step planning, and self-reflection in one small model remains challenging.Existing systems often use large models, increasing modeling complexity, compute needs, and inference time.Source: §1 · p. 2
• Ferret-UI Lite is a 3B end-to-end multimodal agent trained on unified real and synthetic data, with visual tool-use and two-stage SFT and RL.It observes screens and instructions, then predicts reasoning traces and GUI actions.Source: §1 · p. 2, Figure 3 · p. 3
• Experiments find that grounding and navigation data can benefit each other, while inference-time gains are limited and RL is sensitive to reward design.A balanced data mixture achieves the best overall results.Source: §1 · p. 2

2 Supervised Fine-Tuning

• The unified SFT recipe combines human-annotated and synthetic GUI data to broaden interaction, visual-context, and platform coverage.The paper frames comprehensive supervision as especially important for smaller models.Source: §2 · p. 3

2.1 Data Distribution

• Public grounding and navigation datasets plus synthetic mobile and OS data provide supervision across platforms and task types.The collected resources form the basis for training agents to ground and navigate in varied environments.Source: §2.1 · p. 3

2.2 Format Unification

• Dataset-specific formats are unified so heterogeneous sources can train one model without overfitting to individual schemas.The shared interface supports multi-source training across grounding and navigation.Source: §2.2 · p. 3
• Bounding-box targets are converted to center points, while existing point labels stay unchanged and templates reference the resulting points.This provides a shared grounding interface across datasets and instruction styles.Source: §2.2 · pp. 3–4
• Navigation actions across mobile, desktop, and web use predefined functions with constrained parameters, producing structured outputs for interpretation and evaluation.The action space contains eleven representative action types.Source: §2.2 · p. 4

2.3 Synthetic Data Generation

• Synthetic pipelines address limits in public data; concatenated screenshots create high-resolution grounding examples with denser layouts and richer spatial contexts.Annotations are converted to the shared point-based format.Source: §2.3 · p. 4
• CoT navigation data adds separate plan, action-think, and reflection traces, with short- and long-CoT agents using different subsets.GPT-4o generates the traces using visual prompting and conditions them on annotated actions and episode history.Source: §2.3 · p. 4
• Synthetic QA pairs use episode goals and terminal screens, while perturbed trajectories train recovery from navigation errors.For example, an erroneous swipe can replace a correct termination action.Source: §2.3 · p. 4

3 Reinforcement Learning

• RLVR addresses strict imitation by rewarding task success, and grounding RL accepts any predicted location inside the annotated box.For grounding, the model first predicts a location, then crops around it and re-predicts; this zoom-in is especially beneficial for complex or high-resolution interfaces.Source: §3 · p. 5, Figure 5 · p. 5
• Navigation reward sums action-type and parameter matches, giving type matches partial credit when parameters are required and rewarding exact string-parameter matches.A matching action type scores more when no parameters are expected; string parameters receive 1 for an exact match and 0 otherwise.Source: §3 · p. 6
• For tap actions, sparse reward is 1 inside the ground-truth box and 0 otherwise, while dense reward grades normalized center distance with sensitivity set by λ = 0.5.The reward functions balance categorical correctness with parameter precision.Source: §3 · p. 6
• GRPO samples 8 original-image and 4 zoomed-crop predictions for grounding, and 32 candidate outputs for navigation, using normalized within-group advantages.Prompts with identical rewards across all samples are discarded because they provide no meaningful learning signal.Source: §3 · p. 6
• The unified RL strategy is reported to improve fine-grained grounding precision and multi-step navigation robustness while remaining stable and sample-efficient.Grounding and navigation RL use task-specific rewards optimized with GRPO.Source: §3 · p. 6

4.1 GUI Grounding

• 91.6 on ScreenSpot-V2, 53.3 on ScreenSpot-Pro, and 55.3 on OSWorld-G place Ferret-UI Lite-3B ahead of other 3B models and near larger models.On ScreenSpot-Pro, it is slightly below GUI-Owl-7B (54.9); on OSWorld-G, it is competitive with GUI-Owl-7B (55.9).Source: Table 1 · p. 7, §4.1 · p. 7
• RL consistently improves grounding, and zoom-in adds gains by helping the model handle small or cluttered interface elements.The model first predicts a location, then refines its prediction on a crop around that location.Source: Figure 6 · p. 7, §4.1 · p. 8
• Balanced navigation/grounding mixtures perform best, while synthetic high-resolution data yields modest ScreenSpot-V2 gains and larger ScreenSpot-Pro gains.The two data types can mutually benefit each other; high-resolution data is particularly helpful for precise localization.Source: Figure 6 · p. 7, §4.1 · p. 8

4.2 GUI Navigation

• 86.6% success on low-level Android Control tasks and 68.9% on high-level tasks outperform similar-scale models.Low-level tasks use detailed single-step instructions; high-level tasks require planning across multiple steps.Source: §4.2 · p. 8, Table 2 · p. 8
• Short CoT traces improve AndroidWorld success by 2.1%, and longer traces improve it by 4.1% over the model trained without CoT data.These results are averaged over five runs.Source: §4.2 · p. 8, Table 4 · p. 9
• Our 3B model achieves 28% success on AndroidWorld, competitive with 7B models; on OSWorld it reaches 17.3% with 15 max steps and 19.8% with 50, demonstrating test-time scaling.Source: §4.2 · p. 8
• Scaling synthetic online and offline navigation data to 17K trajectories yields nearly 6% additional improvement over the long-CoT agent.The added data includes filtered online multi-agent data and offline data with task rewriting.Source: §4.2 · p. 9
• RLVR raises AndroidWorld from 25% to 28% and OSWorld from 15% to 17.3%; gains persist with less SFT, and dense grounding rewards outperform sparse rewards.Combining action-type and grounding rewards improves performance; action-type reward alone substantially reduces it.Source: §4.2 · pp. 9–10, Figure 7 · p. 9

5 Conclusion

• Ferret-UI Lite combines real and synthetic data curation, inference-time visual tool use, and two-stage SFT–RL training, with competitive performance relative to larger models.The paper also reports limitations, particularly in multi-step navigation.Source: §5 · p. 10

A Related Work

• Prior GUI grounding work uses supervised fine-tuning to predict coordinates and reinforcement learning with automatically verifiable grounding rewards.The passage describes these approaches as advancing grounding accuracy and robustness across platforms.Source: A Related Work · p. 15
• GUI navigation extends single-step grounding to multi-step reasoning and action prediction, with multi-agent and end-to-end approaches discussed as two broad paradigms.Multi-agent systems separate components such as planners and grounding modules, while end-to-end systems jointly learn grounding and navigation.Source: A Related Work · p. 15
• End-to-end GUI agents directly predict action sequences from multimodal inputs and have been applied across multiple platforms and applications.The related-work discussion includes both specialized GUI systems and general-purpose multimodal models with navigation capabilities.Source: A Related Work · p. 15
• The paper studies a 3B end-to-end GUI agent as a complementary direction to larger models, which remain the dominant focus for their reasoning and planning strength.The related-work discussion identifies latency, privacy, and connectivity as important for on-device deployment.Source: A Related Work · p. 15

B.1 Desktop Case Study

• Figure 8 shows successful completion of an OSWorld task to create sparklines for each order ID using January-to-March data in the Chart column.The figure is identified as a desktop case study.Source: Figure 8 · p. 16

B.2 Mobile Case Study

• Figure 9 shows successful completion of an AndroidWorld task to delete three named recipes from the Broccoli app.The recipes are Kale and Quinoa Salad, Baked Cod with Lemon and Dill, and Rasperry Almond Smoothie.Source: Figure 9 · p. 17
• Figure 10 shows successful completion of an AndroidWorld task to create a contact for Lina Muller with the specified phone number.The number shown is +15410831733.Source: Figure 10 · p. 18

C Unified Action Space

• Table 5 presents a unified action space with shared and platform-specific operations.The caption identifies the scope of the table but does not specify its individual operations.Source: Table 5 · p. 19

D SFT Data Mixture

• Figure 11 shows the supervised fine-tuning mixture includes grounding and navigation datasets, with mobile and desktop synthetic datasets generated using the Section 2.3 pipeline.Its legend also identifies web, desktop, and mobile categories.Source: Figure 11 · p. 19
• The listed dataset sources include OS-Atlas (Wu et al., 2025) at 11089.3 K, UGround (Gou et al., 2025) at 9550.0 K, and AGUVIS-Grounding (Xu et al., 2025) at 723.0 K.The same list includes Aria-UI at 173.0 K, WaveUI at 58.7 K, AGUVIS-Planning at 34.3 K, ShowUI at 22.0 K, and GroundUI at 11.6 K.Source: D SFT Data Mixture · p. 19
• Another listed source inventory gives OS-Atlas (Wu et al., 2025) at 1134.6 K and OpenCUA (Wang et al., 2025b) at 420.8 K.That inventory also lists Synthetic at 75.0 K, ShowUI at 8.0 K, Aria-UI at 7.8 K, AGUVIS-Grounding at 7.0 K, GroundUI at 2.5 K, and WaveUI at 1.8 K.Source: D SFT Data Mixture · p. 19
• A further listed inventory gives OS-Atlas (Wu et al., 2025) at 4586.3 K and AGUVIS-Grounding (Xu et al., 2025) at 306.0 K.It also lists AGUVIS-Planning at 259.6 K, UGround at 112.0 K, Synthetic at 70.0 K, GroundUI at 4.0 K, and WaveUI at 3.1 K.Source: D SFT Data Mixture · p. 19

• Figure 12 tracks the evolution of the verifiable reward curve during RLVR training.Source: Figure 12 · p. 20

• Excluding OSAtlas produces the largest ScreenSpot-Pro degradation (-6.80), versus drops of -2.08 and -1.26 when excluding UGround and ShowUI.Source: Table 6 · p. 20, F Grounding Dataset Ablations · p. 20
• Removing UGround and ShowUI causes minor changes on ScreenSpot-V2.Source: F Grounding Dataset Ablations · p. 20
• Removing Aria-UI and Jedi causes noticeable decreases, suggesting complementary contributions across ScreenSpot-Pro and ScreenSpot-V2.Source: F Grounding Dataset Ablations · p. 20

• Table 7 reports fine-grained grounding performance on ScreenSpot-V2.Source: Table 7 · p. 21
• Table 8 reports fine-grained grounding performance on OSWorld-G.Source: Table 8 · p. 22
• Table 9 reports fine-grained grounding performance on ScreenSpot-Pro.Source: Table 9 · p. 22