Source-linked AI summary
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar
TL;DR
VLMs lack sufficient decision-centric data for reliable in-the-wild GUI control, and static demonstrations do not capture real-world stochasticity and non-stationarity. DigiRL combines offline RL initialization with offline-to-online RL, using a scalable Android environment and stochasticity-aware advantage learning. On AitW, it achieves a 49.5% absolute success-rate improvement from 17.7% to 67.2% over supervised fine-tuning and surpasses prior autonomous RL and device agents.
Problem
VLMs lack sufficient decision-centric data, and static demonstrations do not represent the stochastic, non-stationary conditions of deployed GUI control.
Method
DigiRL fine-tunes a pre-trained VLM through offline RL followed by offline-to-online RL with scalable Android interaction, autonomous evaluation, and stochasticity-aware advantage estimation.
Results
49.5% absolute improvement raises success from 17.7% to 67.2% on the AitW Web Shopping subset, surpassing prior device agents and autonomous learning methods.
Takeaways & Limitations
DigiRL establishes state-of-the-art performance for in-the-wild Android device control through autonomous offline-to-online reinforcement learning.
Takeaways & Limitations
Due to computational limitations, training covers AitW tasks rather than all possible device tasks.
Abstract
from arXiv · showhide
Training corpuses for vision language models (VLMs) typically lack sufficient amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal for decision-making tasks such as in-the-wild device control through graphical user interfaces (GUIs). While training with static demonstrations has shown some promise, we show that such methods fall short for controlling real GUIs due to their failure to deal with real-world stochasticity and non-stationarity not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline RL to initialize the model, followed by offline-to-online RL. To do this, we build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator and develop a simple yet effective RL approach for learning in this domain. Our approach runs advantage-weighted RL with advantage estimators enhanced to account for stochasticity along with an automatic curriculum for deriving maximal learning signal. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to 67.2% success rate -- over supervised fine-tuning with static human demonstration data. These results significantly surpass not only the prior best agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent trained with AitW data (38.5%), but also the prior best autonomous RL approach based on filtered behavior cloning (57.8%), thereby establishing a new state-of-the-art for digital agents for in-the-wild device control.
1 Introduction
In-the-wild GUI control requires agents to convert VLM capabilities into goal-directed behavior while handling stochastic, changing environments. DigiRL addresses this with offline initialization followed by autonomous online reinforcement learning, achieving substantially higher Android task success than prior approaches.
- In-the-wild device agents must complete GUI tasks amid stochasticity and distractors across devices, the Internet, and tools.
- Pre-trained VLM reasoning does not reliably transfer to accurate device-control actions, leaving a mechanism for deployable agents lacking.
- DigiRL trains agents in two phases: offline RL on existing task data, then offline-to-online RL using fresh interaction rollouts.
- A scalable Android environment supports up to 64 parallel emulators and uses a VLM evaluator with 2.8% average error against human judgment.
- 28.7% improvement raises success from 38.5% to 67.2% over 18B CogAgent, while DigiRL also exceeds the prior autonomous-learning approach by over 9%.
2 Related Work
Prior device-control work relies on static demonstrations, wrappers, or simplified environments that do not fully capture multi-step GUI control under stochasticity. DigiRL instead scales autonomous offline-to-online RL for real-world device-control tasks.
- Multimodal digital agents: Multimodal device agents act at the pixel level in coordinate-based spaces and face stochastic, unpredictable device and Internet ecosystems.
- Multimodal digital agents: Static imitation learning can maximize single-step accuracy yet produce poor multi-step solutions because agents struggle to recover from unseen states.
- Multimodal digital agents: DigiRL is presented as the first scalable autonomous offline-to-online RL approach for device control, using around 1e3 rather than around 1e6 trajectories in comparison with simplified web-navigation RL.
- Environments for device control agents: Existing device-control environments are often evaluation-oriented, task-limited, deterministic, and stationary, restricting diverse skill acquisition.
- Reinforcement learning for LLM/VLMs: Prior foundation-model RL mainly targets single-turn preference or reasoning tasks, whereas stochastic multi-turn device control requires different optimization.
3 Problem Setup and Preliminaries
The study formulates Android control as pixel-based interaction with language instructions and screenshot observations, emphasizing real-world non-stationarity and stochasticity. Its environment and training pipeline support continuous adaptation through scalable online interaction.
- Problem formulation: The problem is scoped to pixel-based Android-device interaction, which covers broader applications than browser-only or link-based control.
- Problem formulation: At each step, the agent receives a screenshot and acts through taps, slides, and typed text based on normalized screen coordinates.
- Challenges of stochasticity: Real device control faces non-stationary applications, unpredictable distractors, and technical glitches such as incomplete loading or temporary access restrictions.
- Environment details: Figure 3 organizes the environment description around the action-space dynamics and examples of real-world non-stationarity and dynamism.
- Setup for reliable and scalable online RL: Parallelized Android emulators provide a real-time data-collection pipeline for interleaved autonomous RL training and experience gathering.
- Challenges of stochasticity: A frozen policy degrades as conditions change, while DigiRL's autonomous online training retains performance under non-stationarity and stochasticity.
4 DigiRL: Autonomous RL for Building a Strong Device-Control Agent
DigiRL adapts advantage-weighted regression for stochastic, variable-difficulty device-control tasks by combining robust advantage estimates with automatic curriculum filtering. Its two value functions filter trajectories and states before maximum-likelihood actor training.
- MDP formulation: DigiRL frames natural-language device control as a finite-horizon MDP with terminal success rewards and solves it using policy gradients.The reward is 1 for evaluator-judged task completion and 0 otherwise.
- Advantage-weighted regression: Advantage-weighted regression improves the policy by weighting action likelihoods with exponentiated advantages while remaining close to the previous policy.The objective uses past experience and a positive temperature parameter β.
- Advantage-weighted regression: Hard filtering replaces exponentiated advantage weighting to avoid tuning β, training the actor only on selected advantageous data.The resulting fine-tuning loss is maximum likelihood over a filtered distribution.
- Reliable advantage estimates: DigiRL combines Monte-Carlo and bootstrapped estimates in a doubly robust per-step advantage estimator to balance stochasticity-driven variance against value-function bias.The estimator is thresholded at Astep(sh, ah, c) > 1/H to select actions that progress toward the goal.
- Automatic curriculum: An instruction-level value function prioritizes the top-p datapoints with the highest instruction-level advantage, targeting tasks that provide more informative learning signals.The curriculum addresses variable task difficulty and avoids spending data on tasks the agent already performs proficiently.
- Final algorithm: The final algorithm trains trajectory- and step-level value functions with cross-entropy losses, filters trajectories and states, then trains the actor with MLE.Figure 5 summarizes this sequence: value estimation, filtering, and actor optimization.
5 Experimental Evaluation
DigiRL is evaluated against proprietary-VLM wrappers, static-demonstration training, and autonomous RL baselines on Android-in-the-Wild tasks. Its offline-to-online RL, autonomous data collection, evaluator, and ablations show stronger performance, faster adaptation, and improved recovery from mistakes.
- Main Results: DigiRL outperforms prompting, retrieval, and static-demonstration baselines on both AitW subsets, with more than 49.5% absolute improvement.Success rises from 17.7% to 71.9% on General and from 17.7% to 67.2% on Web Shopping, without human supervision.
- Main Results: Under 20% success rates for GPT-4V and Gemini 1.5 Pro show that prompting and Set-of-Marks augmentation do not yield satisfactory device-control performance.Inference-time adaptation gains for these models do not exceed 5%.
- Main Results: Below-20% success on both subsets for supervised imitation and below-40% success for 18B CogAgent show that static demonstrations and increased scale remain insufficient.The predominant failure mode for these agents is inability to rectify errors.
- Main Results: Offline-to-online RL increases General-subset success from 62.8% offline to 75% and outperforms Filtered BC, reaching 71.9% versus 61.5% on General.On Web Shopping, DigiRL improves from 57.8% to 61.4% against Filtered BC and adapts to stochastic, non-stationary interfaces.
- Failure Modes: RL most consistently reduces failures to recover from mistakes by training DigiRL on autonomously collected rollouts.The analysis attributes this improvement to learning from mistakes encountered in previously unseen dynamic states.
- Ablations: Cross-entropy value-function training boosts performance by around 12%, step-level advantages improve efficiency by 12%, and automatic curriculum improves learning speed by around 25%.DigiRL also outperforms vanilla AWR without a doubly robust estimator or curriculum, while hard filtering remains nearly comparable to tuned exponential filtering.
6 Discussion and Limitations
DigiRL combines scalable Android interaction infrastructure with offline and autonomous RL to establish state-of-the-art Android device-control performance. Its main stated limitation is that computational constraints restrict training to AitW tasks rather than all possible device tasks.
- Discussion: DigiRL uses offline RL pre-training followed by autonomous RL fine-tuning with interaction in stochastic real-world Internet and device settings.The approach includes a scalable, parallelizable Android environment with a robust VLM-based evaluator for fast online data collection.
- Discussion: DigiRL establishes new state-of-the-art performance on multiple Android control tasks from the Android-in-the-Wild dataset.The reported result includes a 280% improvement from 17.7% to 68.2% task success rate over prior agents and supervised models.
- Limitations: Computational limitations restrict training to AitW tasks instead of all possible device tasks, although the environment and evaluator can extend to more complicated tasks.The authors position DigiRL’s implementation simplicity as a base for future algorithmic research and broader task coverage.
A.1 Post-processing of AitW
The study selects the AitW General and Web Shopping subsets to examine multi-step Android control while excluding single-step or security-sensitive subsets. The appendix also documents task examples, horizon settings, and rollout-length comparisons.
- Post-processing of AitW: AitW contains five subsets: GoogleApps, Install, Web Shopping, General, and Single; the study selects General and Web Shopping.Single is excluded because its tasks can be completed in one step, while Install and GoogleApps are excluded for security reasons.
- Post-processing of AitW: The General subset covers information search and basic application use, with 545 training tasks, 96 test tasks, and a maximum horizon of 10 steps.Examples include searching news or flights and opening Gmail; offline data comes from rolling out the initial AutoUI policy.
- Post-processing of AitW: The appendix provides examples of task descriptions for the General and Web Shopping subsets.These examples are presented in Tables 2 and 3.
- Post-processing of AitW: The Web Shopping subset contains search tasks across shopping websites, after removing captcha-prone operations and websites to support autonomous evaluation.The resulting tasks involve navigating websites including Costco, Best Buy, and Target.
- Post-processing of AitW: Table 4 compares average rollout lengths for DigiRL and filtered BC across the AitW General and Web Shopping test subsets.Darker green denotes shorter rollout length, and DigiRL consistently produces shorter rollouts.
B Other Quantitative Experiments
Additional experiments examine horizon limits and rollout lengths, showing that longer horizons accelerate learning and that DigiRL produces shorter rollouts than filtered behavior cloning.
- Horizon length: A redundant horizon of H = 20 produces significantly faster learning than H = 10 for both filtered BC and DigiRL on AitW General.The authors attribute this pattern to the greater opportunity to try within a single trajectory.
- Rollout length: DigiRL consistently achieves shorter average rollout lengths than filtered BC across both AitW General and Web Shopping.The comparison holds for all rollouts and for rollouts that eventually succeed, indicating more efficient and directed task solving.
C.1 Random sample of trajectories for different agents
Qualitative trajectories show DigiRL completing in-the-wild device-control tasks more efficiently, avoiding wrong pages and recovering from mistakes where comparison agents may get stuck.
- Random trajectories: DigiRL is less likely than AutoUI and GPT-4V to get stuck or reach a wrong page in sampled test trajectories.Figures 11 and 12 show randomly sampled trajectories from the General and Web Shopping splits.
- Random trajectories: In a Costco task, DigiRL’s trajectory demonstrates searching for “macbook pro” and selecting the first result.The example is one of the sampled device-control trajectories.
- Error recovery: DigiRL recovers from its own mistakes by returning toward the original screen, whereas AutoUI fails to reset and gets stuck on the diverged screen.The authors connect DigiRL’s reset behavior to maximizing state value, since the original screen usually has high value for success.
- Trajectory length: Qualitative examples consistently show DigiRL producing shorter trajectories than filtered BC.This observation is illustrated in Figure 14.
C.4 Reasoning failure of GPT-4V
GPT-4V commonly fails when precise GUI actions diverge from its high-level plan, and it often cannot recognize or recover from the resulting wrong state. These failures include misclicks, mistaken page beliefs, and exceeding interaction limits.
- Failure mechanisms: GPT-4V fails to ground high-level plans in precise GUI actions and often cannot recover after misclicks.The paper identifies action grounding and recovery from mistakes as primary causes of failure.
- Failure mechanisms: After missing a website search button, GPT-4V can mistake Google’s search bar for the target site and continue along the wrong trajectory.In the NewEgg example, it fails to locate the site’s search button, deletes the query, and then assumes it is already on the intended website.
- Failure mechanisms: A misclick on an advertisement can make GPT-4V believe it found search results, causing subsequent scrolling to continue from the wrong page.This failure occurs despite initially reaching the correct Costco website and searching for the requested item.
- Failure mechanisms: GPT-4V can reach the correct alarm settings but fail to complete the task because its precise action sequence exceeds the maximum number of rounds.The final tap also contradicts the model’s stated plan to set the minutes to “00”.
E Experiment machines
The experiments use cloud virtual machines and a distributed emulator architecture to separate policy updates from parallel trajectory collection. An autonomous evaluator assigns rewards from screenshots and task instructions, while prompts support evaluation and Set-of-Marks testing.
- Experiment machines: Experiments run on Google Cloud Platform VMs equipped with one Tesla T4 GPU and 16 Intel Xeon CPUs.These machines provide the hardware resources for the main experiments.
- Experiment machines: The distributed setup keeps emulator processes independent and communicates with the training process through separate UIAutomator servers.This design addresses synchronization inefficiency and fault propagation across parallel emulators.
- Experiment machines: A GPU-equipped host distributes the current policy to workers, which collect trajectories in parallel before the host aggregates them and updates the policy.The architecture supports multi-machine trajectory collection with centralized policy updates.
- Evaluation: The autonomous evaluator receives the current screenshot and task, returns reward 1 for task completion, and terminates the POMDP.The evaluator is used to provide reward signals during interaction.
- Evaluation prompts: Set-of-Marks prompting labels interactive smartphone elements with numeric tags and exposes actions such as tap(element: int).The testing prompt was used to evaluate GPT-4V and Gemini 1.5 Pro.