PaperlayerView exact revision on arXiv

Source-linked AI summary

DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar

arXiv:2406.11896v1cs.LG

TL;DR

VLMs lack decision-centric training data for reliable GUI device control amid real-world stochasticity. DigiRL fine-tunes a pretrained VLM with offline and then offline-to-online RL, outperforming prompting, retrieval, and static-demonstration methods on both AitW subsets by more than 49.5% absolute.

  • Problem

    A mechanism for channeling pretrained VLM abilities into deployable agents that complete tasks and recover from mistakes in GUI environments is lacking.

    Source: §1 · p. 2

  • Method

    DigiRL fine-tunes a pretrained VLM through offline RL initialization followed by offline-to-online RL using an Android learning environment with a VLM-based evaluator.

    Source: §1 · pp. 2–3

  • Results

    DigiRL outperformed prompting, retrieval, and static-demonstration methods on both AitW subsets, with more than 49.5% absolute improvement (17.7% to 71.9% on General; 17.7% to 67.2% on Web Shopping).

    Source: §5.1 · p. 8

  • Takeaways & Limitations

    On the evaluated AitW subsets, DigiRL's improvement was achieved fully autonomously without manually labeled rollouts or hand-written verifiers.

    Source: §5.1 · p. 8

    Due to computational limitations, the agent was trained only on AitW tasks rather than all possible device tasks.

    Source: §6 · p. 11

Abstract

from arXiv · show

Training corpuses for vision language models (VLMs) typically lack sufficient amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal for decision-making tasks such as in-the-wild device control through graphical user interfaces (GUIs). While training with static demonstrations has shown some promise, we show that such methods fall short for controlling real GUIs due to their failure to deal with real-world stochasticity and non-stationarity not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline RL to initialize the model, followed by offline-to-online RL. To do this, we build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator and develop a simple yet effective RL approach for learning in this domain. Our approach runs advantage-weighted RL with advantage estimators enhanced to account for stochasticity along with an automatic curriculum for deriving maximal learning signal. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to 67.2% success rate -- over supervised fine-tuning with static human demonstration data. These results significantly surpass not only the prior best agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent trained with AitW data (38.5%), but also the prior best autonomous RL approach based on filtered behavior cloning (57.8%), thereby establishing a new state-of-the-art for digital agents for in-the-wild device control.

1 Introduction

DigiRL targets the gap between VLM capabilities and reliable GUI control with autonomous offline-to-online RL, achieving 67.2% success on Android-in-the-Wild tasks with a 1.3B model.

Introduction

  • VLMs’ reasoning and common-sense abilities do not reliably produce accurate GUI actions, while wrappers are limited by their base models and stale imitation data can impair adaptation and recovery.In-the-wild device tasks involve stochasticity and distractors, and model deductions about novel scenes may not transfer to accurate control.Source: §1 · pp. 1–2
  • DigiRL’s learning approach uses online interaction data and the agent’s own multi-turn rollouts, with mechanisms to select correct actions while filtering mistakes.Static demonstrations may not represent changing websites and device conditions encountered at deployment.Source: §1 · p. 2
  • DigiRL initializes a VLM with offline RL, then fine-tunes online using AWR, an instruction-level task curriculum, and a step-level value function for learning under stochasticity.Its scalable Android environment supports up to 64 real emulators simultaneously, and its VLM-based evaluator has a 2.8% average error rate against human judgment.Source: §1 · p. 3
  • 67.2% success rate, a 28.7% improvement over 18B CogAgent (38.5%), exceeds Filtered Behavior Cloning [26] by over 9% and proprietary VLM wrappers at 17.7%.The proprietary-model comparison covers GPT-4V and Gemini 1.5 Pro.Source: §1 · p. 3

2 Related Work

The related work highlights the challenges of stochastic, pixel-level GUI control and gaps in existing agents, environments, and RL methods. DigiRL addresses these gaps with autonomous offline-to-online RL for real-world device control.

Multi-modal digital agents

  • Pixel-level, coordinate-based actions and stochastic device ecosystems challenge multi-modal agents; proprietary VLMs with rule-based wrappers lack fine-tuning gains, while stale demonstrations can fail under stochasticity and out-of-distribution states.Prior demonstration-trained VLMs use imitation learning; maximizing single-step accuracy may neglect later consequences [26] and hinder recovery from states absent in demonstrations [8] [12].Source: §2 · pp. 3–4
  • DigiRL is described as the first scaled autonomous offline-to-online RL for device control, outperforming imitation agents and using 1000x fewer trajectories than MiniWob++ on-policy RL (around 1e3 vs around 1e6).The comparison is with prior on-policy RL in simplified web navigation settings; DigiRL operates in real-world GUI navigation tasks.Source: §2 · p. 4

Environments for device control agents

  • Existing device-control environments offer either limited tasks in deterministic, stationary settings or diverse but oversimplified tasks that may not transfer in the wild.DigiRL instead uses Gemini 1.5 Pro for autonomous evaluation [26] across diverse open-ended tasks on parallel actual Android devices, unlike single-threaded Android emulators [26] [39].Source: §2 · p. 4

Reinforcement learning for LLM/VLMs

  • Most prior foundation-model RL targets single-turn preference optimization [25] [58] or reasoning, whereas DigiRL focuses on multi-turn learning from sub-optimal online interaction under stochasticity and non-stationarity.Prior value-based LLM RL methods [57] [50] typically maintain multiple networks and delayed targets, with potential slow convergence and hyperparameter sensitivity; DigiRL emphasizes a simple, effective algorithm for full-scale Android control.Source: §2 · p. 4

3 Problem Setup and Preliminaries

The section formulates pixel-based control of Android devices as a broader and more challenging setting than web-only or link-based environments. It highlights real-world stochasticity and describes an online-RL setup designed to support adaptation and scalable evaluation.

3 Problem Setup and Preliminaries

  • The problem is pixel-based Android interaction, a broader setting than web-only navigation or link-based controls that cannot handle tasks such as games.The web browser is only one application in the Android environment.Source: §3 · p. 4
  • Episodes start at the home screen; agents follow AitW General and Web Shopping instructions using screenshots and taps, swipes, text entry, and system buttons.Actions include normalized screen coordinates and buttons such as HOME, BACK, and ENTER.Source: §3 · p. 4
  • Real-world interaction introduces non-stationarity, unpredictable distractors, and technical glitches that challenge pretrained VLMs, including models fine-tuned for device control.Examples include app or website updates, pop-up ads, login requests, changing search-result order, incomplete page loading, and temporary access restrictions.Source: §3 · pp. 4–5
  • Scalable online RL requires parallel Android emulators and an autonomous evaluator that judges task completion without emulator internals or task-specific rules.The Gemini 1.5 Pro evaluator uses few-shot rollouts with human-labeled outcomes and has a 2.8% average error rate against human annotations.Source: §3 · p. 5
  • Frozen DigiRL policies degrade as websites and applications change, while continuous online updates help retain performance.The comparison uses a checkpoint trained with autonomous data from June 1–3 and fresh data from June 7–11.Source: Figure 4 · p. 5, §3 · p. 5

4 DigiRL: Autonomous RL for Building a Strong Device-Control Agent

DigiRL frames instruction-guided device control as finite-horizon reinforcement learning and adapts off-policy AWR with stochasticity-aware advantage filtering and an instruction-level curriculum.

4 DigiRL: Autonomous RL for Building a Strong Device-Control Agent

  • Device control is modeled as a finite-horizon MDP with binary success reward, a maximum interaction horizon, and states represented by the last two screenshots.The initial state and natural-language instruction are sampled at the start of each task.Source: §4 · p. 6

4.1 Backbone of Our Approach: Off-Policy RL via Advantage-Weighted Regression

  • AWR [29] updates the policy by fitting to actions weighted by exponentiated advantages, while hard advantage filtering avoids tuning β.AWR uses past experience and keeps the updated policy close to the previous policy; DigiRL replaces exponentiation with hard filtering.Source: §4.1 · p. 6

4.2 Obtaining Reliable Advantage Estimates from Doubly-Robust Estimators

  • To handle stochastic rollouts, DigiRL uses a simplified GAE-style doubly robust advantage [40], balancing noisy Monte Carlo and biased value estimates, and keeps actions with A_step > 1/H.The estimator combines a final-step Monte Carlo estimate with a next-step value-based estimate; the simplification uses these terms because the task has no intermediate rewards.Source: §4.2 · p. 6

4.3 Automatic Curriculum using an Instruction-Level Value Function

  • The instruction-level value function filters the top-p datapoints by instruction-level advantage, prioritizing successful rollouts on difficult tasks.Both value functions are trained with cross-entropy losses based on Monte Carlo trajectory rewards.Source: §4.3 · pp. 6–7

5 Experimental Evaluation

The evaluation compares DigiRL with prompting, retrieval, supervised imitation, and filtered behavior cloning on Android device-control tasks, then analyzes its failure modes, components, evaluator, and emulator scaling.

5.1 Main Results

  • DigiRL reaches 71.9% success on AitW General and 67.2% on Web Shopping, versus 17.7% for prior approaches on each subset, with the improvement achieved without human supervision.The paper reports improvements of more than 49.5% absolute and summarizes the main agent comparisons in Table 1.Source: Table 1 · p. 8, §5.1 · p. 8
  • GPT-4V and Gemini 1.5 Pro remain below 20% success, and AppAgent’s gains do not exceed 5% for either model; their primary failure is hallucinatory reasoning that leads to a relevant but wrong page.The paper suggests limited Android representation in pretraining as one possible cause and reports that inference-time prompting and retrieval alone offer limited improvement.Source: §5.1 · pp. 8–9
  • Supervised imitation remains below 20% on both subsets for AutoUI, while 18-billion-parameter CogAgent remains below 40%; these agents often fail to recover from their own errors.The paper reports AutoUI can match or surpass larger generalist VLMs, but model scale and architecture enhancements do not fundamentally resolve the shortfall.Source: §5.1 · p. 9
  • DigiRL exceeds Filtered BC’s 61.5% on General and 57.8% on Web Shopping, reaching 71.9% and 61.4%, respectively; offline-to-online training also reaches 75% versus 62.8% for the top offline run on General.Offline and offline-to-online comparisons use roughly equal total samples; the offline data had become stale, while offline-to-online runs began with around 500 offline trajectories and added 1,000 online trajectories.Source: §5 · p. 8, Figure 6 · p. 9, §5.1 · p. 10

5.2 Analysis and Ablations

  • Offline and offline-to-online RL reduce failures across three categories, with the most consistent reduction in failure to recover from mistakes as DigiRL learns from its autonomous rollouts.The categories are failure to recover from mistakes, getting stuck midway, and arriving at the wrong goal.Source: Figure 7 · p. 9, §5.2 · p. 10
  • Ablations report around 12% improvement from cross-entropy value training, 12% better efficiency from step-level advantages, and around 25% faster learning from automatic curriculum; DigiRL also outperforms vanilla AWR.Hard-filtering causes no observed performance degradation versus the best tuned exponential-filtering run of vanilla AWR.Source: §5.2 · pp. 10–11, Figure 9 · p. 11
  • Autonomous evaluator success rates differ from human assessments by less than 3% across almost all models, with more precise evaluations on Web Shopping than General.The paper attributes the subset difference likely to free-form language ambiguity in General versus narrower language expressions in Web Shopping.Source: Figure 8 · p. 10, §5.2 · p. 11
  • The distributed emulator scales to 64 parallel emulators on 128 CPUs with near-linear speedup, reaching 1.74 trajs/min versus 0.74 trajs/min for the same-server baseline using 64 CPUs.The paper reports that performance improves nearly linearly with worker machines.Source: §5.2 · p. 11, Figure 9 · p. 11

6 Discussion and Limitations

DigiRL combines scalable Android interaction infrastructure with offline-to-autonomous RL fine-tuning and reports improved task success on Android-in-the-Wild tasks. Computational limits restrict its training tasks to AitW, although the environment and evaluator can be extended to more complicated tasks.

6 Discussion and Limitations

  • DigiRL reports a 280% improvement in task success rate, from 17.7% to 68.2%, over prior agents including AppAgent based on GPT-4V and Gemini 1.5 Pro, AutoUI, and CogAgent.The approach combines a scalable, parallelizable Android environment and VLM-based evaluator with offline RL pre-training followed by autonomous RL fine-tuning through interaction.Source: §6 · p. 11
  • Due to computational limitations, DigiRL is trained only on AitW tasks, although its parallel emulator and autonomous evaluator can extend to more complicated tasks.The authors aim for implementation simplicity and position DigiRL as a base for future algorithmic research and expansion of the task space.Source: §6 · p. 11

Appendices · A Environment details · A.1 Post-processing of AitW

The appendix defines the AitW task subsets used for DigiRL and documents how General and Web Shopping tasks were prepared for training and evaluation. It excludes subsets with one-step tasks or account-related security concerns and adapts Web Shopping tasks to avoid captcha-prone operations and websites.

A.1 Post-processing of AitW

  • DigiRL uses the AitW General and Web Shopping subsets, excluding Single because its tasks are one-step and excluding Install and GoogleApps over account-related security concerns.The authors note that parallel emulations of Install and GoogleApps tasks could raise security concerns.Source: §A.1 · p. 16
  • General uses 545 training tasks and the first 96 test tasks, with a maximum of 10 steps per task.The subset covers information searches and basic app use; examples include searching for news or flights and opening Gmail.Source: §A.1 · p. 16
  • Offline data comprises 1,552 General and 1,296 Web Shopping trajectories for offline training, versus 608 and 528 respectively for offline-to-online training.For both subsets, trajectories were collected by rolling out the initial AutoUI policy on training tasks.Source: §A.1 · p. 16
  • Web Shopping tasks were post-processed to exclude captcha-prone websites and operations, leaving five websites and three basic shopping operations.The retained operations are visiting a website, searching it, and selecting items from search results.Source: §A.1 · p. 16
  • Web Shopping uses 438 post-processed training tasks and 96 test tasks, with a maximum of 20 steps per task.The five retained websites are Costco, Best Buy, Target, Walmart, and Newegg.Source: §A.1 · p. 16
  • Table 3 provides example task descriptions for the General and Web Shopping sets.The appendix identifies examples after Web Shopping post-processing; the supplied passages do not include the table entries.Source: Table 3 · p. 16

B Other Quantitative Experiments … C Qualitative Examples

On AitW General, DigiRL learns faster than filtered BC across horizon settings and produces shorter average rollouts across both subsets.

B.1 Horizon Limit

  • At H = 10 and H = 20, DigiRL speeds learning by around 100 trajectories over filtered BC, while the redundant H = 20 horizon accelerates learning for both methods.The paper describes H = 10 as sufficient and H = 20 as redundant.Source: §B.1 · p. 17

B.2 Trajectory Length

  • DigiRL achieves shorter average rollout lengths than filtered BC across both subsets, whether measuring all rollouts or only those that succeed.The paper interprets this as more efficient and directed task solving.Source: §B.2 · p. 17

C.1 Random sample of trajectories for different agents · C.2 Error Recovery · C.3 Trajectory Length

Random trajectory examples show DigiRL performing in-the-wild device-control tasks with fewer stalls or wrong-page outcomes than AutoUI and GPT-4V; it can recover from mistakes, and its trajectories are consistently shorter than filtered BC’s.

C.1 Random sample of trajectories for different agents

C.2 Error Recovery

  • DigiRL recovers from its own mistakes by exploring ways back to the original screen, while AutoUI can remain stuck on a diverged screen.The Best Buy example illustrates this contrast.Source: §C.2 · p. 20, Figure 13 · p. 20
  • DigiRL’s drive to maximize state value usually leads it to reset to the original screen, which has high value for success.Source: §C.2 · p. 20

C.3 Trajectory Length

  • Qualitative trajectory examples compare the number of steps taken by DigiRL and filtered BC.The supplied passage reports consistent cases in which DigiRL’s trajectories are shorter.Source: §C.3 · p. 20
  • DigiRL has shorter trajectory lengths than filtered BC in consistent observed cases.Source: §C.3 · p. 20

C.4 Reasoning failure of GPT-4V · D Fine-grained failure modes

GPT-4V often fails when its GUI actions do not realize its high-level plan, then hallucinates that it reached the right app or page and cannot recover. The fine-grained breakdown identifies six failure modes, spanning action errors, mistaken completion, premature quitting, and technical obstacles.

C.4 Reasoning failure of GPT-4V

  • GPT-4V’s misclicks can lead it to mistake the current app or page, continue under that false belief, and fail to recover.In web shopping, it mistakes Google’s search bar for Newegg’s after failing to reach the site; on Costco, it clicks an advertisement and continues down the wrong path.Source: §C.4 · p. 20, Figure 15 · p. 22, Figure 16 · p. 22
  • A core failure is grounding precise GUI actions in the interface, leaving the model’s actions misaligned with its thoughts and intermediate goals.For example, GPT-4V reaches the clock app but fails to set a 4pm alarm within the available rounds.Source: §C.4 · p. 21, Figure 17 · p. 23

D Fine-grained failure modes

  • The six-mode breakdown includes failure to recover from mistakes and stopping at a wrong but relevant page while believing the task is complete.Table 5 presents the translation between the fine-grained and coarse-grained failure modes.Source: D Fine-grained failure modes · pp. 21, 23
  • Failure to click covers missing the intended element and repeatedly clicking nearby, or failing to begin typing when expected.The breakdown describes these as failure to click on the right link or failure to click.Source: D Fine-grained failure modes · p. 21
  • Failure to take reasonable attempts describes trajectories with no clear reason for failure that never seem on track to complete the task.Source: D Fine-grained failure modes · p. 21
  • Quitting or pressing HOME early means ending or restarting before the task is finished.Source: D Fine-grained failure modes · p. 21
  • Technical issues include impossible tasks, such as opening an uninstalled app, and temporary website blocks caused by frequent visits.Source: D Fine-grained failure modes · p. 23

E Experiment machines … I Hyperparameters

The appendix specifies the compute platform, parallel emulator setup, screenshot-based evaluator, zero-shot prompting, and hyperparameter tuning.

E Experiment machines

F Setup for parallel environment

  • Independent emulator server processes communicate with the trainer through UIAutomotor servers, which translate high-level instructions into UI commands and recover from errors by resetting processes or servers.The trainer stops and resets a UIAutomotor server after its exception to preserve data correctness.Source: F Setup for parallel environment · p. 24
  • A GPU-equipped host distributes its current policy to worker machines for parallel task trajectories, then aggregates synchronized trajectories to update the policy iteratively.The figure describes workers as CPU-only, while the prose specifies one GPU and multiple CPUs per worker.Source: F Setup for parallel environment · p. 24, Figure 19 · p. 24

G Autonomous evaluator details

H Zero-shot Baseline Details

I Hyperparameters

  • The final DigiRL and Filtered BC hyperparameters were tuned by binary search on the General and Web Shopping training sets; DigiRL adds value-function and instruction-value-function training settings and GAE λ.The added settings include iteration counts and learning rates; the final choices are listed in Table 6.Source: I Hyperparameters · p. 27, Table 6

Related papers

Loading 2406.11896v1…