Source-linked AI summary
DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents
Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, Kun Shao
TL;DR
On-device control agents need to handle complex commands, but fine-tuning them online and learning from distributed asynchronous data pose challenges. DISTRL combines centralized training and decentralized data acquisition with A-RIDE. The paper reports superior long-term performance over baselines and identifies broader task generalization as future work.
Problem
On-device agents need improved fine-tuning for complex GUI control, while asynchronous distributed data can be non-stationary and stale relative to policy updates.
Source: §1 · p. 2Method
Results
DISTRL demonstrates superior long-term performance compared with the baselines at convergence or after extended training budgets.
Source: §6.4 · p. 9Takeaways & Limitations
The framework provides a distributed pipeline connecting real-time mobile interactions with training infrastructure for adaptive learning.
Source: §7 · p. 11The authors identify extending generalization to a broader range of tasks as future work.
Source: §7 · p. 11
Abstract
from arXiv · showhide
On-device control agents, especially on mobile devices, are responsible for operating mobile devices to fulfill users' requests, enabling seamless and intuitive interactions. Integrating Multimodal Large Language Models (MLLMs) into these agents enhances their ability to understand and execute complex commands, thereby improving user experience. However, fine-tuning MLLMs for on-device control presents significant challenges due to limited data availability and inefficient online training processes. This paper introduces DistRL, a novel framework designed to enhance the efficiency of online RL fine-tuning for mobile device control agents. DistRL employs centralized training and decentralized data acquisition to ensure efficient fine-tuning in the context of dynamic online interactions. Additionally, the framework is backed by our tailor-made RL algorithm, which effectively balances exploration with the prioritized utilization of collected data to ensure stable and robust training. Our experiments show that, on average, DistRL delivers a 3X improvement in training efficiency and enables training data collection 2.4X faster than the leading synchronous multi-machine methods. Notably, after training, DistRL achieves a 20% relative improvement in success rate compared to state-of-the-art methods on general Android tasks from an open benchmark, significantly outperforming existing approaches while maintaining the same training time. These results validate DistRL as a scalable and efficient solution, offering substantial improvements in both training efficiency and agent performance for real-world, in-the-wild device control tasks.
1 INTRODUCTION
On-device MLLM agents can interpret mobile screens and carry out complex commands, but effective real-world control remains difficult. DISTRL addresses online fine-tuning and distributed-learning challenges with decentralized data acquisition and a tailored RL algorithm.
1 INTRODUCTION
- MLLM-based mobile agents use screen observations to understand context and execute complex commands, but their real-world GUI control remains difficult.Without updating model weights, wrappers and prompting remain limited by the base model; offline datasets also miss dynamic app changes.Source: §1 · pp. 1–2
- Efficient online fine-tuning is needed because static datasets miss app changes, while asynchronous collection creates stale-policy and non-stationary-data challenges.These challenges can hinder convergence and degrade performance.Source: §1 · p. 2
- DISTRL is a scalable Android-agent RL pipeline with centralized training and decentralized data acquisition.It introduces asynchronous data collection across heterogeneous worker devices.Source: §1 · p. 2
- A-RIDE is an off-policy algorithm for distributed asynchronous data that prioritizes significant experiences while encouraging exploration.The authors validate the framework with a 1.3B-parameter T5-based multimodal generation architecture.Source: §1 · p. 2
2 RELATED WORKS
Prior work explores multimodal mobile agents, RL fine-tuning, and distributed training, but the authors identify inefficiencies and mismatches with dynamic, heterogeneous mobile environments. DISTRL and A-RIDE are presented as responses to these limitations.
2.1 MULTI-MODAL ON-DEVICE CONTROL AGENTS
- Mobile-agent approaches include prompted or fine-tuned models, but static demonstrations and synchronous multi-machine collection struggle with real-world variability and uneven task durations.DigiRL workers may wait for slower machines, despite task durations varying by up to 100 times.Source: §2.1 · p. 3
2.2 REINFORCEMENT LEARNING FOR ON-DEVICE AGENT FINE-TUNING
- A-RIDE is a simplified off-policy multi-turn RL algorithm that learns from suboptimal online interactions, addressing prior work focused on single-turn tasks.The authors describe it as reducing complexity and accelerating convergence compared with previous value-based methods.Source: §2.2 · p. 3
2.3 SCALABLE AND DISTRIBUTED RL FRAMEWORK
- General distributed RL systems and algorithms do not directly meet mobile control needs involving heterogeneous devices, fluctuating experiences, and distributed buffer management.The authors therefore build on IMPALA’s concepts while extending them for on-device control.Source: §2.3 · pp. 3–4
3 PROBLEM SETUP AND PRELIMINARIES
The paper models mobile control as a finite-horizon decision process over GUI states and actions, with sparse task-completion rewards. Because trajectories come from behavior policies while training a target policy, learning must address off-policy data; an automated evaluator supplies rewards.
3 PROBLEM SETUP AND PRELIMINARIES
- Mobile control is modeled as a finite-horizon MDP whose states are GUI observations, actions include touch events, and rewards are typically sparse and positive at task completion.The model also specifies state transitions, an initial-state distribution, and an episode horizon.Source: §3 · p. 4
- Asynchronous collection produces trajectories under behavior policies that are used to optimize a target policy, requiring off-policy learning to address policy discrepancies.At each timestep, the agent observes a state, acts, receives a reward, and transitions to the next state.Source: §3 · p. 4
- Gemini-1.5-pro evaluates task completion from the task description and screenshot, assigning reward 1 for success and 0 otherwise.The paper treats this evaluator-based termination assessment as making the problem partially observable and penalizes unexpected repetition.Source: §3 · p. 4
4 SYSTEM DESIGN
DISTRL separates online trajectory collection from policy learning, using heterogeneous workers for Android interactions and a central learner for GPU-based updates. Asynchronous queues, replay, and policy distribution let workers operate at different speeds.
4 SYSTEM DESIGN
- DISTRL decouples trajectory collection from policy learning, assigning CPU-intensive mobile interactions to working machines and policy training to GPU servers.The design is described as improving scalability, resource utilization, and dataset diversity.Source: §4 · p. 4
- A central Host Learner trains the policy while multiple heterogeneous Workers provide Android emulators or connected devices for interaction.The components coordinate through asynchronous data collection and policy updates.Source: §4 · p. 5
- The Host Learner stores worker trajectories in a circular replay buffer and uses priority-based sampling and tailored regularization when updating the policy.A FIFO trajectory queue organizes incoming experiences, and updated policies are sent back to workers.Source: §4 · p. 5
- Independent worker threads send trajectories asynchronously, allowing machines with different speeds and task durations to contribute without waiting for one another.Two 96-vCPU machines support up to 32 concurrent emulators, with scaling described as almost linear with worker performance.Source: §4 · p. 5
5 METHODOLOGY
A-RIDE adapts value-based reinforcement learning to asynchronous, off-policy mobile-agent training, combining robust policy updates with corrections and prioritized replay. Its components address stale or diverse data while balancing informative experience use and exploration.
5 METHODOLOGY
- A-RIDE combines Retrace and DPER to support stable, efficient learning from asynchronous distributed data.The approach is designed for device-control settings with limited on-device resources and distributed constraints.Source: §5 · pp. 5–6
5.1 A-RIDE: THE BACKBONE OF DISTRL
- A-RIDE uses trajectory-level value estimates to label and retain high-value trajectories, and a separate state-value network predicts whether the Monte Carlo return is positive.Advantages are computed from immediate rewards and state-value estimates, then further modified by off-policy corrections.Source: §5.1 · p. 6
- The policy objective incorporates importance sampling, entropy regularization, and a penalty for invalid actions to encourage exploration while constraining inappropriate commands.The invalid-action penalty is assigned using validation through pretrained LLMs such as Gemini.Source: §5.1 · pp. 6–7
5.2 OFF-POLICY CORRECTION: RETRACE
- Retrace(λ) corrects state-value estimates using future rewards and importance sampling, supporting off-policy learning while mitigating variance from importance weights.The correction is applied directly to V(st) to improve value estimation and the stability and convergence of the value network.Source: §5.2 · p. 7
5.3 DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY (DPER)
- DPER prioritizes trajectories using temporal-difference error, importance-sampling ratio, and policy entropy, replaying high-priority experiences more often and periodically updating priorities.The entropy term encourages exploration and is intended to help avoid early convergence to suboptimal policies in dynamic environments.Source: §5.3 · p. 7
6 EXPERIMENTS
Experiments assess evaluator agreement, training efficiency, and trained-agent success across Android task settings. DISTRL improves training and collection performance over DigiRL and achieves higher success rates than the compared agents.
6.2 BENCHMARKS AND BASELINE METHODS
- The experiments use General and Web Shopping tasks, with training instructions drawn from AitW, AndroidWorld, and expert-curated sets and evaluation on corresponding AitW test subsets.The environment used a host learner with four V100 GPUs and two worker machines with eight T4 GPUs and 96 vCPUs each, supporting parallel emulation.Source: §6.1 · pp. 7–8, §6 · p. 7, §6.2 · p. 8
6.3 VALIDATION OF OUR EVALUATOR
- With the last screenshot and two actions as evaluator input, Gemini's output differed from human assessments by less than 2%; adding longer context reduced accuracy and increased cost.The evaluation covered different policy models on General subsets from AiTW.Source: §6.3 · p. 8, Figure 5 · p. 8
6.4 TRAINING PERFORMANCE
- At 6k seconds, DISTRL's success rate was 30% higher than DigiRL (multi) and 40% higher than DigiRL (single); it was also 10% higher than DigiRL-DistRL Async.DISTRL also maintained higher proportions of success rates above 60% and 80% across training phases; Table 1 reports superior long-term performance over baselines.Source: Figure 6 · p. 9, §6.4 · p. 9
- DISTRL collected 800 trajectories in 6k seconds versus DigiRL's 300 in a multi-machine setting, and reached about 7.7 trajectories per minute with 192 CPUs.Collection scaled nearly linearly and approached the stated ideal upper bound.Source: Figure 6 · p. 9, §6.4 · p. 9
6.5 PERFORMANCE EVALUATION OF AGENTS TRAINED WITH DISTRL
- 73.2% success rate on the General test set, about 19.6% above DigiRL (multi); Web Shopping reached 68.5%, about 14.4% above DigiRL (multi).DISTRL's relative improvements over DigiRL (single) were about 22.2% on General and 14.9% on Web Shopping; the table reports means and standard deviations across three runs.Source: Table 1 · p. 10, Figure 7 · p. 10, §6.5 · p. 10
6.6 ABLATION STUDIES
- Removing DPER decreased success rate by 8%; its priority scheme replays trajectories with higher TD errors and smaller policy discrepancies, while entropy promotes exploration.The study identifies DPER as important for accelerating convergence and focusing updates on informative experiences.Source: §6.6 · p. 10
- Ablating Retrace caused a 6% success-rate drop and sharp performance decreases during training, while its off-policy correction supports stable updates from diverse trajectories.The ablation results identify both DPER and Retrace as integral to DISTRL's efficiency and robustness.Source: §6.6 · pp. 10–11
7 CONCLUSION AND FUTURE WORK
DISTRL is presented as a scalable pipeline connecting real-time mobile-device or emulator interactions with distributed training. Future work targets broader task generalization and support for more MLLMs.
7 CONCLUSION AND FUTURE WORK
- DISTRL bridges real-time mobile-device or emulator interactions with distributed training infrastructures through an efficient, adaptive learning pipeline.The conclusion describes the pipeline as robust and scalable.Source: §7 · p. 11
- Future work aims to extend DISTRL’s generalization to more tasks and integrate additional MLLMs for broader applications and benchmark evaluations.The planned work includes changes to both the training pipeline and algorithmic architecture.Source: §7 · p. 11
A.1 CASE STUDY
The case studies contrast DISTRL’s performance with other agents on Android app-operation and web-shopping tasks. DISTRL completes the messaging task and operates smoothly in the shopping case, though that case ends at the step limit.
A.1 CASE STUDY
- In the app-update case, DISTRL performs the intended operation, while DigiRL and AutoUI open the wrong apps and AppAgent with GPT-4V diverges into web searching.DigiRL’s errors are attributed to discrepancies between its older training environment and the updated one; the case study says DISTRL was trained online in the newly updated environment.Source: Figure 8 · p. 15, §A.1 · p. 15
- DISTRL completes the messaging task by reaching the New Conversation screen; DigiRL gets stuck on the updated interface, and AutoUI opens Contacts instead.GPT-4V also reaches the New Conversation screen in this simpler task, though the passage cautions that its general-purpose approach may not scale to more complex tasks.Source: Figure 9 · p. 16, §A.1 · p. 16
- DISTRL progresses smoothly through the web-shopping sequence but reaches the step limit before completing the task.The case study describes its operations as structured, despite early termination.Source: §A.1 · pp. 16–17, Figure 10 · p. 17
- On web shopping, DigiRL loops between pages and exhausts its step budget, while AutoUI wanders into unrelated apps and GPT-4V gets stuck at a cookie-settings pop-up.The discussion characterizes DISTRL and GPT-4V as smoother, but only DISTRL maintains consistently structured progression in this case.Source: §A.1 · p. 17
A.2.1 EVALUATION WITH AUTO-EVALUATOR
The evaluation uses a prompted LLM as an expert judge of task completion from a task description and device screenshot. The example contrasts the expected sent-message screen with a contact list that indicates failure.
A.2.1 EVALUATION WITH AUTO-EVALUATOR
- A prompted pre-trained LLM, Gemini-1.5-Pro, is used as an expert evaluator to generalize task assessment without manual rule definitions.The evaluator compares the expected task outcome with the screenshot’s visual content and reasoning about the task.Source: §A.2.1 · p. 18
- The evaluator judges whether the screenshot shows successful completion by comparing it with the expected outcome for the task.In the example, an open contact list without a sent message is labeled failure for the task of messaging Evelyn.Source: §A.2.1 · p. 18
A.2.2 REWARD PENALTY
The reward-penalty design uses cumulative-return rollouts and penalizes repeated nonsense actions and invalid actions.
A.2.2 REWARD PENALTY
- Repeated nonsense actions in successful AutoUI rollouts sometimes destabilized asynchronous online learning, prompting penalties for repetitions and invalid actions.Monte-Carlo rollouts compute cumulative returns and propagate them backward across transitions.Source: §A.2.2 · p. 18
A.3 SYSTEM DESIGN
This section describes DistRL’s asynchronous collection and centralized training design, along with its evaluation across Android task subsets and AiTW generalization, and discusses adaptations to changing apps, devices, and operating systems.
A.3.1 DETAILED SYSTEM DESCRIPTION
- DISTRL decouples trajectory collection on working devices from policy learning on GPU servers, asynchronously sending interaction data to servers for LLM training.The design assigns inference and CPU-intensive collection to workers and training to GPUs.Source: §A.3.1 · p. 18
A.3.2 COMMUNICATION BETWEEN HOST LEARNER AND WORKERS
- SCP over SSH transfers approximately 100 MB of LoRA weights in under 2 seconds on average, versus worker collection of 6–10 trajectories per minute and roughly 120 seconds per host policy update.The paper reports the transfer overhead as negligible relative to collection and training durations.Source: §A.3.2 · pp. 18–19, Table 2 · p. 19
A.4.1 LIMITATIONS OF TRADITIONAL METHODS
- PPO and A2C require synchronous collection and updates, while V-trace can be suboptimal when behavior and target policies diverge too little or too much.The paper states that clipping can cause this V-trace limitation and reports that its empirically validated weights balance bias and variance in asynchronous learning.Source: §A.4.1 · p. 19
A.4.2 IMPLEMENTATION DETAILS
- DPER prioritizes informative trajectories while retaining entropy-driven exploration; the policy loss also penalizes invalid actions, and trajectory-value estimates filter or prioritize data in sparse-reward settings.The replay buffer retains recent experience, and Enhanced Retrace adjusts importance-sampling corrections to control variance as behavior and target policies diverge.Source: §A.4.2 · pp. 19–20
A.4.3 HYPERPARAMETER TUNING FOR DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY
- DPER priority balances average absolute TD error, average importance-sampling ratio, and average policy entropy; TD error and entropy are normalized to [0,1], while ratio naturally lies [0,1] without scaling.Source: §A.4.3 · pp. 20–21
A.5.1 BASELINE METHODS / A.5.2 TRAINING AND TEST DATA
- The system uses a T5-based MLLM with GPT-2 decoder initialization; General Tasks training contains 600 tasks and testing uses 128 AitW tasks, while Web Shopping training contains 500 tasks and testing uses 128 AitW tasks.Source: §A.5.1 · pp. 21–22, §A.5.2 · pp. 22–23
A.5.3 DETAILED PERFORMANCE COMPARISON
- DISTRL is reported to combine high success rates with low performance variance, while DigiRL’s single- and multi-machine variants face scalability or data-collection inefficiencies and standard deviations up to ±3.1%.The paper characterizes DISTRL as robust across tasks, particularly in large-scale asynchronous settings.Source: §A.5.3 · p. 23
A.6.1 FAILURE MODES ANALYSIS
- On General tasks, DISTRL’s failure rates were 4% for recovery, 12% for getting stuck, and 2% for incorrect goals; on Web Shopping, they were 3%, 7%, and 4%, respectively.The paper reports these rates as at least three times lower than alternatives on General tasks and at least a twofold improvement on Web Shopping; Figure 11 compares failure modes across both subsets.Source: §A.6.1 · pp. 23–24, Figure 11 · p. 24
A.6.2 GENERALIZATION PERFORMANCE ON AITW SUBSETS
- 68.5% on Web Shopping, 73.5% on Install, and 70.2% on Google Apps; DistRL outperforms its counterparts on AiTW subsets.DigiRL and AppAgent show lower generalization, particularly on Install and Google Apps.Source: §A.6.2 · p. 25
A.6.3 DISTRL AGENT PERFORMANCE ON TEST SET
- Periodic evaluations across three trials show continuous improvement on AiTW’s General test set during DistRL fine-tuning.The paper distinguishes these checkpoint results from its main results, which use the final trained agent.Source: §A.6.3 · p. 25, Figure 13 · p. 25
A.7 GENERALIZATIONS TO OTHER DOMAINS
- DistRL maintains high performance on GoogleApps and Install tasks excluded from training, but substantial app variation remains challenging.The paper plans to expand exploration to a wider range of mobile applications.Source: §A.7 · p. 26
- DistRL supports varied devices, OS versions, UI changes, and app updates through ADB, emulators, human-like interactions, and system resets, though major changes may require additional training.Compatibility between AVD and app versions is a key challenge.Source: §A.7 · p. 26
- Adapting DistRL to iOS would require replacing Android-specific tools with Apple counterparts; state resets for trial-and-error learning are a key challenge, and Android remains primary.The proposed tools include XCUITest and simctl.Source: §A.7 · p. 26