PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps

Peitong Duan, Khashayar Gatmiry, Jeffrey Nichols, Samy Bengio, Emmanuel Abbe

arXiv:2610.39264v1cs.AI

TL;DR

Frontier models can be costly for everyday UI navigation, while smaller agents trained from static trajectories or sparse rewards can be brittle. This paper trains a decoupled planner and grounder with online RL on live iOS apps, using dense and terminal rewards. Its 8B agents outperform SFT on evaluated benchmarks, with nearly sevenfold higher success than SFT on AndroidWorld.

  • Problem

    Frontier UI models can cost several dollars per task, while SFT and existing RL approaches can rely on static trajectories or sparse terminal rewards.

    Source: §1 · pp. 1–2

  • Method

    The framework trains a planner with online RL on live iOS apps, pairs it with a specialized UI grounder, and uses LLM-judged dense step-level and terminal rewards.

    Source: §1 · p. 2

  • Results

    On evaluated benchmarks, 8B online RL agents substantially improve over SFT; on AndroidWorld they achieve nearly sevenfold higher success than the SFT baseline.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The AndroidWorld result supports the paper’s conclusion that online RL can teach navigation strategies that generalize beyond iOS training.

    Source: §1 · p. 2

    On AndroidWorld, frontier models still outperform the paper’s RL checkpoints.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation. Furthermore, we introduce GiGPPO, a novel neighbor-conditioned value estimation method that improves critic stability by leveraging reward signals from visually similar states across concurrent trajectories. To support this online training paradigm at scale, we introduce an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices. Evaluations on rigorous benchmarks, including UINavBench, iOSWorld, and AndroidWorld, demonstrate that our online RL trained agents substantially outperform both SFT and terminal-only RL baselines while accelerating training convergence. Notably, our 8B RL agents achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments, while outperforming Gemini-3.5-Flash and tying with Gemini-3.1-Pro on real iOS applications.

1 INTRODUCTION

The paper addresses the cost of frontier UI agents and the brittleness of smaller agents trained on static trajectories or sparse terminal rewards. It proposes online RL on live iOS apps, combining specialized planning and grounding with dense and terminal feedback; its agents improve over baselines and generalize to AndroidWorld.

1 INTRODUCTION

  • The framework trains a Planning Agent on live iOS applications and pairs it with a specialized UI Grounding Model that converts planned steps into executable actions and coordinates.The planner can ask users to specify preferences when tasks are underspecified; computationally expensive RL is reserved for planning.Source: §1 · p. 2
  • An LLM-as-a-Judge supplies dense step-level progress feedback and terminal task-completion rewards, optimized with PPO and step-level advantage estimation.The dense reward also penalizes hallucinations and unproductive loops and is intended to encourage recovery from unexpected states.Source: §1 · p. 2
  • A cloud-device infrastructure with rapid virtual-machine snapshots supports hundreds of parallel live iOS trajectories, enabling at-scale training on real applications.The authors describe this as the first at-scale training of UI agents entirely on live iOS devices.Source: §1 · p. 2
  • On AndroidWorld, 8B online RL agents achieve nearly sevenfold higher success than the SFT baseline; on UINavBench they outperform Qwen3.8-27B and Gemini-3.5-Flash and tie with Gemini-3.1-Pro.The agents also substantially improve over SFT and terminal-only RL baselines.Source: §1 · pp. 2–3
  • An asymmetric discounting strategy balances local process supervision with terminal task completion to address reward hacking from prolonged trajectories.The strategy is described as preventing agents from extending trajectories to collect step-level rewards.Source: §1 · p. 2
  • GiGPPO stabilizes the PPO critic by incorporating reward signals from visually similar states across concurrent trajectories.Source: §1 · pp. 2–3

2 RELATED WORK

Prior UI navigation training approaches include imitation of successful trajectories and RL methods with sparse terminal feedback or localized checks. The paper contrasts its live-iOS, decoupled approach with concurrent work and situates it among benchmarks that assess live-environment task execution.

2.1 UI NAVIGATION AGENTS

  • SFT demonstrations often omit errors and recovery, while many RL approaches rely on golden-trace similarity or terminal rewards that do not credit partial progress.The paper contrasts its online dense-progress and terminal-reward approach with localized hand-curated checks and terminal-only training.Source: §2.1 · p. 3
  • Unlike ClawGUI-RL, this work uses a specialized RL-trained planner with a frozen SFT grounder and trains on hundreds of concurrent live iOS trajectories.ClawGUI-RL trains end-to-end on Android emulators at 64 concurrent environments; the paper also distinguishes its learned value function from GiGPO’s averaging of recurring states.Source: §2.1 · pp. 3–4

2.2 UI NAVIGATION BENCHMARKS

  • UI navigation benchmarks have shifted from static screenshot-action pairs toward live environments, with UINavBench and iOSWorld adding live-device iOS evaluation.The cited benchmarks include workflows requiring planning across multiple applications.Source: §2.2 · p. 4

3 UI NAVIGATION AGENT SYSTEM AND TRAINING

The system frames UI navigation as iterative screen-conditioned action selection, separates planning from coordinate grounding, and trains the planner online on live iOS apps. Its training combines LLM-judged step and terminal rewards with PPO variants and scalable device rollouts.

3.1 PROBLEM FORMULATION

  • The agent repeatedly observes a screenshot, selects a screen-specific action toward a user goal, and revises its sequence when the environment deviates.Episodes end when it selects TASK COMPLETE or reaches the maximum step limit.Source: §3.1 · p. 4

3.2 ARCHITECTURE

  • The architecture separates open-ended planning from grounding, which maps each plan to an action primitive and screen coordinates.The Planning Agent maintains reasoning and a remaining plan; training infrastructure runs 512 simultaneous trajectories on virtual iOS devices.Source: §3.2 · pp. 4–5

3.3 SFT WARM START

  • The Planning Agent is warm-started with supervised fine-tuning on human-annotated trajectories from in-house iOS datasets and Android GUIOdyssey.Gemini generates reasoning and remaining-plan targets because the datasets do not record them natively.Source: §3.3 · p. 5

3.4 ONLINE REINFORCEMENT LEARNING SETUP

  • Online RL trains the agent through live iOS rollouts, with LLM judges providing dense step rewards and terminal task evaluation.The judge-based reward requires no ground-truth action sequence, supporting training on synthetic multi-app tasks; Gemini-3.5-Flash simulates user replies to ASK USER actions.Source: §3.4 · pp. 5–6

3.5 ONLINE REINFORCEMENT LEARNING OPTIMIZATION

  • PPO is used because its critic can exploit dense rewards; discounting dense rewards while keeping terminal reward undiscounted addresses reward hacking and preserves task-completion credit.With γ = 1, the agent can prolong trajectories to collect locally positive dense rewards instead of terminating after completion; the dense component instead uses γ = 1/2.Source: §3.5 · pp. 6–7
  • GiGPPO conditions value estimates on a visually nearest sibling state, its return, and screen distance; critic warm-up precedes joint training.The critic is warm-started for 15 steps, and GAE with λ = 1 uses realized trajectory return minus the critic’s state-value estimate.Source: §3.5 · pp. 7–8

4 EXPERIMENTS

The experiments assess the agents on iOS benchmarks and out-of-distribution Android, using a specialized grounding setup. Online RL improves on SFT, while dense rewards and GiGPPO accelerate training; iOSWorld remains challenging.

4.1 BENCHMARKS

  • UINavBench covers 116 tasks across 36 real iOS apps, iOSWorld 133 tasks across 26 synthetic apps, and AndroidWorld 116 tasks across 20 real applications.iOSWorld includes 46 personalization tasks; both iOS benchmarks mix single-app and multi-app tasks, and success is verified by a VLM.Source: §4.1 · p. 8

4.2 UI GROUNDING MODELS

  • The RL runs used an internal 3B grounding model with over 90% held-out accuracy, routing all swipes to Gemini-3.5-Flash because the internal model lacks location-based swiping.Reported benchmark success rates use the optimal grounding configuration for each benchmark.Source: §4.2 · p. 8

4.3 RESULTS

  • Online RL generally beat SFT across benchmarks; PPO Step 336 reached 32.8% vs. 4.7% on AndroidWorld and tied Gemini-3.1-Pro at 32.4% on UINavBench.The reported AndroidWorld result is nearly seven times the SFT success rate.Source: §4.3 · p. 9, Table 1 · p. 9
  • Standard PPO at Step 168 matched or exceeded terminal-only PPO at Step 336 across benchmarks; GiGPPO also learned faster than standard PPO on UINavBench.The authors suggest GiGPPO’s gains there may reflect value estimates using signals from visually similar screens, since UINavBench most closely matches the real-iOS training environment.Source: §4.3 · p. 9
  • iOSWorld gains were modest except GiGPPO Step 336, which reached 9.8% vs. 4.5% for SFT; synthetic layouts differ from training apps, and tasks include complex workflows.The cited task types include personalization and multi-app workflows; both are out-of-distribution for the agent’s training.Source: §4.3 · pp. 9–10
  • On AndroidWorld, frontier models still outperformed the RL checkpoints; the authors suggest public Android training data may contribute to the gap.They contrast the availability of large public Android UI datasets with the scarcity of public iOS datasets.Source: §4.3 · p. 10

5 CONCLUSION

The conclusion presents online RL on live iOS apps as a way to train UI navigation agents beyond SFT’s brittleness and frontier models’ high inference costs. The agents improve on SFT, accelerate convergence, and transfer to Android.

5 CONCLUSION

  • Across UINavBench, iOSWorld, and AndroidWorld, online RL agents substantially outperform SFT, while dense rewards and GiGPPO accelerate convergence against terminal-only rewards.On real iOS applications, the 8B agents surpass multiple frontier models and match Gemini-3.1-Pro; on Android they achieve nearly seven times the SFT baseline.Source: §5 · p. 10
  • The framework trains a planner online on live iOS apps, pairs it with a frozen SFT grounder, and uses dense step-level and terminal rewards with PPO and GiGPPO.GiGPPO estimates values using visually similar states across concurrent trajectories; the conclusion also notes asymmetric discounting to prevent reward hacking.Source: §5 · p. 10
  • The authors frame the framework as a pathway toward robust, generalizable, and economically viable UI navigation agents.The reported scope includes evaluation on UINavBench, iOSWorld, and AndroidWorld.Source: §5 · p. 10

AI USE STATEMENT

The statement describes generative AI use in data generation, grounding, reward judging, software development, literature identification, and manuscript editing.

AI USE STATEMENT

  • Generative AI was used to synthesize data, perform UI grounding, and provide LLM-as-a-Judge reward functions during RL training.Source: AI USE STATEMENT · p. 10
  • The authors also used generative AI tools to write software code, identify relevant literature, and edit the paper for readability.Source: AI USE STATEMENT · p. 10
  • The authors report reviewing AI-assisted work and verifying and testing all LLM-generated code for correctness.Source: AI USE STATEMENT · p. 10

A.1 PPO TRAINING DETAILS

The appendix specifies the PPO training setup, including model initialization, rollout sampling, advantage estimation, optimization, and distributed infrastructure.

A.1 PPO TRAINING DETAILS

  • The planner starts from an 8B vision-language SFT checkpoint; the critic uses that checkpoint with a fresh value head and LoRA rank 32, excluding visual modules.The neighbor-conditioned network adds about 0.9M trainable parameters.Source: §A.1 · p. 14
  • Each batch samples 16 tasks with 8 trajectories per task, for 128 trajectories, and episodes are capped at 30 agent steps.Rollouts use temperature 1.0; evaluation uses temperature 0.7 with five samples per task.Source: §A.1 · p. 14
  • Advantages use step-level GAE with γ = λ = 1 and dense discount γ_dense = 1/2; the dual-clipped objective uses a KL penalty of 0.01.The clipping parameters are ϵ_low = 0.18, ϵ_high = 0.22, and c = 3; value updates are clipped at 0.5.Source: §A.1 · p. 14
  • The policy uses AdamW at 1 × 10^-6 and the critic at 1 × 10^-4, with weight decay 0.01 and gradient-norm clipping 1.0.The policy takes two optimizer steps per batch; the critic takes one, and the critic trains alone for the first 15 steps.Source: §A.1 · p. 14
  • Training runs on 8 GPUs under FSDP, with 16 generator shards each driving up to 32 concurrent virtual iOS devices.Source: §A.1 · p. 14

A.2 GRPO

The GRPO setup trains the planning agent from an 8B SFT checkpoint using group-relative returns, while the reward design combines dense step rewards with a terminal reward. Because all rewards are nonnegative, wrong actions are not explicitly penalized and accumulated partial credit can outweigh task completion.

A.2 GRPO

  • The agent starts from an 8B vision-language SFT checkpoint and uses GRPO without a value network, computing advantages from relative returns among same-task rollouts.Each step samples 32 tasks with four trajectories per task; episodes are capped at 30 steps.Source: §A.2 · p. 14
  • Each trajectory sums dense per-step rewards and a terminal reward, with returns in [0, 40] and every reward nonnegative.Successful steps receive +1; terminal success receives +10 and blocked completion +5.Source: §A.2 · p. 14
  • Because wrong actions and inaction both receive zero, harmful steps are not distinguished from neutral ones, and partial credit can out-earn task completion.Dense credit is bounded by the 30-step limit, while terminal success contributes 10.Source: §A.2 · p. 14

A.3 GROUNDING EXPERIMENTS

The experiment compares three grounding configurations paired with the same SFT planning agent. The best grounding setup differs across benchmarks: the RL setup leads on UINavBench, while Gemini-3.5-Flash is identified as optimal for iOSWorld and AndroidWorld.

A.3 GROUNDING EXPERIMENTS

  • The comparison tested three grounding configurations paired with the SFT planning model: the internal 3B model, the RL setup, and Gemini-3.5-Flash alone.Source: §A.3 · p. 15, Table 2 · p. 15
  • The RL training setup achieved the best performance on UINavBench among the grounding configurations paired with the SFT planning agent.The authors attribute this to the 3B model’s training on real iOS screenshots and routing swipes to Gemini-3.5-Flash.Source: §A.3 · p. 15, Table 2 · p. 15
  • Gemini-3.5-Flash was the optimal grounding model for iOSWorld and was also concluded to be best for AndroidWorld.The authors relate this choice to the synthetic iOS and Android interfaces deviating from the 3B model’s training distribution.Source: §A.3 · p. 15, Table 2 · p. 15

A.4 AGENT TRACES

The example traces contrast the RL-trained agents’ ability to recover or make progress with SFT or terminal-only agents that fail to complete the illustrated tasks. The examples span UINavBench, iOSWorld, and AndroidWorld.

A.4 AGENT TRACES

  • The traces illustrate examples from UINavBench, iOSWorld, and AndroidWorld.Source: §A.4 · p. 15
  • On the Apple Maps task, GiGPPO corrects its mistaken destination and restarts, while SFT repeatedly tries to set the starting point until the step limit.The workflow requires selecting the destination before the starting location, despite the prompt naming the start first.Source: Figure 2 · p. 16
  • On the multi-app iOSWorld task, GiGPPO finds the requested order, receipt, and bank details, while SFT opens the apps but fails to search for them.Source: Figure 3 · p. 17
  • On the AndroidWorld recipe task, PPO with dense rewards searches for and deletes the recipes, while terminal-only PPO scrolls inefficiently and times out.Source: Figure 4 · p. 18

Related papers

Loading 2610.39264v1…