Source-linked AI summary
UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning
Zhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen, Haiyang Xu, Ziwei Zheng, Weiming Lu, Ming Yan, Fei Huang, Jun Xiao, Yueting Zhuang
TL;DR
GUI automation must reconcile stable offline RL, which struggles with multi-step execution, with online RL, which captures long-horizon signals at high deployment cost. Semi-online RL simulates online rollouts on offline trajectories using adaptive patching and long-term advantages, while SOP provides a stronger proxy for online performance. UI-S1-7B achieves state-of-the-art results among open-source 7B models across multi-turn benchmarks, including gains of +12.0% on AndroidWorld and +23.8% on AITW.
Problem
Offline RL trains stably but struggles with multi-step execution, whereas online RL captures trajectory-level signals through interaction at prohibitive cost.
Method
Semi-online RL simulates online rollout dynamics on static trajectories with a Patch Module, discounted future returns, and weighted step-level and episode-level advantages.
Results
UI-S1-7B achieves state-of-the-art performance among open-source 7B models across multi-turn benchmarks, improving +12.0% on AndroidWorld and +23.8% on AITW compared with its base model.
Takeaways & Limitations
Semi-online RL bridges offline training efficiency and online multi-turn reasoning while maintaining competitive single-turn performance.
Abstract
from arXiv · showhide
Graphical User Interface (GUI) agents have demonstrated remarkable progress in automating complex user interface interactions through reinforcement learning. However, current approaches face a fundamental dilemma: offline RL enables stable training on pre-collected trajectories, but struggles with multi-step task execution for lack of trajectory-level reward signals; online RL captures these signals through environment interaction, but suffers from sparse rewards and prohibitive deployment costs. To address it, we present Semi-online Reinforcement Learning, a novel paradigm that simulates online RL on offline trajectories. During each rollout process, we preserve the original model output within the multi-turn dialogue, where a Patch Module adaptively recovers the divergence between rollout and expert trajectories. To capture long-term training signals, Semi-online RL introduces discounted future returns into the reward computation and optimizes the policy with weighted step-level and episode-level advantages. We further introduce Semi-Online Performance (SOP), a metric that aligns better with true online performance, serving as a practical and effective proxy for real-world evaluation. Experiments show that ours Semi-online RL achieves SOTA performance among 7B models across four dynamic benchmarks, with significant gains over the base model (e.g., +12.0% on AndroidWorld, +23.8% on AITW), demonstrating significant progress in bridging the gap between offline training efficiency and online multi-turn reasoning. The code is available at https://github.com/X-PLUG/MobileAgent/tree/main/UI-S1.
1 INTRODUCTION
GUI automation faces a trade-off between stable offline RL with weak multi-step execution and costly online RL with sparse rewards. Semi-online RL simulates online dynamics on static trajectories, while SOP better tracks online performance and UI-S1-7B achieves strong multi-turn results without sacrificing single-turn capability.
- Motivation: Offline RL trains stably with step-wise supervision but often fails on multi-step tasks, whereas online RL captures interaction dynamics at prohibitive deployment cost.Offline methods face a training–evaluation mismatch, while online methods encounter sparse and delayed GUI rewards.
- Evaluation metric: SOP correlates more strongly with online AndroidWorld performance than traditional offline metrics, with R2=0.934 versus AC-High at R2=0.470.Figure 2 presents the correlation comparison and identifies superior UI-S1-7B performance on both metrics.
- Results: +12.0% on AndroidWorld and +23.8% on AITW-Gen versus Qwen2.5VL-7B, while UI-S1-7B reaches state-of-the-art among open-source 7B models across multi-turn benchmarks.The reported gains cover dynamic AndroidWorld and AITW-Gen settings, with SOP included among the evaluated multi-turn settings.
- Semi-online RL: Semi-online RL simulates online rollout dynamics using static trajectories and uses a Patch Module to recover action mismatches by injecting expert actions.The approach preserves model outputs and improves trajectory utilization during multi-turn learning.
- Policy optimization: Semi-online RL incorporates discounted future returns and dual-level advantages to balance step-level accuracy with trajectory-level task completion.The optimization combines step-level and episode-level signals rather than relying only on immediate step supervision.
2 RELATED WORK
Prior GUI-agent research used supervised learning for strong single-step accuracy, while newer work explored multi-turn reinforcement learning through online environment interaction. These approaches still involve challenges such as sparse rewards, infrastructure demands, and limited exploration diversity.
- GUI Agents with Reinforcement Learning: Earlier GUI agents relied on large annotated datasets and supervised fine-tuning to achieve impressive single-step accuracy.The related work discusses AGUVIS, OS-Atlas, UGround, SeeClick, and UI-TARS as examples.
- Multi-Turn Reinforcement Learning: Recent multi-turn RL methods use online environment interaction to address the limitations of single-step optimization.The cited examples include ARPO and MobileGUI-RL.
- Multi-Turn Reinforcement Learning: Online multi-turn approaches face sparse-reward and scaling challenges, including extensive parallel infrastructure and limited exploration diversity.These constraints are explicitly associated with ARPO in the related-work discussion.
3 METHOD
Semi-online RL simulates online interaction using offline trajectories while preserving policy-generated context and recovering action mismatches. Its optimization combines hierarchical rewards, discounted future returns, and dual-level advantages for multi-turn task learning.
- 3.2 Semi-online Rollout: Semi-online rollout maintains policy-generated history while using offline expert trajectories to approximate environment dynamics and continue interaction after matching actions.The process conditions subsequent actions on the agent’s own generated history rather than only expert context.
- 3.3 Patch Module for Trajectory Recovery: The Patch Module replaces a mismatched action with the expert action and inserts synthetic reasoning so rollout learning can continue through the remaining trajectory.This avoids discarding later trajectory steps after early divergence; three strategies generate the synthetic reasoning differently.
- 3.3 Patch Module for Trajectory Recovery: Thought-Free, Off-Policy Thought, and On-Policy Thought patches trade minimal intervention, auxiliary-model reasoning, and policy-consistent reasoning, respectively.Off-Policy Thought Patch may introduce distribution shift, whereas On-Policy Thought Patch uses the current policy with expert action hints.
- 3.4 Semi-online Policy Optimization: Dual-level advantages combine step-level comparisons across rollouts with episode-level trajectory returns, balancing local accuracy and global task completion in policy optimization.The resulting objective uses Patch Module-enhanced trajectories, importance sampling, a clipped policy term, KL regularization, and dynamic sampling for minimum advantage variance.
- 3.4 Semi-online Policy Optimization: The step reward combines formatting, action-type, and exact-match accuracy, while discounted future returns propagate later consequences through the rollout.The reward weights are 0.1 for formatting, 0.4 for action type, and 0.5 for exact-match accuracy, with indicator conditions on later terms.
4 EXPERIMENT
Experiments evaluate Semi-online RL on multi-turn and single-turn GUI benchmarks, analyze patching, scaling, SOP, discounting, and training-paradigm choices. UI-S1-7B achieves strong multi-turn results while retaining single-turn capability.
- Multi-turn Performance: UI-S1-7B establishes state-of-the-art performance among open-source 7B/8B models across evaluated multi-turn benchmarks, including AndroidWorld and AITW-Gen.It improves over Qwen2.5VL-7B by +19.1% on AndroidWorld and +23.8% on AITW-Gen.
- Single-turn Performance: +15.5% AndroidControl-High SR and +7.1% GUI Odyssey SR show that Semi-Online RL retains competitive single-turn performance.The reported results indicate gains over the base model despite the method targeting multi-turn execution.
- Patch Module Analysis: Increasing patch threshold ϵ improves SOP and AndroidWorld results, with Thought-Free Patch SOP rising from 22.3 to 25.7 at 1000 samples.The reported increase is a 15% relative improvement and is associated with greater exposure to later trajectory steps.
- Patch Module Analysis: On-Policy Thought Patch reaches SOP 26.1 at ϵ=∞, while Thought-Free Patch reaches 25.7 at lower computational cost and Off-Policy Thought Patch reaches 22.6.The final configuration adopts Thought-Free Patch with ϵ=1 based on performance-efficiency considerations.
- Training Dynamics: Larger ϵ values improve data efficiency, while γ=0 performs worst and γ=0.5 gives the best performance in the reported setting.The discount analysis associates future reward discounting with improved long-horizon optimization for multi-turn tasks.
- Semi-Online Performance Metric: SOP achieves R2=0.934 correlation with AndroidWorld, exceeding AndroidControl-High at R2=0.470 while requiring minimal evaluation time.SOP evaluates model-generated history across multi-turn tasks and is positioned as a proxy for online evaluation.
- Training Paradigm: 34.0% AndroidWorld performance from SFT combined with Semi-Online RL exceeds 30.4% for Semi-Online RL only and 21.7% for SFT only.The combined approach also reduces average completion steps, while episode-level advantages and multiple historical images contribute to performance.
5 CONCLUSION
The paper presents Semi-online RL as a bridge between offline training stability and online long-horizon GUI policy optimization. Experiments report strong multi-turn performance without compromising single-turn capability.
- 5 CONCLUSION: Semi-online RL bridges offline and online reinforcement learning for GUI agents by enabling stable yet long-horizon-capable policy optimization.UI-S1-7B achieves state-of-the-art results among open-source 7B-scale models across dynamic and static multi-turn benchmarks.
A.1 NOTATION DEFINITION
The appendix provides notation definitions used in Section 3.
- A.1 NOTATION DEFINITION: Table 5 defines the notation used throughout Section 3.
A.2 PATCH MODULE
The Patch Module rollout initializes an offline trajectory state, samples policy outputs, and either continues, patches mismatches, or terminates after reaching the threshold.
- A.2 PATCH MODULE: The rollout initializes history H1, an empty trajectory, and patch counter c before beginning from the offline trajectory state.The algorithm then samples outputs from the current policy conditioned on state and history.
- A.2 PATCH MODULE: When the sampled action mismatches the expert action, the algorithm applies the patch function while c remains below threshold ϵ.The patch counter increments after each recovery operation.
- A.2 PATCH MODULE: Once the maximum number of patches is reached, the rollout terminates instead of producing the next state.Otherwise, history and state are updated and the rollout proceeds to the next step.
A.3 SOP
SOP evaluates GUI agents across success, trajectory-step coverage, and alignment with online performance. Compared with offline metrics, SOP shows stronger correlation with online metrics while requiring minimal evaluation time.
- Metric definition: SOP combines task success and trajectory-step coverage into a semi-online evaluation score.For each task, PG measures successful steps relative to expert-trajectory steps, while TSR captures task-level success; the overall score is Score = PG+TSR.
- Alignment with online evaluation: SOP shows stronger correlation with online metrics than AC-High and GUI Odyssey.Figure 10 compares online metrics with offline metrics and SOP; SOP reaches R2=0.934 in the reported comparison.
- Correlation analysis: R2 measures the proportion of variance explained by a regression model, with higher values indicating a better fit.The metric is defined from residual and total sums of squares and ranges from 0 to 1.
A.4 MODEL SIZE SCALING
Semi-online RL improves performance across 3B, 7B, and 32B models, indicating generalization across model sizes. However, relative gains diminish as model size increases.
- Model-size scaling: Relative performance improvements diminish as model size increases.
A.5 OTHER ABLATIONS
The ablations examine future-reward and training hyperparameters, patch-related behavior, and evaluation examples. Results favor natural trajectory endpoints for future rewards and larger patch thresholds, while qualitative cases include both successes and a multi-step arithmetic failure.
- Hyperparameters: The final training configuration selects γ=0.5, ω=1, and η=0.3 from the tested hyperparameter settings.The ablation varies future-reward discount γ, advantage weight ω, and DAPO threshold η while reporting SOP.
- Future reward: Setting the future-reward endpoint to the last step of a natural trajectory segment outperforms using the final step of the entire trajectory.
- Qualitative cases: The reported qualitative cases include successful tasks across AITW-Gen, AndroidWorld, and MiniWob++, plus an AndroidWorld failure caused by an incorrect product calculation.The failed case reports calculating 9 · 10 · 9 · 5 · 5 as 2250 at step 11.
- Prompt and implementation: The agent prompt integrates global objectives, action history, thought history, screenshot history, and manager planning into end-to-end reasoning.The prompt specifies separate thought and action formats and requires historical context to guide subsequent decisions.