PaperlayerView exact revision on arXiv

Source-linked AI summary

GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning

Haolong Yan, Yeqing Shen, Xin Huang, Jia Wang, Kaijun Tan, Zhixuan Liang, Hongxin Li, Zheng Ge, Osamu Yoshie, Si Li, Xiangyu Zhang, Daxin Jiang

arXiv:2512.02423v1cs.CV

TL;DR

Proprietary GUI environments make it difficult to obtain the information needed for systematic navigation training and evaluation. The paper introduces GE-Lab and a staged training approach, reporting that SFT supports basic navigation, ST-RL generalizes to unseen scenarios, and MT-RL promotes exploration and further navigation gains.

  • Problem

    Complex, proprietary GUI environments make comprehensive environment information difficult to obtain, hindering systematic investigation and benchmarking of navigation capabilities.

    Source: §1 · p. 2

  • Method

    GE-Lab provides a simulation engine with flexibly defined screens, icons, and navigation graphs, and the paper evaluates SFT, ST-RL, and MT-RL training.

    Source: §1 · pp. 2–3

  • Results

    SFT memorizes fundamental navigation knowledge, ST-RL improves generalization to unseen scenarios, and MT-RL encourages exploration and further improves navigation performance.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors report that their systematic benchmarking clarifies reinforcement learning's strengths in GUI navigation and offers guidance for building more generalizable agents.

    Source: §1 · p. 3

    GE-Lab's interaction logic does not yet capture the full diversity and complexity of real-world GUI interactions, including varied icons, substantial text, and visual noise.

    Source: §A.2 · p. 18

Abstract

from arXiv · show

With the rapid development of Large Vision Language Models, the focus of Graphical User Interface (GUI) agent tasks shifts from single-screen tasks to complex screen navigation challenges. However, real-world GUI environments, such as PC software and mobile Apps, are often complex and proprietary, making it difficult to obtain the comprehensive environment information needed for agent training and evaluation. This limitation hinders systematic investigation and benchmarking of agent navigation capabilities. To address this limitation, we introduce GUI Exploration Lab, a simulation environment engine for GUI agent navigation research that enables flexible definition and composition of screens, icons, and navigation graphs, while providing full access to environment information for comprehensive agent training and evaluation. Through extensive experiments, we find that supervised fine-tuning enables effective memorization of fundamental knowledge, serving as a crucial foundation for subsequent training. Building on this, single-turn reinforcement learning further enhances generalization to unseen scenarios. Finally, multi-turn reinforcement learning encourages the development of exploration strategies through interactive trial and error, leading to further improvements in screen navigation performance. We validate our methods on both static and interactive benchmarks, demonstrating that our findings generalize effectively to real-world scenarios. These findings demonstrate the advantages of reinforcement learning approaches in GUI navigation and offer practical guidance for building more capable and generalizable GUI agents.

1 Introduction

GUI agents increasingly face the challenge of navigating complex screen transitions, while proprietary real-world interfaces make the environment information needed for systematic training and evaluation difficult to obtain. The introduction presents GE-Lab and a staged training approach in which SFT builds basic navigation, ST-RL generalizes to unseen paths, and MT-RL encourages exploration.

1 Introduction

  • The introduction identifies complex screen navigation—not basic form completion or icon selection—as a central challenge for GUI agents.Agents must understand the navigation map and select icons that lead to the target page.Source: §1 · pp. 1–2
  • GE-Lab flexibly defines screens, icons, and navigation graphs while exposing environment information for comprehensive agent training and evaluation.It also enables low-cost trajectory collection and systematic ablation studies.Source: §1 · p. 2
  • SFT provides a foundation for screen navigation by learning accurate transitions between arbitrary screens from sufficiently large trajectory datasets.The introduction notes that existing SFT approaches depend heavily on expert trajectories, which are costly to collect on real-world platforms.Source: §1 · p. 2
  • SFT memorizes fundamental navigation knowledge, ST-RL generalizes to unseen paths, and MT-RL promotes exploration and further improves task success.The authors also report validation on static and interactive benchmarks, with findings that generalize effectively to real-world scenarios.Source: §1 · pp. 2–3, Figure 1 · p. 2

2 Related work

Prior GUI-agent work has strengthened visual grounding and GUI interaction, but complex screen transitions and costly annotated trajectories remain challenges. Related reinforcement-learning work includes single-step GUI action correction and interactive, multi-turn training approaches.

2.1 GUI Agents

  • GUI-agent research moved from HTML and accessibility-tree methods toward LVLMs, which support visual grounding, while complex screen transitions and costly annotated trajectories remain challenges.Static benchmarks assess execution through ground-truth matching, whereas interactive benchmarks evaluate task completion in virtual machines.Source: §2.1 · p. 3

2.2 Reinforcement Learning

  • GUI reinforcement-learning methods use preference optimization or rule-based rewards to correct single-step actions, while interactive multi-turn RL work motivates training for multi-step reasoning and environmental interaction.The cited GUI methods focus on single-step correction, whereas real-world interactions often involve longer action chains.Source: §2.2 · pp. 3–4

3 Method

The method pairs a configurable GUI simulation environment with a partially observable navigation model and reinforcement-learning protocols. It contrasts single-step training on pre-constructed histories with multi-step online interaction.

3.1 GUI Exploration Lab

  • GE-Lab constructs customizable screen-transition graphs with randomized icons, enabling trajectory synthesis, interactive rewards, and static or end-to-end evaluation.The graph determines screen relationships, while generated metadata supports tasks between arbitrary screens and shortest- or redundant-path trajectories.Source: §3.1 · p. 4

3.2.1 Problem Formulation

  • Navigation is modeled as a partially observable process in which task, actions, states, observations, and probabilistic transitions define the decision context.The agent uses current observations to select actions, and the transition model maps state-action pairs to distributions over next states.Source: §3.2.1 · p. 4

3.2.2 Agent Protocol

  • The agent combines task instructions, textual observation/action history, and current screenshots, then emits ReAct reasoning and executable actions within a 12-round, under-1k-token context.The protocol caps interaction rounds at 12; the instruction, history, and observation together remain below 1k tokens.Source: §3.2.2 · pp. 4–5

3.3.1 Single-Turn Reinforcement Learning

  • ST-RL trains current-step actions against pre-constructed observation/action histories, maximizing single-step return with a composite interaction reward.The reward explicitly supervises GUI interaction.Source: §3.3.1 · p. 5

3.3.2 Multi-Turn Reinforcement Learning

  • MT-RL generates histories through online multi-step interaction and maximizes final-step reward, sparse +1 for reaching the target and 0 otherwise.The task specification and observation/action sequences are generated dynamically during interaction with the environment.Source: §3.3.2 · p. 6

4 Experiment

The experiments use a configurable tree environment and compare supervised fine-tuning, single-turn reinforcement learning, and multi-turn reinforcement learning. Reinforcement learning improves out-of-distribution performance, while MT-RL performs best on interactive benchmarks.

4.1 Environment Configuration

  • Env-Base is a tree with maximum depth 7; OOD variants change icon images, names, or positions, or add noise icons.Edge traversal represents single-step transitions and path traversal represents multi-step navigation; SFT data includes edge examples from all subtrees, including the test subtree.Source: §4.1 · pp. 6–7

4.2 ST-RL Generalizes on Out-of-Distribution Tasks

  • ST-RL scores 63.06 overall on OOD benchmarks versus SFT’s 55.45, with interactive Pass@1 of 17.22 and Pass@5 of 22.34.Figure 4 covers static accuracy by setting and difficulty, and interactive success rates across Pass@N and difficulty.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 MT-RL Explores in Interactive Environments

  • MT-RL reaches 63.25 overall on OOD tasks versus ST-RL’s 63.06 and SFT’s 55.45, and leads the interactive benchmark with Pass@1 17.47 and Pass@5 25.16.The reported navigation example shows the agent exploring alternatives and backtracking after a suboptimal state.Source: §4.3 · p. 8

5 Analysis

The analysis examines performance as task difficulty and OOD conditions vary, and tests how SFT initialization affects later ST-RL training. RL methods retain advantages on complex tasks, while the degree of SFT before RL presents a trade-off.

5.1 Performance Comparison on Tasks of Varying Difficulty

  • MT-RL has the highest interactive Pass@1 and Pass@5 success rates, while ST-RL and MT-RL outperform SFT especially on static paths of 5–7 steps.Performance drops as path length increases across the evaluated scenarios.Source: §5.1 · p. 8

5.2 Generalization in Different Out-of-Distribution Scenarios

  • Across OOD settings, ST-RL and MT-RL outperform SFT; icon or label changes reduce performance more sharply than position changes or added noise icons.In Env-Base, RL methods reach around 63% success versus 55% for SFT; under Env-Position and Env-Noise, MT-RL retains 42–44% overall success.Source: §5.2 · p. 9

5.3 Influence of SFT on the ST-RL

  • ST-RL initialized from early SFT stages learns faster and reaches higher cumulative rewards, but the authors identify a moderate SFT extent as the preferred initialization trade-off.The authors associate early-stage checkpoints with greater diversity and plasticity during subsequent RL.Source: §5.3 · p. 9

6 Generalization to Real-World

The paper evaluates simulated-environment agents on real-world GUI tasks, both without additional fine-tuning and after continual training with real-world data. The reported outcomes include non-trivial accuracy on actions unseen during training and the strongest continual-training average for MT-RL.

6 Generalization to Real-World

  • The real-world evaluation includes zero-shot static testing and continual training with real-world data to assess transferability and scalability.The continual-training experiments incorporate 24k samples from public GUI datasets.Source: §6 · p. 9, §6.2 · p. 10

6.1 Zero-Shot Evaluation on Real-World Static Benchmark

  • On the 1,569-instance static benchmark, the agent achieves 96.09% on COMPLETE and 92.25% on TYPE, while CLICK accuracy is 64.06%.The benchmark includes CLICK, COMPLETE, WAIT, SCROLL, and TYPE; the agent had not encountered WAIT, SCROLL, or TYPE during training.Source: §6.1 · pp. 9–10, Table 4 · p. 9

6.2 Evaluation on Real-World Benchmarks with Continual Training

  • 72.03% average performance is achieved by MT-RL-Continue-Train, which also scores 86.08% on ScreenSpot, 76.70% on VWB AG, and 15.51% on AndroidWorld.After continual training, the reported hierarchy is MT-RL above ST-RL above SFT across simulated and real-world GUI tasks.Source: Table 5 · p. 10, §6.2 · p. 10

7 Conclusion

The conclusion presents GE-Lab as a configurable simulation engine for GUI navigation research and describes a training progression from SFT to ST-RL and MT-RL.

7 Conclusion

  • GE-Lab supports modular construction of screens, icons, and navigation graphs, with comprehensive environment information available for training and evaluation.The authors present it as a simulation environment engine for GUI agent navigation research.Source: §7 · p. 10
  • The conclusion describes SFT as a foundation for memorizing navigation knowledge, ST-RL as improving generalization, and MT-RL as fostering exploration and stronger navigation performance.The authors also report reduced dependence on annotated data with MT-RL.Source: §7 · p. 10
  • The authors state that their benchmarking highlights reinforcement learning’s strengths for GUI navigation and offers guidance for building more capable, generalizable agents.Source: §7 · p. 10

A.1 Experimental Details

The experiments use a partitioned navigation graph, separate interactive and out-of-domain evaluations, and training configurations for SFT and RL. The section also compares training costs and sampling efficiency across the three methods.

Dataset & Benchmark

  • Env-Base is a depth-7 graph partitioned into five subtrees in a 2:2:1 ratio for SFT training, RL training, and testing, with path and edge data for navigation learning.Each subtree contains 12,439 path instances and 274 edge instances; the graph has node counts [5, 3, 2, 2, 1, 1, 0] across levels.Source: §A.1 · p. 15, Figure 6 · p. 16
  • The interactive benchmark contains 2,162 tasks formed by randomly pairing start and end nodes in a test environment isolated from training and validation.Separate icon-captioning and icon-grounding datasets each contain 2,320 instances.Source: §A.1 · p. 15, Table 6 · p. 15

Details of the Out-of-domain Test Environment

  • The OOD suite compares Env-Base with four controlled variants that shift icon appearance, labels, positions, or visual complexity while preserving task structure.One subtree is reserved for OOD evaluation with no overlap with training trajectories; the variants test robustness to visual, semantic, and spatial shifts and distractors.Source: §A.1 · pp. 15–16, Figure 7 · p. 17

Experimental Settings

  • RL hyperparameters match SFT settings unless Table 8 specifies otherwise, and simulation feedback uses cached screen metadata rather than real-time rendering.Feedback takes milliseconds per sample, compared with approximately 13 seconds for model inference.Source: §A.1 · pp. 16–17, Table 7 · p. 17, Table 8 · p. 17

Training Time and Sampling Efficiency

  • SFT trains in 3–4 GPU hours on 60,864 samples, achieving highest sample efficiency; ST-RL takes approximately 48 GPU hours on 12,439 samples, while MT-RL takes 36 GPU hours on 2,162 interactive tasks.SFT exhibits superior sample efficiency for in-distribution scenarios.Source: §A.1 · p. 16
  • ST-RL records 99,512 total interactions and MT-RL 89,939, with both using 8 generations per training round.The paper links the increased interaction volume to more comprehensive state-space exploration and improved robustness in out-of-distribution and interactive scenarios.Source: §A.1 · p. 17

A.2 Limitations

GE-Lab does not yet capture the full diversity and complexity of real-world GUI interactions, and its current action space is intentionally restricted.

A.2 Limitations

  • The current simulation does not fully capture real-world interface diversity, including broader varieties of icons, substantial text, and visual noise.The authors plan to enrich the simulation with these elements to better approximate real-world scenarios.Source: §A.2 · p. 18
  • The current action space is restricted to focus analysis on navigation strategies, whereas real-world GUI interactions also include gestures such as swiping and long-pressing.Supporting more complex interactions is identified as future work.Source: §A.2 · p. 18

A.3 Broader Impact

The paper presents staged training and simulation-based analysis as contributions to developing GUI agents. It also notes potential productivity and accessibility benefits alongside ethical concerns about autonomous interaction with real-world systems.

A.3 Broader Impact

  • The staged training paradigm moves agents from acquiring essential knowledge to generalizing to novel scenarios and developing exploration strategies through interactive trial and error.The stages are supervised fine-tuning, single-turn reinforcement learning, and multi-turn reinforcement learning.Source: §A.3 · p. 18
  • GE-Lab enables controlled analysis while circumventing the complexities and proprietary constraints of real-world GUI platforms.The paper presents simulation-based training and evaluation as a safer pathway for testing agent capabilities before real-world deployment.Source: §A.3 · p. 18
  • More capable GUI agents could enhance productivity and accessibility through routine-task automation, but real-world autonomy raises privacy, unintended-action, and transparency concerns.The paper frames these as ethical considerations for deployment.Source: §A.3 · p. 18

A.4 Reward Function Design

The reward signal combines action-type, coordinate, intent-matching, and format-matching rewards. The authors use equal weights and report that adding format matching improves explanation standardization, navigation success, and generalization in GE-Lab.

A.4 Reward Function Design

  • The complete reward signal R combines four complementary components for action type, coordinate accuracy, intent matching, and format matching.The components are summed with equal weights.Source: §A.4 · p. 18
  • Equal weights are used because type and format rewards optimize early as syntactic constraints, while coordinate and intent rewards are strongly correlated.In 1,000 sampled interactions, P(R_coord=1 | R_intent=1) ≈ 96.3% and P(R_intent=1 | R_coord=1) ≈ 94.2%; the authors say equal weighting avoids extra hyperparameter tuning.Source: §A.4 · pp. 18–19
  • The action-type reward checks whether the generated action is the optimal type; the action space contains click and complete.The paper describes this as helping the agent identify the action type required by the task context.Source: §A.4 · p. 19
  • The coordinate reward checks whether a click falls within the correct interactive region, while complete actions receive reward 1 without spatial coordinates.The region is defined by the bounding box of the interactive element.Source: §A.4 · p. 19
  • The intent reward checks whether the explanation names the clicked element exactly or, for completion, indicates the target page.The paper describes exact-name matching for clicks and a target-page check for complete actions.Source: §A.4 · p. 19
  • The format reward checks whether both the generated action and explanation match their respective reference formats.Format matching can concern structure, key fields, or style.Source: §A.4 · p. 19
  • Adding format matching makes generated explanations more standardized and further improves navigation success and generalization in GE-Lab.The reward is described as supporting standardized explanation expression alongside action selection, localization, and semantic understanding.Source: §A.4 · p. 19

A.5 Real-World Benchmark and Training Data

This section covers real-world evaluation and training data, training stability and reward-hacking mitigation, navigation behavior and difficulty, and prompt instructions for classifying GUI tasks, analyzing screens, and formatting responses and coordinate-based actions.

A.10 System Prompt

  • The system prompt directs the agent to classify navigation, icon-grounding, or icon-understanding requests and respond using the task-specific output format.Inputs may include the user request, optional task history, and current screen image; navigation actions include clicking, typing, scrolling, waiting, or completing the task.Source: §A.5 · p. 20

A.5 Real-World Benchmark and Training Data

  • The 1,569-instance real-world GUI benchmark spans four datasets, five action types, and diverse domains; WAIT, SCROLL, and TYPE are unseen during simulated training.The benchmark uses original-dataset screenshots and GPT-4o-generated instructions and action annotations, followed by human quality assurance. Eight additional benchmarks cover static grounding and interactive Android task completion; 24,000 samples from the same sources support continual training under matched settings.Source: §A.5 · pp. 20–21

A.6 Training Stability

  • Across seeds, ST-RL reward rises from 3.302±0.192 to 3.760±0.175, and MT-RL reward from 0.394±0.081 to 0.745±0.079, with standard deviations below 0.2.Evaluation scores increase from 55.13 to 63.67 for ST-RL and from 63.46 to 64.40 for MT-RL; both methods show smooth, monotonic convergence.Source: §A.6 · pp. 21–22, Table 10 · p. 22

A.7 Reward Hacking in MT-RL

  • MT-RL reward hacking arises when final-turn-focused rewards and a “complete” action encourage premature task termination instead of effective click sequences.Reducing action-space diversity, primarily retaining click actions, mitigates the shortcut: the no-hack model shows stable output length and progressively increasing reward.Source: §A.7 · p. 22

A.8 Case Study

  • In three navigation cases, SFT fails after deviations, ST-RL succeeds on simpler routes but fails on a complex one, and MT-RL recovers from errors and discovers an unseen route.In the third case, MT-RL reaches the target by a novel 8-step path absent from training data; collectively, the cases illustrate exploration and error recovery.Source: §A.8 · pp. 23–25

A.9 Performance Comparison on Tasks of Varying Difficulty

  • As path length increases, success rates decline across methods, while ST-RL and MT-RL outperform SFT and remain relatively stable at 5–7 steps.The passage identifies path length as a critical indicator of task difficulty.Source: §A.9 · p. 25, Table 11 · p. 25

A.10 System Prompt

  • The prompt asks the agent to classify the request as navigation, grounding, or understanding and thoroughly analyze the current screen state.The screen state may be provided as a description or image.Source: §A.10 · p. 26
  • For coordinate-based clicks, the prompt specifies a coordinate system ranging from (0,0) to (1000,1000).Source: §A.10 · p. 26
  • The prompt requires strict adherence to the task-specific output format, including tab separators where indicated, and specifies direct-answer text and an icon-name-or-description placeholder.Source: §A.10 · p. 26
Loading 2512.02423v1…