PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training

Yiming Wang, Da Yin, Yuedong Cui, Ruichen Zheng, Zhiqian Li, Zongyu Lin, Di Wu, Xueqing Wu, Chenchen Ye, Yu Zhou, Kai-Wei Chang

arXiv:2510.14969v1cs.CLcs.AIcs.LG

TL;DR

Digital-agent training is limited by the expense and scarcity of diverse, high-quality UI trajectories. UI-Simulator synthesizes structured UI states and guided rollouts at scale, while UI-Simulator-Grow targets high-impact tasks and trajectory variants. On WebArena and AndroidWorld, UI-Simulator rivals or surpasses real-environment training with weaker teachers, and UI-Simulator-Grow matches Llama-3-70B-Instruct using Llama-3-8B-Instruct with 66% of the original data.

  • Problem

    Large-scale, high-quality UI trajectories are costly to collect, limiting scalable training for digital agents.

  • Method

    UI-Simulator generates structured UI states and transitions, guided rollouts, and training trajectories; UI-Simulator-Grow prioritizes high-impact tasks and synthesizes variants.

  • Results

    UI-Simulator rivals or surpasses real-environment training on WebArena and AndroidWorld with weaker teachers, while UI-Simulator-Grow matches Llama-3-70B-Instruct using Llama-3-8B-Instruct and 66% of the original training data.

  • Takeaways & Limitations

    Simulation-driven synthesis supports robust, data-efficient improvement of digital agents across web and mobile UI benchmarks.

  • Takeaways & Limitations

    The paper identifies extending the simulator beyond current UI domains and toward pixel-level simulation to narrow the sim-to-real gap as future work.

Abstract

from arXiv · show

Digital agents require diverse, large-scale UI trajectories to generalize across real-world tasks, yet collecting such data is prohibitively expensive in both human annotation, infra and engineering perspectives. To this end, we introduce $\textbf{UI-Simulator}$, a scalable paradigm that generates structured UI states and transitions to synthesize training trajectories at scale. Our paradigm integrates a digital world simulator for diverse UI states, a guided rollout process for coherent exploration, and a trajectory wrapper that produces high-quality and diverse trajectories for agent training. We further propose $\textbf{UI-Simulator-Grow}$, a targeted scaling strategy that enables more rapid and data-efficient scaling by prioritizing high-impact tasks and synthesizes informative trajectory variants. Experiments on WebArena and AndroidWorld show that UI-Simulator rivals or surpasses open-source agents trained on real UIs with significantly better robustness, despite using weaker teacher models. Moreover, UI-Simulator-Grow matches the performance of Llama-3-70B-Instruct using only Llama-3-8B-Instruct as the base model, highlighting the potential of targeted synthesis scaling paradigm to continuously and efficiently enhance the digital agents.

1. Introduction

UI-Simulator addresses the scarcity and cost of diverse UI training trajectories by synthesizing structured environments and guided rollouts at scale. Across WebArena and AndroidWorld, it achieves competitive or stronger results with weaker teachers, while UI-Simulator-Grow improves data efficiency through targeted synthesis.

  • Motivation and approach: Environment diversity is central to robustness and generalization, but parallel real UIs face high resource demands, network instability, and limited distributed support.
  • Motivation and approach: UI-Simulator synthesizes scalable training trajectories from diverse structured UI states and transitions, reducing dependence on costly real-environment data collection.Its simulator generates future states, while guided rollouts produce coherent exploration for agent training.
  • UI-Simulator-Grow: UI-Simulator-Grow targets tasks with greater learning potential and synthesizes diverse variants to accelerate improvement with fewer trajectories.
  • Results: Across WebArena and AndroidWorld, UI-Simulator achieves very competitive open-source-agent performance, greater robustness and adaptability, and can outperform real-environment training despite using GPT-4o-mini.
  • Results: UI-Simulator-Grow matches Llama-3-70B-Instruct using Llama-3-8B-Instruct and only 66% of the original training trajectories.

2. Related Works

Prior work studies world models and scalable synthetic data generation as ways to reduce the cost of training digital agents. Existing approaches convert indirect knowledge or interactions into demonstrations, motivating UI-Simulator’s simulation-based alternative.

  • World Models: World-model research models environment dynamics for decision-making and increasingly explores video- and language-based representations.
  • Synthetic Data for Digital Agent Training: Synthetic-data methods address scarce UI trajectories by converting tutorials, manuals, or environment interactions into direct task demonstrations.

3. Digital World Models in UI-Simulator

UI-Simulator represents UI environments as structured textual states and predicts transitions after agent actions with an LLM-based world model. Its pipeline combines hierarchical state generation, structured rendering, rule-based handling of deterministic actions, and optional retrieval grounding.

  • 3.1. Formulation: UI environments are modeled as states, observations, actions, and transitions, with observations extracted from state elements intersecting the viewport.The transition function may be LLM-based or rule-based.
  • 3.2. (Retrieval-Free) Simulation: The simulator predicts a next-state overview, expands it into a rich natural-language draft, and converts that draft into structured UI elements with coordinates.
  • 3.2. (Retrieval-Free) Simulation: Rule-based transitions handle actions with relatively fixed outcomes, such as scrolling, while model-based transitions generate diverse page-level changes.
  • 3.3. Retrieval-Augmented Simulation: Retrieval-augmented simulation conditions next-state generation on relevant offline transitions retrieved from limited target-environment experience.The simulator combines the current interaction context with the retrieved state to ground generation while allowing novel states.
  • 3.3. Retrieval-Augmented Simulation: The retrieval pipeline narrows candidates using BM25 and GPT-4o semantic retrieval over action-history queries.

4. Scalable Synthetic Training Trajectory Collection in Simulated World

UI-Simulator collects diverse training trajectories by letting an LLM teacher explore an LLM-based digital world simulator through guided, instruction-free rollouts. A trajectory wrapper then converts these explorations into coherent, filtered training instances with inferred instructions and reasoning.

  • Overview and Formulation: Scaling rollouts across environments and teachers yields diverse data, while guided controls and wrapping address trajectory validity and goal-consistent instruction generation.These components directly target the two collection challenges posed by unconstrained rollouts.
  • Overview and Formulation: Instruction-free teacher rollouts interact with the simulated environment until a coherent task is completed, after which the underlying user instruction is retrospectively derived.Each trajectory records observations and actions across the rollout.
  • Step-Wise Guided Rollout Process: Step-wise task controls counter homogeneous model behavior by steering exploration toward diverse, reasonable directions and iteratively proposing semantically meaningful sub-goals.The teacher proposes an initial control, checks completion, and updates the next control from the current observation.
  • Step-Wise Guided Rollout Process: The teacher generates a reasoning thought, action, and step summary under the current control and rollout history, preserving a coherent rationale for each interaction.These elements are recorded in the evolving rollout history.
  • Trajectory Wrapping: The trajectory wrapper infers valid user instructions by summarizing what the agent accomplished, while optional intermediate thoughts support information-query and analysis tasks.Low-quality trajectories are filtered for valid element targeting and meaningful state transitions.

5. UI-Simulator-Grow: UI-Simulator-Powered Targeted Scaling

UI-Simulator-Grow accelerates agent improvement by selecting high-learning-potential tasks and synthesizing targeted trajectory variants instead of merely increasing dataset size. Iterative validation updates and replay-based continual learning support continued gains without forgetting earlier capabilities.

  • Targeted Scaling: UI-Simulator-Grow achieves faster gains with fewer synthesized trajectories by targeting tasks that offer greater learning potential.It is designed as a targeted alternative to blindly increasing trajectory volume.
  • Target Task Selection: Target tasks are selected using teacher-forcing loss so they are neither trivial for the current agent nor beyond its present capabilities.Tasks already mastered provide limited learning signal, while excessively difficult tasks may not produce meaningful progress.
  • Target Task Selection: The validation set evolves with the agent, using newly synthesized data in later iterations to promote continual improvement and avoid overfitting evaluation to earlier iterations.The first validation set is independently synthesized; later sets are split from upcoming iteration data.
  • Synthesizing Diverse Target Task Trajectory Variants: For selected tasks, lightweight instruction rewriting creates diverse trajectories while preserving the original task structure, logic, and overall reasoning flow.Environment states, thoughts, and actions are adjusted to match the rewritten instruction.
  • Continual Learning: Replay-based continual learning incorporates new trajectories while revisiting representative tasks from prior iterations to address forgetting.Representative prior tasks are selected using embedding-based similarity and row-sum scores.

6. Experiments

UI-Simulator is evaluated on WebArena and AndroidWorld using success rate, with comparisons spanning simulated-only training, retrieval-augmented simulation, larger proprietary models, and open-source synthesis baselines. The results show strong performance and adaptation despite limited real-environment exposure and a weaker teacher model.

  • Experimental Setup: For WebArena, agents use Llama-3-8B-Instruct as the base model, while AndroidWorld uses Qwen-2.5-7B-Instruct because its tasks can exceed Llama-3-8B-Instruct’s context length.GPT-4o-mini is used for both state simulation and guided rollout.
  • UI-Simulator-F: UI-Simulator-F raises AndroidWorld success rate from 0% to 9% without exposure to real-world test environments and outperforms OS-Genesis on WebArena.The result supports simulated-only training as a viable alternative when real environments are costly or difficult to access.
  • UI-Simulator-R vs. Larger & Proprietary Models: UI-Simulator-R performs on par with Gemini-Pro on WebArena and GPT-4o on AndroidWorld despite using a much smaller 8B-scale LLM.The comparison is presented as evidence of strong generalization with limited target-environment exposure.
  • Overall Performance: UI-Simulator-R surpasses NNetNav and OS-Genesis by 2.2% and 0.9% on WebArena, respectively, and surpasses OS-Genesis by 3.8% on AndroidWorld.Unlike those baselines, UI-Simulator-R restricts exposure to the real downstream environments to a much smaller scope.

7. Analysis

The analysis finds that UI-Simulator improves robustness and adaptation by generating diverse trajectories, while UI-Simulator-Grow accelerates performance gains through targeted scaling and reaches increasingly difficult tasks.

  • Robustness and Real-Environment Experience: Around 4 and 2.5 times higher performance than OS-Genesis on WebArena and AndroidWorld, respectively, is achieved by UI-Simulator-R under equal test-environment experience.This comparison highlights adaptation with limited exposure to real environments.
  • Rollout and Simulation Process Design: Removing step-wise task controls drops performance by around 4.7% on WebArena and 7.7% on AndroidWorld, while task-diversity dimension falls from 153 to 118.The ablation supports fine-grained control as important for trajectory quality and diversity.
  • Rollout and Simulation Process Design: Replacing multi-step with single-step simulation drops performance by approximately 2.4% on WebArena and 3.8% on AndroidWorld.Single-step simulation is cost-saving but generates common, biased content, whereas multiple steps produce richer and more diverse trajectories.
  • UI-Simulator-Grow vs. Standard UI-Simulator Scaling: UI-Simulator-Grow produces steeper improvement than standard scaling, matches Qwen-1.5-72B-Instruct, surpasses Llama-3-70B-Instruct, and uses only 66% of UI-Simulator-R trajectories by iteration three.Figure 3 presents the comparison between standard and targeted scaling.
  • UI-Simulator-Grow vs. Standard UI-Simulator Scaling: Successful tasks rise across most major WebArena categories, with final UI-Simulator-Grow iterations solving code-repository tasks that standard scaling and earlier iterations could not.Figure 4 tracks successful task numbers across five categories over three iterations.
  • Analysis on Targeted Task Selection in UI-Simulator-Grow Paradigm: UI-Simulator-Grow selects intermediate-difficulty tasks by excluding validation tasks below the 25th percentile and above the 75th percentile of teacher-forcing loss.The middle range is synthesized for the next training iteration because lower-loss tasks are already learned and higher-loss tasks may be overly challenging or ambiguous.

8. Conclusions

UI-Simulator synthesizes diverse UI trajectories at scale through multi-step simulation, guided rollouts, and trajectory wrapping, while UI-Simulator-Grow prioritizes high-impact tasks for data-efficient improvement. The framework supports common web and mobile interactions and points toward broader text- and pixel-level environment simulation.

  • 8. Conclusions: UI-Simulator synthesizes diverse UI trajectories at scale through multi-step simulation, guided rollouts, and final trajectory wrapping.
  • 8. Conclusions: UI-Simulator-Grow prioritizes high-impact tasks for more data-efficient continuous improvement and achieves faster gains with 66% of the training data while matching 70B-scale models.
  • 8. Conclusions: Future work extends the approach to desktop and other text-representable environments, then toward pixel-level simulation to narrow the sim-to-real gap.
  • 8. Conclusions: The simulator supports common WebArena and AndroidWorld interactions, including click, type, and scroll, with rule-based transitions for deterministic actions.

C. Key Statistics and Hyperparameters of Step-Wise Rollout Process

The step-wise rollout process uses domain-specific trajectory lengths and task-control counts to guide exploration, collecting thousands of web and mobile trajectories at low estimated cost.

  • C. Key Statistics and Hyperparameters of Step-Wise Rollout Process: The rollout process specifies collected trajectory steps and step-wise task-control counts across web and mobile domains.
  • C. Key Statistics and Hyperparameters of Step-Wise Rollout Process: The collection includes 2K WebArena trajectories averaging 3.3 steps and 1.3K AndroidWorld trajectories averaging 5 steps.
  • C. Key Statistics and Hyperparameters of Step-Wise Rollout Process: Human evaluation uses eight dimensions with scores defined as the proportion of trajectories satisfying each criterion.
  • C. Key Statistics and Hyperparameters of Step-Wise Rollout Process: Estimated cost is $0.02 per retrieval-free web trajectory and $0.05 per retrieval-augmented web trajectory, with AndroidWorld training trajectories costing twice as much.

D. Training and Evaluation Details

Training uses Llama-3-8B-Instruct for WebArena and Qwen-2.5-7B-Instruct for AndroidWorld with fixed optimization settings, while inference uses a higher generation temperature and bounded output length.

  • D. Training and Evaluation Details: The world simulator and teacher agents both use decoding temperature 0.5 during trajectory synthesis.
  • D. Training and Evaluation Details: Llama-3-8B-Instruct and Qwen-2.5-7B-Instruct are trained for WebArena and AndroidWorld, respectively.
  • D. Training and Evaluation Details: Training uses batch size 48, learning rate 1 × 10^-5, two epochs, and four 48GB A6000 GPUs with Liger-Kernel.
  • D. Training and Evaluation Details: Downstream inference uses generation temperature 0.6 with a maximum output length of 1024 tokens.

E. Human Evaluation of Training Trajectories Synthesized by UI-Simulator

Human evaluation assesses synthesized trajectories across eight dimensions using trained annotators and a dedicated annotation interface. Agreement is strong, and satisfaction rates consistently reach or exceed 90% across dimensions.

  • E. Human Evaluation of Training Trajectories Synthesized by UI-Simulator: Each evaluation score is the proportion of trajectories satisfying its criterion across eight dimensions.
  • E. Human Evaluation of Training Trajectories Synthesized by UI-Simulator: Three computer-science annotators evaluated 40 trajectories from each simulator variant using a front-end annotation website.
  • E. Human Evaluation of Training Trajectories Synthesized by UI-Simulator: Pairwise agreement on 30 overlapping trajectories was 0.876, 0.890, and 0.976, indicating strong consistency.
  • E. Human Evaluation of Training Trajectories Synthesized by UI-Simulator: Satisfaction rates consistently reach, and often exceed, 90% across the human-evaluation dimensions.

F. UI Simulation Issue Analysis

UI-Simulator can produce realistic UI transitions but still exhibits specific failures when irrelevant context or retrieved reference states override the active interaction context.

  • The simulator remains limited by minor discrepancies in capturing certain real-world UI state transitions.Figures 7 and 8 provide concrete examples of these simulation failures.
  • UI-Simulator-F incorrectly fuses irrelevant context into the next simulated page instead of following the requested transition.In the Reddit example, clicking Forums should show all available forums, but the generated page incorporates information from the current forum.
  • The failure cases highlight a tension in retrieval-augmented simulation between using prior UI experience and preserving the current interaction context.The two examples show opposite errors: irrelevant current context contaminates retrieval-free simulation, while reference retrieval dominates the active query.
  • UI-Simulator-R can over-rely on a retrieved reference state and ignore the user’s current search context.For the Byte Blaze query, it simulates results from the reference state rather than results relevant to the active keyword.
Loading 2510.14969v1…