PaperlayerView exact revision on arXiv

Source-linked AI summary

Simulating Environments with Reasoning Models for Agent Training

Yuetai Li, Huseyin A Inan, Xiang Yue, Wei-Ning Chen, Lukas Wutschitz, Janardhan Kulkarni, Radha Poovendran, Robert Sim, Saravan Rajmohan

arXiv:2511.01824v1cs.AIcs.LG

TL;DR

LLM agents remain brittle in broad, complex contexts where success depends on robustness across diverse environments and tools. Simia-SFT synthesizes trajectories from small seed sets, and Simia-RL trains using LLM-simulated feedback. The authors report consistent benchmark improvements and conclude that smaller open models surpass much larger proprietary models.

  • Problem

    LLM agents struggle with individually simple tasks in broad, messy, or dynamic contexts, while generating diverse training trajectories at scale is non-trivial.

    Source: §1 · p. 1

  • Method

    Simia-SFT amplifies small seed sets into diverse trajectories, and Simia-RL enables training with LLM-simulated feedback instead of real environment implementations.

    Source: §1 · p. 2

  • Results

    Fine-tuning open models on simulated trajectories yields consistent improvements across diverse benchmarks, and the conclusion reports smaller open models surpassing much larger proprietary models.

    Source: §1 · p. 2, §4 · p. 9

  • Takeaways & Limitations

    The authors present environment simulation as a practical pathway for agent training across diverse tasks without environment engineering.

    Source: §1 · p. 2, §4 · p. 9

    Experiments cover a finite set of domains, and LLM-simulated environments may introduce distributional bias relative to real deployments.

    Source: Limitations · p. 9

Abstract

from arXiv · show

LLM agents excel in compact environments requiring deep reasoning but remain brittle when operating in broader, more complex contexts that demand robustness across diverse tools and schemas. Building bespoke environments for training is heavy, brittle, and limits progress. In this paper, we demonstrate that LLMs can simulate realistic environment feedback without access to actual testbed data or APIs. Inspired by this capability, we propose two frameworks: Simia-SFT, a pipeline that synthesizes SFT data by amplifying small seed sets into diverse trajectories in an environment-agnostic manner, and Simia-RL, a framework that enables RL training without real environment implementations through LLM-simulated feedback. Fine-tuning open models yields consistent improvements across multiple benchmarks, surpassing GPT-4o and approaching o4-mini on $\tau^2$-Bench. Together, Simia-SFT and Simia-RL enable scalable agent training without environment engineering, replacing heavy and brittle implementations with flexible LLM-based simulation.

1 Introduction

LLM agents can reason in compact environments but remain brittle across broad, messy contexts. The paper proposes Simia-SFT and Simia-RL to train agents using LLM simulation instead of real environment implementations.

1 Introduction

  • Agents that manage difficult reasoning in compact, well-specified environments remain brittle on individually simple tasks in broad, messy, or dynamic contexts.Success in these settings depends on robustness across diverse environments, tools, and edge cases.Source: §1 · p. 1
  • Existing synthetic trajectory pipelines rely on environment-specific implementations and APIs, making each new environment costly to set up and broad coverage impractical.The paper argues that diverse trajectories are needed for simple-task/complex-environment challenges, while building them at scale is non-trivial.Source: §1 · pp. 1–2
  • The paper demonstrates that LLMs can generate coherent state transitions and tool interactions without access to real environments.This capability motivates using LLMs as simulators for agent training.Source: §1 · p. 2
  • Simia-SFT amplifies small seed sets into diverse trajectories, while Simia-RL trains with LLM-simulated feedback, both without real environment implementations.Together, the frameworks aim to replace heavy, brittle implementations with flexible LLM-based simulation.Source: §1 · p. 2
  • Fine-tuned open models show consistent gains across diverse benchmarks, and the conclusion reports smaller open models surpassing much larger proprietary models.On τ 2-Bench, the 32B model surpasses GPT-4o and xLAM-2-70B, while the 8B model outperforms Qwen2.5-32B-Instruct on Airline and Retail.Source: Figure 1 · p. 1, §1 · p. 2
  • The authors frame LLM simulation as a practical training recipe that replaces environment-specific code and treats environment engineering as prompt-and-schema design.They argue this reframing can support broader progress in agentic LLM training.Source: §1 · p. 2

2 LLM Simulator for Agent Training

The paper shows that LLMs can simulate coherent agent-environment interactions without real testbeds, then builds Simia-SFT and Simia-RL on that capability. These frameworks synthesize SFT trajectories and provide simulated feedback for RL without per-environment implementations.

2.1 LLMs Can Reason to Simulate Agent Environments

  • LLMs can simulate realistic environment feedback from interaction history, tool specifications, reference trajectories, and response formats, producing coherent state transitions and tool interactions without testbed access.The paper uses this capability as the basis for its SFT and RL frameworks.Source: §2.1 · pp. 2–3

2.2 Simia-SFT: SFT with LLM simulated Traj

  • Simia-SFT generates complete trajectories from seed data without real environment implementations, while Simia-RL enables RL in LLM-simulated environments.The two frameworks target SFT data synthesis and RL training, respectively.Source: §2 · p. 2, §2.2 · p. 3

2.2.1 End-to-end Agent Trajectory Synthesis Pipeline

  • The Simia-SFT pipeline filters seeds, embeds tool specifications and a reference example in prompts, then simulates diverse interactions and checks trajectory structure.Seed checks assess completion, reasoning/action consistency, and formatting; generation varies task formulations, reasoning strategies, and action sequences.Source: §2.2.1 · p. 3
  • Simia-SFT expands task distributions and enforces structural fidelity, but remains bounded by the synthesizer’s semantic correctness.Source: §2.2.1 · p. 4

2.3 Simia-RL: RL on LLM Simulated Env

  • Simia-RL addresses the per-task engineering required by conventional agent RL; its simulator generates tool outputs and errors, then rewards trajectories against task success criteria.The framework uses multi-turn interactions to support policy optimization.Source: §2.3 · p. 4

3 Experiments

Across τ²-Bench, OfficeBench, and AgentBench, models fine-tuned on simulated trajectories perform strongly against the reported baselines, while synthetic-data scaling and simulated-environment RL show additional benefits in the evaluated settings. The experiments also include trajectory-simulator comparisons and joint training on combined datasets.

3.2 Results

  • For Pass^k, Simia-Tau (Qwen2.5-32B) leads xLAM-2-70B on Airline across k=1–3 but trails it on Retail as k increases.Pass^k requires each task to succeed on all k retries.Source: Figure 5 · p. 5, §3.2 · p. 6

3.1 Setup

  • The setup synthesizes trajectories from APIGen-MT, AgentTuning, and OfficeBench seeds, fine-tunes Qwen and Llama families, and evaluates across three agentic benchmark suites.RL uses GRPO in OfficeBench and τ²-Bench configurations.Source: §3.1 · p. 5

3.2 Results

  • 58.9 average: Simia-Tau (Qwen2.5-32B) exceeds GPT-4o by 4.7 points and xLAM-2-70B by 2.6 points on τ²-Bench.Simia-Tau (Qwen3-8B) averages 49.3, exceeding GPT-4 by 3.9 points and xLAM-2-8B by 4.6 points.Source: §3.2 · p. 6
  • On OfficeBench, Simia-OB (Qwen3-8B) averages 44.0, 12.9 points above GPT-4; it also exceeds the Qwen3-8B seed-data baseline by 14.4 points.The reported Qwen3-8B scores are 58.8 on 2-apps and 29.1 on 3-apps.Source: §3.2 · pp. 6–7, Table 2 · p. 6
  • On AgentBench, Simia-AB (Qwen3-8B) averages 42.6, above GPT-4o’s 38.1 and close to GPT-4’s 44.2.The Qwen2.5-7B model scores 69.2 on WebShop, above GPT-4’s 61.1 and GPT-4o’s 48.9.Source: Table 3 · p. 6, §3.2 · p. 7
  • At identical dataset sizes, synthesized trajectories match real-environment trajectories on OfficeBench and AgentBench and do better on τ²-Bench; larger synthesized sets outperform seed-based ones.The comparison uses Qwen2.5-7B-Instruct fine-tuned on the respective trajectories.Source: §3.2 · p. 7

3.3 RL Results

  • On OfficeBench, simulated-environment RL scores 64.7/34.5 versus 60.8/28.6 in real settings for 2-apps/3-apps; its case study shows richer feedback identifying a lunch-break conflict.On τ²-Bench, SFT followed by RL yields slight additional gains, but the paper does not directly compare simulated and real environments there.Source: §3.3 · p. 7

3.4 More Ablations

  • The ablations compare trajectory simulators in Appendix E and report joint training of Qwen3-8B on three combined datasets.The supplied results do not state an outcome for the simulator comparison or combined-data training.Source: §3.4 · p. 7

4 Conclusion

The conclusion presents Simia-SFT and Simia-RL as LLM-based approaches to training agents without laborious environment engineering. It reports comparable or better results for simulated trajectories in an ablation, while noting finite domain coverage and possible simulator bias.

4 Conclusion

  • The conclusion presents Simia-SFT as a trajectory-synthesis framework and Simia-RL as a way to conduct RL without real agent-environment implementations.The Simia-SFT pipeline combines LLM-based pre-filtering, simulation, and rule-based sanity checks to produce trajectories across heterogeneous environments.Source: §4 · pp. 7, 9
  • At identical dataset sizes, simulated trajectories match real-environment trajectories on OfficeBench and AgentBench and perform better on τ 2-bench; scaling dataset size improves performance.The figure also highlights larger, more diverse simulated datasets as a way to address real-world seed-data collection limits.Source: Figure 6 · p. 8
  • A case study contrasts fixed failure messages in the real environment with simulated feedback that explains conflicts and lets the model adjust until it gets the reward.The example conflict is an action overlapping with a lunch break.Source: Figure 8 · p. 8

Limitations

  • The experiments cover a finite set of domains, and LLM simulation may introduce distributional bias relative to real deployments.The listed domains include airline, retail, web navigation, and office scenarios; extending coverage to more domains with different tool schemas is identified as important.Source: Limitations · p. 9

Ethical Considerations

  • The paper reports no human subjects, user studies, or personally identifiable information collection, and says its experimental datasets and benchmarks are publicly available.It states that the work presents no ethical concerns.Source: Ethical Considerations · p. 9

A Related Work

Prior work explores tool-using LLMs and synthetic agentic datasets, with varied APIs, generation pipelines, and verification strategies. This paper’s complementary direction is to simulate complete trajectories integrating reasoning, tool use, and environment feedback.

A Related Work

  • Tool-using LLM research spans web browsing, Python execution, and information retrieval, while agents still face tasks requiring real-time access, accurate computation, or environment interaction.The passage points readers to Qu et al. (2025) for a comprehensive survey.Source: A Related Work · p. 11
  • Synthetic agentic dataset efforts use varied approaches, including instruction–API generation, multi-agent simulation, real-API solutions, and trajectory generation across task domains.The passage also describes pipelines using multi-stage verification, tool evolution, and diverse prompt templates.Source: A Related Work · p. 11
  • Unlike prior efforts centered on APIs and generation pipelines, this work simulates end-to-end trajectories with reasoning, tool use, and environment feedback for extension across domains and tasks.The paper frames this as complementary to, rather than a replacement for, existing datasets.Source: A Related Work · p. 11

B.1 Supervised Fine-tuning

The supervised fine-tuning appendix identifies the training software, hardware, parameter source, and hyper-parameter table.

B.1 Supervised Fine-tuning

B.2 GRPO

The GRPO appendix describes RL training on a simulated environment using RAGEN and VeRL across eight A100 GPUs, with specified rollout and interaction settings.

B.2 GRPO

  • RL training on the simulated environment uses eight A100 GPUs and the RAGEN and VeRL frameworks.Source: §B.2 · p. 11
  • The RL configuration uses a rollout of 16 and 64 training steps.Source: §B.2 · p. 11
  • Each task allows up to 40 interaction rounds, with a maximum context length of 12,000 tokens and temperature 0.7.Source: §B.2 · p. 11

C Full Prompt for Synthetic Trajectory Generation

The appendix identifies prompts for synthesizing APIGen-MT and Agenttuning trajectories and OfficeBench trajectories. It also points to OfficeBench prompts for simulated RL feedback and reward computation, alongside RL hyper-parameters.

C Full Prompt for Synthetic Trajectory Generation

D Full Prompt for RL on the simulated environment

E More Results of Synthetic Data

Jointly fine-tuned Qwen3-8B models score higher than GPT-4 on average across benchmarks, while GPT-5 and o4-mini produce similar results across most tasks with domain-specific differences.

F More Results of RL on Simulated Environment

RL on simulated environments improves OfficeBench performance across training steps and slightly improves τ 2-Bench on APIGen-MT. The section also reports simulator comparisons and joint-dataset SFT results.

  • Figure 11 is the prompt for simulating OfficeBench environment feedback, and Figure 12 is the prompt for computing OfficeBench RL reward.Source: Figure 11 · p. 15, Figure 12 · p. 15
  • Figure 13 is the prompt for simulating APIGen-MT environment feedback, and Figure 14 is the prompt for computing APIGen-MT RL reward.Source: Figure 13 · p. 16, Figure 14 · p. 16
  • GPT-5 and o4-mini yield comparable results as trajectory synthesizers across most OfficeBench, τ 2-Bench, and AgentBench tasks.The comparison uses 15k synthetic trajectories generated by each model.Source: Figure 15 · p. 17
  • Jointly fine-tuned Qwen3-8B SFT models average higher scores than GPT-4 across benchmarks.The models were trained on three datasets generated by the pipeline.Source: Figure 16 · p. 17
  • OfficeBench 2-apps rises from around 57.8 to approximately 64.7 and 3-apps from 27.3 to 34.5; simulated-environment RL is always better than real-environment RL on 3-apps and slightly improves τ 2-Bench on APIGen-MT.Source: F More Results of RL on Simulated… · p. 14, Figure 17 · p. 17, Figure 18 · p. 18

Related papers

Loading 2511.01824v1…