PaperlayerView exact revision on arXiv

Source-linked AI summary

GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators

Jiacheng Guo, Ling Yang, Peter Chen, Qixin Xiao, Yinjie Wang, Xinzhe Juan, Jiahao Qiu, Ke Shen, Mengdi Wang

arXiv:2512.19682v2cs.CL

TL;DR

GenEnv addresses the high cost and static nature of real-world interaction data by using a difficulty-aligned co-evolutionary game between an agent and a generative environment simulator. Across five benchmarks, it improves performance over 7B baselines and uses less synthetic data than Gemini 2.5 Pro-based offline augmentation.

  • Problem

    Training LLM agents is constrained by the high cost, static nature, and limited adaptability of real-world interaction data and pre-collected datasets.

  • Method

    GenEnv co-evolves an agent policy and environment policy, using an α-Curriculum Reward to generate tasks aligned with the agent’s target success zone.

  • Results

    GenEnv delivers consistent gains over strong 7B baselines across five benchmarks and matches or surpasses Gemini 2.5 Pro-based static augmentation while using 3.3× less synthetic data.

  • Takeaways & Limitations

    Adaptive, difficulty-aligned simulation provides a data-efficient pathway for training LLM agents beyond larger static datasets.

  • Takeaways & Limitations

    The theoretical analysis assumes that the squared norm of the score function does not vary too dramatically when conditioned on the binary outcome.

Abstract

from arXiv · show

Training capable Large Language Model (LLM) agents is critically bottlenecked by the high cost and static nature of real-world interaction data. We address this by introducing GenEnv, a framework that establishes a difficulty-aligned co-evolutionary game between an agent and a scalable, generative environment simulator. Unlike traditional methods that evolve models on static datasets, GenEnv instantiates a dataevolving: the simulator acts as a dynamic curriculum policy, continuously generating tasks specifically tailored to the agent's ``zone of proximal development''. This process is guided by a simple but effective $α$-Curriculum Reward, which aligns task difficulty with the agent's current capabilities. We evaluate GenEnv on five benchmarks, including API-Bank, ALFWorld, BFCL, Bamboogle, and TravelPlanner. Across these tasks, GenEnv improves agent performance by up to \textbf{+40.3\%} over 7B baselines and matches or exceeds the average performance of larger models. Compared to Gemini 2.5 Pro-based offline data augmentation, GenEnv achieves better performance while using 3.3$\times$ less data. By shifting from static supervision to adaptive simulation, GenEnv provides a data-efficient pathway for scaling agent capabilities.

1. Introduction

GenEnv addresses the cost and rigidity of real-world and static training data by using a generative simulator to produce adaptive tasks. Its difficulty-aligned curriculum improves agent performance and data efficiency across benchmarks.

  • Motivation: Static expert datasets capture a fixed world snapshot and cannot adapt efficiently to agents’ evolving weaknesses.Synthetic-data methods can likewise produce large but static corpora that fail to target current agent requirements.
  • Approach: GenEnv uses an LLM environment simulator to generate diverse, relevant scenarios at lower interaction cost than real-world training.The simulator produces an adaptive curriculum tailored to the agent’s performance.
  • Approach: The simulator and agent co-evolve: agent performance rewards curriculum tasks near its current capability, while task success updates the agent.This creates a self-contained training loop rather than relying on static supervision.
  • Approach: GenEnv uses the α-Curriculum Reward to target tasks within the agent’s zone of proximal development.The mechanism is presented as an automated curriculum that aligns generated-task difficulty with learning progress.

2. GenEnv: Difficulty-Aligned Co-Evolution

GenEnv treats agent training as a two-player game in which an agent solves simulator-generated tasks and the simulator adapts task difficulty to the agent. Online interaction produces rewards, evolving training pools, and dual policy updates.

  • 2.1. Data-Evolving Paradigm: GenEnv co-evolves an Agent Policy that solves tasks with an Environment Policy rewarded for generating tasks near the agent’s target difficulty.The simulator targets the agent’s zone of proximal development rather than a fixed environment.
  • 2.1. Data-Evolving Paradigm: The data distribution is generated online from agent performance, allowing the simulator to search for learning-relevant breaking points instead of defeating the agent.This replaces a fixed offline distribution with a feedback loop conditioned on historical performance.
  • 2.2.1. Agent Task Reward: The agent reward uses exact equality for structured actions and similarity for free-form answers, with rewards scaled to [0, 1].Structured actions can be evaluated by execution, whereas unstructured outputs use task-dependent similarity.
  • 2.2.2. Environment Difficulty-Alignment Reward: The environment reward targets an empirical success-rate band around α, with a bell-shaped signal that penalizes mastered or hopeless tasks.A difficulty filter excludes batches whose success rate is more than k_min from α before environment updates.
  • 2.3. Data Structures and How New Training Data Is Produced: Each epoch generates new task batches and interaction traces, then appends valid traces to the agent pool while weighting environment examples by environment reward.The evolving pools combine fresh on-policy experience with accumulated data and shift future task difficulty toward α.
  • Algorithm: The loop updates the agent with GRPO and the simulator with reward-weighted regression, using filtered batches and KL regularization for environment stability.The algorithm separates online generation, dual updates, and aggregation of valid traces.

3. Theoretical Analysis of Difficulty-Aligned GenEnv

The analysis explains why intermediate-difficulty tasks provide the strongest agent learning signal and why the α-Curriculum Reward can consistently rank tasks near the target difficulty. Together, these results justify GenEnv’s difficulty-aligned co-evolution mechanism.

  • 3.1. Intermediate Difficulty Maximizes Agent Learning Signal: Intermediate-difficulty tasks maximize the expected squared gradient signal, because it is proportional to p(τ)(1−p(τ)) and peaks at p(τ)=1/2.The result assumes bounded score variation and uses the on-task expected reward as the REINFORCE baseline.
  • 3.1. Intermediate Difficulty Maximizes Agent Learning Signal: The α-Curriculum Reward transforms distance from the target success rate into a reward, encouraging task types whose empirical success rate remains close to α.GenEnv uses finite-rollout estimates ˆp(τ), so the environment policy aligns task generation with observed rather than true success probabilities.
  • 3.2. Ranking Consistency of the α-Curriculum Reward: The reward ranking becomes statistically consistent as rollout counts grow, favoring tasks whose true success probabilities are closest to α with exponentially increasing reliability.The theorem’s mis-ranking probability decays exponentially in the minimum number of rollouts.
  • 3.2. Ranking Consistency of the α-Curriculum Reward: The consistency result provides a formal basis for using noisy empirical success rates to identify and up-weight task families in the agent’s target difficulty zone.Increasing the rollout budget makes the environment policy more reliable at ranking task difficulty.

4. Experiments

GenEnv improves agent performance across five benchmarks while using adaptive simulation to co-evolve task difficulty with agent capability. Its training is stable, produces an emergent curriculum, and achieves better data efficiency than static augmentation.

  • 4.2. RQ1: Does GenEnv Improve Downstream Task Performance?: 54.5% accuracy on ALFWorld versus 14.2% for the base model, while GenEnv achieves 79.1% success on API-Bank and 41.8% on BFCL.GenEnv consistently outperforms general-purpose and specialized 7B baselines across the five-benchmark comparison.
  • 4.2. RQ1: Does GenEnv Improve Downstream Task Performance?: 53.6 average score makes GenEnv competitive with, and often better than, 14B–72B models without difficulty-aligned simulation.The comparison supports the importance of aligned data generation beyond simply increasing model size or static dataset volume.
  • 4.6. Training Dynamics: Training remains stable: reward and batch-level accuracy rise, validation improves before saturation, and no reward collapse, hacking, or pathological oscillation appears.The co-evolutionary process can be optimized jointly with standard policy gradients.
  • 4.3. RQ2: Does GenEnv Induce an Emergent Curriculum?: 137 to 204 tokens (+49%) by epoch 6 shows that generated reasoning challenges become harder as the agent becomes more capable without success-rate collapse.The simulator creates progressively more complex tasks without a hand-specified difficulty schedule.
  • 4.4. RQ3: Is GenEnv More Data-Efficient Than Gemini-Based Augmentation?: 0.458 validation score with 1x original data exceeds Gemini-Offline (3.3x) at 0.438, while GenEnv-Random trails GenEnv by 12.3%.These results compare dynamic difficulty-aligned simulation with both larger offline augmentation and online random task generation.
  • 4.5. RQ4: Does the Environment Reward Produce Well-Calibrated Difficulty?: Success rate converges from 0.138 toward the target band around α=0.5, remaining approximately within [0.4, 0.6] for most training.The simulator adapts task generation toward intermediate difficulty and continues moving the frontier as the agent learns.

5. Related Work

Prior work advances agent reasoning, memory, tool use, synthetic trajectories, and environment simulation, but much of it relies on fixed or offline data. GenEnv extends this trend with a simulator that generates tasks conditioned on agent performance and target difficulty.

  • Agent Training and Tool Use: Earlier agent systems improve reasoning, planning, memory, and tool use, but primarily rely on imitation learning or fixed datasets.The related methods include ReAct, Reflexion, Voyager, KnowAgent, MemBench, Toolformer, and WebGPT.
  • Synthetic Data and Adaptive Trajectory Synthesis: Synthetic-trajectory methods increase diversity, self-reflection, exploration, or alignment, while newer work increasingly emphasizes adaptive, self-improving experience generation.These approaches motivate closing the loop between simulation, exploration, evaluation, and learning.
  • Environment Simulation: Existing simulators support costly-interaction domains, from robotics to tool-based and societal environments, emphasizing reliable and diverse simulated experiences.Prior work includes scalable digital environments, configurable user archetypes, and large-scale social interaction simulations.
  • GenEnv’s Position: GenEnv differs by generating entire tasks and goals conditioned on recent agent performance while explicitly optimizing for a target difficulty band.Environment design becomes a learnable policy with its own reward rather than a fixed transition or response model.

6. Conclusion

GenEnv reframes agent training as a dynamic, data-evolving game between an agent and an environment policy. Its difficulty-aligned co-evolutionary loop delivers strong performance and data efficiency across diverse benchmarks.

  • 6. Conclusion: GenEnv replaces static model evolution with a dynamic data-evolving game between Agent Policy and Environment Policy.The environment adapts task generation as the agent learns.
  • 6. Conclusion: GenEnv achieves superior performance and data efficiency across a diverse suite of agent benchmarks through difficulty-aligned co-evolution.The conclusion advocates adaptive, self-calibrating simulation for domains where real-world exploration is costly.

A.1. Proofs for Section 3

The appendix provides detailed proofs for the theoretical results stated in Section 3.

  • A.1. Proofs for Section 3: The appendix contains detailed proofs for the theoretical results introduced in Section 3.

A.1.1. Proof of Proposition 1

The proof shows that, under the Bernoulli reward model and baseline choice b(τ)=p(τ), the expected squared gradient norm is bounded by terms proportional to p(τ)(1−p(τ)). This quantity is maximized at p(τ)=1/2, so intermediate-difficulty tasks provide the strongest gradient signal.

  • The result establishes Proposition 1 by bounding the gradient norm between constant multiples of p(τ)(1−p(τ)).
  • p(τ)(1−p(τ)) governs the expected squared gradient norm up to constant factors under b(τ)=p(τ).The reward is Bernoulli with success probability p(τ), giving variance p(τ)(1−p(τ)).
  • p(τ)=1/2 uniquely maximizes the expected squared gradient norm, corresponding to tasks of intermediate difficulty.The proof uses the concavity of p(1−p) on [0,1].

A.1.2. Proof of Theorem 1

The proof analyzes whether the environment ranks two tasks correctly when their true success probabilities differ in distance from the target α. It shows that mis-ranking requires an empirical estimation error and is therefore exponentially bounded using Hoeffding’s inequality and a union bound.

  • The environment ranks tasks by empirical distance from α, with mis-ranking occurring when the task closer in true distance is assigned no larger reward.
  • Correct ranking follows on the good event that both empirical success rates remain sufficiently close to their true probabilities.
  • Hoeffding’s inequality bounds each empirical-rate deviation, and a union bound yields the theorem’s exponential mis-ranking bound.

A.2. Hyperparameter Details

The experiments use GRPO for the agent and Reward-Weighted Regression for the environment, alongside dynamically generated, random, and static environment variants. The static variant pre-generates a fixed augmentation dataset from the original training samples.

  • The Agent Policy uses GRPO, while the Environment Policy uses Reward-Weighted Regression.
  • The implementation section specifies the environment variants used in the experiments.
  • GenEnv-Random dynamically generates four prompt variations per epoch without updating environment weights, whereas GenEnv-Static fixes 3,264 samples from 544 originals.The static variant pre-generates five variations for each original sample.
Loading 2512.19682v2…