GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators

Motivation

• Static expert datasets capture a fixed world snapshot and cannot adapt efficiently to agents’ evolving weaknesses.Synthetic-data methods can likewise produce large but static corpora that fail to target current agent requirements.Source: §1 · p. 2

Approach

• GenEnv uses an LLM environment simulator to generate diverse, relevant scenarios at lower interaction cost than real-world training.The simulator produces an adaptive curriculum tailored to the agent’s performance.Source: §1 · pp. 2–3
• The simulator and agent co-evolve: agent performance rewards curriculum tasks near its current capability, while task success updates the agent.This creates a self-contained training loop rather than relying on static supervision.Source: §1 · p. 3
• GenEnv uses the α-Curriculum Reward to target tasks within the agent’s zone of proximal development.The mechanism is presented as an automated curriculum that aligns generated-task difficulty with learning progress.Source: §1 · p. 3

2.1. Data-Evolving Paradigm

• GenEnv co-evolves an Agent Policy that solves tasks with an Environment Policy rewarded for generating tasks near the agent’s target difficulty.The simulator targets the agent’s zone of proximal development rather than a fixed environment.Source: §2 · p. 3
• The data distribution is generated online from agent performance, allowing the simulator to search for learning-relevant breaking points instead of defeating the agent.This replaces a fixed offline distribution with a feedback loop conditioned on historical performance.Source: §2.1 · p. 3

2.2.1. Agent Task Reward

• The agent reward uses exact equality for structured actions and similarity for free-form answers, with rewards scaled to [0, 1].Structured actions can be evaluated by execution, whereas unstructured outputs use task-dependent similarity.Source: §2.2.1 · p. 4

2.2.2. Environment Difficulty-Alignment Reward

• The environment reward targets an empirical success-rate band around α, with a bell-shaped signal that penalizes mastered or hopeless tasks.A difficulty filter excludes batches whose success rate is more than k_min from α before environment updates.Source: §2.2.2 · pp. 4–5

2.3. Data Structures and How New Training Data Is Produced

• Each epoch generates new task batches and interaction traces, then appends valid traces to the agent pool while weighting environment examples by environment reward.The evolving pools combine fresh on-policy experience with accumulated data and shift future task difficulty toward α.Source: §2.3 · p. 5

Algorithm

• The loop updates the agent with GRPO and the simulator with reward-weighted regression, using filtered batches and KL regularization for environment stability.The algorithm separates online generation, dual updates, and aggregation of valid traces.Source: §2.3 · p. 5, §2.4 · p. 6

3.1. Intermediate Difficulty Maximizes Agent Learning Signal

• Intermediate-difficulty tasks maximize the expected squared gradient signal, because it is proportional to p(τ)(1−p(τ)) and peaks at p(τ)=1/2.The result assumes bounded score variation and uses the on-task expected reward as the REINFORCE baseline.Source: §3.1 · p. 7
• The α-Curriculum Reward transforms distance from the target success rate into a reward, encouraging task types whose empirical success rate remains close to α.GenEnv uses finite-rollout estimates ˆp(τ), so the environment policy aligns task generation with observed rather than true success probabilities.Source: §3.1 · p. 7

3.2. Ranking Consistency of the α-Curriculum Reward

• The reward ranking becomes statistically consistent as rollout counts grow, favoring tasks whose true success probabilities are closest to α with exponentially increasing reliability.The theorem’s mis-ranking probability decays exponentially in the minimum number of rollouts.Source: §3.2 · p. 8
• The consistency result provides a formal basis for using noisy empirical success rates to identify and up-weight task families in the agent’s target difficulty zone.Increasing the rollout budget makes the environment policy more reliable at ranking task difficulty.Source: §3.2 · p. 8

4.2. RQ1: Does GenEnv Improve Downstream Task Performance?

• 54.5% accuracy on ALFWorld versus 14.2% for the base model, while GenEnv achieves 79.1% success on API-Bank and 41.8% on BFCL.GenEnv consistently outperforms general-purpose and specialized 7B baselines across the five-benchmark comparison.Source: Table 1 · p. 9, §4.2 · p. 11
• 53.6 average score makes GenEnv competitive with, and often better than, 14B–72B models without difficulty-aligned simulation.The comparison supports the importance of aligned data generation beyond simply increasing model size or static dataset volume.Source: §4.2 · p. 11

4.6. Training Dynamics

• Training remains stable: reward and batch-level accuracy rise, validation improves before saturation, and no reward collapse, hacking, or pathological oscillation appears.The co-evolutionary process can be optimized jointly with standard policy gradients.Source: §4.2 · p. 11

4.3. RQ2: Does GenEnv Induce an Emergent Curriculum?

• 137 to 204 tokens (+49%) by epoch 6 shows that generated reasoning challenges become harder as the agent becomes more capable without success-rate collapse.The simulator creates progressively more complex tasks without a hand-specified difficulty schedule.Source: §4.3 · p. 12

4.4. RQ3: Is GenEnv More Data-Efficient Than Gemini-Based Augmentation?

• 0.458 validation score with 1x original data exceeds Gemini-Offline (3.3x) at 0.438, while GenEnv-Random trails GenEnv by 12.3%.These results compare dynamic difficulty-aligned simulation with both larger offline augmentation and online random task generation.Source: §4.4 · pp. 12–13

4.5. RQ4: Does the Environment Reward Produce Well-Calibrated Difficulty?

• Success rate converges from 0.138 toward the target band around α=0.5, remaining approximately within [0.4, 0.6] for most training.The simulator adapts task generation toward intermediate difficulty and continues moving the frontier as the agent learns.Source: §4.5 · pp. 13–14

Agent Training and Tool Use

• Earlier agent systems improve reasoning, planning, memory, and tool use, but primarily rely on imitation learning or fixed datasets.The related methods include ReAct, Reflexion, Voyager, KnowAgent, MemBench, Toolformer, and WebGPT.Source: §5.1 · pp. 14–15

Synthetic Data and Adaptive Trajectory Synthesis

• Synthetic-trajectory methods increase diversity, self-reflection, exploration, or alignment, while newer work increasingly emphasizes adaptive, self-improving experience generation.These approaches motivate closing the loop between simulation, exploration, evaluation, and learning.Source: §5.2 · p. 15

Environment Simulation

• Existing simulators support costly-interaction domains, from robotics to tool-based and societal environments, emphasizing reliable and diverse simulated experiences.Prior work includes scalable digital environments, configurable user archetypes, and large-scale social interaction simulations.Source: §5.3 · pp. 15–16

GenEnv’s Position

• GenEnv differs by generating entire tasks and goals conditioned on recent agent performance while explicitly optimizing for a target difficulty band.Environment design becomes a learnable policy with its own reward rather than a fixed transition or response model.Source: §5.3 · p. 16

6. Conclusion

• GenEnv replaces static model evolution with a dynamic data-evolving game between Agent Policy and Environment Policy.The environment adapts task generation as the agent learns.Source: §6 · p. 16
• GenEnv achieves superior performance and data efficiency across a diverse suite of agent benchmarks through difficulty-aligned co-evolution.The conclusion advocates adaptive, self-calibrating simulation for domains where real-world exploration is costly.Source: §6 · p. 16

A.1. Proofs for Section 3

• The appendix contains detailed proofs for the theoretical results introduced in Section 3.Source: §A.1 · p. 21

• The result establishes Proposition 1 by bounding the gradient norm between constant multiples of p(τ)(1−p(τ)).Source: §A.1.1 · p. 21
• p(τ)(1−p(τ)) governs the expected squared gradient norm up to constant factors under b(τ)=p(τ).The reward is Bernoulli with success probability p(τ), giving variance p(τ)(1−p(τ)).Source: §A.1.1 · p. 21
• p(τ)=1/2 uniquely maximizes the expected squared gradient norm, corresponding to tasks of intermediate difficulty.The proof uses the concavity of p(1−p) on [0,1].Source: §A.1.1 · p. 21

• The environment ranks tasks by empirical distance from α, with mis-ranking occurring when the task closer in true distance is assigned no larger reward.Source: §A.1.2 · pp. 21–22
• Correct ranking follows on the good event that both empirical success rates remain sufficiently close to their true probabilities.Source: §A.1.2 · p. 22
• Hoeffding’s inequality bounds each empirical-rate deviation, and a union bound yields the theorem’s exponential mis-ranking bound.Source: §A.1.2 · p. 22

• The Agent Policy uses GRPO, while the Environment Policy uses Reward-Weighted Regression.Source: §A.2 · p. 23
• The implementation section specifies the environment variants used in the experiments.Source: §A.3 · p. 23
• GenEnv-Random dynamically generates four prompt variations per epoch without updating environment weights, whereas GenEnv-Static fixes 3,264 samples from 544 originals.The static variant pre-generates five variations for each original sample.Source: §A.3 · p. 23