Source-linked AI summary
Environment Evolution for Terminal Agents
Zhiyuan Fan, Tinghao Yu, Yuanjun Cai, Jiang Zhou, Jiangtao Guan, Jincheng Liu, Yun Yang, Dingxin Hu, Zhuo Han, Xing Wu, Feng Zhang, Lilin Wang
TL;DR
Terminal-agent RL needs environments that remain challenging and provide useful learning signals as models improve, but synthesized environments can become too easy and on-policy co-evolution can limit generalization and continuity. The paper introduces off-policy environment evolution, which derives difficulty factors and incrementally schedules harder environments. It reports consistent difficulty increases and 14.4- and 18.0-percentage-point Terminal-Bench 2.1 improvements for two Qwen models.
Problem
Synthesized environments can become too easy for frontier models, while on-policy co-evolution is constrained by rollout models and initial distributions, limiting generalization and sustained learning signals.
Method
Environment evolution derives scenario novelty, skill rarity, and execution length from the multi-turn objective, then incrementally modifies environments off-policy along these directions.
Results
14.4 and 18.0 percentage points improve Terminal-Bench 2.1 performance for Qwen3.6-27B and Qwen3.6-35B-A3B, respectively, while evolved environments consistently become harder across models.
Takeaways & Limitations
Scheduling evolved environments along lineages provides continuously effective learning signals and achieves higher performance than co-evolution and ensemble baselines.
Takeaways & Limitations
Effectiveness beyond 15 generations cannot be assessed reliably from rollout outcomes once a lineage enters the zero-pass regime, and comparisons fix Claude Opus 5 as the synthesis model.
Abstract
from arXiv · showhide
Scaling interactive and verifiable environments is critical for training terminal agents. As frontier models become more capable, environments synthesized from scratch become less challenging and thus provide limited learning signals. Recent co-evolution methods iteratively synthesize environments near the model's learnable frontier based on weaknesses exposed during rollouts. However, their dependence on on-policy rollouts limits generalization and the continuous provision of learning signals as the model becomes stronger. In this paper, we propose environment evolution, which incrementally increases environment difficulty off-policy and schedules the evolved environments generation by generation during training to provide continuous learning signals. We derive three evolution directions that influence environment difficulty from the multi-turn learning objective and then implement evolution along these directions through a loop-engineered multi-agent harness. Quantitative rollout experiments with Hy4 preview, Claude Opus 5, and GPT-5.6 Sol show that environment evolution consistently produces more difficult environments. We validate its effectiveness on Qwen3.6-27B and Qwen3.6-35B-A3B through simple long-horizon RL training, improving their performance by 14.4 and 18.0 percentage points on Terminal-Bench 2.1, respectively.
1 INTRODUCTION
The paper addresses environments that are too easy for frontier models and co-evolution methods whose on-policy dependence limits generalization and sustained learning signals. It proposes off-policy environment evolution, derives difficulty factors, and reports harder environments and substantial Terminal-Bench gains.
- Motivation: Frontier models often solve synthesized terminal environments repeatedly, leaving insufficiently discriminative learning signals for RL.These environments are discarded when they cannot distinguish better from worse trajectories.
- Motivation: On-policy co-evolution exposes weaknesses for synthesis but remains constrained by the rollout model and initial environment distribution.The paper links these constraints to limited generalization and fewer learning signals as training saturates.
- Approach: Environment evolution increases difficulty generation by generation without relying on rollout models.Its paradigm evolves the environment itself rather than using on-policy weaknesses to construct each new environment.
- Approach: The method derives scenario novelty, skill rarity, and execution length as difficulty factors and modifies existing environments along selected directions.A loop-engineered multi-agent harness constructs lineages of increasing difficulty with diverse variations around the original distribution.
- Results: 14.4 and 18.0 percentage points improve Terminal-Bench 2.1 performance for Qwen3.6-27B and Qwen3.6-35B-A3B, respectively.The experiments use 200-step long-horizon RL and report longer-lasting learning signals than co-evolution and ensemble baselines.
2 RELATED WORK
Related work frames environment scaling as a need for continual, solvable challenges while noting that open-source terminal environments can provide low-quality or insufficiently difficult reward signals. Prior approaches include POET's co-evolution of environment–agent populations.
- Terminal agents: Terminal-agent environments can suffer from instruction–verification misalignment, corruption, and insufficient difficulty for frontier models.These issues undermine the quality and usefulness of their reward signals.
- Environment Scaling: Open-ended reinforcement learning requires a continual stream of solvable yet challenging environments that retain learning potential.POET addresses this setting by co-evolving environment–agent pairs and transferring agents across environments.
3 PRELIMINARIES
The paper distinguishes model-specific weakness from environment difficulty by decomposing multi-turn trajectories into scenario novelty, skill rarity, and execution length. A reference distribution converts these factors into an off-policy, model-agnostic difficulty measure, clarifying why on-policy co-evolution has limited scope.
- Trajectory difficulty: A model-specific negative log-likelihood measures how difficult a high-level trajectory is for model θ.The trajectory likelihood decomposes over reached scenarios and applied skills.
- Trajectory difficulty: Multi-turn trajectory difficulty combines solver-turn count, scenario novelty, and rarity of the required skill under each scenario.The latter two terms depend on the model’s training-data distribution and learned policy.
- Off-policy difficulty: Reference-distribution probabilities measure scenario and skill commonness within the environment family, yielding model-agnostic environment difficulty.The reference distribution T is grounded in broad world knowledge rather than a particular policy.
- Weakness versus environment difficulty: Agent weakness is the excess trajectory difficulty for a model after subtracting the environment-family difficulty.Thus, weakness reflects what is unusually difficult for a particular model rather than the environment’s intrinsic difficulty.
- Limits of on-policy co-evolution: On-policy co-evolution mainly targets skill-selection errors in seed-environment scenarios reached during current-model rollouts.It does not explicitly control solver-step count or systematically increase scenario novelty under the reference distribution.
- Limits of on-policy co-evolution: Environment evolution instead operates across the full difficulty space, providing a more general route to continuous learning signals.This distinction motivates moving from model-specific rollout weaknesses toward environment-level difficulty control.
4 APPROACH
The approach evolves environments through two gated feedback loops that refine an execution-sequence plan and then modify candidates until solvability and quality checks pass. An evolution-effort parameter controls mutation scope, while lineage scheduling exposes environments in an ordered progression during training.
- Loop-engineered harness: Each generation uses sequence-guided plan refinement followed by plan-conditioned environment refinement with rubric review and strict candidate verification.The two loops generate and revise an evolution plan, then apply and repair its changes until checks pass.
- Loop-engineered harness: The harness starts from the latest accepted environment and incrementally modifies it according to the expected execution trajectory at scenario and skill levels.The proposer extracts and updates an interleaved scenario–skill sequence before the modifier applies the resulting residual change.
- Environment verification: Three parallel verifiers require a successful reference solution, failure of an empty or no-op solution, and acceptable environment quality.These checks respectively assess solvability, verification reliability, and adaptive general-rubrics quality.
- Mutation control: Evolution direction selects the sequence edit, while evolution effort restricts its scope to one pair, one contiguous span, or an unrestricted sequence portion.The three effort levels are low, high, and max; failed review or exhausted repair budgets trigger fallback to another direction.
- Evolution-Lineage Scheduler: The Evolution-Lineage Scheduler presents environments from the earliest generation onward and advances within a generation before moving to the next generation.With B = 8 and τ = 6/8, it advances when the current environment exceeds the threshold, moving generations only after the current generation is exhausted.
5 EXPERIMENTS
The experiments evaluate environment evolution across environment construction, difficulty progression, evolution directions, scheduling, training dynamics, and benchmark performance. Results show that evolved environments become harder over generations and support stronger long-horizon RL outcomes.
- 5.1 EXPERIMENTAL SETUP: The experimental setup uses the Claude Code harness, Terminal-Bench 2.1, GRPO training, and rollout-based difficulty monitoring across selected terminal environments.Terminal-Bench 2.1 is evaluated with five-run averages, while difficulty uses pass rate over eight independent rollouts and average assistant turns.
- 5.1 EXPERIMENTAL SETUP: 127 environments are retained from 47,678 candidates after quality, solvability, executable-oracle, and difficulty filtering.Retained seeds require pass rate at most 4/8 and average turns at least 30 under Claude Opus 5.
- 5.2 EVOLUTION EFFORT: High and max evolution monotonically reduce pass rate to zero while increasing average turns; max reaches zero earlier and produces the larger difficulty change.Low effort increases average turns but has fluctuating, nonzero pass rates; high effort is selected as the default because it balances effective and stable evolution with controllability.
- 5.3 EVOLUTION DIRECTION: All three evolution directions reduce pass rate and increase average turns, with length producing the strongest pass-rate reduction and scenario and skill larger turn increases.The agreement between one-step and 15-step measurements indicates that these direction-specific profiles persist along longer lineages.
- 5.4 RL TRAINING DYNAMICS: The EL Scheduler exposes each lineage generation only after the previous generation reaches a pass-rate threshold, preserving partially solved rollout groups for more informative GRPO signals.The scheduler is used throughout the 200-step RL run, while evolved environments progressively increase trajectory turns and tokens.
- 5.5 COMPARISON: 71.5% and 64.9% peak Terminal-Bench 2.1 accuracy are achieved by environment evolution on Qwen3.6-27B and Qwen3.6-35B-A3B, versus 62.9% and 55.1% for Co-evolution.The corresponding Ensemble peaks are 60.0% and 52.8%.
6 CONCLUSION
The paper presents environment evolution as an off-policy paradigm that incrementally synthesizes harder environments and schedules them along lineages for terminal-agent RL. Across models, it reports increasing environment difficulty and higher long-horizon RL performance than co-evolution and ensemble baselines.
- 6 CONCLUSION: Environment evolution incrementally synthesizes environments of increasing difficulty for RL training.The method is presented as a general paradigm for scaling terminal-agent environments.
- 6 CONCLUSION: Evolved environments consistently increase in difficulty generation by generation despite being synthesized off-policy.The conclusion attributes this pattern to evaluations across different models.
- 6 CONCLUSION: Scheduling environments along lineages provides continuously effective learning signals and higher performance than co-evolution and ensemble baselines.The reported scope is long-horizon RL training on Qwen3.6-27B and Qwen3.6-35B-A3B.
A VERSION NOTE
The paper notes that additional details will be provided in future versions, including results where the same model constructs and learns from the environments.
- Additional details about the paper will be provided in future versions.
- Future versions will report RSI results.
- These RSI results will use the same model to construct and learn from the environments.