PaperlayerView exact revision on arXiv

Source-linked AI summary

Environment Duels: A Microcosm of LLM Co-Evolution

Shibo Jin, Zhiqiu Xu, Yiling Zou, Shangyu Gong, Mayur Naik

arXiv:2610.30116v1cs.CLcs.AI

TL;DR

Models increasingly learn from tasks and synthetic data created by other models, raising the question of how well each model authors challenges as well as solves them. Environment Duels stages these interactions through hinted, executable environments, and finds that authoring and solving differ while hint lift tracks downstream training gains.

  • Problem

    As models learn from challenges and synthetic data created by predecessors and competitors, it matters how well models can author problems worth learning from as well as solve them.

    Source: §1 · p. 1

  • Method

    Environment Duels has frontier models author executable interactive environments with privileged hints, solve one another’s environments, and make a counter-move after observing play.

    Source: §1 · p. 1, §2.2 · p. 4

  • Results

    Authoring and solving rank models differently, and a learner’s hint lift on an author’s environments tracks how much training on them improves held-out benchmark performance.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The duel’s hint-lift signal tracks training gains for both self-improvement and co-evolution, offering a measure of how much a learner benefits from an author’s environments.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Through a tangle of real-world interactions, from synthetic training pipelines to data shared on the open web, frontier LLMs absorb what their predecessors and competitors create. How well does each model play its role in this ecosystem, as a creator and as a solver? We introduce Environment Duels, a dynamic agentic benchmark that evaluates both capabilities on interactive environments. Each model writes environments, each with a privileged hint, aiming for tasks that capable solvers fail without the hint and pass with it. Across nine frontier LLMs, solving and authoring prove to be distinct skills: the strongest solvers are not the strongest authors, while smaller models learn more from how others played and write the hardest environments and the most useful hints. Beyond the scores, the pairwise interactions form a microcosm of how models affect one another. Head-to-head outcomes are transitive and so yield a stable ranking; independent authors converge on the same designs; and in a reinforcement-learning study with Qwen3.8-27B, the learner's hint lift on an author's environments tracks how much training on those environments improves it on held-out reasoning and agentic benchmarks, whether the environments are its own or another model's. Each of these patterns has a counterpart in how models shape one another outside the duel. By staging these interactions under controlled conditions, Environment Duels offers a window onto how LLMs teach, challenge and learn from one another. Code is available at https://github.com/LLM-Duels/EnvDuels.

1 INTRODUCTION

Models increasingly learn from challenges and synthetic data created by other models, making task authoring part of how they shape one another’s training. Environment Duels stages this interaction by having frontier models create hinted interactive environments, solve others’ environments, and adapt after observing play.

1 INTRODUCTION

  • Models’ training distributions are shaped by challenges and synthetic data that predecessors and competitors create, including data shared through public repositories and benchmarks.The introduction describes deliberate within-lab curriculum creation as recursive self-improvement.Source: §1 · p. 1
  • As existing human-authored challenges become harder to curate and the supply of unsolved problems thins, posing new, well-specified challenges becomes a bottleneck for training.The introduction presents calibrated challenge creation as a capability that could drive autonomous scientific research.Source: §1 · p. 1
  • Environment Duels has each model write executable interactive environments with privileged hints and solve environments written by other models.Authors aim for tasks capable solvers fail without the hint and pass with it; after observing play, each author makes one counter-move.Source: §1 · pp. 1–2
  • Across nine frontier models, solving and authoring are distinct skills: smaller models learn more from others’ play and write the hardest environments and most useful hints.Hint lift on an author’s environments also anticipates how much training on those environments improves a learner.Source: §1 · p. 2

2 METHODOLOGY

Environment Duels evaluates models as both authors and solvers: models create executable environments with hints, play one another’s environments, and revise their designs after seeing solver trajectories. The evaluation uses matched hinted and unhinted plays, separate scoring axes, and a counter-move round.

2.1 CONSTRUCTING AN ENVIRONMENT

  • Environment authors plan a design and hint before implementation, then validate and review each environment before admission to the duel.A failed execution check or review can trigger repairs; flagged instances judged invalid by both audits are excluded from downstream metrics.Source: §2.1 · p. 3

2.2 THE DUEL

  • Each model plays every environment on the same seeds with and without its hint; scores average cell quantities separately across solvers and environments, with no composite score.The three scoring axes are presented as the same averaging procedure applied to different cell quantities.Source: §2.2 · pp. 3–4, Table 1 · p. 4
  • Hint lift measures plays the hint fixes minus plays it breaks, using matched hinted and unhinted outcomes for each solver–environment cell.The paper reports hint lift as an author score and reports solver hint-use ability separately as rescue and harm.Source: §2.2 · p. 4
  • A single counter-move round of co-evolution raises difficulty and hint lift: no-hint pass rate falls from 84.9% to 78.5%, while hint lift rises from 6.3 to 13.4 points.Authors revise the environment after observing solver play while preserving the task, interface, reward contract, action budget, and seed-to-instance mapping.Source: §2.2 · p. 4

3 RESULTS

Authoring and solving emerge as distinct skills: smaller models lead on environment difficulty and hint usefulness, while Claude-Opus-5 and GPT-5.6-Sol lead as solvers. The duel matrix also yields a stable combined-capability ranking and identifies authors with similar designs.

3.1 EXPERIMENTAL SETUP

  • Nine frontier models authored 90 environments, and every solver played each on four seeds, with and without hints, before and after counter-moves, for 12,960 episodes.Solving used a 24-action budget, and model calls ran in each model’s own agent harness.Source: §3.1 · pp. 4–5

3.2 EVALUATION

  • Three compact models lead the authoring axes, while Claude-Opus-5 and GPT-5.6-Sol lead solving.DeepSeek-V4.1-Flash, Gemini-3.8-Flash and Qwen3.8-27B rank highest on Difficulty and Hint lift; the two strongest solvers rank no higher than fourth on Difficulty and seventh on Hint lift.Source: §3.2 · p. 5, Figure 2 · p. 5
  • Qwen3.8-27B writes the most useful hints, while the three strongest solvers’ hints turn about three times as many correct attempts wrong as other authors’ hints.Top-tier hints turn 17.7% of bottom-tier correct attempts wrong, versus at most 7.7% for other hints, but break no more than 3.1% of top-tier attempts.Source: §3.2 · p. 6, Figure 3 · p. 6
  • After the counter-move, the two strongest solvers change their environments least; Qwen3.8-27B and GLM-5.3 lower pass rates by 13.1 and 10.8 points.The authors suggest that strong solvers may receive weaker feedback because the other solvers expose fewer shortcuts in their environments.Source: §3.2 · p. 6, Figure 4 · p. 6

3.3 INSIDE ENVIRONMENT DUELS

  • Duel outcomes are transitive across all 84 model triples, yielding a stable combined-capability ranking rather than the solving ranking.No model loses to one ranked below it; DeepSeek-V4.1-Flash ranks second and Qwen3.8-27B third in the duel ranking.Source: §3.3 · p. 7, Figure 5 · p. 7
  • Design-plan similarity identifies Claude-Opus-5, DeepSeek-V4.1-Flash and GLM-5.3 as alike, although their solving trajectories are no more alike than other pairs’.The same pairs emerge from code and word-free measures such as program size and plan length.Source: §3.3 · p. 7, Figure 5 · p. 7

4 REINFORCEMENT LEARNING EXPERIMENT

The experiment tests whether pre-training hint lift predicts Qwen3.8-27B’s gains from reinforcement learning on different authors’ environments. Gains follow hint lift for both the model’s own environments and other models’ environments.

4 REINFORCEMENT LEARNING EXPERIMENT

  • The experiment asks whether duel dynamics predict actual improvement from reinforcement learning on a larger pool of environments.Source: §4 · p. 8

4.1 SETUP

  • Qwen3.8-27B is trained separately on 200 environments from each of four authors, then evaluated across five reasoning, agentic, coding, and general benchmark suites.Each training run uses GRPO for one epoch; duel hint lift is measured before training.Source: §4.1 · p. 8

4.2 RESULTS

  • GPT-5.6-Sol’s environments pair the highest hint lift (+17.5) with the largest mean gain (+2.0), while DeepSeek-V4.1-Flash’s pair the lowest (+5.0) with almost no gain (+0.1).The model’s own environments rank second on both hint lift and mean gain, supporting the pattern for self-improvement as well.Source: §4.2 · p. 8, Table 2 · p. 8

5 RELATED WORK

Related work spans synthetic task and environment generation, feedback-driven adaptation, evaluation of generated data and model interactions, and hint-guided reinforcement learning. Environment Duels relates these strands by connecting author–solver behavior with downstream learning outcomes.

Synthetic Data and Environments

  • Synthetic-data methods generate examples, executable environments, or adaptive tasks, applying feedback to task selection, environment behavior, or validation.Examples include SPADE, which uses hinted–unhinted returns, and EnvHarness, which revises environments from policy trajectories.Source: §5 · p. 9

Recursive Self-Improvement

  • Recursive self-improvement methods differ in whether feedback changes curricula or learners, and in whose behavior drives updates.The resulting progress targets may be harder challenges, stronger solving, or downstream learning.Source: §5 · p. 9

Evaluating Synthetic Data and Model Interactions

  • Evaluation work tests generated data through fine-tuning or data-preparation outcomes, while interaction benchmarks test models as authors and solvers.These approaches motivate relating author–solver behavior to downstream reinforcement-learning outcomes, not only rankings.Source: §5 · p. 9

Hints for Reinforcement Learning

  • Hint-guided RL methods use privileged guidance during optimization, whereas Environment Duels measures hint effects in matched plays before hint-free RL.This allows rescue and harm to be compared across author–solver pairs.Source: §5 · p. 9

6 CONCLUSION

Environment Duels stages models as authors and solvers, with counter-moves shaped by observed play. Its records support reproducibility, and the paper reports no human subjects or personal data.

6 CONCLUSION

  • Environment Duels has models author hinted executable environments, solve other models’ environments, and respond with counter-moves after observing play.The benchmark’s ceiling rises with its strongest author, while authoring and solving rank models differently.Source: §6 · p. 9

Reproducibility statement

  • The supplementary materials provide environments, plans, hints, seeds, action traces, and analysis code that recomputes scores without model calls.Seeded Python environments permit deterministic replay of recorded play.Source: §6 · p. 10

Ethics statement

  • The work uses no human subjects or personal data, and runs model-written environments in a sandbox without file-system or network access.Source: §6 · p. 10

A ENVIRONMENTS VERIFICATION

The authors audit final environment versions when every model fails in either hint condition on a seed. They exclude three instances judged invalid from downstream metrics, while retaining other seeds of the same environments.

A ENVIRONMENTS VERIFICATION

  • Three of 360 environment–seed instances were flagged because all nine models failed in either hint condition.The 360 instances comprise 90 environments evaluated on four seeds each.Source: A ENVIRONMENTS VERIFICATION · p. 14
  • Both audit models found replayable wins for all three flagged instances but judged each environment invalid, as did manual review.Issues included hidden delivery deadlines and a temperature-update rule that differed from the code.Source: A ENVIRONMENTS VERIFICATION · p. 14
  • The three invalid instances are excluded from downstream metrics in both hint conditions, while other seeds of those environments remain eligible.Source: A ENVIRONMENTS VERIFICATION · p. 14

B EVALUATED MODELS AND RUN HANDLING

The evaluation uses each model’s maximum available reasoning effort, with provider-served and OpenRouter-served models run through their configured harnesses. Plays interrupted by external errors or timeouts are retried, and reported averages use a balanced design.

B EVALUATED MODELS AND RUN HANDLING

  • All nine models use their maximum available reasoning effort, with Claude-Opus-5 and GPT-5.6-Sol provider-served and the other seven served through OpenRouter.Gemini CLI reaches OpenRouter through a LiteLLM 1.101.0 protocol bridge.Source: B EVALUATED MODELS AND RUN HANDLING · p. 14
  • External API errors and timeouts trigger retries, and hint lift includes only pairs in which both plays finished.Every author writes the same number of environments per domain, and each environment is played by every model on the same seeds, making reported figures unweighted averages.Source: B EVALUATED MODELS AND RUN HANDLING · p. 14

C CONVERGENT ENVIRONMENT DESIGNS

Independent authors reuse mechanics and design patterns across environments, including parity or XOR rules, modular arithmetic, similar cross-author designs, and repeated title vocabulary.

C CONVERGENT ENVIRONMENT DESIGNS

  • Parity or XOR appears in 14 environments from eight authors, while modular arithmetic appears in 24 environments from eight authors.The authoring prompt does not request either mechanic.Source: C CONVERGENT ENVIRONMENT DESIGNS · p. 15
  • DeepSeek-V4.1-Flash’s Ashlock Concordance and GLM-5.3’s Bellfounders’ Vault use different objects but share single-use switches whose effects combine by parity.They are the most similar pair by code; the same two authors also produced the most similar pair by plan, both coolant-loop diagnosis tasks.Source: C CONVERGENT ENVIRONMENT DESIGNS · p. 15
  • All 20 most-similar cross-author pairs share a domain; DeepSeek-V4.1-Flash appears in 13, GLM-5.3 in nine, and Claude-Opus-5 in eight.Similarity is measured by TF-IDF cosine similarity of design plans; the five closest pairs are listed in Table 5.Source: C CONVERGENT ENVIRONMENT DESIGNS · p. 15, Table 5 · p. 15
  • Vault and loop each occur in eight titles by five authors, while relay and grid occur in five titles by four authors, and cascade in five titles by three.The paper describes this repeated vocabulary as evidence that model authors explore a narrower space of ideas than their population size suggests.Source: C CONVERGENT ENVIRONMENT DESIGNS · p. 15

D REINFORCEMENT LEARNING HYPERPARAMETERS

The referenced tables identify shared reinforcement-learning settings and report the full downstream results, including the evaluation splits and averaging conventions.

D REINFORCEMENT LEARNING HYPERPARAMETERS

E EFFECT OF THE COUNTER-MOVE

After one counter-move, environments become harder without substantially reducing success when hints are available, increasing hint lift. The accompanying prompts specify how models solve, design, implement, repair, and harden these environments.

E EFFECT OF THE COUNTER-MOVE

  • Adaptation varies by author, and its magnitude is unrelated to solve rate across the nine authors (Spearman ρ = 0.10; exact permutation p = 0.80).Qwen3.8-27B, GLM-5.3, Inkling-Small, and DeepSeek-V4.1-Flash lower no-hint pass rates by 9–13 points; Gemini-3.8-Flash mainly improves hinted pass rate, while GPT-5.6-Sol leaves environments essentially unchanged. Changes below about five points are within noise.Source: E EFFECT OF THE COUNTER-MOVE · p. 17
  • No-hint pass rate falls from 84.9% to 78.5%, while hint lift rises from 6.3 to 13.4 points after the counter-move.Hinted pass rate stays near 91% (91.2% before and 91.9% after); the share of no-hint failures fixed by hints rises from 69.6% to 79.0%, while hint-breaking remains near 5%. Figure 6 compares authors before and after.Source: Figure 6 · p. 17, E EFFECT OF THE COUNTER-MOVE · p. 17

F PROMPTS

  • The solver prompt directs models to use the latest observation and prior feedback to plan within remaining turns, incorporating any hint alongside observations.Each play is independent, and the solver cannot inspect environment source or access another play’s trajectory or outcome.Source: F PROMPTS · p. 17
  • Repair instructions use checker or review findings to revise the plan, code, or hint while preserving the intended challenge and interface.A failed execution check returns errors to implementation; a review rejection returns to planning for a revised plan and fresh implementation.Source: F PROMPTS · pp. 17–18
  • The planning prompt requires a fixed, precise hint and a difficult but fair multi-turn design, with private accounting for hinted feasibility and checks against plausible unhinted strategies.Necessary rules must be public or discoverable before irreversible decisions; the hint must remain accurate across seeds without giving the exact answer or a complete action sequence.Source: F PROMPTS · p. 18
  • The implementation prompt requires a deterministic environment that preserves the plan’s causal traps and hinted feasibility within the supplied budgets.It specifies the environment interface, public rules, action feedback, and standard-library-only constraint.Source: F PROMPTS · p. 18
  • Hardening compares matched hinted and unhinted attempts to diagnose bypasses, ineffective hints, or misleading guidance, then proposes evidence-based revisions.Revisions preserve the core task, interface, rewards, turn budget, seed mapping, and instance values, and are tested against plausible bypasses.Source: F PROMPTS · pp. 18–19

G TOKENS

The section reports estimated token usage by phase and model, alongside estimated costs for environment design and solving.

G TOKENS

  • Table 8 reports average token usage by phase, calculated over completed calls with reported token counters.M denotes millions of tokens.Source: Table 8 · p. 19
  • Table 9 reports token usage by model and estimated total cost across all listed models.M denotes millions and B denotes billions of tokens.Source: Table 9 · p. 19
  • Estimated environment-design costs were $821.79 (15.2%), versus $4,578.66 (84.8%) for solving, out of $5,400.45 total.Approximately $4,000 was spent generating the 200 environments used for reinforcement learning.Source: G TOKENS · p. 19

Related papers

Loading 2610.30116v1…