Source-linked AI summary
Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis
Yucheng Shi, Zhenwen Liang, Kishan Panaganti, Dian Yu, Wenhao Yu, Haitao Mi
TL;DR
Existing zero-data reasoning RL lacks durable, reusable environments whose rewards remain stable as the learner improves. EvoEnv synthesizes and validates such environments, improving three model families, including an already-strong checkpoint where fixed public-data and hand-crafted environment RLVR reduce average performance.
Problem
Zero-data reasoning RL lacks durable, reusable training environments that can be validated once, sampled repeatedly, calibrated, and reused as the learner changes.
Method
EvoEnv synthesizes reusable executable environments and curates them through staged validation, semantic review, solver-relative difficulty calibration, novelty control, and pool rotation.
Results
EvoEnv improves three model families, including an already-strong thinking-mode checkpoint where fixed public-data and hand-crafted environment RLVR both reduce average performance.
Takeaways & Limitations
Stable self-improvement may depend on models constructing executable environments that can be validated, frozen, sampled, and reused rather than generating more synthetic data.
Takeaways & Limitations
The study is limited to zero-data RLVR for reasoning with deterministic Python environments and does not establish the same claims for open-ended judgment, preference modeling, physical simulation, or interactive tool use.
Abstract
from arXiv · showhide
We pursue a vision for self-improving language models in which the model does not merely generate problems or traces to imitate, but constructs the environments that train it. In zero-data reasoning RL, this reframes self-improvement from a data-generation loop into an environment-construction loop, where each artifact is a reusable executable object that samples instances, computes references, and scores responses. Whether this vision sustains improvement hinges on a single property: the environments must exhibit stable solve--verify asymmetry, the model must be able to write an oracle once that it cannot reliably execute in natural language on fresh instances. This asymmetry takes two complementary forms. Some tasks are algorithmically hard to reason through but trivial as code: a dynamic program or graph traversal, compiled once, yields unboundedly many calibrated instances. Others are intrinsically hard to solve but easy to verify, like planted subset-sum or constraint satisfaction. Both create a durable gap between proposing and solving that the policy cannot close by gaming the verifier, and it is this gap that keeps reward informative as the learner improves. We instantiate this view in EvoEnv, a single-policy generator, solver method that synthesizes Python environments from ten seeds and admits them only after staged validation, semantic self-review, solver-relative difficulty calibration, and novelty checks. The strongest evidence comes from the already-strong regime: on Qwen3-4B-Thinking, fixed public-data RLVR and fixed hand-crafted environment RLVR reduce the average, while EvoEnv improves it from 72.4 to 74.8, a relative gain of 3.3%. Stable self-improvement, we suggest, depends not on producing more synthetic data, but on models learning to construct worlds whose difficulty stays structurally beyond their own reach.
1 Introduction
The paper reframes zero-data reasoning self-improvement from generating more data to synthesizing reusable executable environments with stable solve–verify asymmetry. EvoEnv implements this through staged environment validation and solver-relative calibration, improving performance where fixed RLVR distributions have saturated.
- Motivation: Self-improvement requires interactive environments that generate situations, impose constraints, and provide execution-based feedback, not only additional examples to imitate.
- Limitations: Fixed RLVR distributions lose reward variation as policies improve, while policy-derived labels can compromise stability and one-off verifiers lack reusable value.
- Stable solve–verify asymmetry: Generated environments provide reusable prompt distributions and reward sources because authoring or checking executable procedures can remain easier than solving fresh instances.
- EvoEnv: EvoEnv alternates generator and solver roles, admitting Python environments only after interface, semantic correctness, difficulty, and novelty checks.The method curates a self-generated pool through staged validation, semantic self-review, solver-relative calibration, novelty control, and pool rotation.
- Evidence: EvoEnv improves an already-strong thinking-mode checkpoint where fixed public-data and hand-crafted-environment RLVR reduce average performance.The result supports using self-evolving environments to maintain stable, frontier-calibrated reward after static distributions saturate or become misaligned.
2 Related Work
EvoEnv is positioned at the intersection of verifier-backed RL, adaptive self-generated curricula, executable task checks, and environment-level training. Its distinguishing setting is zero-data reasoning RL, where the learner authors reusable executable environments admitted through validation, solver-relative calibration, and novelty filtering.
- EvoEnv’s positioning: EvoEnv combines stable verifiers, adaptive curricula, executable checks, and environment-level training by having the learner author reusable executable environments.Each environment is admitted only after validation, solver-relative calibration, and novelty filtering.
- Fixed-distribution RLVR: Fixed-distribution RLVR uses reliable verifier-based outcome rewards but can saturate, over-specialize, or forget when prompts become too easy or too difficult.Group-relative rewards lose useful variation when examples are almost always solved or failed.
- Self-generated curricula: Self-generated curricula adapt through model-proposed tasks and self-estimated correctness, but their reward signals are often policy-coupled.Methods use majority vote, self-consistency, internal feedback, or co-evolving agents to estimate correctness.
- Executable grounding: Executable-grounding methods replace purely self-believed correctness with code or tests, including execution-based program verification and generated Code-as-Task verifiers.Examples include Absolute Zero Reasoner, Self-Challenging Agents, SPC, and agentic systems for tool use and software engineering.
- Environment-level training: Environment-level training treats environments as training objects, using externally given suites, offline-synthesized tool or embodied environments, or learned simulators.In learned-simulator approaches, reward depends on the simulator’s beliefs.
3 Method
EvoEnv trains one policy to solve verifiable tasks and construct reusable executable environments, admitting candidates only after validation, semantic review, solver-relative calibration, and novelty filtering. Its environments implement standardized Python generation, prompting, and scoring interfaces, while pool admission preserves semantic correctness, nontrivial difficulty, and diversity.
- Method overview: EvoEnv trains a single policy to solve verifiable reasoning tasks and construct reusable executable environments for sampling and scoring fresh instances.Candidate environments influence solver training only after passing mechanical validation, semantic review, solver-relative difficulty calibration, and novelty filtering.
- Environment interface: Each candidate is a sandboxed Python subclass whose _generate, _prompt_generate, and _process/scorer methods implement instance generation, prompt rendering, and verification.Candidates use an approved standard-library subset and execute in sandboxed subprocesses with wall-clock timeouts.
- Validation and review: Five validation layers test parseability, interface execution, determinism, non-triviality, and higher-level correctness before candidates can enter the active pool.Mechanical checks alone cannot establish that code matches the task described in the prompt, motivating an additional conservative semantic-review filter.
- Validation and review: Semantic review controls pool admission but is excluded from generator reward, which instead uses mechanical validation, solver-relative difficulty, and novelty.A rejected candidate may still contribute mechanically computed generator rollout reward, but cannot become a reward source for solver training.
- Difficulty calibration: Candidates target solver-relative pass rate a⋆= 0.3, retaining instances that are solvable enough to yield positive examples but difficult enough to preserve reward variation.Admission requires 0 < ˆam(e; πθ) < 1; candidates with pass rate 0 are too hard or underspecified, while pass rate 1 indicates saturation or permissiveness.
- Novelty and admission: Two-view prompt and code novelty embeddings discourage duplicate environments, while final admission requires L5 validation, review approval, intermediate solver pass rate, and similarity below the gate.The admission rule is At(e) = I[ℓ(e) = 5] · I[review(e) = 1] · I[0 < ˆam(e; πθ) < 1] · I[simt(e) < τgate].
4 Experiments
Across three model families, EvoEnv improves average reasoning performance, sustains a frontier signal through training, and expands a validated, diverse environment pool. Ablations show that both quality and diversity rewards are necessary for strong gains and transfer.
- Main results: EvoEnv is the only method improving all three model families, raising average pass@1 from 49.2 to 53.1, 72.4 to 74.8, and 71.0 to 73.2.These correspond to relative gains of 7.9%, 3.3%, and 3.1%, respectively, for Qwen3-4B-Instruct, Qwen3-4B-Thinking, and Nemotron-Cascade-8B.
- Training dynamics: 72.4% to 80.4%: held-out accuracy rises while EvoEnv’s solver training score falls from 0.88 to 0.61 as generated environments become harder.The fixed-environment baseline stagnates near 72%, whereas EvoEnv’s declining training score preserves a frontier signal.
- Environment growth: 840 accepted environments and 45 tag prototypes accumulate within 100 steps from ten seeds, indicating continued growth and structurally distinct environment families.A new prototype requires Jaccard similarity below τ=0.5 to every existing prototype, so the prototype count lower-bounds distinct families.
- Environment diversity: 68.2% of generated environments involve number theory, with modular and sequence structure appearing in about 38% each alongside varied executable task mechanisms.The pool also includes feasibility, minimization, subset, graph/path, traversal, set, and sampling mechanisms.
- Reward ablations: Removing quality reward cuts the average gain from +2.4 to +0.5, while removing diversity cuts it to +0.6 and particularly harms out-of-distribution benchmarks.Quality removal most affects HMMT, GPQA, and AIME 2025; diversity removal particularly hurts GPQA and LiveCodeBench.
5 Conclusion
EvoEnv frames zero-data reasoning RL as an environment-construction loop in which models build executable training environments with stable reward. Its mechanism is compile-once, solve-many: structures are validated, frozen, sampled, and reused rather than answers being trusted.
- 5 Conclusion: EvoEnv reframes zero-data reasoning RL from a problem-generation loop into an environment-construction loop.The model produces executable structure that can be validated, frozen, sampled, and reused.
- 5 Conclusion: Compile-once, solve-many is the mechanism for building reusable training environments.The approach compiles executable structures once, then samples and reuses them across instances.
A Detailed positioning against nearby self-improvement methods … D Illustrative planted subset-sum environment
EvoEnv is positioned by the unit it synthesizes and by whether its reward source remains stable under policy updates. Its environments preserve a code-mediated verification path, undergo staged validation, and can test planted subset-sum solutions against exact constraints and multiplicities.
- A Detailed positioning against nearby self-improvement methods: EvoEnv differs from nearby self-improvement families by synthesizing a reusable environment rather than a single problem, verifier, or trajectory.The comparison also distinguishes methods by reward-source stability against further updates of the same policy.
- A Detailed positioning against nearby self-improvement methods: A reward source may support useful curricula without providing a durable, sample-many object that remains stable as the policy updates.The passage identifies policy-coupled, learned, and per-instance reward sources as examples that lack this guarantee by themselves.
- B Anatomy of a verifiable environment: Reference answers are computed by deterministic Python execution and reach the scorer through the code path, while the solver sees only the rendered natural-language prompt.Figure 5 presents this separation as the data flow inside one environment.
- B Anatomy of a verifiable environment: The validation pipeline must preserve the intended meaning of every arrow connecting generation, reference computation, and scoring.The passage attributes this protection to safeguards across validation layers L1–L5 in Appendix C.
- C Additional interface and validator details: Candidates enter the pool only after validation layers progress from cheap syntactic checks to more expensive semantic and solver-facing tests.This ordering prevents early failures from consuming solver rollout budget.
- D Illustrative planted subset-sum environment: The planted subset-sum generator embeds a solution and distractors, requiring the solver to find any valid subset.The scorer verifies both the target sum and multiset multiplicities.
E Semantic-review audit against an external reviewer · F Seed environments
The semantic reviewer closely tracks a stronger external reviewer and serves as a pool-admission gate rather than a generator-reward signal. Ten fixed executable seeds provide broad coverage of downstream interfaces, algorithmic families, scorer types, and difficulty controls.
- E Semantic-review audit against an external reviewer: Because reviewer verdicts affect pool admission rather than generator reward, false positives lose solver-pool slots while false negatives must still pass solver-relative calibration.Residual reviewer disagreement therefore does not directly contaminate the generator gradient.
- E Semantic-review audit against an external reviewer: The audit uses 79 mechanically validated environments labeled by GPT-5.4 and applies the deployed Qwen3-4B reviewer with the same four-step structured prompt.GPT-5.4 labels 35 environments as buggy and 44 as correct.
- E Semantic-review audit against an external reviewer: F1 = 87.0% under the deployed any-reject configuration, with P = 85.7%, R = 88.2%, and accuracy 88.6%.This exceeds the always-correct baseline’s 55.7% accuracy and the always-has_bugs baseline’s 44% precision.
- E Semantic-review audit against an external reviewer: The reviewer catches greedy substitutions, hardcoded or trivial references, mathematical violations, and most incorrect code, but misses subtle generation pathologies and hallucinated edge cases.These three latter modes account for most false negatives and false positives in Table 5.
- F Seed environments: All experiments use the same ten executable seed environments, which define file layout, parameter, prompt-rendering, and scorer conventions rather than supplying external problem–answer data.The seeds are both interface examples and initial pool items, and the exact set remains fixed across experiments.
- F Seed environments: Each seed is a Python class implementing VerifiableEnvironment, with parameter knobs filled through self.parameter and a scorer-family designation.Table 6 lists the seeds individually and the paper expands on four scorer families below it.
- F Seed environments: The ten seeds span seven algorithmic families, four scorer families, and primitives including arrays, graphs, recurrences, dynamic programming, and game-theoretic state.Four seeds are adapted from Luogu OI archives, while six are written from scratch.
G Generated-environment data audit · G.1 Qualitative examples within the 100-step run
The qualitative audit examines environments generated within the first 100 steps, measuring their distance from the ten seeds while treating validation—not inspection—as the correctness gate. It shows movement toward solver-calibrated number-theoretic, modular, and sequence-structured problems rather than trivial RLVE copies.
- G.1 Qualitative examples within the 100-step run: The gallery is restricted to environments generated no later than step 100, matching the Section 4.4 ablation study.
- G.1 Qualitative examples within the 100-step run: Distances are measured by Jaccard distance from each environment’s tag set to the nearest of the ten original seeds.
- G.1 Qualitative examples within the 100-step run: The audit makes the generator’s distributional shift concrete at the environment level.
- G.1 Qualitative examples within the 100-step run: The examples are illustrative and are not intended to certify correctness by inspection.
- G.1 Qualitative examples within the 100-step run: Every accepted environment still passes the validation and self-review pipeline described in Section 3.2.
- G.1 Qualitative examples within the 100-step run: The generator moves quickly from the seed pool toward number-theoretic, modular, and sequence-structured problems.
- G.1 Qualitative examples within the 100-step run: These problems are solver-calibrated rather than trivially copied from RLVE.
H Limitations and Risks
The paper’s claims are limited to zero-data RLVR with deterministic Python environments and do not yet extend to several open-ended or real-world settings. Generated executable code requires sandboxed validation, while autonomous curriculum construction offers auditability but may amplify reasoning capabilities for beneficial and dual-use purposes.
- Scope of the environment interface: The method is scoped to deterministic Python environments for zero-data reasoning RLVR, not open-ended judgment, human preferences, or long-horizon physical simulation.The interface best fits tasks with compact, automatically validated samplers, oracles, renderers, and scorers.
- Executable-code safety: Generated environments must be treated as untrusted code and rejected when they violate interface or execution constraints.The implementation uses sandboxed subprocesses, restricted imports, resource limits, deterministic seeds, and wall-clock timeouts.
- Broader impact: Verifiable environment synthesis makes curricula more inspectable and rewards easier to audit, but autonomous curriculum construction may amplify beneficial and dual-use reasoning capabilities.Accepted environments are concrete code artifacts rather than opaque pseudo-labels or hidden preference judgments.
I Hyperparameters
The paper reports headline hyperparameters in Table 8 and the complete deployed training configuration in Table 9. The full configuration is organized by component and specifies which settings are shared across backbone variants versus rescaled for 8B.
- Headline hyperparameters: Table 8 reports the training and evaluation hyperparameters highlighted in the main text.Table 8 is the headline hyperparameter reference.
- Full training configuration: Table 9 gives the full configuration of the deployed 4B-Instruct training run, grouped by generator, solver, novelty, pool, review, sandbox, and optimizer components.The configuration is organized by component rather than presented only as headline settings.
- Cross-backbone settings: The generator, solver, novelty, pool, review, and sandbox settings are shared across backbone variants.Only the optimizer block changes when scaling to 8B.
- 8B scaling: For 8B scaling, only optimizer batch sizes and context lengths are rescaled.This distinction is specified in the full deployed-run configuration.