Source-linked AI summary
RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
Zhiyuan Zeng, Hamish Ivison, Yiping Wang, Lifan Yuan, Shuyue Stella Li, Zhuorui Ye, Siting Li, Jacqueline He, Runlong Zhou, Tong Chen, Chenyang Zhao, Yulia Tsvetkov, Simon Shaolei Du, Natasha Jaques, Hao Peng, Pang Wei Koh, Hannaneh Hajishirzi
TL;DR
Static RL problem distributions can become too easy or too hard as language-model capabilities evolve, weakening learning signals. RLVE uses procedurally generated, verifiably rewarded environments whose difficulty adapts during training, and its 400-environment suite improves reasoning performance beyond continued original RL training. The paper’s future work remains focused on adaptive non-verifiable environments and more reliable automatic environment engineering.
Problem
RL training can stall on static problem distributions when problems become too easy or too difficult for the evolving policy model.
Method
RLVE uses verifiable environments that procedurally generate problems, algorithmically verify outputs, and adapt their difficulty distributions to policy capabilities.
Results
Across six reasoning benchmarks, RLVE improves ProRL-1.5B-v2 by 3.37% absolute on average, versus 0.49% for continued original RL training using over 3× the compute.
Takeaways & Limitations
Scaling the collection of training environments consistently improves performance on held-out environments and supports generalizable reasoning capabilities.
Takeaways & Limitations
Adaptive non-verifiable environments remain an open direction because their rewards cannot be algorithmically defined and their difficulty is harder to control.
Abstract
from arXiv · showhide
We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs). RLVE enables each verifiable environment to dynamically adapt its problem difficulty distribution to the policy model's capabilities as training progresses. In contrast, static data distributions often lead to vanishing learning signals when problems are either too easy or too hard for the policy. To implement RLVE, we create RLVE-Gym, a large-scale suite of 400 verifiable environments carefully developed through manual environment engineering. Using RLVE-Gym, we show that environment scaling, i.e., expanding the collection of training environments, consistently improves generalizable reasoning capabilities. RLVE with joint training across all 400 environments in RLVE-Gym yields a 3.37% absolute average improvement across six reasoning benchmarks, starting from one of the strongest 1.5B reasoning LMs. By comparison, continuing this LM's original RL training yields only a 0.49% average absolute gain despite using over 3x more compute. We release our code publicly.
1. Introduction
RLVE addresses stalled language-model reinforcement learning by combining procedurally generated, verifiably rewarded problems with difficulty distributions that adapt to policy capabilities. Across six reasoning benchmarks, RLVE improves a strong 1.5B reasoning model more than continued original RL training while using less compute.
- Motivation: RL training can stall when problems are too easy to provide learning signals or too difficult to produce useful rewards for gradient updates.Static problem distributions cannot adjust as the policy’s capabilities evolve.
- Approach: RLVE uses verifiable environments that procedurally generate unbounded problems, algorithmically verify outputs, and dynamically adjust difficulty to the policy model.For array sorting, difficulty can increase through longer arrays while correctness is checked against a sorting program.
- Empirical motivation: 3.37% absolute average improvement across six reasoning benchmarks was achieved by RLVE from ProRL-1.5B-v2, versus 0.49% from continued original RLVR training using over 3× the compute.The comparison concerns continued training beyond the model’s original RL training.
- RLVE-GYM: RLVE-GYM contains 400 manually engineered verifiable environments designed as pedagogical tools for developing reasoning capabilities.The environments support scalable problem generation and output verification.
- Scaling environments: Environment scaling consistently improves performance on unseen environments, while joint training across all 400 environments evaluates mathematics, code generation, and logical reasoning.The paper frames expanding the environment collection as a route to generalizable reasoning.
2. Methodology
RLVE represents verifiable environments as programmable generators and verifiers, then adapts each environment’s difficulty range as policy performance improves. Joint training samples among environments while monitoring challenge and reward diversity to maintain useful learning signals.
- Verifiable Environment: A verifiable environment E = (I, P, R) combines an input template, a procedural problem generator, and an algorithmic verifier or reward function.An integer difficulty level controls the expected reasoning complexity of generated problems.
- Adaptive difficulty: RLVE maintains a difficulty range [ℓπ, hπ] and increments hπ when the policy performs well at the current upper bound.Problems are generated by sampling difficulty within the range and then sampling environment-specific parameters.
- Adaptive difficulty: The upper difficulty bound has no predefined cap, while a sliding window lowers ℓπ after updates to preserve exposure to harder problems.The window is controlled by the hyperparameter d∆.
- Joint training: RLVE jointly trains across environments by sampling an environment and maintaining independent difficulty ranges and performance statistics for each one.The same adaptive procedure is applied separately within the selected environment.
- RL Algorithm: RLVE is compatible with RLVR algorithms, and the paper adopts DAPO, a GRPO variant, for its RL training.The methodology therefore changes environment generation and adaptation rather than requiring a new RL optimizer.
- RL Algorithm: The effective prompt ratio measures prompts with non-identical rollout rewards; higher ratios indicate appropriately challenging problems and reduce wasted inference rollouts.DAPO discards prompts whose outputs all receive identical rewards, making reward diversity relevant to training efficiency.
3. RLVE-GYM: A Suite of 400 Environments Created through Environment Engineering
RLVE-GYM is a suite of 400 verifiable environments engineered to develop reasoning through manually solved tasks and algorithmic verification. Its environments define progressively harder problems so adaptive training can provide scalable supervision.
- RLVE-GYM comprises 400 verifiable environments designed through environment engineering for developing language-model reasoning capabilities.The environments are intended as pedagogical tools rather than substitutes for the reasoning process.
- Models solve tasks manually while environments exploit program execution to verify outputs, separating reasoning from reliable computation.This design uses the environment’s computational advantages for verification while restricting the LM from executing programs.
- Some environments make verification substantially easier than solving, enabling supervision signals without implementing costly task solvers.For example, a verifier can check whether an integral answer matches a generated function without computing the integral itself.
- Difficulty levels are designed so solving a higher-level problem entails solving lower-level subproblems within the same environment.Sorting longer arrays and integrating larger expression trees illustrate this monotonic difficulty design.
4. Analyzing Components of RLVE
The component analyses show that adaptive difficulty avoids both saturation and inefficiently challenging training distributions, while expanding the environment collection improves held-out performance. These conclusions are evaluated with controlled held-out environments and progressively larger training collections.
- 4.1. Adaptivity for Unstalled and Efficient Learning: Static environments with low upper-bound difficulty eventually reach an effective prompt ratio of zero after the model masters their hardest problems.Once the environment stops providing differing rewards, learning signals vanish and training saturates.
- 4.1. Adaptivity for Unstalled and Efficient Learning: Static environments with high upper-bound difficulty retain a nonzero effective prompt ratio but provide fewer appropriately challenging problems than adaptive difficulty, impairing ID and OOD performance.Adaptive difficulty maintains a higher proportion of useful prompts as the policy changes.
- 4.1. Adaptivity for Unstalled and Efficient Learning: A static [0, 20] distribution has an oracle advantage because it matches the ID evaluation distribution, yet RLVE achieves comparable or superior ID performance.The paper notes that finding such an oracle distribution is infeasible in realistic settings.
- 4.1. Adaptivity for Unstalled and Efficient Learning: Adaptive difficulty avoids stalled learning from easy problems and inefficient learning when most static problems are too challenging.The conclusion covers both failure modes identified in the adaptive-versus-static comparison.
- 4.1. Adaptivity for Unstalled and Efficient Learning: Joint training across 256 environments requires individually tuned difficulty because environments reach a broad range of upper-bound levels, and a covering static range still underperforms RLVE.The adaptive upper-bound distribution spans 0 to 12 at step 400, while static [0, 20] training is consistently worse.
- 4.2. Environment Scaling as a Key Driver of Generalizable Reasoning Capabilities: Expanding the collection of training environments consistently improves performance on 50 held-out environments across model types.The collections contain 1, 4, 16, or 256 environments under identical training setups, with each larger collection containing the smaller ones.
5. Scaling Up RL Training with RLVE
RLVE scales reinforcement-learning training in both data-saturated and compute-constrained settings, outperforming continued RLVR training and a strong RLVR dataset across six reasoning benchmarks.
- Experimental Scenarios: The evaluation tests RLVE in data-saturation and compute-constrained scenarios using joint training across all 400 RLVE-GYM environments.The first scenario starts from a model saturated on ProRL; the second starts from a model without prior reasoning RL.
- Evaluation: The experiments cover six benchmarks spanning mathematics, code generation, and logical reasoning.The benchmarks include AIME 2024/2025, OMEGA-500, OlympiadBench, LiveCodeBench, and BBEH.
- 5.1. Scaling beyond Data Saturation: 3.37% absolute average improvement across six reasoning benchmarks with RLVE exceeded the 0.49% gain from continued ProRL training, despite using over three times less compute.RLVE used approximately 1,100 H100 GPU hours, versus 3,600 for continued original RL training.
6. Related Work
Prior work applies procedural generation, curriculum learning, and adaptive filtering to LM training, but RLVE differs through controllable, pre-rollout adaptation of verifiable environments.
- RL training on procedurally generated data: Procedurally generated LM training environments commonly use static difficulty distributions, leading to early saturation or inefficient learning.The cited contrast motivates adaptive difficulty rather than fixed problem distributions.
- Adaptive data for LM training: Curriculum-learning methods reorder finite datasets after construction, whereas RLVE predefines difficulty levels and adapts generated problems during training.RLVE therefore changes the environment’s problem distribution rather than only reordering existing examples.
- Adaptive data for LM training: Post-rollout filtering complements RLVE because it removes unhelpful prompts after inference, while RLVE adapts problems before they reach the inference engine.The two approaches operate at different stages of the RL pipeline.
7. Discussion on Future Work
The discussion identifies adaptive non-verifiable environments and automated environment engineering as important but unresolved directions for scaling RL.
- Scaling up RL with adaptive non-verifiable environments: Adaptive non-verifiable environments remain an open direction because rewards cannot be algorithmically defined and their weak structure complicates difficulty control.Examples include creative writing and deep research.
- Model-based automatic environment engineering: Automatic environment engineering did not reliably maintain input clarity, generator validity and diversity, or verifier robustness without human intervention.The authors therefore regard manual engineering of all 400 environments as worthwhile.
- Implementation details: RLVE’s default implementation uses an accuracy threshold of 0.9, a minimum sample threshold of eight times the rollout count, and a sliding-window size of 4.These are implementation defaults rather than general conclusions of the discussion.
B.1. Representative Sources of Verifiable Environments with Example Environments in RLVE-GYM
RLVE-GYM combines diverse procedurally generated, verifiable tasks whose difficulty controls and reward functions are tailored to mathematical, algorithmic, optimization, and logical problems.
- Environment framework: RLVE-GYM contains 400 verifiable environments, each using a problem generator and algorithmic verifier to produce problems and rewards.The environments span multiple sources, including mathematical operations, programming competition problems, optimization, classical algorithms, and logical puzzles.
- RLVE algorithm: The RLVE loop samples environments uniformly, generates rollouts, computes verifier rewards, updates the policy, and then checks whether each environment’s difficulty should change.Per-environment accuracy and sample counts support the difficulty-update logic.
- Mathematical operations: The integral environment generates a function by expression-tree size, differentiates it with SymPy, and verifies whether the predicted antiderivative differentiates to the supplied derivative.This avoids generating arbitrary integrals whose antiderivatives may lack elementary closed forms.
- Optimization problems: PolynomialMinimum samples degree 2(d + 1) polynomials with positive leading coefficient and smoothly rewards predictions approaching the analytically or numerically computed minimum.Its reward raises normalized progress toward the true minimum to the fifth power.
- Classical algorithmic problems: Sorting difficulty increases with array length, which is sampled roughly proportional to 3 × 1.1^d; rewards scale with the fraction of correctly placed elements.Malformed outputs receive −1.0 and incorrect-length arrays receive −0.5.
- Logical puzzles: Hamiltonian-path difficulty uses N = d + 3 vertices, while the verifier rewards the fraction of valid consecutive edges and assigns 1.0 exactly to valid Hamiltonian paths.The construction guarantees at least one Hamiltonian path, exploiting efficient verification despite difficult search.
C. RL Training Details
RL training uses the slime framework with DAPO, dynamic sampling, and partial rollouts. Runs use model-specific optimization settings and require approximately 350–1,500 H100 GPU hours.
- Training uses slime with DAPO, a GRPO variant, and omits KL regularization and entropy loss.
- Dynamic sampling combines oversampling and filtering, while partial rollouts cache unfinished generations for later resumption.Training uses batches of 128 and 384, with 16 rollouts per prompt at temperature 1.0.
- The four 1.5B models use a 2 × 10−6 learning rate and 24,576-token rollout limit, while Qwen2.5-7B-Base uses 1 × 10−6 and 8,192 tokens.Models use their default chat templates, whereas Qwen2.5-7B-Base uses a DeepSeek-AI-inspired prompt format.
- Each run uses 8× NVIDIA H100 GPUs and takes approximately 2–8 days, or roughly 350–1,500 H100 GPU hours.
- RLVE-specific training details are provided in Appendix A.
D. Evaluation Details
Evaluation uses SGLang with fixed sampling settings and model-dependent response limits. Benchmark protocols vary by task, while the held-out Dood set contains 2,500 procedurally generated problems across 50 environments.
- SGLang evaluation uses temperature 0.6, top-p 0.95, and maximum response lengths of 32,768 or 16,384 tokens depending on the model.
- AIME uses 64 samples with Avg@64, three benchmarks use four samples with Avg@4, and LiveCodeBench uses 16 samples with Pass@8.Pass@8 is defined as the probability that at least one of eight uniformly sampled outputs passes all test cases.
- The held-out environment list is drawn from the RLVE-Gym suite and includes named environments such as BAJBytecomputer, CRT, and ConvexHull.
- Dood contains 2,500 problems generated across 50 held-out environments, with difficulty evenly distributed from 0 to 4.The environments are randomly sampled from the full suite of 400 RLVE-Gym environments.
- In-distribution and Dood evaluations use one model output per problem.
E. Details of Training Environment Collection
Training environment collections are nested subsets of RLVE-Gym that scale from one to 256 environments. Each larger collection adds environments while retaining all environments in the preceding collection.
- The collections follow C1 ⊂ C4 ⊂ C16 ⊂ C256, with each drawn from the 400-environment suite after excluding 50 held-out environments.
- The section lists each collection’s composition and its incremental difference from the preceding collection.
- C1 contains only the Multiplication environment.
- C4 contains four environments, expanding C1 with Division, EuclidGame, and Sorting.
- C16 contains 16 environments, adding 12 to C4, including HamiltonianPath, SAT, and ShortestPath.
- C256 contains 256 environments, extending C16 with 240 additional environments.