Source-linked AI summary
REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, Andreas Köpf
TL;DR
Reasoning research is constrained by fixed, noisy, or quickly memorized benchmarks and scarce high-quality training data. Reasoning Gym provides over 100 procedurally generated, algorithmically verifiable environments with controllable difficulty for RLVR training and evaluation. Experiments show improved reasoning capabilities that transfer within and across domains and to established external benchmarks.
Problem
Reasoning research faces a data scarcity bottleneck because existing benchmarks can be fixed in size, quickly memorized, or noisy, while high-quality training data is costly or unreliable.
Method
RG uses over 100 procedurally generated, algorithmically verifiable reasoning tasks with controllable difficulty and structural variation for RLVR training and evaluation.
Results
RLVR training on RG tasks produces improved reasoning capabilities that transfer within and across domains and to established external benchmarks.
Takeaways & Limitations
RG provides a framework for evaluating reasoning across difficulty levels without fixed-benchmark memorization concerns.
Takeaways & Limitations
RG currently focuses on single-turn, text-based reasoning and does not yet include multi-turn or multimodal tasks.
Abstract
from arXiv · showhide
We introduce Reasoning Gym (RG), a library of reasoning environments for reinforcement learning with verifiable rewards. It provides over 100 data generators and verifiers spanning multiple domains including algebra, arithmetic, computation, cognition, geometry, graph theory, logic, and various common games. Its key innovation is the ability to generate virtually infinite training data with adjustable complexity, unlike most previous reasoning datasets, which are typically fixed. This procedural generation approach allows for continuous evaluation across varying difficulty levels. Our experimental results demonstrate the efficacy of RG in both evaluating and reinforcement learning of reasoning models.
1 Introduction
Reasoning Gym addresses the scarcity and scalability limits of existing RLVR data with procedurally generated, verifiable tasks. Its experiments report benefits for reasoning-model training and evaluation.
- Current RLVR approaches depend on costly human-curated or unsustainable, unreliable internet-scraped data, creating a scalability bottleneck.
- RG provides over 100 algorithmically verifiable tasks that generate unlimited instances with controllable difficulty and structural variation across diverse reasoning domains.
- RG reduces memorization concerns, supports adaptive curricula, removes fixed-dataset limits, and enables controlled study of specific reasoning capabilities.
- Zero-shot performance is low on many RG tasks involving visual concepts represented in text, including ARC, cognition, and games.
- Curriculum RLVR improves final models, while RLVR transfers within domains, across domains, and to external benchmarks.
- The complete library, training infrastructure, and experimental configurations are released as open-source resources.
2 REASONING GYM (
Reasoning Gym is an open-ended, procedurally generated playground designed to overcome fixed, noisy, and memorization-prone benchmarks. Its generators span diverse reasoning categories and expose controllable task complexity for training and evaluation.
- RG is motivated by benchmarks that are fixed in size, quickly memorized, or noisy, limiting evaluation and training beyond the dataset ceiling.
- Every task supports automatic verification without human judgment, enabling reliable RLVR training and objective evaluation.
- RG partitions generators into mathematical, algorithmic, logical, pattern-recognition, and constraint-satisfaction categories.
- The taxonomy lets practitioners target specific abilities while retaining a procedurally generated mix of reasoning challenges.
- Generators vary difficulty, structure, and presentation through parameters controlling complexity, problem properties, and style.
- Figure 3 compares zero-shot model ability on hard configurations with the effect of dataset difficulty on per-category accuracy.
3 Zero-shot performance of LLMs
Reasoning-optimized models outperform general-purpose models across RG categories, but performance remains fragile as task difficulty increases. Visual-spatial tasks are especially challenging, and difficulty cliffs vary substantially by domain.
- Model Capabilities Across Reasoning Domains: Reasoning-optimized systems form a distinct leading group, substantially outperforming general-purpose alternatives across RG tasks.
- Model Capabilities Across Reasoning Domains: 63.5% is o3-mini’s performance, compared with 41.5% for Llama 4 Maverick on the reported RG evaluation.
- Performance Patterns Across Domains: Mathematical domains show relatively strong performance, whereas text-based visual-spatial tasks in cognition and games are particularly challenging.
- Performance Patterns Across Domains: Algorithmic tasks are intermediate, with systematic decomposition benefiting significantly from specialized reasoning training.
- The Difficulty Cliff Phenomenon: Difficulty increases commonly degrade performance across domains and model families, suggesting that current reasoning competencies remain fragile.
- The Difficulty Cliff Phenomenon: −71.9% is o3-mini’s code decline when moving from easy to hard configurations; its corresponding drops are −33.8% in graphs, −33.1% in geometry, and −25.6% in algorithms.
- Performance Patterns Across Domains: Text-based spatial reasoning remains difficult for all models, while difficulty-cliff magnitudes vary across domains.
4 Skill Transfer and Generalization
The experiments test whether RLVR training on procedurally generated tasks transfers within domains, across domains, and to established benchmarks. Results show consistent intra-domain gains, substantial cross-domain transfer, and improvements on several external benchmarks.
- Intra-Domain Transfer: RLVR training on tasks within a reasoning category improves performance on held-out tasks from the same domain.The design uses composite category training, held-out same-domain evaluation, and three independent runs.
- Intra-Domain Transfer: Intra-domain gains range from arithmetic +6.3% and cognition +2.0% to algebra +11.7% and algorithmic reasoning +7.4%.The largest reported gains occur in more challenging domains, while arithmetic starts from stronger base-model competence.
- Intra-Domain Transfer: Games accuracy rises from zero for the base model to 3.3% after RLVR training.The passage interprets this as measurable capability emerging in a category where the base model initially has no accuracy.
- Cross-Domain Transfer: Cross-domain experiments train on one RG category and evaluate on held-out tasks from different domains, isolating transfer across reasoning types.Evaluation domains are unseen during training, with three independent runs per cross-domain evaluation.
- Cross-Domain Transfer: Algorithmic training improves algebra by +29.1% and geometry by +22.3%, while logic training improves cognition by +13.3% and graph reasoning by +9.1%.Games training also yields selective gains in algebra (+21.8%) and cognition (+13.1%).
- Transfer to External Benchmarks: RG-Math improves MATH by +9.7%, Big-Bench Hard by +7.7%, and GSM8K by +0.5% over the baseline.Both RG-Math and RG-Algorithmic also significantly outperform their respective baselines on several MMLU-Pro tasks.
5 Curriculum RLVR
The curriculum experiments compare progressively increasing difficulty with fixed-difficulty sampling during RLVR training. Curriculum training generally achieves higher final performance, although progression may be less useful when difficulty paths are ambiguous or hard to formalize.
- Experimental Setup: Curriculum training starts at the easiest level and increases difficulty after performance exceeds 70% over 20 training steps.The comparison uses Qwen2.5-3B-Instruct trained with GRPO for one epoch against uniform sampling across all difficulty levels.
- Results: Curriculum training generally achieves superior performance across difficulty levels compared with non-curriculum training.Table 5 compares a model trained with an adjustable curriculum against a model trained on all levels concurrently.
- Training Dynamics: Difficulty increases are often followed by sharp reward drops, while the curriculum model ultimately encounters increasingly difficult examples.In Spell Backwards, the curriculum model’s lower terminal reward reflects exclusive exposure to maximum-difficulty examples, unlike uniform difficulty sampling.
- Results: Curriculum-trained models outperform non-curriculum equivalents in all environments and difficulty levels, including gains of +40.67% on Spell Backwards, +13.33% on Mini Sudoku, and +26.27% on Count Primes.These examples cover different environments and difficulty settings.
- Limitations: Curriculum learning may be less valuable in environments where progression paths are ambiguous or difficult to formalize.The stated boundary concerns whether a meaningful sequence of increasingly difficult examples can be defined.
6 Related Work
Related work includes fixed reasoning benchmarks, RLVR environments, procedural puzzle systems, and libraries of reasoning environments. RG differs by generating unlimited examples with controllable characteristics while exposing the data-generating process.
- Reasoning Benchmarks: Fixed benchmarks such as GSM8K, MATH, OlympiadBench, BIG-Bench, GPQA, and coding datasets cover mathematical, general, and programming reasoning.
- Reasoning Benchmarks: Fixed datasets can encourage overfitting, while internet-scraped benchmark data may contain errors.
- RLVR Environments: RLVR environments use objectively verifiable answers and automatic rewards, with prior systems combining curated examples and automatically gradable tasks.
- RLVR Environments: Related systems include text-based games, procedurally generated logic puzzles, and methods for synthesizing open-ended logic puzzles.
- Reasoning Environment Libraries: KORGym, Reasoning Core, and GEM provide libraries or frameworks for games, formal domains, and diverse LLM reasoning environments.
7 Discussion and Future Work
The discussion identifies limitations in procedural reasoning environments and outlines future work beyond RG’s current single-turn, text-based, independently sampled setting.
- Limitations: Procedural generators struggle with domains requiring extensive domain knowledge or creativity, especially when answers are unstructured.
- Limitations: Verification functions may miss aspects of solution quality that humans consider important, preserving a role for human feedback and human-gathered datasets.
- Future Work: RG currently excludes multi-turn and multimodal reasoning tasks, which are identified as valuable for agentic and vision-language models.
- Future Work: Experiments sample tasks uniformly under an independent-and-identically-distributed assumption, motivating study of continual learning with nonstationary data streams.
- Future Work: Future continual-learning studies could examine regularization, model merging, and replay buffers under catastrophic forgetting.
8 Conclusion
The conclusion presents Reasoning Gym as an open library of procedurally generated, verifiable environments that addresses reasoning-data scarcity. It reports improved transfer and positions RG as both a training and evaluation framework.
- Reasoning Gym provides over 100 procedurally generated and algorithmically verifiable tasks across algebra, algorithms, logic, games, and geometry.
- RG addresses reasoning-data scarcity through unlimited, controllable task generation for reinforcement learning with verifiable rewards.
- RLVR training on RG tasks improves reasoning capabilities within domains, across domains, and on established external benchmarks.
- RG supports evaluation across difficulty levels without the memorization concerns associated with fixed benchmarks.
- The complete library is released as an open-source resource for studying reasoning development without fixed datasets or expensive human curation.
Checklist
The checklist reports that the paper addresses claims, limitations, assets, reproducibility, ethics, and experimental reporting requirements. It also states that no human-subject or crowdsourcing data were used.
- The paper states that its main claims and contributions accurately reflect the work and that its limitations are discussed in Section 7.
- Potential negative societal impacts are marked not applicable because the work is limited to dataset generators for problem-solving tasks with minimal societal implications.
- The paper reports no theoretical results, so assumptions and complete proofs are marked not applicable.
- The authors report including code, data, instructions, training details, and hyperparameters needed to reproduce the experiments in the linked repository.
- The paper reports error bars and identifies the relevant experimental sections where they are provided.
- The authors report the compute amount and resource type used for the experiments.
- Existing assets are cited and licensed, while newly released assets are included through a URL.
- The checklist marks human-data consent, personally identifiable information, offensive-content review, and human-subject procedures as not applicable because no such data or studies were used.
A.1 Dataset categories
REASONING GYM presents procedurally generated reasoning tasks across diverse domains, with adjustable difficulty and verifiable outputs. Representative environments cover computation, cognition, geometry, graph search, and logical or causal inference.
- A.1 Dataset categories: REASONING GYM organizes environments into categories such as arithmetic, computation, cognition, geometry, graph theory, and logic.The overview describes a broad dataset collection, while representative tasks illustrate several of these categories.
- A.1 Dataset categories: Complex arithmetic tasks generate operations over bounded real and imaginary components, with weighted addition, subtraction, multiplication, and division.The example subtracts (7.0 - 7.0i) - (-5.0 + 2.0i) and returns 12.0 - 9.0i.
- A.1 Dataset categories: Algorithmic computation tasks include spiral-matrix traversal, prime factorization, and execution of Brainf*ck programs.These tasks specify output formats and, where shown, configurable input ranges or fixed difficulty.
- A.1 Dataset categories: Cognitive and symbolic tasks include one-dimensional ARC rule induction, figlet-font reading, mini Sudoku, and sage-or-fool classification.The examples require inferring transformations, decoding visual text, satisfying Sudoku constraints, or evaluating truthfulness statements.
- A.1 Dataset categories: Geometry and graph tasks ask models to solve coordinate problems and find shortest paths through blocked grids using four-directional movement.The examples provide an orthocenter answer and a path answer of up right right.
- A.1 Dataset categories: Other environments test causal detector activation and truth-telling classification using prior observations and explicit response labels.Blicket tasks distinguish on, off, and undetermined outcomes, while island puzzles distinguish sages from fools.