Source-linked AI summary
Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li, Richard Fan, Jianshu She, Chengqian Gao, Abulhair Saparov, Haonan Li, Taylor W. Killian, Mikhail Yurochkin, Zhengzhong Liu, Eric P. Xing, Zhiting Hu
TL;DR
RL reasoning research has concentrated on math and code and lacks reliable reward data across diverse domains. The paper introduces GURU, a curated six-domain corpus with verifiable rewards and evaluates RL across domains. Results show domain-sensitive RL effects and state-of-the-art open-model performance, while training duration and the absence of an initial SFT phase remain scope boundaries.
Problem
RL reasoning work largely focuses on math and code, while diverse domains lack reliable, scalable reward signals and curated training data.
Method
The paper constructs GURU, a 92K-example corpus spanning six reasoning domains using domain-specific verification, deduplication, and heuristic and model-based filtering.
Results
RL effects are domain-sensitive, with cross-domain gains in Math, Code, and Science, in-domain requirements for Logic, Simulation, and Tabular, and GURU models outperforming best baselines by 7.9% and 6.7% across 17 tasks.
Takeaways & Limitations
RL can elicit pretrained knowledge in well-covered domains and is more likely to foster genuine reasoning capabilities in domains underrepresented during pretraining.
Takeaways & Limitations
Training lasted only three epochs for 7B models and two epochs for 32B models because of computing resource constraints.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has emerged as a promising approach to improve large language model (LLM) reasoning, yet most open efforts focus narrowly on math and code, limiting our understanding of its broader applicability to general reasoning. A key challenge lies in the lack of reliable, scalable RL reward signals across diverse reasoning domains. We introduce Guru, a curated RL reasoning corpus of 92K verifiable examples spanning six reasoning domains--Math, Code, Science, Logic, Simulation, and Tabular--each built through domain-specific reward design, deduplication, and filtering to ensure reliability and effectiveness for RL training. Based on Guru, we systematically revisit established findings in RL for LLM reasoning and observe significant variation across domains. For example, while prior work suggests that RL primarily elicits existing knowledge from pretrained models, our results reveal a more nuanced pattern: domains frequently seen during pretraining (Math, Code, Science) easily benefit from cross-domain RL training, while domains with limited pretraining exposure (Logic, Simulation, and Tabular) require in-domain training to achieve meaningful performance gains, suggesting that RL is likely to facilitate genuine skill acquisition. Finally, we present Guru-7B and Guru-32B, two models that achieve state-of-the-art performance among open models RL-trained with publicly available data, outperforming best baselines by 7.9% and 6.7% on our 17-task evaluation suite across six reasoning domains. We also show that our models effectively improve the Pass@k performance of their base models, particularly on complex tasks less likely to appear in pretraining data. We release data, models, training and evaluation code to facilitate general-purpose reasoning at: https://github.com/LLM360/Reasoning360
1 Introduction
The paper broadens RL reasoning research beyond math and code with GURU, then shows that RL effects vary by domain and supports strong cross-domain reasoning performance.
- Most RL reasoning studies train and evaluate exclusively on math and code, limiting understanding of whether findings generalize across domains.
- GURU provides 92K curated examples across Math, Code, Science, Logic, Simulation, and Tabular, each paired with a verifiable reward signal.Its pipeline combines sourcing and synthesis, deduplication, domain-specific reward design, and heuristic or model-based filtering.
- RL effects are domain-dependent: Math, Code, and Science benefit from cross-domain training, whereas Logic, Simulation, and Tabular require in-domain training for meaningful gains.The pattern suggests easier elicitation in pretrained domains and greater likelihood of new skill learning in less familiar domains.
- GURU-7B and GURU-32B establish state-of-the-art results among open models trained with publicly available data across 17 tasks and six domains.They outperform the best baselines by 7.9% and 6.7%, respectively.
- The models also expand base-model Pass@k reasoning boundaries on complex tasks less likely to appear in pretraining data.The paper reports this effect on a synthetic Zebra Puzzle task for both model scales.
2 Data Construction
The data-construction process addresses narrow-domain coverage and quality problems by assembling six reasoning domains and applying deduplication, domain-specific verification, and multi-stage filtering.
- Existing RL reasoning datasets often focus narrowly on Math and Code, overfit domain-specific structures, and contain redundant, noisy, or poorly calibrated samples.
- GURU curates six domains—Math, Code, Science, Logic, Simulation, and Tabular—to provide diverse and verifiable reasoning data.The collection includes sourced, synthesized, and table-based tasks tailored to each domain.
- 27.2% of Math samples and 7.5% of Code samples are removed using conservative substring-based deduplication.The strategy targets repeated or near-duplicate questions arising from shared upstream sources.
- GURU uses rule-based matching, execution-based verification, and model-based verification to produce domain-appropriate rewards.These methods cover Math, Logic, Simulation, Tabular, Code, and Science, respectively, with binary correctness rewards across domains.
- Heuristic and difficulty-aware filtering removes truncation risks, invalid references, noisy or trivial examples, and samples with unstable reward signals.Difficulty is assessed using weak- and strong-model empirical pass rates over N = 16 runs per sample.
1. Overly easy samples: Pweak ≥15
The filtering criteria identify samples that provide little learning signal, appear noisy, or contradict expected weak-versus-strong model difficulty patterns.
- Samples consistently solved by the weak model are removed because they provide limited signal for further RL improvement.
- Samples consistently failed by the strong model are treated as potentially noisy, ambiguous, incorrectly labeled, or unsuitable for RL training.
- Samples with Pweak > Pstrong are flagged as anomalous because the weak model may match an incorrect memorized label.The authors suggest such cases can mislead RL training.
- The difficulty gap Pstrong − Pweak serves as a proxy for sample learnability, with larger gaps indicating greater potential for RL improvement.
- Math filtering removes samples with Pstrong − Pweak ≤ 6 16 when Pstrong ≥0.75, focusing RL on challenging problems where improvement remains achievable.
3 Analysis of Cross-Domain RL for Reasoning
The analysis evaluates how RL transfers across six reasoning domains and finds that transferability depends on domain identity, task difficulty, training mixture, and data difficulty.
- Experimental Setup: RL experiments compare single-domain and mixed-domain training across six reasoning domains using GURU-18K subsets and Qwen2.5 base models.The setup samples 3K examples per domain, forming an 18K mixed dataset.
- Differential Transferability: Math, Code, and Science gain substantially from cross-domain RL, whereas Logic, Simulation, and Tabular improve mainly through in-domain training.The authors associate this asymmetry with differences in pretraining exposure.
- Task Difficulty: Easier Math and Code tasks show positive cross-domain transfer, but harder tasks such as AIME24, LiveCodeBench, ARC-AGI, and CodeI/O show limited gains.The hardest tasks with the lowest baseline scores exhibit marginal to negligible improvement from non-native data.
- Mixed-Domain Training: Mixed-domain RL performs comparably to or better than single-domain training across individual benchmarks, with minimal apparent interference among the six domains.The authors note that interference may require further study as domain count and diversity increase.
- Reward and Response-Length Dynamics: Reward rises steeply within the first few hundred gradient steps during joint GURU-18K training across all domains.Response-length changes remain domain-dependent: some domains contract outputs, others become more verbose, and Simulation is largely unchanged.
- Training Data Difficulty: Harder Math training improves in-domain AMC and AIME24 performance by +6.3 and +5.9, but degrades easier cross-domain HumanEval and HiTab performance by -9.2 and -3.0.The results favor balanced difficulty distributions or explicit cross-domain data for transfer.
4 Main Experiment
The main experiments train GURU models directly from Qwen2.5 base models and evaluate them across 17 benchmarks, finding strong balanced performance and task-dependent changes in Pass@k.
- Training and Evaluation: GURU-7B and GURU-32B are trained on the complete GURU dataset with RL and evaluated alongside directly RL-trained open baselines from Qwen2.5.The evaluation suite contains 17 benchmarks across six domains plus unseen generalization tasks.
- Results: GURU-7B achieves 43.29% and GURU-32B achieves 54.24%, outperforming the best baselines by 9.0% and 6.7%, respectively.The models show superior and well-balanced performance across all six evaluated domains.
- Generalization: Domain-specific baselines perform poorly outside their specialties, while GURU remains competitive on unseen IFEval and LiveBench tasks.Math-focused and STEM-focused baselines particularly weaken on coding and logic tasks.
- Pass@k Analysis: Pass@k curves vary by task rather than following a uniform pattern: AIME curves cross at k = 64, while Zebra Puzzle shows expanded reasoning boundaries after RL.The AIME pattern aligns with prior evidence that math gains can exploit capabilities already present in the base model.
- Pass@k Analysis: Pass@k behavior also changes with model scale: RL consistently outperforms the base model on AIME at 32B, while Zebra Puzzle continues rising for GURU-32B.GURU-7B plateaus around k=32 on Zebra Puzzle, whereas GURU-32B keeps increasing across the sampled range.
- Pass@k Analysis: Higher sampling temperature or top-p broadens exploration and improves Pass@k at larger k after RL-induced entropy collapse.The authors recommend treating Pass@k as informative but not exhaustive because it is sensitive to decoding hyperparameters.
5 Related Work
Related work has advanced RL-based reasoning models while also questioning whether RL primarily elicits latent capabilities rather than acquiring new reasoning skills.
- RL for LLM Reasoning: Recent RL reasoning models have achieved strong results, with open efforts commonly using mathematical data to train specialized reasoning systems.Examples include Open-Reasoner-Zero, Skywork-OR1, DeepScaler, and SimpleRL.
- Understanding RL for Reasoning: A parallel research line studies RL mechanisms and argues that RL may primarily elicit abilities already latent in the base model.This work examines whether apparent reasoning improvements reflect capability acquisition or more effective sampling.
6 Conclusion
GURU is a curated 92K-example RL dataset spanning six reasoning domains. The authors find that RL’s effects on reasoning are highly domain-sensitive, including effective elicitation of pretrained knowledge in Math, Code, and Science.
- GURU contains 92K high-quality RL examples spanning Math, Code, Science, Logic, Simulation, and Tabular reasoning.The dataset is designed to address a gap in RL-based reasoning research.
- The empirical analysis finds that RL produces highly domain-sensitive effects on reasoning behaviors.
- RL can effectively elicit pretrained knowledge in well-covered domains such as Math, Code, and Science.
A Limitations
The reported improvements are promising, but the experiments are limited by short RL training and the absence of an initial supervised fine-tuning stage.
- Training lasted only three epochs for 7B models and two epochs for 32B models because of computing resource constraints.The authors state that longer RL training might produce more pronounced results.
- The approach starts directly from base models with RL, bypassing an initial SFT phase to avoid dependence on specific SFT data.The authors expect an SFT-then-RL pipeline could further improve performance.
B Training Data Curation Details
GURU’s training data are assembled through open-source sourcing, provenance tracing, selection, and reward verification. The curation process targets reliable, varied, and sufficiently challenging examples.
- The curation section documents sourcing and curation details for datasets used in GURU.
- All newly created and repurposed assets are intended to be fully open-source and publicly available.The authors frame this as supporting collaboration and open science.
- Table 4 lists public source links for the datasets used in each GURU domain.
- Math: The math subset prioritizes OR1, DAPO, and DeepScaler, while excluding lower-quality or more duplicate-heavy sources such as BigMath.The original sources are traced in the math-data provenance diagram.
- Math: Math data are curated to provide a robust and varied foundation for training and evaluating mathematical reasoning.
- Math: Math answers receive reward when normalized or numerically equivalent to the ground truth, or when Sympy2 simplifies their difference to zero.
B.2 Code
The code subset combines verified programming problems from online platforms and competitions with execution-based reward evaluation. The surrounding curation details also define structured logic, simulation, and tabular reasoning data and identify a Python-only scope.
- Code: The code subset draws from LeetCode, TACO-Verified, PrimeIntellect, and LiveCodeBench, using pre-filtered DeepCoder subsets for two sources.
- Code: Code tasks require Python program synthesis in either pure-function or stdin/stdout format, with execution against tests under a uniform 30-second limit.
- Code: The code-data provenance is traced with a Sankey diagram covering all code data included in GURU.
- Code: The code setup currently supports only Python, leaving Java, C++, Go, and other languages for future work.
- Logic: Logic tasks use structured constraints and exact parsed answers, including Zebra, ordering, and graph puzzles with configurable complexity.Zebra puzzles are generated from sampled ground-truth tables and backsolved constraints; the logic corpus includes difficulty and size filtering.
- Simulation: Simulation data repurpose Code I/O examples containing verified programs and input-output or output-input pairs generated through actual function execution.
C Evaluation Benchmark Details
The evaluation suite combines offline and online benchmarks across all six reasoning domains, with selected subsets used in some online evaluations for efficiency or capability focus.
- Online evaluation samples up to 200 examples per dataset, except ARC-AGI, which uses examples with prompts shorter than 4,096 tokens.The full offline ARC-AGI set contains 400 examples, creating a distributional discrepancy that partly explains offline–online performance differences.
- Large datasets containing more than 1,000 samples are subsampled to 1,000 examples for efficient evaluation.
- Evaluation benchmarks are reported across all six reasoning domains in separate offline and online tables.
D.1 Cross-Domain Transferability
Cross-domain RL transfer in Qwen-32B largely mirrors the 7B pattern: pretraining coverage and task difficulty shape transfer, while weaker-prior domains remain harder to improve out of domain.
- Cross-domain transferability in Qwen-32B closely mirrors the 7B variant, reaffirming the roles of pretraining priors, task difficulty, and training composition.
- Math, Code, and Science continue to gain substantially from cross-domain RL, while weaker-pretraining domains remain difficult to improve through out-of-domain supervision alone.Tabular shows modest additional transfer in the 32B setting, likely reflecting greater capacity and stronger instruction following.
- Easier tasks such as MATH500, AMC, HumanEval, and MBPP consistently benefit from cross-domain training, unlike more challenging benchmarks.
- The analysis uses source domains as rows and target domains as columns, with in-domain results outlined and each cell showing normalized validation-accuracy gains.
D.2 Pass@k Dynamics during Training
GURU-7B Pass@k follows a two-phase training dynamic: early optimization raises Pass@1 while sharply reducing the maximum-k score, followed by slower joint improvement after 160 steps.
- Figure 9 plots Pass@k curves across training steps for AIME24, ARC-AGI, and Code I/O.
- Before 160 steps, Pass@1 increases while Pass@max_k decreases sharply as training approaches the end of the first epoch.Here, max_k denotes the largest evaluated k.
- After 160 steps, both Pass@1 and Pass@max_k increase steadily but slowly, producing a horizontal shift in the Pass@k curve.
- Generation entropy drops rapidly before 160 steps and then plateaus, aligning with the two phases observed in Pass@k behavior.The authors suggest entropy dynamics may help interpret Pass@k and that entropy control could shape learning dynamics.