Source-linked AI summary
General-Reasoner: Advancing LLM Reasoning Across All Domains
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, Wenhu Chen
TL;DR
Existing RL reasoning methods focus mainly on mathematics and coding because those domains provide abundant data and easy rule-based verification, leaving broader-domain reasoning underserved. General-Reasoner combines a diverse verifiable-question dataset with a generative model-based verifier for Zero-RL training, and evaluations show robust cross-domain performance while retaining strong mathematical reasoning. The method’s demonstrated scope is primarily Zero RL, while combining its components with supervised fine-tuning or distillation remains future work.
Problem
Existing RL reasoning methods mainly target mathematics and coding because broader domains have scarcer data and more diverse answer representations that are harder to verify.
Method
General-Reasoner trains base LLMs with GRPO in the Zero-RL setting using WebInstruct-verified data and a generative model-based answer verifier.
Results
Across 12 benchmarks, General-Reasoner generally improves broad reasoning performance, typically by approximately 10% on MMLU-Pro and SuperGPQA, while remaining strong on mathematics.
Takeaways & Limitations
General-Reasoner provides a broad-domain Zero-RL approach that achieves competitive reasoning performance while preserving mathematical effectiveness.
Takeaways & Limitations
The study focuses on Zero RL; the effects of combining its verifier and data with supervised fine-tuning or distillation are left for future work.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has recently demonstrated strong potential in enhancing the reasoning capabilities of large language models (LLMs). Particularly, the "Zero" reinforcement learning introduced by Deepseek-R1-Zero, enables direct RL training of base LLMs without relying on an intermediate supervised fine-tuning stage. Despite these advancements, current works for LLM reasoning mainly focus on mathematical and coding domains, largely due to data abundance and the ease of answer verification. This limits the applicability and generalization of such models to broader domains, where questions often have diverse answer representations, and data is more scarce. In this paper, we propose General-Reasoner, a novel training paradigm designed to enhance LLM reasoning capabilities across diverse domains. Our key contributions include: (1) constructing a large-scale, high-quality dataset of questions with verifiable answers curated by web crawling, covering a wide range of disciplines; and (2) developing a generative model-based answer verifier, which replaces traditional rule-based verification with the capability of chain-of-thought and context-awareness. We train a series of models and evaluate them on a wide range of datasets covering wide domains like physics, chemistry, finance, electronics etc. Our comprehensive evaluation across these 12 benchmarks (e.g. MMLU-Pro, GPQA, SuperGPQA, TheoremQA, BBEH and MATH AMC) demonstrates that General-Reasoner outperforms existing baseline methods, achieving robust and generalizable reasoning performance while maintaining superior effectiveness in mathematical reasoning tasks.
1 Introduction
Existing RL reasoning work is concentrated in mathematics and coding because those domains offer abundant data and reliable rule-based answer verification. General-Reasoner addresses this gap with diverse verifiable data and a model-based verifier, then evaluates Zero-RL models broadly.
- Motivation: Mathematics and coding dominate current RL reasoning because their data is abundant and answers are easy to verify with rule-based methods.This narrow focus limits generalizability to domains with diverse answer representations.
- Approach: General-Reasoner targets broader-domain reasoning while preserving mathematical reasoning strengths.The proposed paradigm is designed for diverse, non-mathematical domains.
- Approach: WebInstruct-verified provides approximately 230K high-quality, verifiable reasoning questions spanning disciplines including physics, chemistry, social sciences, and finance.Questions are crawled, filtered, and selected for reliable answer verification.
- Approach: General-Verifier replaces rule-based verification with a compact generative model-based verifier for diverse answer types and reasoning domains.The contribution is framed as an answer-verification component for chain-of-thought-based training.
- Evaluation: Across 12 challenging benchmarks, General-Reasoner typically improves general benchmarks by approximately 10% and can slightly outperform math-focused RL frameworks on mathematics.The best General-Reasoner-Qw3-14B model matches or beats GPT-4o on various benchmarks.
2 Related Works
Reasoning-focused LLM research combines chain-of-thought and test-time scaling with supervised fine-tuning and reinforcement learning. Zero RL offers a direct alternative, but existing applications have largely remained mathematical.
- Reasoning Methods: Chain-of-thought reasoning improves LLM performance by explicitly decoding thought processes during inference.Later work studies scaling the reasoning process at test time.
- Reasoning Models: Commercial and open-source reasoning models have achieved competitive results on reasoning-intensive evaluations, narrowing their performance gap.Examples include OpenAI’s O1 series, Qwen, QWQ, and Deepseek R1.
- Training Paradigms: Typical reasoning pipelines use supervised fine-tuning on chain-of-thought data before reinforcement learning, whereas Zero RL applies reinforcement learning directly to a strong base model.The direct approach avoids an initial supervised fine-tuning stage.
- Training Paradigms: Zero RL can collect verifiable question–answer pairs without complete reasoning-chain targets, but existing work has mostly focused on mathematical reasoning.This reduces the complexity of training-data collection relative to full reasoning supervision.
3 General Reasoner
General-Reasoner builds a diverse, verifiable reasoning dataset through web crawling, extraction, annotation, and quality filtering. It pairs this resource with a compact generative verifier that assesses answer equivalence using chain-of-thought and supports reinforcement-learning training across domains.
- Diverse Verifiable Reasoning Tasks: The dataset-building pipeline mines web-based questions, extracts verifiable answer pairs, annotates them, and filters noisy or unsuitable examples.Questions are traced to source pages, screened for human-written answers, annotated with answer type, subject, and difficulty, and tested with eight candidate solutions.
- Diverse Verifiable Reasoning Tasks: Approximately 230k curated reasoning questions span multiple answer formats and disciplines, including mathematics, physics, chemistry, finance, humanities, and social sciences.The answer formats include multiple-choice, numerical expressions, and matrices, while the domain distribution is balanced across fields.
- Model-Based Verifier for GRPO: The compact verifier replaces rigid rule-based checking, which struggles with semantically equivalent answers, varied representations, and cross-domain scalability.The verifier is trained from generated candidate answers and verification annotations, and empirical analysis reports high agreement with Gemini-2.0-Flash while outperforming traditional rule-based approaches.
- Model-Based Verifier for GRPO: The model-based verifier evaluates whether an extracted student answer is equivalent to a ground-truth reference in the question context.It generates a chain-of-thought reasoning process followed by a binary true/false equivalence prediction.
4 Experiments
The experiments evaluate General-Reasoner using Zero RL with Qwen2.5 and Qwen3 base models across general and mathematical reasoning benchmarks. Results show consistent gains over base, supervised fine-tuned, and other RL baselines, alongside shorter responses and faster inference in one comparison.
- Training Setup: General-Reasoner applies GRPO directly to Qwen2.5 and Qwen3 base models without intermediate supervised fine-tuning.The models use general Qwen base models rather than Qwen-math models to maximize general performance.
- Evaluation: The evaluation covers 12 challenging benchmarks spanning general reasoning, multiple disciplines, and mathematical tasks.Examples include MMLU-Pro, SuperGPQA, BBEH, GPQA, MATH-500, GSM8K, and Olympiad.
- Main Results: General-Reasoner with Zero RL consistently outperforms base and supervised fine-tuned models across Qwen2.5 and Qwen3 backbones.For Qwen2.5-7B-Base, it reaches 58.9% on MMLU-Pro versus 47.7% for the base model and 57.0% for the instructed model.
- Main Results: General-Reasoner outperforms SimpleRL and Nemotron-CrossThink across MMLU-Pro, GPQA, SuperGPQA, and BBEH, with the trend holding for 14B models.General-Reasoner-Qw3-14B reaches 56.1% on GPQA and 54.4% on TheoremQA, matching or exceeding the cited commercial-model comparisons.
- Efficiency: On MMLU-Pro computer science, General-Reasoner-4B reaches 61% accuracy in 1.5 minutes, compared with DeepScaleR-1.5B-Preview’s 35% in 18 minutes.The paper also reports average response lengths around 1,000 tokens, versus around 32k tokens for DeepScaleR.
5 Analysis and Ablation Study
The ablations examine training-data diversity and verifier choice. Diverse-domain data improves general reasoning while preserving mathematical performance, and the model-based verifier yields stronger learning and substantially higher agreement with Gemini-2.0-Flash than the rule-based verifier.
- Impact of Data Abundance: Math-only training gives the 7B model a one-point gain on Math-Related, but lowers MMLU-Pro, GPQA, and SuperGPQA by roughly two points each.The full diverse-domain dataset therefore provides stronger overall benchmark performance for the 7B comparison.
- Impact of Data Abundance: Training on diverse reasoning domains enhances general reasoning while maintaining or improving mathematical reasoning.This conclusion is supported by comparisons between full-data and math-only Zero RL variants on 7B and 14B backbones.
- Verifier Ablation: With identical Qwen3-4B-Base training conditions, the model-based verifier achieves higher accuracy across all evaluated benchmarks than the rule-based verifier.Both verifier variants are trained for 120 steps.
- Verifier Ablation: By step 120 on MMLU-Pro, the model-based verifier reaches about 60% accuracy, while the rule-based verifier plateaus near 58% around step 60.The model-based verifier continues improving after the rule-based verifier reaches its early plateau.
- Verifier Analysis: Among 50k answer-verification pairs judged correct by Gemini-2.0-Flash, agreement is 78.7% for the model-based verifier versus 22.2% for the rule-based verifier.The analysis attributes many rule-based false negatives to unstructured answers and textual variations in multiple-choice responses, especially in non-math STEM fields.
6 Conclusion
The paper introduces a Zero-RL paradigm that combines diverse, verifiable reasoning data with a generative model-based verifier. It reports strong cross-domain generalization while preserving mathematical reasoning effectiveness and competitive performance against models using supervised fine-tuning.
- Conclusion: General-Reasoner applies GRPO directly to base LLMs using a curated verifiable dataset and a generative model-based verifier.The paradigm is designed to enhance general reasoning across diverse domains.
- Conclusion: The approach demonstrates competitive reasoning performance against models requiring an additional supervised fine-tuning stage.The conclusion reports strong generalization across challenging domains while preserving mathematical effectiveness.
A.1 Limitation
The work is limited to Zero RL training directly from base language models, leaving combinations with supervised fine-tuning or distillation for future study. It also targets general language reasoning rather than code reasoning or olympiad-level mathematics.
- The study focuses on Zero RL, training models directly from base language models without intermediate supervised fine-tuning.
- The proposed data and verifier integration is not tied to Zero RL, but its combination with supervised fine-tuning or distillation remains unexplored.
- The study targets general language reasoning across diverse domains rather than specifically focusing on code reasoning or olympiad-level mathematics.
A.2 Broader Impact
The work aims to improve LLM general reasoning so models are more applicable to real-world, multi-domain reasoning scenarios. It connects this broader applicability to decision-making, analysis, and problem-solving across diverse fields.
- The approach aims to make LLMs more applicable to real-world scenarios requiring complex, multi-domain reasoning.
- By extending reasoning beyond math and code, the approach is intended to support decision-making, analysis, and problem-solving across diverse fields.
A.3 Detailed Results
The detailed-results section reports evaluation conventions and organizes accuracy comparisons by domain or subtask across MMLU-Pro, SuperGPQA, and BBEH. The referenced tables provide per-domain or per-subtask comparisons among models.
- MMLU-Pro results use micro-average accuracy, while SuperGPQA and BBEH results use macro-average accuracy across subtasks or domains.
- Table 6 compares different models by per-domain accuracy on MMLU-Pro.
- Table 7 compares different models by per-domain accuracy on SuperGPQA.
- Table 8 compares different models by per-subtask accuracy on BBEH.
A.4 Verifier Input-Output Example
The verifier example shows a question-and-answer verification workflow for a Bernoulli equation. The verifier compares the student answer with the ground truth and returns a binary final decision.
- The verifier receives a request to solve a Bernoulli equation and assess whether the student answer matches the ground truth.
- The verifier is instructed to output “Final Decision: Yes” for a correct student answer and “Final Decision: No” otherwise.
- In the example, the verifier rewrites and compares the ground truth and student answers before deciding they are equivalent.
- The appendix also provides hyperparameter settings for General-Reasoner variants in Table 9.