Source-linked AI summary
Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, Dong Yu
TL;DR
RLVR has mainly succeeded on structured math and coding tasks, leaving its effectiveness on broader free-form domains uncertain. This paper tests cross-domain reference-based verification, introduces generative soft rewards and compact reward models, and reports improved performance across free-form tasks. The approach also examines how conservative soft scoring behaves under ambiguous evaluation.
Problem
RLVR has primarily been demonstrated on structured mathematics and coding tasks, while its extension to diverse domains with free-form reference answers remains underexplored.
Method
The paper uses generative reference-based verifiers to provide binary or probability-based soft rewards and trains compact cross-domain reward models from exploration data.
Results
31.2% accuracy versus 27.3% is reported for soft versus binary rewards with REINFORCE and RM-7B on multi-subject tasks, while the framework improves performance across diverse free-form domains.
Takeaways & Limitations
Generative model-based rewards provide a practical route for extending RLVR to free-form, multi-domain reasoning without extensive domain-specific annotation.
Takeaways & Limitations
The work does not use chain-of-thought rationales for verification and does not consider format-based rewards.
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) has demonstrated significant success in enhancing mathematical reasoning and coding performance of large language models (LLMs), especially when structured reference answers are accessible for verification. However, its extension to broader, less structured domains remains unexplored. In this work, we investigate the effectiveness and scalability of RLVR across diverse real-world domains including medicine, chemistry, psychology, economics, and education, where structured reference answers are typically unavailable. We reveal that binary verification judgments on broad-domain tasks exhibit high consistency across various LLMs provided expert-written reference answers exist. Motivated by this finding, we utilize a generative scoring technique that yields soft, model-based reward signals to overcome limitations posed by binary verifications, especially in free-form, unstructured answer scenarios. We further demonstrate the feasibility of training cross-domain generative reward models using relatively small (7B) LLMs without the need for extensive domain-specific annotation. Through comprehensive experiments, our RLVR framework establishes clear performance gains, significantly outperforming state-of-the-art open-source aligned models such as Qwen2.5-72B and DeepSeek-R1-Distill-Qwen-32B across domains in free-form settings. Our approach notably enhances the robustness, flexibility, and scalability of RLVR, representing a substantial step towards practical reinforcement learning applications in complex, noisy-label scenarios.
1 Instruction
The paper extends RLVR beyond structured mathematics and coding to diverse domains with free-form answers. It combines cross-model binary verification consistency with generative soft rewards and compact cross-domain reward models.
- RLVR is extended to medicine, chemistry, psychology, economics, and education, beyond traditional structured-answer settings.
- Binary judgments show high agreement across closed- and open-source LLMs when expert-written reference answers are available.
- 60.3% of mathematical problems and 45.4% of complex multi-domain queries have single-term numerical answers verifiable by rule-based methods.
- The framework introduces generative model-based soft rewards to provide richer verification signals for unstructured tasks.
- Performance improves by up to 8.0% accuracy, surpassing Qwen2.5-72B-Instruct and DeepSeek-R1-Distill-Qwen-32B on diverse free-form reasoning tasks.
- 7B-scale cross-domain reward verifiers can be trained without extensive domain-specific annotation.
2 Related Work
Prior RLVR work concentrates on narrow tasks with short, structured references and rule-based verification. This paper studies reference-based RLVR for diverse domains with free-form answers and generative verification.
- Reference-based correctness labels are commonly binary, although some studies use values in [0, 1] to represent graded correctness.
- Existing reference-based reward models can be restricted to one domain and require large-scale data, including 800k math instances.
- Previous RLVR studies primarily target mathematics, code generation, and logic puzzles with well-structured reference answers.
- Math benchmarks such as GSM8K and MATH typically use reference answers shorter than two words, enabling straightforward rule-based verification.
- The paper explores generative reference-based verifiers across diverse domains and trains them without annotated or synthetic step-by-step rationales.
3 Method
The method uses expert references and generative verifiers to assign binary or probability-based rewards, then normalizes rewards for policy-gradient training. Reward models are fine-tuned from exploration data produced by the evolving actor.
- Each prompt x is paired with an expert-written reference answer a for reference-based verification.
- 3.1 Reward Estimation: A generative verifier is instructed to output 0 or 1, producing a binary model-based reward from its final-step judgment.
- 3.1 Reward Estimation: The verifier also supplies a soft reward from the probability of the judgment token, 0 or 1.
- 3.1 Reward Estimation: The resulting reward remains bounded within [0, 1], matching the widely used binary reward scale.
- 3.2 Reward Normalization: Rewards are z-score normalized within each batch to stabilize gradients and encourage improvement among above-average samples.
- 3.2 Reward Normalization: When batch reward variance is zero, all normalized rewards are set to zero because samples are too difficult or too easy for the current policy.
- A 72B reward model reaches 62.7% on math data versus 58.8% for a 7B model, motivating reward-model training.
- During RL, exploration triples are labeled by a fixed LLM and used to fine-tune reward models with supervised learning.
4 Experiments
Experiments evaluate RLVR on free-form mathematics and multi-subject data using binary and soft rewards, comparing rule-based, model-based, and supervised approaches. Results favor model-based rewards, especially the distilled 7B verifier, with stronger performance, scaling, and robustness across settings.
- Data: 773k Chinese QA pairs provide free-form mathematics references spanning elementary, middle, and high school levels.The evaluation reserves 3,000 QA pairs from each educational level, with reference answers averaging 33.7, 36.3, and 53.9 words.
- Data: 638k college-level ExamQA instances cover at least 48 subjects, while 6,000 randomly sampled questions form the test set.GPT-4o-mini assigns subjects; 15.8% of test instances are unclassified.
- Evaluation: 0.81 ≤κ ≤1.00 measures near-perfect agreement between Qwen-based majority voting and single GPT-4o judgments across evaluations.κ exceeds 0.86 for mathematics and 0.88 for multi-subject college problems; subsequent evaluations use m = 1.
- Main Results: SFT reaches 45.7% on mathematics, below rule-based RLOO at 58.8% and RM-7B at 63.0%.The results compare performance with no high-quality Chain-of-Thought training data.
- Main Results: 63.0% accuracy for RM-7B with RLOO exceeds 58.5% for rule-based reward and 61.6% for Qwen2.5-72B-Instruct binary reward on average.RM-7B is trained as a reward model on distilled data and uses model-based judgments.
- Main Results: 31.2% versus 30.3% accuracy shows RM-7B outperforming the 72B reward model on multi-subject REINFORCE evaluations.The comparison uses model-based rewards and highlights competitive performance despite the parameter disparity.
- Main Results: 31.2% versus 27.3% shows soft rewards outperforming binary rewards for RM-7B on a multi-subject REINFORCE setting.Soft scoring is more conservative in ambiguous cases, where reference-label diversity increases judgment complexity.
- Scaling Experiments: Learned rewards improve consistently as training scales, whereas rule-based rewards fluctuate and eventually degrade.The scaling experiments use progressively larger datasets on mathematical and multi-subject tasks.
5 Discussions and Conclusions
The paper simplifies verification by using generative reward models that output binary judgments without chain-of-thought, while excluding format-based rewards and constraints.
- The generative reward model outputs 1 or 0 without requiring chain-of-thought reasoning.The paper leaves the necessity of detailed rationales for semantic-equivalence assessment as an open question.
- The framework does not consider format-based rewards, avoiding format constraints on reference answers and model responses.This reduces reliance on human effort for data standardization and pattern design.
A.1 Template
The appendix identifies templates for grading and classifies subjects into four broad disciplinary groups.
- Table 6 classifies subjects into STEM, Social Sciences, Humanities, and Applied Sciences.
- Table 4 provides the template for the grading task.
A.2 Agreement
GPT-4o supplies a single decision, whereas majority voting offers a more stable and deterministic grading outcome by reducing randomness.
- Majority voting provides a more stable and deterministic grading outcome than a single sampled decision when m > 1.The comparison concerns GPT-4o's single decision and the reward model's majority vote.
A.4 Hyper parameters
The appendix lists templates, subject-classification materials, agreement tables, and training hyperparameters used in the experiments.
- Table 5 provides the classification-task template with subject names and IDs.
- Table 6 groups subjects into STEM, Social Sciences, Humanities, and Applied Sciences.
- Agreement: Table 7 reports Cohen’s Kappa agreement between GPT-4o and majority voting across education levels of math problems.
- Agreement: Figure 3 identifies Cohen’s Kappa as the measure of agreement between GPT-4o and majority voting with m graders.
- Agreement: Table 8 reports Cohen’s Kappa agreement between GPT-4o and majority voting across college-level multi-subject problems.
- Hyper parameters: Table 9 lists the training hyperparameters, with other hyperparameters using OpenRLHF defaults.
- Table 10 provides example question and reference-answer pairs from ExamQA.