Source-linked AI summary
JudgeRLVR: Judge First, Generate Second for Efficient Reasoning
Jiangshan Duo, Hanyu Li, Hailin Zhang, Yudong Wang, Sujian Li, Liang Zhao
TL;DR
RLVR’s focus on final-answer correctness can yield verbose, trial-and-error reasoning, while existing length controls risk accuracy losses. JudgeRLVR first trains the model to judge solution validity, then initializes generating RLVR from that judge, achieving a better quality–efficiency trade-off across in-domain and out-of-domain evaluations.
Problem
Final-answer-focused RLVR often produces verbose backtracking, while length penalties can reduce verbosity at the cost of essential reasoning or accuracy.
Method
JudgeRLVR sequentially trains a model to classify candidate solutions as correct or incorrect, then applies Vanilla RLVR generation initialized from the judge.
Results
JudgeRLVR improves the quality–efficiency trade-off, with about +3.7 points average in-domain math accuracy, -42% average generation length, and about +4.5 points out-of-domain accuracy.
Takeaways & Limitations
Learning to discriminate valid reasoning before generation is associated with more direct, reliable reasoning and reduced explicit backtracking without explicit length penalties.
Takeaways & Limitations
On IFEval and ZebraLogic, JudgeRLVR yields higher accuracy but longer outputs when explicit checks are needed for format compliance or rule verification.
Abstract
from arXiv · showhide
Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for reasoning in Large Language Models. However, optimizing solely for final-answer correctness often drives models into aimless, verbose exploration, where they rely on exhaustive trial-and-error tactics rather than structured planning to reach solutions. While heuristic constraints like length penalties can reduce verbosity, they often truncate essential reasoning steps, creating a difficult trade-off between efficiency and verification. In this paper, we argue that discriminative capability is a prerequisite for efficient generation: by learning to distinguish valid solutions, a model can internalize a guidance signal that prunes the search space. We propose JudgeRLVR, a two-stage judge-then-generate paradigm. In the first stage, we train the model to judge solution responses with verifiable answers. In the second stage, we fine-tune the same model with vanilla generating RLVR initialized from the judge. Compared to Vanilla RLVR using the same math-domain training data, JudgeRLVR achieves a better quality--efficiency trade-off for Qwen3-30B-A3B: on in-domain math, it delivers about +3.7 points average accuracy gain with -42\% average generation length; on out-of-domain benchmarks, it delivers about +4.5 points average accuracy improvement, demonstrating enhanced generalization.
1 Introduction
RLVR can improve correctness but often produces verbose, backtracking-heavy reasoning, motivating JudgeRLVR’s two-stage strategy of learning to judge solutions before generating them. The approach targets more direct reasoning without explicit length or quality penalties and reports improved quality–efficiency trade-offs.
- Motivation: RLVR’s final-answer optimization often produces long, low-information reasoning with frequent backtracking, while length penalties can reduce accuracy.Examples include explicit self-correction and exploratory phrases such as “but” and “let’s try again.”
- Method: JudgeRLVR proposes training discriminative judgment before generation so the model can identify and prune unpromising reasoning paths early.The paradigm is motivated by expert reasoning that filters low-value paths before expanding them.
- Method: The judging stage classifies candidate solution responses as correct or incorrect, after which Vanilla RLVR generation is initialized from the judge-trained model.The two stages are shown as a sequential pipeline rather than an interleaved objective.
- Design rationale: JudgeRLVR imposes no explicit CoT length or quality penalties, attributing reduced verbosity and refined reasoning to discriminative priors transferred from judging.This design separates the proposed discriminative training signal from heuristic length control.
- Results: +3.7 points average accuracy and -42% average generation length are reported on in-domain math, while out-of-domain benchmarks gain about +4.5 points average accuracy.These results are compared with Vanilla RLVR trained on the same data and support a better quality–efficiency trade-off.
- Results: JudgeRLVR is presented as improving the quality–efficiency trade-off and producing higher-quality thinking patterns through judge-then-generate training.The contributions also include interpretable linguistic evidence consistent with reduced explicit backtracking.
2 Related Work
Prior work establishes RLVR as an effective but imperfect approach for reasoning: it can encourage verbose or suboptimal traces and leaves quality–efficiency trade-offs underexplored. Related research uses discriminative supervision and evaluators to steer outputs, but rarely adopts the proposed two-stage transfer of judging capability into generation.
- RLVR limitations: RLVR often outperforms supervised fine-tuning on math and logic tasks, but its gains can occur under spurious rewards and mainly reweight existing high-reward paths.These limitations motivate attention to reasoning-process quality beyond final correctness.
- Mathematical reasoning: Mathematical reasoning studies report that extended RLVR and synthetic-data scaling improve capability while leaving verbose, redundant, and low-information reasoning traces.The broader challenge is improving correctness and efficiency together.
- Discriminative supervision: Discriminative fine-tuning, stronger evaluators, verifiers, and task-specific validation metrics can help separate or assess good and bad generations.These approaches provide precedents for using evaluation signals to steer reasoning quality.
3 Method
JudgeRLVR trains one reasoning policy in two sequential stages: first to recognize correct solutions, then to generate solutions using Vanilla RLVR initialized from the judge. The design aims to transfer discriminative error awareness into generation so low-quality branches are down-weighted early without explicit length penalties.
- Two-stage paradigm: JudgeRLVR first trains discriminative error awareness, then optimizes solution generation from the judging-stage weights.The two stages have different roles but are intended to transfer capability within the same policy.
- Mechanism hypothesis: The proposed mechanism is that internalized validity judgments prune erroneous branches before extensive textual search, reducing explicit backtracking without imposed CoT penalties.The paper links style transfer during judging with reduced backtracking during generation, but presents this as a hypothesis tested in later experiments.
- Judging stage: In the judging stage, the policy receives a problem and candidate solution, produces commentary, and emits a binary verdict marking correctness.The verdict is 0 for incorrect and 1 for correct, while commentary is optimized by the same policy-gradient signal.
- Judging stage: Judge data pairs problems, candidate responses, and verifiable correctness labels obtained by comparing parsed answers with gold answers.Candidate responses are sampled from several models, with hard negatives and balanced positive and negative examples used in construction.
- Generating stage: In the generating stage, the judge-initialized policy produces CoT trajectories and solution responses under Vanilla RLVR with a sparse final-answer correctness reward.The generation stage retains the final-answer reward rather than introducing a separate process reward.
4 Experimental Setup
The experiments compare Base SFT, Vanilla RLVR, and sequential JudgeRLVR under matched training conditions across in-domain mathematics and diverse out-of-domain benchmarks. They measure both accuracy and generation length, while ablations and linguistic analyses examine whether stage ordering and reduced backtracking explain the quality–efficiency trade-off.
- Evaluation design: JudgeRLVR is evaluated against Base SFT and Vanilla RLVR on five in-domain math tasks and diverse out-of-domain reasoning, coding, knowledge, and instruction-following benchmarks.The setup is designed to test quality–efficiency improvements beyond Vanilla RLVR across domains.
- Evaluation design: Vanilla RLVR and JudgeRLVR use the same training and evaluation hyperparameters and the same total training steps.This controls training budget and configuration when comparing the staged paradigm with final-answer-only RLVR.
- Evaluation design: Accuracy and average generation length are reported together to assess correctness alongside unnecessary computation and reasoning information density.Table 1 defines accuracy as Acc and generation length as average tokens, with JudgeRLVR deltas measured against Vanilla RLVR.
- Ablations: Ablations compare Judge Only with full JudgeRLVR to test whether judging alone improves generation or whether generating optimization is necessary.The Judge Only setting directly probes whether judging is a prerequisite but not a substitute for generation training.
- Ablations: The Mixed Strategy interleaves judge and generative updates before generation training to test whether role ordering and separation matter.This probes whether simultaneous optimization interferes with consolidating a clean internal decision policy.
- Mechanism verification: Mechanism analyses use Base SFT perplexity to track linguistic-style shifts and transition-word statistics to examine explicit backtracking and reflection.The transition markers include contrastive and self-correction words such as “but,” “however,” and “let’s try again.”
5 Results
JudgeRLVR generally improves or preserves accuracy while shortening generations, with benefits across in-domain and out-of-domain tasks. Ablations and linguistic analyses support sequential staging as a mechanism for reducing unproductive backtracking without explicit length penalties.
- Main results: JudgeRLVR achieves higher or comparable accuracy while producing substantially shorter generations than Vanilla RLVR on most benchmarks.Table 1 reports accuracy and average generation length; Table 2 defines Acc, Len, and comparison deltas for ablations.
- In-domain math: On in-domain math, JudgeRLVR improves accuracy over Vanilla RLVR while markedly reducing generation length on AIME24/25, HMMT_feb_2025, and BeyondAIME.The larger gains on HMMT and BeyondAIME are associated with problems requiring longer reasoning chains and stronger strategy selection.
- In-domain math: On MATH500, JudgeRLVR is slightly less accurate than Vanilla RLVR but drastically reduces generation length.The passage attributes the limited accuracy headroom to dataset saturation while noting that reasoning cost can still fall substantially.
- Out-of-domain: On GPQA Diamond, LiveCodeBenchv6, and MMLU-Redux, JudgeRLVR improves accuracy while reducing length, indicating transfer beyond math.The reported transfer spans science reasoning, coding, and broad knowledge tasks.
- Out-of-domain: On IFEval and ZebraLogic, JudgeRLVR produces higher accuracy but longer outputs, showing a task-dependent quality–efficiency trade-off.The authors frame extra explicit checking as potentially necessary for format compliance, constraint satisfaction, or discrete rule verification.
- Ablations: Judge Only and Mixed Strategy underperform the sequential pipeline’s stability: judging alone can lengthen and hurt generation, while interleaving objectives causes instability.Judge Only does not reliably convert discrimination into concise generation, and Mixed Strategy shows task-specific drops and often longer outputs.
- Mechanism verification: JudgeRLVR judging shifts output style and its generating stage reduces explicit backtracking markers over training.Base SFT PPL increases during judging, while transition/backtracking counts and frequencies decrease during generation.
6 Conclusion
The paper presents JudgeRLVR as a precursor to efficient reasoning by training validity discrimination before generation. Its experiments report improved in-domain math accuracy, approximately 42% shorter generations, and generalization to out-of-domain tasks.
- Conclusion: JudgeRLVR is a two-stage paradigm that trains solution-validity discrimination before applying generation optimization.The conclusion describes the approach as acting as a precursor to efficient reasoning.
- Conclusion: The paper argues that generation can improve when the model internalizes what good reasoning looks like through judging.The stated interpretation is that later generation can proceed with fewer false starts after judge training.
A Prompt Templates of Judge and Generate
The prompt templates separate judging candidate answers from generating solutions. The judge analyzes correctness and emits a binary boxed verdict, while the generator produces a step-by-step solution with a boxed final answer.
- Judge prompt: The judge prompt supplies a question and a proposed solution response for correctness analysis.Its inputs are represented as {question} and {solution response}.
- Judge prompt: The judge must output only a boxed binary judgment: 1 for correct or 0 for incorrect.The prompt requests concise analysis followed by the final judgment in boxed form.
- Generate prompt: The generating prompt asks the model to think step by step and place the final answer within a box.This template is labeled “Generating and Rollout Prompt.”
B Case Study
The case study converts the rectangular point (0, 3) into polar coordinates by calculating its radius and identifying its angle on the positive y-axis. The resulting coordinate pair is (3, π/2), satisfying the specified constraints and reproducing the original point.
- The task is to convert (0, 3) from rectangular coordinates to polar coordinates (r, θ), with r > 0 and 0 ≤ θ < 2π.
- The radius is 3, obtained from the distance formula using x = 0 and y = 3.
- The polar coordinate (3, π/2) converts back to x = 0 and y = 3, matching the original point.
- The final response should be written as a coordinate pair and boxed as requested.
- Because x = 0 and y = 3 > 0, the point lies on the positive y-axis, so θ = π/2.
- The result satisfies r = 3 > 0 and θ = π/2 within [0, 2π), while negative-radius and alternate-angle representations are excluded by the stated conditions.
C Evaluate Details
The evaluation section introduces Table 3 as the setup for the evaluation benchmarks.
- Table 3 presents the metric setup of the evaluation benchmarks.