Source-linked AI summary
JudgeLRM: Large Reasoning Models as a Judge
Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, Bingsheng He
TL;DR
LLM judges offer scalable evaluation, but SFT struggles as judgment requires deeper reasoning. JudgeLRM addresses this with RL and judge-wise, outcome-driven rewards, outperforming SFT baselines and leading reasoning models while improving reasoning-heavy evaluation.
Problem
SFT-trained judges have limited generalization and reasoning depth, especially for evaluation tasks with complex reasoning demands.
Method
JudgeLRM trains judgment-oriented LLMs with RL using judge-wise, outcome-driven rewards that combine structural and content-based signals.
Results
JudgeLRM consistently outperforms comparable SFT and RL baselines, with an average 8.14% F1 improvement over SFT counterparts and gains against GPT-4 and DeepSeek-R1.
Takeaways & Limitations
Judgment is inherently reasoning-intensive, and RL-based outcome rewards support structured reasoning and flexible generalization in judge models.
Takeaways & Limitations
The experiments focus mainly on pair-wise open-ended QA evaluation rather than closed-domain or expert-level tasks.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are increasingly adopted as evaluators, offering a scalable alternative to human annotation. However, existing supervised fine-tuning (SFT) approaches often fall short in domains that demand complex reasoning. Judgment is inherently reasoning-intensive: beyond surface-level scoring, it requires verifying evidence, identifying errors, and justifying decisions. Through the analysis of evaluation tasks, we find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios. To address this, we introduce JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models: notably, JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score, with particularly strong gains on reasoning-heavy tasks. Our findings underscore the value of RL in unlocking reasoning-aligned LLM judges.
1 INTRODUCTION
LLM judges offer scalable evaluation, but SFT gains decline as reasoning demands increase. JudgeLRM addresses this limitation with RL and judge-wise, outcome-driven rewards.
- LLMs provide a scalable alternative to costly human annotation, while prior judges such as JudgeLM and PandaLM rely on large-scale SFT.
- SFT performance improvements show an inverse relationship with the proportion of reasoning-required tasks across five question-source categories.Figure 1 reports a negative trend: y = -0.41x + 16.72, R² = 0.53.
- JudgeLRM trains LLM judges with RL using judge-wise, outcome-driven rewards that combine structural and content-based components.Structural rewards target formatted reasoning and answers, while content rewards align judgments with ground-truth preferences.
- 8.14% average F1 improvement over SFT counterparts is reported for JudgeLRM across models ranging from 3B to 14B parameters.
- JudgeLRM-3B/4B surpasses GPT-4, while JudgeLRM-7B/8B matches or surpasses DeepSeek-R1 in benchmark judgment performance.
2 RELATED WORK
Related work positions LLM-as-a-Judge as a scalable evaluation paradigm and highlights reasoning as central to reliable judgment. Large reasoning models provide evidence that structured reasoning benefits evaluation tasks.
- Chain-of-Thought and related prompting or architectural techniques enable LLMs to perform multi-step reasoning across diverse domains.
- Large reasoning models such as DeepSeek-R1 demonstrate strong evaluation performance by leveraging structured reasoning paths.
- LLM-as-a-Judge seeks scalable, potentially cost-effective evaluation because human assessment is time-consuming, resource-intensive, and subjective.
3 JUDGE-WISE OUTCOME REWARD AND RL TRAINING
JudgeLRM applies GRPO to judge tasks with rewards that combine output structure, comparative accuracy, score calibration, and confidence. Group-wise advantage normalization stabilizes learning across tasks with different reward levels.
- 3 JUDGE-WISE OUTCOME REWARD AND RL TRAINING: JudgeLRM applies RL to judge tasks through judge-wise outcome rewards, motivated by SFT’s insufficiency for reasoning-heavy evaluation.
- 3 JUDGE-WISE OUTCOME REWARD AND RL TRAINING: The judge receives a query and two responses, assigns each a 1–10 score, and aims to match human win, tie, or loss judgments.
- 3 JUDGE-WISE OUTCOME REWARD AND RL TRAINING: The total reward combines structural and content-based components for formatted reasoning, valid answer scores, ranking accuracy, score accuracy, and confidence.
- 3 JUDGE-WISE OUTCOME REWARD AND RL TRAINING: Relative-order rewards prioritize correct response ranking, while hierarchical absolute rewards refine score calibration and address sparse reinforcement-learning signals.
- 3 JUDGE-WISE OUTCOME REWARD AND RL TRAINING: Confidence rewards encourage polarized judgments only when the predicted relative order is correct, avoiding blind variance expansion.
- 3 JUDGE-WISE OUTCOME REWARD AND RL TRAINING: GRPO removes the need for a separate value model by normalizing within-group advantages, reducing variance and improving learning stability on difficult tasks.
4 EXPERIMENTS
The experiments evaluate JudgeLRM against SFT, RL, proprietary, and specialized judgment baselines across established benchmarks, while testing reward components, reasoning-heavy performance, reliability, and reasoning behavior. JudgeLRM generally outperforms comparable baselines, with gains linked to outcome-driven rewards and stronger performance on reasoning-intensive tasks.
- Experimental Setup: JudgeLRM is evaluated on JudgeLM and PandaLM, using GPT-4-aligned and human-annotated pairwise judgments with precision, recall, and F1 metrics.The experiments also compare against base, SFT, RL, proprietary, and specialized judgment models.
- Judge Performance: Reasoning-heavy domains show larger JudgeLRM-7B gains relative to SFT baselines, supporting the paper’s focus on reasoning-oriented judgment training.Figure 3 relates F1 improvement to the proportion of reasoning-demanding tasks across domains.
- Ablation Study: Approximately 3% degradation follows from incentivizing slightly longer answers, suggesting that increased output length alone does not improve judging.The paper notes that repetition may contribute to the degradation.
- Reliability of Model Judgment: JudgeLRM improves self-consistency, reduces first-answer bias, and maintains comparable second-answer bias under answer-order permutations.These tests assess self-consistency, positional bias, and the bias gap.
- Case Study: JudgeLRM responses exhibit verification, subgoal setting, reflection, error identification, and criterion-linked decision justification.The observed three-stage structure evaluates Assistant 1, evaluates Assistant 2, and then synthesizes a comparative assessment.
5 CONCLUSION
The paper frames LLM judgment as inherently reasoning-intensive and proposes JudgeLRM to generate structured, trustworthy reasoning through reinforcement learning. It reports strong performance across model sizes and against leading baselines.
- Judgment requires verification, sub-goal planning, and justification rather than mere scoring.
- JudgeLRM treats evaluation as an RL problem with judge-specific, outcome-driven rewards.
- JudgeLRM scales effectively from 3B to 14B parameters while outperforming GPT-4, DeepSeek-R1, and comparable baselines.
- Reinforcement learning elicits diverse reasoning behaviors that were not explicitly present in the training data.
DETAILS OF PANDALM CATEGORY ANALYSIS
The category analysis organizes PandaLM motivation-app tasks into five source categories and compares model F1 performance across them. It also specifies prompts for single-case judgment and reasoning evaluation.
- Category organization: The five categories are Entertainment Media, Office Productivity, Life Utility, Search Information Retrieval, and Social Professional Networking.
- Category results: Table 8 compares model F1 performance by category.
- Category results: The analysis reports selected PandaLM subtest results using accuracy, precision, recall, and F1-score.
- Judgment prompts: The single-case prompt asks a model to score one AI response from 1–10 using helpfulness, relevance, accuracy, and detail criteria.
- Judgment prompts: The reasoning-evaluation prompt presents two responses for an instruction and input and asks the model to judge their performance.
PROMPT FOR ACCESSING THE NECESSITY OF REASONING WHEN JUDGING
The paper uses a prompt to rate how much reasoning is needed to judge AI responses, illustrating that reasoning demands vary by task. GPT-4-generated scores were checked against manual labels for scalability.
- The prompt rates the level of reasoning ability needed to judge two responses for a given instruction and input.
- Judging a math problem is presented as not requiring reasoning, whereas judging a writing problem is presented as requiring reasoning.
- GPT-4 generated the reasoning-required scores, and manual labels for 100 samples had a Kappa coefficient of 0.82 with GPT-4 scores.
PROMPT FOR BASELINES
The paper distinguishes prompts for inference by JudgeLRM, JudgeLM, and PandaLM. The baseline evaluation prompt asks models to assess two responses and generate a reference answer.
- Figure 2 contains the JudgeLRM inference prompt, while Figures 8 and 9 contain the JudgeLM and PandaLM inference prompts.
- The baseline prompt provides two responses for a task defined by an instruction and contextual input.
- The baseline task asks the model to evaluate the responses and generate a reference answer.
FURTHER REWARD DESIGN ANALYSIS
The analysis finds that modest changes to exact reward values do not significantly affect performance when their relative ordering is preserved, whereas changing the relative reward relationship reduces performance.
- Reward Magnitudes: Reward exact values are insensitive under the paper’s RL rule-based training setting when their relative order remains unchanged.The analysis modifies relation, absolute, and confidence reward magnitudes without disrupting their relative order.
- Reward Magnitudes: GRPO may improve training stability by grouping samples and normalizing intra-group advantages, encouraging relative judgments between good and bad responses.This mechanism reduces variance in strategy updates rather than requiring pursuit of absolute reward values.
- Relative Reward Relationships: Changing the relative relationship among relation, absolute, and confidence rewards decreases performance.The altered reward settings use different relative weights under the 3B configuration.
IMPROVEMENTS BY DIFFERENT CATEGORIES
The section presents category-level improvement visualizations and qualitative case studies comparing JudgeLRM outputs with DeepSeek-R1 and GPT-4 reference responses. It also tracks response length by training steps for JudgeLRM models.
- Category Improvements: JudgeLRM-7B’s average F1 gains over SFT persist in categories with many reasoning-heavy tasks.The cited discussion presents this pattern as evidence for the necessity of RL for judges.
- Case Studies: The qualitative analysis includes a case question followed by JudgeLRM-3B reasoning and answer examples, alongside DeepSeek-R1 reasoning and a GPT-4 gold response.These materials are identified for Case 1.
- Case Studies: A second case presents JudgeLRM-7B reasoning, DeepSeek-R1 thinking and answer, and a GPT-4 gold response.The section identifies these materials for Case 2.
- Response Length: Figure 18 tracks the response lengths of JudgeLRM-3B and JudgeLRM-7B by training steps.The cited text specifies the models and the step-based response-length comparison but does not report numerical values.