PaperlayerView exact revision on arXiv

Source-linked AI summary

J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning

Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, Swarnadeep Saha

arXiv:2505.10320v3cs.CLcs.AIcs.LG

TL;DR

LLM judges need better reasoning processes because judgment quality depends on chain-of-thought reasoning. J1 uses reinforcement learning with unified verifiable training to optimize thinking, producing state-of-the-art judges across benchmarks, with some 32B models surpassing much larger systems.

  • Problem

    LLM-as-a-Judge performance depends on chain-of-thought reasoning, but methods for effectively optimizing judges’ reasoning remain needed.

  • Method

    J1 converts verifiable and non-verifiable judgments into a unified verifiable-reward format and uses online reinforcement learning to optimize judges’ reasoning and decisions.

  • Results

    J1 achieves state-of-the-art performance across multiple benchmarks, with J1-Qwen-32B-MultiTask reaching 76.8 accuracy on PPE Correctness and outperforming prior methods.

  • Takeaways & Limitations

    Pairwise supervision can produce pointwise judges that mitigate position bias, while multitask training supports both pointwise and pairwise evaluation.

Abstract

from arXiv · show

The progress of AI is bottlenecked by the quality of evaluation, making powerful LLM-as-a-Judge models a core solution. The efficacy of these judges depends on their chain-of-thought reasoning, creating a critical need for methods that can effectively optimize this reasoning process. In this work, we introduce J1, a reinforcement learning framework for teaching LLM judges to think before making decisions. Our core contribution lies in converting all judgment tasks for non-verifiable and verifiable prompts into a unified format with verifiable rewards, enabling direct optimization of evaluation quality while mitigating positional bias. We then use RL to train thinking-judges at scales of 8B, 32B, and 70B and show that they obtain state-of-the-art performance across multiple benchmarks. In particular, J1-Qwen-32B, our multitasked pointwise and pairwise judge also outperforms o1-mini, o3, and a much larger 671B DeepSeek-R1 on some benchmarks, while only training on synthetic data. Through comprehensive ablations of pairwise, pointwise, and multitask J1 variants, we demonstrate the effectiveness of our approach across seed prompts, reward strategies, and training recipes. Qualitative analysis reveals that J1 develops systematic evaluation strategies, including dynamic criteria generation, reference answer creation, iterative self-correction of initial assessments, and feedback generation for low-quality responses.

1 INTRODUCTION

J1 frames LLM-as-a-Judge evaluation as a reasoning problem and introduces reinforcement-learning methods that unify verifiable and non-verifiable judgment tasks under verifiable rewards. The resulting judges support pointwise and pairwise evaluation and achieve state-of-the-art performance across multiple benchmarks, with ablations analyzing training and design choices.

  • Framework: J1 converts verifiable and typically subjective, non-verifiable judgment tasks into a unified format optimized with reinforcement learning from verifiable rewards.This recipe is designed to train evaluation models through explicit thinking rather than direct score prediction.
  • Reasoning patterns: J1’s learned reasoning patterns include outlining criteria, generating reference answers, re-evaluating correctness, and comparing responses before producing a verdict or score.Pairwise models output a final verdict, while pointwise models generate real-valued scores.
  • Framework: J1 develops inherently consistent pointwise judges from pairwise supervision and unifies both capabilities in a single multitask model.The multitask model performs both pointwise and pairwise evaluations.
  • Results: At 8B, 32B, and 70B scales, J1 judges built on Llama-3.1-Instruct and Qwen3 achieve state-of-the-art performance across PPE, RewardBench, JudgeBench, RM-Bench, and FollowBenchEval.The reported benchmark set spans five evaluation benchmarks.
  • Ablations: Ablations compare pairwise, pointwise, and joint multitask designs, score usage, bias mitigation, reward modeling, and seed thinking prompts, finding the joint model superior to separately trained counterparts.The analysis also examines the flexibility of the joint model across pointwise and pairwise evaluations.

2 J1: THINKING-LLM-AS-A-JUDGE VIA REINFORCEMENT LEARNING

J1 trains LLM judges to generate chain-of-thought reasoning before producing response scores or preference verdicts, using synthetic verifiable tasks and reinforcement learning. Its formulations include pairwise, pointwise, and multitask judges, with position-agnostic data and rewards for correctness and consistency.

  • Core framework: J1 generates intermediate thought tokens before scoring responses or selecting a preferred response in pairwise evaluation.The pairwise setup conditions on an instruction and two responses, producing thought tokens followed by a final verdict.
  • Synthetic data generation: The method uses synthetic preference pairs to create verifiable reinforcement-learning tasks for training generalist judges across verifiable and non-verifiable prompts.The final training set contains 22K examples: 17K WildChat prompts and 5K MATH prompts with corresponding preference pairs.
  • Synthetic data generation: Both response orderings, (x, a, b) and (x, b, a), are processed in the same batch to make training position-agnostic and support consistency rewards.This design targets position bias in pairwise judges and requires correctness across both orderings for the consistency reward.
  • Reward design: +1 is awarded for a correct final verdict, while the consistency reward gives +1 only when both response orderings receive correct verdicts.Incorrect verdicts receive 0 under each corresponding reward; format-based <think> rewards showed no noticeable performance benefit.
  • Training formulations: GRPO jointly optimizes thought generation and final judgments across pairwise verdict, score, and score-and-verdict formulations.The score-based variant generates real-valued scores for both responses, whereas the score-and-verdict variant uses the final verdict as the rewarded output.
  • Training formulations: The pointwise judge scores one response and is trained by distant supervision from pairwise data, while the multitask model jointly supports pairwise and pointwise evaluation.Pointwise judges are inherently consistent, and the multitask model is evaluated pairwise because pairwise judgments are generally superior.

3 EXPERIMENTAL SETUP

J1 is trained on 22K synthetic preference pairs and evaluated across five pairwise judgment benchmarks spanning verifiable and non-verifiable tasks. The setup uses task-specific accuracy protocols and compares J1 with non-thinking and thinking LLM judges as well as scalar reward models.

  • Training: J1 variants are implemented with verl and trained on 22K synthetic preference pairs.The paper refers to additional experimental details in Appendix C.
  • Evaluation: Five pairwise judgment benchmarks evaluate J1 on verifiable and non-verifiable tasks with multilingual instructions and responses from diverse LLMs.The evaluation suite includes PPE, RM-Bench, FollowBenchEval, RewardBench, and JudgeBench.
  • Evaluation: PPE includes 10.2K human-preference samples spanning 20 LLMs in 121+ languages and 12.7K correctness samples from four models across verifiable benchmarks.The correctness subset covers MMLU-Pro, MATH, GPQA, MBPP-Plus, and IFEval.
  • Evaluation: Accuracy uses random response ordering for PPE, RewardBench, and RM-Bench, but position-consistent accuracy for JudgeBench and FollowBenchEval.A JudgeBench or FollowBenchEval sample is correct only when the verdict is correct under both response orders; model selection uses overall RewardBench accuracy.
  • Baselines: Baselines comprise zero-shot non-thinking judges, thinking judges, and state-of-the-art scalar reward models.Examples include Llama-3.3-70B-Instruct and GPT-4o; DeepSeek-R1, Qwen3-32B, OpenAI-o1-mini, and o3; and DeepSeek-BTRM-27B, Armo, and Skywork-Reward-Gemma-2-27B.

4 RESULTS

J1 achieves state-of-the-art reward-modeling performance across PPE Correctness and RewardBench, including strong gains over generative reward models and thinking-LLM baselines. Ablations show that multitasking improves both pointwise and pairwise judgment quality, while J1 remains robust to prompt choices and develops structured reasoning behavior.

  • PPE Correctness: 76.8 overall accuracy makes J1-Qwen3-32B-MultiTask state of the art on PPE Correctness, outperforming EvalPlanner by 6.8% and DeepSeek-GRM-27B by 17%.It also improves over base Qwen3-32B by 10.3%, from 66.5 to 76.8.
  • RewardBench: 93.6 overall score makes J1-Qwen-32B-MultiTask the top method on RewardBench, with comparable performance across all four categories.This supports using J1 as a generalist judge for verifiable and non-verifiable prompts across LLM-development stages.
  • Thinking-LLM comparison: J1-Llama models outperform their same-base DeepSeek-R1-Distilled-Llama counterparts across all benchmarks, while J1-Qwen-32B-MultiTask beats R1 and o3 on three of five benchmarks.The Qwen model achieves this using a 32B model trained only on synthetic data.
  • Position consistency: Multitask J1 outperforms separately trained pointwise and pairwise judges in both random-order and position-consistency accuracy.Pointwise-J1 is stronger under the stricter position-consistent metric, whereas Pairwise-J1 performs better with random response ordering.
  • Reasoning analysis: J1 remains robust across thinking prompts, while simpler prompts produce richer traces containing criteria, reference answers, re-evaluations, and detailed comparisons.During training, pairwise judges converge near 500 thought tokens, whereas pointwise judges typically generate 300–400 tokens.

5 RELATED WORK

Prior work spans scalar reward models, generative LLM judges, and reinforcement-learning approaches for reasoning judges. J1 builds on verifiable-reward reinforcement learning and reports superior performance with significantly less data than concurrent reasoning-judge methods.

  • Reward Models: Traditional reward models use the Bradley-Terry objective to produce scalar response rewards, but often generalize and calibrate poorly across prompts and responses.They have supported both training-time and test-time LLM alignment.
  • LLM-as-a-Judge and Generative Reward Models: LLM-as-a-Judge and Generative Reward Models use language-modeling heads to generate chain-of-thought critiques before preference judgments or rewards.Their rewards may be obtained through training a mechanism described in the continuation of the passage.
  • Reinforcement-Learning Reasoning Judges: J1 compares against DeepSeek-GRM, JudgeLRM, RM-R1, and Reward Reasoning Model, achieving superior performance with significantly less data.These concurrent methods also use reinforcement learning to build reasoning judge models.
  • Reinforcement Learning with Verifiable Rewards: J1 draws on reinforcement learning with verifiable rewards, using preference pairs and rewards based on judgment correctness to improve reasoning.The approach is motivated by results showing that online optimization algorithms such as GRPO can elicit enhanced reasoning when paired with accurate, robust rewards.

6 CONCLUSION

J1 is an online reinforcement-learning recipe that makes judgment tasks verifiable across both verifiable and non-verifiable prompts, enabling optimization of judges’ thoughts and judgments. The authors trained models across multiple scales and formulations, using open-weight foundations and open-source training infrastructure.

  • Method: J1 converts verifiable and non-verifiable judgment tasks into verifiable objectives and optimizes thoughts and judgments with online reinforcement learning.This is the framework’s central methodological innovation.
  • Experiments: J1 models were trained at 8B, 32B, and 70B scales using pointwise, pairwise, and multitask formulations.The generalist models outperformed all baselines at their respective sizes.
  • Reproducibility: All J1 models use open-weight Llama and Qwen foundations, with training code based on the open-source Verl repository.Prompts, hyperparameters, and reproducibility details are provided in the appendices.

A J1’S THINKING TRACES

J1’s thinking traces show step-by-step evaluation for both verifiable math and non-verifiable prompts. In the math example, J1 identifies an error and provides detailed feedback, while the writing example begins by clarifying the user’s requirements.

  • Representative examples: The section provides representative thinking-trace examples for verifiable math and non-verifiable prompts.The examples are shown in Figures 7 and 8.
  • Verifiable math prompt: For a verifiable math prompt, J1 analyzes Assistant B’s answer step by step, detects an error in simplification, and provides detailed feedback.This trace is presented in Figure 7, with the feedback highlighted in blue.
  • Non-verifiable writing prompt: For a non-verifiable writing prompt, J1 first identifies the requested topic, safety requirements, target audience, and applicable approach boundaries.The prompt concerns arc flash PPE maintenance, use, inspections, shock protection, and qualified persons.

B PROMPT TEMPLATES

The paper provides seed prompt templates for pairwise and pointwise J1 judges, including verdict, score, and plan-plus-execution variants. It also describes synthetic data construction using noisy instructions to create rejected responses.

  • Pairwise prompt templates: Pairwise-J1 training uses verdict, score, and combined score-and-verdict seed prompt templates, including an EvalPlanner-style plan-plus-execution variant.The plan-plus-execution prompt first asks the model to plan the evaluation recipe and then execute it during thinking.
  • Pointwise prompt template: Pointwise-J1 adapts the pairwise prompts to instruct the model to think and assign real-valued scores between 0 and 10.The pointwise training template is shown in Figure 13.
  • Synthetic training pairs: Synthetic training pairs generate a noisy instruction and a response to it, then select that response as rejected for the original instruction.Because the response is good for the noisy instruction, it is bad for the original instruction.

C EXPERIMENTAL SETUP

J1 training uses sampled rollouts, fixed optimization and sequence-length settings, and model-size-dependent hardware configurations. The setup also tunes KL regularization by model scale and applies distinct inference sampling parameters for Llama and Qwen3.

  • Training configuration: Training generates 5 rollouts per prompt with temperature 1.0, uses a 1e−6 learning rate, batch size 512, and 4096-token maximum sequences.The learning rate decays to 3e−7 at later steps for pairwise J1-Llama-70B.
  • Training configuration: KL coefficients are selected by model scale: 0.01 for J1-Llama-8B and 0 for J1-Llama-70B, while entropy bonuses degraded performance through longer, repetitive outputs.The 0.01 value was chosen using development-set accuracy, whereas the 70B setting encourages exploration.
  • Hardware and inference: Training uses 8×A100, 32×A100, and 64×A100 GPUs for J1-Llama-8B, J1-Qwen-32B, and J1-Llama-70B, respectively, with 8-way tensor parallelism.Inference uses 8×A100 GPUs and a maximum generation length of 4096 tokens.
  • Hardware and inference: Inference-time scaling samples with top-p 0.95 and temperature 1 for Llama, while Qwen3 uses the default temperature 0.6.These settings apply during inference-time scaling.

D ADDITIONAL RESULTS

Additional results show that J1 is robust across output formats, decoding choices, and training regularization, while outperforming a scalar reward model on most evaluated benchmarks. The studies also examine position-bias mitigation and test-time scaling for pairwise and pointwise judges.

  • Pairwise-J1 variants: Pairwise-J1 models predicting only the final verdict perform as well as variants also producing response scores, while scores support preference quantification and ranking.The comparison covers verdict-only, score-only, and verdict-plus-score output formats.
  • Decoding hyperparameters: J1 models show consistent performance with negligible variance under greedy decoding and temperature sampling.This indicates robustness to the tested decoding hyperparameters.
  • Training regularization: More exploration through the GRPO KL penalty and entropy bonus generally degrades Pairwise-J1 performance.The ablation studies this effect in a Pairwise J1-Llama-8B model.
  • Comparison with scalar reward models: J1 outperforms a scalar reward model trained on the same base model and data on four out of five benchmarks, with the largest improvement on RM-Bench.The comparison uses Llama-3.1-8B-Instruct for both models.
  • Bias mitigation and test-time scaling: Additional experiments evaluate position-bias mitigation for Pairwise-J1 and test-time scaling for pointwise and pairwise models on PPE Correctness.Pointwise judgments use average response scores, whereas pairwise judgments use majority voting over multiple verdicts.
Loading 2505.10320v3…