Source-linked AI summary
Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, Tianlu Wang
TL;DR
LLM judges lack human-annotated CoTs, leaving effective evaluation-trace structure understudied. EvalPlanner separates unconstrained planning from execution and judgment, then self-trains on synthetic preference pairs, achieving strong results across benchmarks, including 93.9 on RewardBench. Its synthetic training data can reflect stereotypes, biases, and other negative traits inherited from seed models.
Problem
The lack of human-annotated CoTs makes it difficult to train LLM-as-a-Judge models and leaves effective reasoning-trace structure understudied.
Method
EvalPlanner generates unconstrained evaluation plans, executes them step by step, and iteratively optimizes synthetic plan-execution-verdict preference pairs with DPO.
Results
EvalPlanner achieves a new state-of-the-art 93.9 on RewardBench and improves performance across RewardBench, PPE, RM-Bench, JudgeBench, and FollowBenchEval.
Takeaways & Limitations
Planning and reasoning support robust, data-efficient Thinking-LLM-as-a-Judge models across diverse evaluation benchmarks.
Takeaways & Limitations
Synthetic training data comes from seed Llama models and may reflect stereotypes, biases, and other negative traits in their pre-training data.
Abstract
from arXiv · showhide
LLM-as-a-Judge models generate chain-of-thought (CoT) sequences intended to capture the step-bystep reasoning process that underlies the final evaluation of a response. However, due to the lack of human annotated CoTs for evaluation, the required components and structure of effective reasoning traces remain understudied. Consequently, previous approaches often (1) constrain reasoning traces to hand-designed components, such as a list of criteria, reference answers, or verification questions and (2) structure them such that planning is intertwined with the reasoning for evaluation. In this work, we propose EvalPlanner, a preference optimization algorithm for Thinking-LLM-as-a-Judge that first generates an unconstrained evaluation plan, followed by its execution, and then the final judgment. In a self-training loop, EvalPlanner iteratively optimizes over synthetically constructed evaluation plans and executions, leading to better final verdicts. Our method achieves a new state-of-the-art performance for generative reward models on RewardBench (with a score of 93.9), despite being trained on fewer amount of, and synthetically generated, preference pairs. Additional experiments on other benchmarks like RM-Bench, JudgeBench, and FollowBenchEval further highlight the utility of both planning and reasoning for building robust LLM-as-a-Judge reasoning models.
1 Introduction
EvalPlanner addresses the difficulty of training LLM judges without human-annotated reasoning traces by separating evaluation planning from plan execution. It uses self-training over synthetic reasoning data and achieves strong benchmark performance with fewer preference pairs.
- LLM-as-a-Judge models are promising evaluators, but training detailed evaluation reasoning is difficult because human-annotated CoTs are unavailable.
- EvalPlanner separates evaluation into a detailed plan, step-by-step execution, and final verdict for instruction-specific response assessment.
- The method samples multiple plans and executions, then iteratively preference-optimizes correct and incorrect plan-execution-verdict CoTs.
- 93.9 on RewardBench establishes a new state-of-the-art score for generative reward models while using fewer, synthetically generated preference pairs.
2 EvalPlanner
EvalPlanner models evaluation as latent planning followed by plan execution and a verdict, then trains this structure using synthetic preference pairs and iterative optimization. Its data recipe samples diverse plans and executions, while its self-training loop repeatedly improves the resulting evaluator.
- The judge takes an instruction and two responses, generating a preference judgment alongside a CoT intended to capture evaluation reasoning.
- Method overview: EvalPlanner structures each CoT into an evaluation plan, plan execution, and final verdict, with execution analyzing responses according to the plan.
- Synthetic training data generation: The method generates synthetic response pairs and samples multiple, domain-varying evaluation plans instead of manually fixing plan components.
- Synthetic training data generation: The preference dataset includes both response orders, yielding 2 × |P| × |E| CoTs per instruction to address position bias.
- Synthetic training data generation: For each plan, multiple executions produce correct and incorrect judgments that become chosen and rejected CoT pairs.
- Preference optimization: Training begins with SFT on chosen thoughts and continues with two DPO iterations over CoT preference pairs from successive model versions.
3 Experimental Setup
EvalPlanner is evaluated across multiple pairwise reward-modeling benchmarks, using standard protocols and comparisons against zero-shot judges, critique-generating reward models, and generative reward models.
- Benchmarks: The experiments cover RewardBench, PPE, RM-Bench, JudgeBench, and FollowBenchEval across chat, safety, code, math, and fine-grained constraint-following tasks.PPE includes subjective preference and correctness subsets, while FollowBenchEval tests multi-level constraint checking.
- Benchmarks: RewardBench contains prompt–chosen–rejected triples across chat, chat-hard, safety, and reasoning categories.
- Benchmarks: FollowBenchEval contains 205 paired samples where one response satisfies all constraints and the other violates them across five constraint types.Responses are sampled from the same model to reduce superficial style differences in preference judgments.
- Evaluation protocol: For RewardBench, PPE, and RM-Bench, results use accuracy under one random response ordering, while JudgeBench and FollowBenchEval use position-consistent accuracy.Position-consistent accuracy requires a correct judgment in both response orders.
- Baselines: EvalPlanner is compared with zero-shot open- and closed-source judges, reward models with critiques, and state-of-the-art generative reward models that produce rationales.
- Reported comparisons: Table 1 compares generative reward models on RewardBench, while Table 2 compares EvalPlanner with state-of-the-art LLM-as-a-Judge and reward models on PPE.The RewardBench comparison uses 22K synthetically constructed preference pairs for EvalPlanner.
4 Results
EvalPlanner performs strongly across reward-modeling and judge benchmarks, including at smaller scale and with fewer synthetic preference pairs. Ablations indicate that iterative thought optimization, preference optimization, and unconstrained plans contribute to its performance.
- Benchmark results: EvalPlanner-8B matches the performance of larger LLM-as-a-Judge models on RewardBench.The comparison includes Llama-3.1-70B-Instruct and Claude-3.5-Sonnet.
- Data efficiency and iteration: 92.3 on RewardBench with 5K preference pairs rises to 93.9 after a second DPO iteration, versus 92.5 with one iteration.The iterative recipe trains on newer data points augmented with chain-of-thoughts from an updated model.
- Benchmark results: EvalPlanner outperforms Skywork-Critic-Llama-3.1-70B by 13% on FollowBenchEval's multi-level constraint evaluations.FollowBenchEval tests whether responses satisfy up to five objective constraints.
- Benchmark results: EvalPlanner improves RM-Bench performance by up to 8% and remains equally performant across all subsets, including the hard subset.On JudgeBench, the Llama-3.3-70B-Instruct version performs comparably to Skywork-Critic-Llama-3.1-Instruct while using less synthetic preference data.
- Ablations and analysis: Preference optimization of plans and executions significantly improves results over final-verdict-only, seed-model, and chosen-example SFT baselines.Unconstrained plans also outperform criteria-list and verification-question prompts by generating more detailed plans.
5 Related Work
Related work frames EvalPlanner within LLM-based evaluation, self-alignment, and methods that train models to think, plan, and reason. These approaches address scalable evaluation and reduced reliance on human annotations, while EvalPlanner extends them to self-trained evaluation traces.
- LLM-as-a-Judge: LLM-as-a-Judge methods use language models as scalable, cost-effective alternatives to slow, expensive, and noisy human evaluation.Unlike scalar reward models, these evaluators generate reasoning alongside judgments.
- Self-Alignment: Self-alignment methods reduce dependence on expensive human annotations across instruction following, reasoning, and evaluation.This work places evaluation-oriented self-training within that broader research direction.
- Training to Think, Plan, and Reason: EvalPlanner builds on work that equips language models to generate additional thought tokens, but uses bootstrapped self-training rather than ground-truth thoughts.Prior ground-truth-thought methods include algorithmic reasoning, mathematics, and planning.
6 Conclusion
EvalPlanner is presented as a robust, data-efficient approach for Thinking-LLM-as-a-Judge models, achieving new state-of-the-art performance with substantially less synthetic training data.
- EvalPlanner builds robust and data-efficient Thinking-LLM-as-a-Judge models.
- EvalPlanner achieves a new state-of-the-art with significantly less, synthetically generated training data across four benchmarks.
Impact Statement
EvalPlanner aims to improve evaluation by having LLM-as-a-Judge models think before producing judgments, while its synthetic seed-model data may inherit stereotypes, biases, and other negative traits.
- EvalPlanner’s synthetic training data comes from seed Llama models that may reflect stereotypes, biases, and other negative traits from pre-training data.The authors state that they do not control these inherited traits and encourage further responsible research and discussion.
- EvalPlanner’s broader goal is to improve evaluation by allowing LLM-as-a-Judge models to think before producing a judgment.
A More Analysis
Additional analyses find that jointly optimizing diverse plans and executions generally improves performance across categories, while instruction sources benefit different capabilities unevenly.
- Scaling Number of Plans & Executions during Training: Jointly optimizing diverse evaluation plans and executions generally increases performance across all categories.This scaling study varies the number of latent plans and executions, which determines thought preference pairs per instruction during DPO training.
- Effect of Source of Input Instructions: WildChat instructions help the Chat-Hard category more, whereas MATH instructions particularly enhance reasoning performance.
B Prompts
The appendix presents separate prompts for generating initial evaluation plans and executing those plans from the seed model.
- Figure 3 shows the planning prompt used to generate initial evaluation plans from the seed model.
- Figure 4 shows the plan-execution prompt used to generate initial executions from the seed model.
C Examples of plans generated by EvalPlanner
EvalPlanner generates diverse evaluation plans tailored to user instructions, then executes those plans through dedicated prompts before producing judgments. Examples span coding, math, and safety tasks, with plans containing task-specific tests, solution steps, criteria, rubrics, and reference answers.
- Examples of plans: EvalPlanner generates diverse plans conditioned on the user instruction across coding, math, and safety problems.The examples illustrate plans specialized to different evaluation domains.
- Plan execution: A dedicated execution prompt instructs the judge to follow the plan step-by-step, explain its execution, and avoid position bias.The execution template also tells the judge not to copy the plan and to remain within its scope.
- Final judgment: A separate judgment prompt asks the judge to compare responses and select the assistant that better follows the instruction and answers the question.The template names helpfulness, relevance, accuracy, depth, creativity, and detail as evaluation factors.
- Coding problem: Coding plans generate multiple test cases, including invalid inputs and edge cases, to check code correctness.
- Math problem: Math plans combine a step-by-step solution with multiple evaluation criteria for comparing responses.
- Safety question: Safety plans include evaluation criteria, compliance-focused steps, a scoring rubric, and a high-quality reference answer.