PaperlayerView exact revision on arXiv

Source-linked AI summary

JudgePanel: A Compact Judge with Panel Deliberation via Adaptive Multi-Reward Reinforcement Learning

Yiyue Qian, Shinan Zhang, Huan Song, Hannah Marlowe

arXiv:2608.29168v1cs.AI

TL;DR

LLM judges face model bias, while multi-agent deliberation is costly to deploy. JudgePanel trains a compact model on panel traces, improves it with adaptive multi-reward RL, and specializes it through lightweight adaptation. Its 14B model outperforms judge-specialized models up to 70B across four benchmarks and adapts with few hundred samples.

  • Problem

    Single-model judges suffer systematic bias and inconsistent cross-domain judgments, while multi-agent protocols are prohibitively expensive for large-scale deployment.

  • Method

    JudgePanel trains a compact judge on multi-agent deliberation traces, applies AdaReward to adaptively rebalance multi-reward RL weights, and uses lightweight domain specialization.

  • Results

    JudgePanel with a 14B backbone outperforms judge-specialized models up to 70B across four evaluation benchmarks and demonstrates strong position consistency.

  • Takeaways & Limitations

    JudgePanel provides a compact, scalable approach for reliable domain-specific evaluation without deploying large-scale models or multi-agent systems in production.

  • Takeaways & Limitations

    JudgePanel generates more tokens and has higher latency than standard chain-of-thought judges, trading throughput for judgment quality.

Abstract

from arXiv · show

The LLM-as-a-Judge paradigm has emerged as a scalable alternative to human evaluation. However, single-model judges are limited by their inherent model biases, while multi-agent evaluation protocols that mitigate this through diverse deliberation are prohibitively expensive at inference time. To this end, we propose \textbf{\modelname}, which equips a compact \underline{Judge} model with multi-agent \underline{Panel} deliberation capability. Specifically, we first train on panel deliberation traces from an ensemble of strong evaluators, capturing structured patterns of discussion, disagreement, and resolution. To further improve judgment quality beyond SFT, we introduce \textit{AdaReward}, an adaptive multi-reward RL algorithm that dynamically rebalances reward component weights as different objectives saturate at different rates during RL training. For practical deployment, we further design a lightweight domain specialization module for rapid adaptation to new evaluation domains with few hundred labeled samples. As a result, (i) \textit{Novel}: the first framework to equip a single compact judge with multi-agent panel deliberation capability at single-model inference cost; (ii) \textit{Effective \& Reliable}: JudgePanel with a 14B backbone outperforms judge-specialized models up to 70B across four evaluation benchmarks, demonstrates strong position consistency, and rapidly specializes to new domains with few hundred samples.

1 Introduction

JudgePanel addresses biased and expensive LLM evaluation by training one compact judge to deliberate like a multi-agent panel at single-model inference cost.

  • Human evaluation is costly and difficult to scale, motivating LLM-as-a-Judge approaches for evaluating model outputs.
  • Single-model judges exhibit model biases that produce systematic evaluation bias and inconsistent judgments across domains.
  • Multi-agent protocols reduce individual biases through discussion and consensus but require multiple large models and multi-round inference.
  • JudgePanel trains a compact judge on panel deliberation traces containing structured discussion, disagreement, and resolution patterns.
  • JudgePanel introduces AdaReward, which dynamically rebalances multi-reward RL objectives as components progress at different rates.
  • A 14B JudgePanel outperforms judge-specialized models up to 70B across four benchmarks, maintains position consistency, and adapts rapidly with few hundred samples.

2 Preliminary

The preliminary formulation models an LLM judge as generating reasoning traces and verdicts, then uses GDPO to optimize multiple reward functions through normalized advantages and fixed weighting.

  • Given an evaluation input x_i, an LLM judge π_θ generates a reasoning trace t_i followed by a verdict ŷ_i.
  • During RL training, the method samples G rollouts from a reference policy, with each rollout containing a reasoning trace and predicted verdict.
  • GDPO optimizes K reward functions by normalizing each reward before aggregation to address reward-signal collapse in GRPO.
  • Per-reward advantages are standardized using each reward’s batch mean and standard deviation.
  • The normalized advantages are aggregated with fixed weights, and the policy is optimized using a clipped surrogate objective.

3 JudgePanel: Panel Deliberation via Adaptive Multi-Reward RL

JudgePanel combines panel-trace training, adaptive multi-reward RL, and lightweight domain specialization to produce compact judges with panel-style reasoning.

  • Framework: JudgePanel has three stages: panel deliberation for training traces, AdaReward for multi-reward RL, and LoRA-based domain specialization.
  • Panel Deliberation: PanelBench addresses single-model reasoning diversity limits by using multi-round collaborative evaluation traces for judge training.
  • Panel Deliberation: The panel protocol proceeds through independent evaluation, discussion when consensus is absent, and final judgment after convergence or d_max rounds.
  • Panel Deliberation: Only panel traces whose verdict matches the ground-truth label are retained, yielding approximately 8,700 correct training traces.
  • AdaReward: AdaReward adapts reward weights because format compliance saturates quickly while outcome correctness, discussion quality, and efficiency continue learning.
  • Adaptive Weighting: When rewards saturate, fixed weights cannot redirect learning toward harder components, whereas AdaReward increases weights for lagging components and eases them for saturated ones.
  • Adaptive Weighting: The adaptive learner uses asymmetric scaling and clips target weights, with EMA updates balancing responsiveness against stability.
  • Domain Specialization: Domain specialization uses 200–500 human-labeled samples to generate domain-specific traces, then applies LoRA to preserve general deliberation capabilities.

4 Experiments

JudgePanel is evaluated on four judge benchmarks, reliability under position swaps, ablations, and efficiency against multi-agent and chain-of-thought baselines. Results show strong cross-benchmark performance, position consistency, useful contributions from domain adaptation and adaptive rewards, and practical specialization with limited data.

  • Evaluation Setup: 8,500 panel deliberation traces are retained from Chatbot Arena after filtering incorrect verdicts for SFT and RL training.The training traces contain no overlapping queries or responses with evaluation benchmark test sets or domain-adaptation data.
  • Main Results: JudgePanel with domain adaptation achieves the highest average rank across JudgeBench, RewardBench, RMBench, and PPE, while the no-DA model surpasses all reported baselines.The evaluation compares general LLMs and judge-specialized models, including models up to 70B.
  • Reliability: JudgePanel achieves the highest position consistency and stable accuracy on all evaluated benchmarks, surpassing 70B models despite being 5 × smaller.Consistency measures whether verdicts correctly track response-position swaps; stable accuracy requires correctness in both orderings.
  • Ablation: 1.86/2.71 performance points are lost without domain adaptation on RewardBench/RMBench, while fixed adaptive weights cause an additional 1.73/2.30 drop.Removing RL entirely produces a 3.80/6.22 gap below the full system on the same benchmarks.
  • Panel Deliberation Analysis: 91.0%/85.7% Constructive Change Rate on RewardBench/RMBench exceeds the ∼50% random baseline, indicating vote changes tend to produce correct answers.Jaccard Distance is 0.629/0.639, supporting diverse reasoning across the two benchmarks.
  • Efficiency: 91.33/45.61 accuracy on RewardBench/PPE approaches the multi-agent protocol’s 94.54/46.28 at half the latency, 1.5 versus 2.9 minutes per 100 samples.Think-J-32B is faster at 1.0 minute but has lower accuracy, 90.50/40.15.
  • Domain Specialization: 200 domain-specialization samples improve performance over the no-DA baseline, with diminishing returns beyond 500 samples.The same analysis reports that Panel Trace SFT outperforms label-only, single-COT, and majority-voting supervision.

5 Related Work

LLM-as-a-Judge methods use language models as automated evaluators, but general-purpose judges exhibit systematic biases. Dedicated SFT and RL approaches address this need, while SFT methods may rely on pattern memorization rather than flexible reasoning.

  • LLM-as-a-Judge: LLM-as-a-Judge uses LLMs as automated evaluators and has become widely used in LLM-related applications.The related-work discussion situates judge training within the broader growth of LLM applications.
  • Judge Training: General-purpose LLM judges suffer from systematic biases, motivating dedicated judge training.The cited SFT approaches curate synthetic data, while RL-based work investigates judgment reasoning.
  • Judge Training: SFT-based judge approaches rely on pattern memorization rather than flexible reasoning.This limitation is attributed to the cited discussion of SFT-based evaluator training.

6 Conclusion

JudgePanel equips a compact judge with multi-agent panel deliberation through adaptive multi-reward RL. Across four benchmarks, its 14B model outperforms judge-specialized models up to 70B, maintains strong position consistency, and adapts rapidly with few hundred samples.

  • JudgePanel equips a single compact judge model with multi-agent panel deliberation via adaptive multi-reward RL.
  • Across four established judge benchmarks, JudgePanel (14B) consistently outperforms judge-specialized models up to 70B while maintaining strong position consistency.
  • JudgePanel rapidly specializes to new evaluation tasks using a few hundred samples, supporting practical deployment.
  • The framework provides domain-specific evaluation without deploying large-scale models or multi-agent systems in production.

Limitations

JudgePanel improves judgment quality at additional inference-token cost and retains a dependency on panel models during domain specialization. Its training pipeline comprises panel-data construction, SFT, AdaReward RL, and LoRA specialization.

  • Limitations: JudgePanel generates more tokens than standard chain-of-thought judges because independent evaluations, discussion rounds, and final judgment simulate multi-speaker deliberation.
  • Limitations: 1.5 min vs. 1.0 min latency on 100 samples accompanies higher accuracy than Think-J on RewardBench and PPE with a smaller model.The reported accuracy pairs are 91.33 vs. 90.50 on RewardBench and 45.61 vs. 40.15 on PPE; the comparison is for Think-J at 32B versus JudgePanel at 14B.
  • Limitations: Domain specialization can use as few as 200 labeled samples but requires panel-protocol execution to generate traces for LoRA fine-tuning.
  • Limitations: The current specialization pipeline therefore depends on panel models during the specialization phase.
  • Training pipeline: The training pipeline includes panel-data construction, SFT, AdaReward reinforcement learning with adaptive weight updates, and lightweight domain specialization.
  • Training pipeline: Domain specialization uses LoRA fine-tuning on domain-specific panel traces with 200–500 samples.

C Experiment Details

The experiments document implementation settings for SFT, AdaReward RL, and LoRA-based domain adaptation. Training uses eight NVIDIA H200 GPUs, with reported runtimes for each stage.

  • Training configuration: SFT runs for 3 epochs with learning rate 1 × 10^-6, cosine scheduling, batch size 32, and maximum sequence length 8,192.
  • Training configuration: AdaReward RL runs for 5 epochs with 8 rollouts per prompt, learning rate 5 × 10^-7, clip ratio ε=0.2, and KL penalty 0.08.
  • Training configuration: The adaptive learner uses specified reward targets, scaling factors, momentum, bounds, and updates every 5 steps.The initial effective outcome weight is 2.0, with targets ρout=0.95, ρdis=0.85, ρeff=0.85, and ρfmt=0.99.
  • Training configuration: LoRA domain adaptation uses r=16, α=32, learning rate 1 × 10^-4, early stopping patience 10, and 3 epochs.
  • Resource requirements: Experiments use 8× NVIDIA H200 GPUs; SFT takes approximately 2 hours, RL 22 hours, and LoRA adaptation about 5 minutes per task.

C.2 Benchmark and DA Data Details

The evaluation covers four judge benchmarks spanning correctness, reward-model preferences, subtle content sensitivity, and pairwise preference alignment.

  • JudgeBench contains 620 challenging response pairs covering knowledge, reasoning, math, and coding, evaluated by objective correctness.
  • RewardBench contains 2,985 prompt-chosen-rejected trios covering chat, reasoning, and safety.
  • RMBench contains 3,981 samples testing sensitivity to subtle content differences and resistance to style biases.
  • PPE contains 16,038 samples evaluating pairwise preference alignment.

C.3 Evaluation Metric Definitions

The section defines metrics for ranking, position robustness, deliberation change, linguistic diversity, novelty, and discussion length, then illustrates single-model and panel reasoning on a coding task.

  • Metric Definitions: Avg. Rank is the mean model rank across four benchmarks, with rank 1 indicating the highest accuracy and lower averages being better.Consistency measures whether verdicts correctly track response-position swaps, while Stable Accuracy requires correctness in both orderings.
  • Metric Definitions: Constructive Change Rate measures how often vote changes during discussion lead to the correct final answer; values well above 50% indicate functional deliberation.Jaccard Distance captures speaker-level phrasing diversity, while 4-gram Overlap measures repetition of initial-round content.
  • Qualitative Comparison: A qualitative comparison shows that multi-agent deliberation can resolve disagreements that single-model chain-of-thought reasoning cannot.The example contrasts panel trace generation with single-model reasoning on the same evaluation task.
  • Single-Model Trace: The single-model trace reaches an incorrect verdict on the coding task while evaluating two candidate solutions.The task asks for the minimum operations needed to make num1 equal to 0, or −1 if impossible.
  • Single-Model Trace: The single-model analysis identifies the shared popcount condition but concludes that Response A captures the essential logic without unnecessary checks.It treats Response B’s k <= target condition as redundant under the stated constraints.

D.2 Panel Trace Example

The panel trace corrects an initially favored but flawed concise solution through discussion of negative-number behavior and the k <= target constraint, producing consensus for Response B.

  • Discussion: The trace demonstrates disagreement resolution by showing Speaker 2 change from model_a to model_b after hearing Speaker 1’s argument.The panel protocol is contrasted with the single COT reasoning on the same coding task.
  • Independent Evaluation: Speaker 1 favors Response B because negative targets can make Python’s bin().count('1') falsely satisfy the popcount condition, while B skips negatives.Speaker 1 also identifies the safe bound k <= target.
  • Independent Evaluation: Speaker 2 initially favors Response A because both solutions share the core algorithm and B’s extra condition appears redundant.The initial evaluation also criticizes B’s unnecessary math import and added complexity.
  • Discussion: After discussion, Speaker 2 revises to Response B, acknowledging that Python’s handling of negative numbers creates correctness vulnerabilities.The revision explicitly states that conciseness was initially overweighted.
  • Consensus: All three speakers reach consensus on model_b with confidence 5 after recognizing that Response B handles both the Python-specific pitfall and the k <= target bound.Speaker 3 reports Response B’s final total score as 44 versus 33 for Response A.
Loading 2608.29168v1…