PaperlayerView exact revision on arXiv

Source-linked AI summary

RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator

Zhenwei Tang, Zhaoyan Liu, Rasa Hosseinzadeh, Tongzi Wu, Keyvan Golestan, Jesse C. Cresswell

arXiv:2605.21748v1cs.CL

TL;DR

Existing LLM-judge benchmarks inadequately represent multi-turn, reference-grounded conversations and require scalable, diagnostic evaluation. RankJudge generates paired conversations with controlled single-turn flaws and jointly evaluates verdict, turn, and failure type. Across 21 judges and three domains, it produces broad, stable rankings while supporting synthetic scaling and difficulty-based curation.

  • Problem

    Existing judge benchmarks largely focus on isolated single-turn responses, limiting evaluation of multi-turn, reference-grounded conversations.

  • Method

    RankJudge independently generates better and flawed multi-turn conversations from shared reference documents, injects one targeted failure, and scores joint verdict, turn, and failure-type predictions.

  • Results

    Across 21 judges and three domains, RankJudge produces leaderboards spanning nearly 1200 Elo points, with rankings stable under subsampling and other evaluation perturbations.

  • Takeaways & Limitations

    Synthetic generation, automated verification, and Bradley–Terry Elo curation enable scalable judge evaluation without per-item human annotation while preserving diagnostic comparisons.

  • Takeaways & Limitations

    gpt-5.5 shows strong self-preferential bias because it generated and verified the conversations, changing rank substantially under an alternative generator.

Abstract

from arXiv · show

As interactive LLM-based applications are created and refined, model developers need to evaluate the quality of generated text along many possible axes. For simpler systems, human evaluation may be practical, but in complicated systems like conversational chatbots, the amount of generated text can overwhelm human annotation resources. Model developers have begun to rely heavily on auto-evaluation, where LLMs are also used to judge generation quality. However, existing LLM-as-a-judge benchmarks largely focus on simple Q\&A tasks that do not match the complexity of multi-turn conversations. We introduce RankJudge, a benchmark generator for evaluating LLM-as-a-judge on multi-turn conversations grounded in reference documents. RankJudge creates pairs of conversations where one conversation has a single flaw injected into one turn. This construction allows paired conversations to be labeled unambiguously as better or worse, and precisely isolates failure categories to individual turns, enabling a strict joint correctness criterion for judging. We implement RankJudge across the domains of machine learning, biomedicine, and finance, evaluate 21 frontier LLM judges, and rank those judges via the Bradley-Terry model. Our formulation also allows ranking each conversation pair with difficulty ratings, which we use to dynamically curate the evaluation slice to reduce label noise, as confirmed via human annotation. We find that judge rankings are stable under partial observability, coarser correctness criteria, and an alternative random-walk rating algorithm.

1 Introduction

RankJudge addresses gaps in LLM-judge evaluation by generating multi-turn, reference-grounded conversation pairs with controlled single-turn flaws. Its synthetic construction and joint verdict-turn-type scoring support scalable, diagnostic evaluation across domains and judge families.

  • Existing judge benchmarks often score isolated single-turn responses rather than multi-turn, reference-grounded dialogues.
  • RankJudge generates paired conversations sharing reference documents, with one targeted assistant failure injected into exactly one turn of the worse branch.
  • The generator uniquely determines the better conversation, flawed turn, and failure category before evaluation, enabling strict joint correctness.
  • A fully synthetic pipeline uses automated verification and can regenerate benchmark pools without per-item human labels.
  • Across Machine Learning, Biomedicine, and Finance, RankJudge evaluates 21 judges and produces stable rankings spanning proprietary and open-weight families.

2 Related Work

Prior work shows that single-turn evaluation misses interactive dynamics and that scalar judging lacks error localization. RankJudge extends judge meta-evaluation with controlled, document-grounded, multi-turn benchmark generation and diagnostic labels.

  • Multi-turn LLM Benchmarks: Multi-turn benchmarks target user-model-environment dynamics that single-turn benchmarks miss, while single-turn ability does not necessarily transfer to multi-turn success.
  • LLM-as-a-Judge: LLM-as-a-judge methods offer scalable evaluation, but scalar reward models provide limited ability to localize or categorize specific errors.
  • Benchmarking LLM-as-a-Judge: Earlier judge meta-evaluation benchmarks include pairwise, perturbation-based, factuality, correctness, and ranking approaches, often centered on single-turn settings.
  • Benchmarking LLM-as-a-Judge: RankJudge is presented as the first automated pipeline combining multi-turn evaluation, external-document grounding, controlled user behavior, and injected error types.

3 Methodology

RankJudge constructs synthetic, reference-grounded conversation pairs with one targeted flaw localized to a single turn, enabling joint evaluation of verdict, turn, and failure type. It verifies and ranks these pairs and judges, then curates difficulty to reduce label noise.

  • Benchmark representation: Each benchmark item pairs conversations grounded in the same reference documents, with one conversation containing a single injected failure at a specified turn and category.The ground-truth tuple identifies the better conversation, flawed turn, and failure type.
  • Joint correctness: The joint correctness criterion credits a judge only when it identifies the better conversation, flawed turn, and failure category exactly.Choosing the correct conversation while mislocalizing or misclassifying the flaw is not credited.
  • Failure taxonomy: The taxonomy focuses on multi-turn assistant failures, excluding coarse single-turn categories when more specific multi-turn manifestations capture them.For example, later contradiction is categorized as self_contradiction rather than generic hallucination.
  • Conversation generation: Pairs are generated under independent assistant-failure and user-behavior conditions, while the better branch is staged to face the same relevant conversational pressure benignly.This structural matching prevents judges from relying on topic or question-type shortcuts instead of assessing the failure.
  • Automated verification: A three-layer automated cascade checks coherence, adherence, and grounding before retaining a synthetic pair.The pipeline checks the blueprint, conversation adherence, and support for atomic claims.
  • Ranking and curation: Bradley–Terry ratings jointly score judges and conversation-pair difficulty, allowing partial-observability ranking and difficulty-based curation.The published slice drops the hardest tail because human audits found greater ambiguity and label noise there.

4 Experiments

Across three domains, RANKJUDGE evaluates 21 judges using Elo, compute-efficiency, failure-type bias, robustness, and prompt-intervention analyses. The experiments show wide capability differences, systematic bias in weaker judges, stable rankings under several perturbations, and limited gains from prompt revisions.

  • Experimental Settings: RANKJUDGE instantiates standalone benchmarks in machine learning, biomedicine, and finance, evaluating 21 proprietary and open-weight judges.The same generation, verification, and ranking components are applied across the three knowledge-intensive domains.
  • RankJudge Leaderboard: The leaderboard spans nearly 1200 Elo points, with gemini-3.1-pro leading at 1959 and gpt-oss-20b at 762.Gemini-3.1-pro leads gpt-5.5 by 92 Elo points, while per-domain rankings broadly preserve the overall order.
  • Elo vs. Compute: The compute frontier contains opus-4.7, gpt-5.5, and gemini-3.1-pro on tokens, while open-weight judges occupy most of the cost frontier.Mid-Elo judges using 7–10k tokens per match sit well off the token frontier, showing that verbosity alone does not buy ranking accuracy.
  • Class Bias: As judge strength declines, evasion over-prediction rises to +41.6pp, while no_clarification and unnecessary_refusal are under-predicted by roughly −15pp and −18pp.The strongest judge’s class errors remain within 2pp of zero across all classes.
  • Failure-type Criterion: Including failure-type correctness yields high ranking agreement with the coarser criterion, with Spearman ρ = 0.968 and Kendall τ = 0.895, while widening Elo separation.The taxonomy further tests whether judges understand the reasons for failure.
  • Error Analysis and Mitigation: None of four opus-4.7 prompt revisions reaches opus-4.6 or sonnet-4.6 accuracy, suggesting a capability ceiling rather than a prompting deficiency.The revisions target category hints, candidate comparison, decision rules, and chain-of-thought.
  • Robustness and Benchmark Cost: Spearman ρ exceeds 0.95 with 10% of judgment results, and Kendall τ exceeds 0.95 at 50%, enabling substantially cheaper ranking with negligible distortion.The ordering also remains stable when noisy top-Elo pairs are included and when BT-Elo is replaced by EIP.

5 Conclusion

RANKJUDGE evaluates LLM judges on synthetic, multi-turn, reference-grounded conversation pairs with unambiguous flaws and strict joint correctness. Across 21 judges and three domains, it produces a broad, robust leaderboard while exposing systematic failure-type bias in weaker judges.

  • Conclusion: RANKJUDGE constructs synthetic pairs with an unambiguously worse conversation, enabling strict joint correctness under automated verification and Bradley–Terry Elo curation.Human auditing and a held-out fine-tuning experiment cross-check verification and label quality.
  • Conclusion: The three-domain leaderboard spans nearly 1200 Elo points, surfaces open-weight judges above proprietary models, and exposes sharp failure-type bias in weaker judges.These findings cover 21 frontier judges across machine learning, biomedicine, and finance.
  • Conclusion: Judge rankings remain stable across curation setups, alternative rating with EIP, and subsampling that reduces benchmarking cost.The conclusion reports robustness under multiple evaluation perturbations.

A.1 Detailed Judge Elo Ratings

Detailed Elo ratings provide combined and per-domain results with cluster-robust confidence intervals. Difficulty varies substantially by assistant failure type, whereas user behavior has comparatively little effect.

  • Elo Ratings: Table 4 reports each judge’s combined Elo and per-domain breakdown with cluster-robust 95% confidence intervals.Intervals tighten near the leaderboard middle and widen at the extremes because contested matches provide more discriminative information.
  • Per-domain Results: Finance is the lowest-Elo domain for most mid-leaderboard judges, consistent with the heavier numerical reasoning load of 10-K filings.gpt-5.4 is the sole judge whose Finance Elo exceeds its machine-learning and biomedical scores.
  • Pair Difficulty: No_clarification and disorganized pairs are hardest, with median Elos of 1525 and 1475, while unnecessary_refusal and fabricated_answer are easiest at 849.Refusals stand out in context, and fabrications are directly checkable against the grounding document.
  • Pair Difficulty: User-behavior categories span only 979 to 1247 in median Elo and have heavily overlapping interquartile ranges.The passage contrasts this narrow variation with the clearer separation produced by assistant failure types.

A.3 Model Bias and Self-Preference

A generator swap reveals strong self-preference for the generator models but nearly preserves the ordering of the other judges. Removing the two generators further increases cross-condition rank agreement.

  • Setup: The parallel experiment replaces gpt-5.5 with opus-4.7 while holding documents, prompts, verification gates, and the 21-judge panel fixed.The parallel set contains 131 verified pairs from 150 candidates.
  • Findings: gpt-5.5 falls from rank 2 to rank 7 when opus-4.7 generates the pairs, while opus-4.7 rises from rank 11 to rank 3.This pattern exposes self-preference in the generator models.
  • Findings: The other 19 judges are ordered almost identically across generators, with gemini-3.1-pro ranked first under both conditions.Gemini-3.1-pro does not generate either evaluation set.
  • Findings: After removing the two generators, rank agreement rises from Spearman ρ = 0.917 to 0.979 and Kendall τ from 0.810 to 0.906.The authors localize the bias to whichever model produces the conversations.

A.4 Filtering Cascade

The filtering cascade retains 652 of 1,200 generated pairs, with survival determined mainly by the injected assistant failure type. Disorganized is the hardest flaw to verify because its structural defect must remain subtle while preserving all facts.

  • 652 of 1,200 generated pairs survive every verification layer, yielding 54.3% overall retention.
  • Disorganized loses 130 of 174 generated pairs at adherence verification and appears in only 30 published pairs.Its survival rate is 17.2%, the lowest among failure types.
  • Unnecessary_refusal and fabricated_answer pass adherence verification without losses, unlike disorganized.
  • Domain and user-behavior survival rates remain comparatively narrow, so injected failure type sets the cascade shape.Rates span 48.5–60.0% across domains and 46.4–61.8% across user behaviors.
  • Disorganized requires reshuffling correct facts without dropping or distorting them, creating a narrow band where verification accepts and judges still disagree.Too mild a reshuffle fails verification; too severe a reshuffle becomes fabrication or evasion.
  • The example pair preserves the same three corrections in both responses, differing only in their ordering.The verifier accepts the pair because the lead answer is buried mid-paragraph while all facts remain present.
  • Weak judges systematically relabel diverse failures as evasion, while evasion over-prediction and no_clarification under-prediction recur across domains.The bias pattern grows as Elo decreases, and weak judges show a single evasion dump bin.

A.7 Failure-type Correctness by Domain

Dropping the failure-type label from correctness widens Elo differences within every domain but does not reorder local judge leaderboards.

  • Spearman ρ ranges from 0.969 in ML to 0.993 in Finance when comparing full and reduced correctness rankings.
  • Kendall τ ranges from 0.885 to 0.959 across domains under the same comparison.
  • The seven-way failure taxonomy widens the Elo range within each domain without reordering its local leaderboard.

A.8 Subsample Stability by Domain

Judge rankings remain highly stable under uniform subsampling within each domain, although Biomedical requires more observations to stabilize.

  • Spearman ρ exceeds 0.98 from the 50% subsample onward in all three domains.
  • Kendall τ exceeds 0.9 at the 50% fraction for Machine Learning and Finance.
  • Biomedical reaches ρ = 0.786 at 10% but joins the other domains by the 50% fraction.
  • Per-domain stability transfers the match-cost reduction, with Biomedical needing a marginally larger minimum sample.

A.10 Ranking Methods: BT vs. EIP

Bradley–Terry Elo and difficulty-aware EIP produce closely aligned judge rankings across slices, while pointwise and pairwise protocols agree less strongly because they measure different capabilities.

  • BT vs. EIP: Spearman ρ ranges from 0.944 to 0.953 between Bradley–Terry Elo and EIP across all four slices.
  • BT vs. EIP: Kendall τ ranges from 0.820 to 0.852, with top-10 overlap of 80% to 90% between the two ranking methods.
  • Pointwise vs. pairwise: Pairwise judging tests discrimination between conversations, whereas pointwise judging tests absolute quality calibration without a reference.
  • Pointwise vs. pairwise: Pointwise and pairwise rankings correlate at Spearman ρ = 0.809 and Kendall τ = 0.660, but their judge orderings differ.

B Reproducibility

RankJudge combines synthetic pair generation, automated verification, and judge-ranking procedures to evaluate multi-turn conversation judges reproducibly. Its prompts define failure categories, benign behavior, and strict joint correctness across verdict, flawed turn, and failure type.

  • Bradley–Terry Elo Details: Bradley-Terry estimates judge strengths from pairwise correctness outcomes, with MM updates, normalization, convergence checks, and Elo conversion using R0 = 1500.The implementation reports Ri ± CI95(i) and uses clustered sandwich errors because outcomes from the same conversation pair are correlated.
  • Bradley–Terry Elo Details: Cluster-robust variance groups residuals by conversation pair because ambiguous pairs can systematically affect many judges.The score vectors accumulate match residuals within each pair cluster before forming the sandwich estimator.
  • Empirical Interaction Propagation: EIP models judge quality and pair difficulty as a PageRank-style random walk over judges and conversation pairs.The walk links pairs to judges that solve them and judges to pairs they fail, concentrating mass on judges solving hard pairs and pairs fooling strong judges.
  • Empirical Interaction Propagation: The EIP implementation uses damping, teleportation, renormalization, convergence limits, and min–max rescaling to [0, 100].A degenerate convention assigns scoreEIP(i) = 50 when the stationary-vector range is below 10^-12.
  • Failure Taxonomy: The discovery prompt labels assistant and user behaviors, identifies taxonomy gaps, proposes evidence-grounded refinements, and generates downstream insights.The taxonomy definitions include assistant behaviors and user behaviors, while the guidelines require precise, data-grounded descriptions.
  • Failure Taxonomy: The failure specifications distinguish subtle self-contradiction from benign consistency, while requiring non-flawed turns to remain clean.Self-contradiction may shift qualifiers or implications without lexical reversal; missed clarification requires genuine ambiguity whose resolution changes the answer.
  • Judge Prompt Variants: Judge guidance distinguishes no_clarification from evasion and treats contradictions in tangential claims as self_contradiction rather than evasion.The no_clarification criterion requires both genuine ambiguity and an answer that would change with its resolution.

C.2 Annotation Tool

The annotation tool consolidates pair metadata, plans, conversations, verification, and judge outputs, then guides auditors through a targeted noisy-slice review. Auditors assign clean, ambiguous, or noise labels using explicit checks of the injected flaw and competing errors.

  • Interface: The Streamlit interface presents filters and tabs for overview metadata, plans, conversations, verification, and the 21 judges’ verdicts and reasoning.The Conversations tab opens and tags the injected-flaw turn, while the Judges tab shows each judge’s verdict, selected turn, behavior type, and reasoning.
  • Audit Procedure: Annotators inspect 20 candidate noisy pairs rather than auditing every pair end-to-end.The slice is surfaced by the difficulty-curation procedure and divided across two parts.
  • Audit Procedure: Judge disagreement prioritizes pairs that nearly all judges miss or split close to 50/50, while consensus pairs receive a quick skim.Auditors then verify that the declared bad_round_index realizes the planned weakness and that the flaw category is neither different nor milder.
  • Audit Procedure: When strong judges miss while weaker judges succeed, auditors inspect the strong judges’ suspected competing flaw in the better conversation.The specified strong judges are gemini-3.1-pro, gpt-5.5, and opus-4.7; the example weaker judges are gpt-oss-20b and gemma-4-31b.
  • Audit Procedure: Fact-checking a suspected competing flaw uses the disputed turn, matching source span, and preceding user question, followed by manual source verification.This is described as the most expensive audit step.
  • Disposition Rubric: The rubric labels pairs clean, ambiguous, or noise according to whether the intended single flaw is isolated, multiple types fit, or any intended label component deviates.Noise includes a flawed better conversation, no clear flaw in the worse conversation, or a flaw in the wrong turn.
  • Examples: A full audited example exposes the joint correctness criterion, while a second example summarizes how it partitions all 21 judges.The examples are labeled clean by the audit.

D.1 Full Example: Unnecessary Refusal in ML

This example contrasts two ML-grounded conversations that differ in turn 2, where one assistant unnecessarily refuses an answerable question about discriminator flags. The judge must identify the better conversation, flawed turn, and unnecessary-refusal type jointly.

  • Turn 2: Turn 2 concerns whether the flag variables v_i,m and u_j,m are learned; the correct answer is that k-max discriminator scores set them and they are not backpropagated.The flags select which discriminators handle each sample, while the balance loss prevents selection from collapsing onto one discriminator.
  • Turn 3: Turn 3 explains that GMAN is the closest comparison for testing discriminator specialization, using a matched DCGAN setup alongside broader mode-collapse comparisons.The comparison includes GMAN variants and methods such as Inclusive GAN.
  • Turn 1: The pair tests whether MCL-GAN was evaluated beyond small settings, including unconditional generation, image-to-image translation, and text-to-image synthesis.The coverage includes high-resolution conditional synthesis and comparisons with multiple baselines, including a state-of-the-art model.
  • Turn 2: Conversation A makes an unnecessary refusal by saying it cannot determine whether the flags participate in backpropagation, although the reference explicitly answers that question.Its explanation of k-max selection is otherwise correct, but the unsupported uncertainty creates the injected flaw.
  • Joint evaluation: The joint criterion requires judges to get the better conversation, flawed turn, and failure type all correct, exposing judges that identify the verdict but miss the type.In the example, all five frontier judges identify unnecessary_refusal, while the weakest judge mislabels it as evasion despite choosing the correct conversation and turn.
Loading 2605.21748v1…