Source-linked AI summary
SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition
Dae Lee, Mihai Delgeanu, Adel Youssef
TL;DR
Holistic LLM-as-judge evaluation does not show which quality dimensions drive a preference or make disagreements auditable. SESSE decomposes judgments into automatically derived A/B/NA sub-questions and achieves near-parity with chain-of-thought holistic evaluation while remaining training-free, with interpretable per-criterion diagnostics.
Problem
Holistic A/B judgment and unstructured reasoning do not decompose quality into auditable dimensions or isolate the aspects driving preferences.
Method
SESSE mines criteria from judge error cases, clusters them into reusable A/B/NA sub-questions, and aggregates independent per-criterion votes into a final preference.
Results
SESSE achieves near-parity with chain-of-thought holistic evaluation and performs in the same range as fine-tuned specialist evaluators on RewardBench while remaining training-free.
Takeaways & Limitations
Per-criterion vote distributions provide interpretable diagnostic evidence that surfaces judge failure modes and benchmark quality issues left opaque by holistic scoring.
Takeaways & Limitations
A/B/NA decomposition cannot capture end-to-end execution evaluation such as code compilation or derivation correctness, limiting performance on reasoning and code subsets.
Abstract
from arXiv · showhide
LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions mined directly from the judge's own error cases; requiring no oracle responses, task-specific rubrics, or fine-tuning. On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist, while remaining fully training-free. Per-criterion vote evidence provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes unavailable from a single holistic output token.
1 Introduction
SESSE addresses the limitations of holistic LLM-as-judge evaluation by automatically decomposing judgments into reusable, auditable sub-questions mined from the judge’s own error cases. Its training-free pipeline provides independently verifiable per-criterion evidence without oracle responses, manual rubrics, or fine-tuning.
- Motivation: Holistic A/B judging offers no mechanism to decompose quality, identify preference-driving aspects, or detect systematic annotation noise.Even chain-of-thought reasoning remains unstructured free text produced in a single pass.
- Limitations of prior work: Criterion-based methods decompose quality using predefined dimensions but require manually specified criteria or human-designed rubrics.Examples include G-Eval, FActScore, UniEval, and CheckList.
- Method: SESSE mines evaluation criteria from dev-set error cases, clusters them into k groups, and generates a reusable A/B/NA sub-question bank.Independent per-criterion voting yields the final preference, while randomized ordering mitigates positional bias.
- Interpretability: Each SESSE sub-question can be independently re-checked by an annotator, another model, or the same judge, localizing disagreements without re-deriving the holistic judgment.This makes the audit trail useful for diagnosing which quality dimensions drove a preference.
- Contributions: SESSE is a fully automated, training-free five-stage pipeline that avoids oracle responses, task-specific rubrics, and fine-tuning while producing diagnostic evidence unavailable from holistic judgment.The introduction positions SESSE as competitive with fine-tuned specialist evaluators on RewardBench.
2 Method
SESSE uses a five-stage pipeline to mine evaluation criteria from the judge’s own errors, organize them into a diverse active bank, and apply only relevant criteria at inference. Its two-tier evaluation preserves concrete error-derived questions while excluding inapplicable votes.
- Error Collection and Sketch: SESSE collects holistic judge errors, then generates conditional evaluation criteria as sub-questions from the judge’s own failure distribution rather than manually specified dimensions.This contrasts with Prometheus and G-Eval, which require manually specified dimensions.
- Expand and Sort: Candidate sub-questions are embedded, clustered into k thematic groups, and diversity-ranked by greedy farthest-first traversal into an active bank of depth n ≤nmax.The ranked bank enforces criterion orthogonality and supports inference-time depth ablation.
- Summarize: One generalized protasis labels each cluster, while near-duplicate clusters are merged through LLM binary equivalence judgments before pooled sub-questions are re-selected.Merged clusters pool their sub-questions and select the top-n items.
- Evaluate: Tier 1 selects relevant clusters for each example, and Tier 2 answers their original error-derived questions, reducing cost while preserving instance-specific phrasing.Tier 1 skips inapplicable clusters before Tier 2 evaluation.
- Evaluate: When a criterion is inapplicable, the judge returns NA, and NA votes are excluded from the A/B tally.The passage illustrates this with a code cluster applied to a poetry prompt.
3 Experiments
Experiments evaluate SESSE on RewardBench using a dev/validation split and held-out pairwise preference accuracy. Results show model-dependent depth behavior and harder SESSE tie cases, especially for Gemini 2.5 Flash.
- Experimental setup: RewardBench contains 2,985 pairwise preference examples, with 1,985 dev examples for error collection and 1,000 held-out validation examples for reported accuracy.The dataset spans chat, chat-hard, safety, and reasoning subsets.
- Depth ablation: Depth saturates at n=2 for Gemini 2.5 Flash, peaks at n=5 for Flash Lite, and peaks at n=3 for Qwen2-VL-7B before degrading.The per-tier pattern indicates that optimal sub-question depth depends on judge capability.
- Depth ablation: Qwen2-VL-7B’s accuracy degrades past n=3 because 51.9% of instances are covered by a single dominant protasis, causing redundant sub-questions to consume the depth budget.The passage attributes the decline to bank quality rather than simply to increased depth.
- Tie-row analysis: Gemini 2.5 Flash holistic accuracy falls from 94.5% on non-tie rows to 69.5% on tie rows, indicating that tie cases are harder and may reflect genuine ambiguity.Tie-row accuracy is consistently lower than non-tie accuracy across all three evaluated models.
4 Discussion
SESSE preserves near-parity with holistic chain-of-thought judging while adding auditable per-criterion evidence without training. Its benefits vary by domain, and its vote trails expose judge reliability and failure modes for deployment and auditing.
- Training-free parity with fine-tuned judges: SESSE reaches near-parity with CoT holistic for Gemini 2.5 Flash while providing structured, auditable per-criterion evidence without training.CoT holistic already outperforms RISE-Judge, so SESSE preserves the base model’s advantage without fine-tuning.
- Domain-specific signal recovery: +20.6% relative gain on donotanswer improves Gemini 2.5 Flash from 75.9% →91.5% when explicit safety-refusal criteria isolate domain-specific signals.On xstest-should-respond, SESSE instead drops 27.1% relative because Tier-1 false positives on borderline-safe prompts make precision the key failure mode.
- When to prefer SESSE: SESSE is preferable when interpretability matters because per-criterion vote trails support annotation auditing and expose label noise despite added inference overhead.The audit trail provides a rationale for using SESSE beyond raw accuracy.
- Tie rate as a deployment-time capability signal: 21.3% Tie% for Qwen2-VL-7B versus 2.6–3.5% for Gemini models signals reliability differences and predicts its accuracy gap.Qwen2-VL-7B scores only 66.7% when its roughly one-in-five ties are forced into holistic decisions; high Tie% on a held-out probe can indicate capability mismatch or domain shift without labels.
- Failure modes: Per-criterion vote distributions diagnose capability-dependent failure modes and reveal that nearly half of Gemini degradation rows reflect stylistic rather than substantive quality differences.This makes SESSE a lightweight benchmark-audit tool independent of accuracy measurement and exposes information absent from monolithic A/B scoring.
5 Conclusion
SESSE is a training-free framework that decomposes holistic LLM-as-judge evaluation into automatically derived sub-questions mined from the judge’s own errors. On RewardBench, it approaches capable judges’ holistic CoT performance and matches the range of a fine-tuned specialist while providing interpretable diagnostic evidence.
- SESSE is a training-free framework for decomposing holistic LLM-as-judge evaluation.
- It automatically derives A/B/NA sub-questions from the judge’s own error distribution.
- On RewardBench, SESSE has a non-significant gap to CoT holistic for capable judges and performs in the same range as fine-tuned RISE-Judge 32B.The framework remains training-free while achieving this performance range.
- Per-criterion vote distributions provide interpretable diagnostic evidence for surfacing judge failure modes and benchmark quality.
Limitations
SESSE’s limitations concern portability, benchmark scope, inference cost, and the expressiveness of its decomposition. Its sub-questions cannot fully evaluate end-to-end execution, while reducing n can substantially lower cost without sacrificing accuracy for capable judges.
- Bank portability: Reusing the question bank across judge models or domains requires rerunning Stages 0–4, and amortization strategies are unexplored.The bank is derived from one judge model’s error distribution on one development set.
- Cluster count & benchmark scope: k=25 is fixed by silhouette analysis, while ablations over k and evaluation beyond RewardBench are deferred.Deferred settings include open-ended generation, multi-turn, and non-English evaluation.
- Inference cost: ∼282 LLM calls per example are required at n=10 versus 1 holistic call, but capable judges saturate at n=2 with no accuracy loss.Reducing n proportionally reduces inference cost.
- Decomposition limits: A/B/NA sub-questions cannot capture end-to-end execution evaluation, causing SESSE’s primary degradation on reasoning and code subsets.Examples include code compilation and derivation correctness.
Ethical Considerations
SESSE inherits biases from the LLM judge’s error distribution, so high-stakes deployments should audit its generated question bank for bias. The experiments use publicly available models and datasets.
- Bias and Audit: SESSE inherits any biases present in the LLM judge’s error distribution.Criteria mined from judge failures may encode systematic biases.
- Bias and Audit: High-stakes annotation pipelines should audit SESSE’s generated question bank for bias before use.The experiments use publicly available models and datasets.
A Depth Ablation
The depth ablation finds that accuracy stabilizes after only a few sub-questions per cluster, with little benefit from larger question budgets. Qwen2-VL-7B instead degrades slightly at larger depths because its self-generated bank has limited cluster diversity.
- A Depth Ablation: Accuracy stabilizes within the first few values of n, with little change beyond them across all three self-generated banks.Table A.1 sweeps n=1–10 sub-questions per cluster, indicating that a small question budget is sufficient once the most informative sub-questions are exhausted.
- A Depth Ablation: Qwen2-VL-7B shows slight degradation at larger n because its self-generated bank has limited cluster diversity.This is the exception to the general plateau pattern observed in the ablation.
B Cross-Model Bank Transfer
With the judge model fixed, replacing Qwen2-VL-7B’s self-generated question bank with one generated by Gemini 2.5 Flash does not change peak accuracy. Both banks achieve 66.3%.
- B Cross-Model Bank Transfer: 66.3% peak accuracy is achieved by both Qwen2-VL-7B’s self-generated bank and Gemini 2.5 Flash’s question bank with the judge model fixed.The comparison indicates no meaningful evaluation-performance change from substituting the stronger model’s questions.
C Failure Mode Taxonomy
SESSE’s degradation rows are analyzed only for high-accuracy configurations, where disagreements with the holistic baseline can be organized into three failure signatures. These signatures distinguish genuine ambiguity, confidently wrong decomposition, and semantic equivalence.
- Scope: The taxonomy covers degradation rows where SESSE and the holistic baseline disagree, restricted to configurations with SESSE accuracy above 80%.The analyzed configurations are Flash Lite and Gemini 2.5 Flash; lower-accuracy settings produce too many degradation rows for distinct signatures.
- Failure buckets: C1 (Ambiguous) captures genuine label difficulty when sub-question votes are evenly split.Table C.1 describes this category as genuine ambiguity with high vote entropy.
- Failure buckets: C2 (Decomposition failure) identifies confidently wrong judgments with ≥65% of non-NA votes opposing the ground-truth label.This bucket represents cases where SESSE is consistently wrong against the label.
- Failure buckets: C3 (Semantic equivalence) covers cases where high NA rates correctly indicate that responses are too similar to discriminate.The bank declines to pick a winner when the responses are semantically equivalent.