PaperlayerView exact revision on arXiv

Source-linked AI summary

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Laurène Vaugrante, Thilo Hagendorff

arXiv:2610.05094v1cs.CLcs.AI

TL;DR

LLM-judge design lacks standardized protocols, despite the risk that design choices affect evaluations. This study compares designs across scalar ratings and binary accuracy classification, finding that performance is broadly stable while model identity and task subjectivity shape judge behavior and design sensitivity.

  • Problem

    LLM-judge protocols lack standardization, and systematic comparisons of prompt designs remain rare despite known sensitivity to judge design.

    Source: §1 · p. 2

  • Method

    The study compares judge designs across scalar sentiment and toxicity ratings and binary answer-accuracy classification, using human annotations as ground truth for scalar tasks.

    Source: §1 · p. 2

  • Results

    Most design choices have limited impact on performance, but model identity is the dominant source of variance and robustness decreases as task subjectivity increases.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors recommend aggregating results across two to three models and designs to mitigate model-specific biases.

    Source: §5 · p. 9

    The experiments cover two rating tasks, one accuracy-classification task, a limited set of design choices, and a small number of non-LLM classifiers.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (over 500 items per category), as well as a binary accuracy classification of question-answer pairs (n=600). For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are rarely impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.

1 INTRODUCTION

LLM judges are increasingly used for tasks requiring nuanced judgment, but evaluation protocols remain unstandardized and judge outputs may be sensitive to design. This study compares judge designs across scalar ratings and binary accuracy classification, finding broad reliability alongside meaningful behavioral differences.

1 INTRODUCTION

  • LLM judges are used because nuanced evaluation is costly to obtain from human annotators at scale and poorly captured by traditional metrics.Prior work validated LLM judges against human judgment on open-ended chat quality and other NLG tasks.Source: §1 · p. 1
  • There is no consensus on standardized evaluation protocols, and systematic comparisons of LLM-judge prompt designs remain rare.Prior work also documents position, verbosity, and self-preference biases, alongside sensitivity to prompt formulation.Source: §1 · p. 2
  • Across scalar ratings and binary accuracy classification, most design choices have limited impact on performance, but equally accurate judges can behave differently.Model identity is the dominant source of variance across both tasks, and no single model is consistently best.Source: §1 · p. 2

2 RELATED WORK

Prior research reports both strengths and weaknesses in LLM-judge reliability, while finding that prompts, evaluator models, and output formats can affect evaluation. This study addresses the limited within-study comparison of judge designs by varying prompts and scales across models and comparing scalar ratings with human annotations and classifiers.

2 RELATED WORK

  • Prior findings on judge reliability are mixed: some report lower agreement than humans, while others find human-level agreement in selected tasks.Surveys report stronger performance on some tasks and weaker performance on toxicity and safety judgments; judges may also rely on superficial cues.Source: §2 · p. 2
  • Prompt design and evaluator-model choice have been reported to affect robustness, while complex prompts can reduce accuracy and reference-guided grading can reduce failures on math and reasoning tasks.Prior work also leaves scalar-scoring reliability less explored than pairwise response comparisons.Source: §2 · p. 3
  • Output-format constraints and answer-extraction methods can affect performance, with effects depending on task; scoring granularity and range have also been linked to bias.Inconsistent evaluator choices may make results across papers incomparable.Source: §2 · p. 3
  • The study addresses limited within-study comparisons by varying prompt designs and scale formats across 10 model judges, comparing scalar ratings with human annotations and classifiers.The passage identifies prompt detail and evaluator-model choice as design choices often not varied together in prior work.Source: §2 · p. 3

3 METHODS

The methods evaluate LLM judges on scalar sentiment and toxicity ratings and binary answer-accuracy classification. They vary judge designs and compare scalar ratings with human ground truth, while defining separate metrics and ground-truth procedures for the two task types.

3.1 BENCHMARKS AND DATA

  • The benchmarks cover scalar sentiment and toxicity ratings and binary accuracy classification of question-answer pairs.The scalar items were generated in sets of 600 per dimension, with human ratings establishing ground truth; evaluation splits contain 501 sentiment and 521 toxicity items. The classification benchmark contains 600 question-answer pairs from Humanity’s Last Exam.Source: §3.1 · pp. 3–4

3.2 LLM-AS-A-JUDGE DESIGNS

  • Judge designs vary prompt detail, rating scale, model, and, for classification, reasoning effort.The rating scales are 1–7, -3–3, 0–1, and 0–100; prompt detail ranges from a minimal base prompt to longer in-context instructions, principles, and examples. Ten reasoning models are evaluated.Source: §3.2 · p. 4

3.3.1 SCALAR RATING

  • For scalar ratings, judges return JSON scores that are rescaled to 1–7 and compared with human ground truth; bias, mean absolute error, and rank-based measures characterize results.Popular classifiers are also rescaled for comparison, and rank-biserial correlation is used to express effect strength rather than relying on statistical significance alone.Source: §3.3.1 · pp. 4–5

3.3.2 ACCURACY CLASSIFICATION

  • For classification, judges return binary verdicts, and the study measures accuracy, leniency, and switch rate against a ground truth partly determined by judge consensus.Items are divided into full-agreement, near-agreement, and ambiguous subsets; chi-square and Cochran’s Q tests assess impacts alongside effect measures.Source: §3.3.2 · p. 5

4 RESULTS

The results compare rating judges with human annotations and classifiers, then examine how judge-design choices affect rating and classification outcomes. Across both tasks, aggregate performance is strong, but model identity and selected design choices can still shift judgments.

4.1.1 DO THE LLM JUDGES DISAGREE WITH HUMANS AND CLASSIFIERS?

  • Across 240 rating designs, judges averaged 0.11-point deviation from human ground truth, although 67% differed significantly and 7% deviated by over 0.3 points.Only 24% of designs showed a medium-to-large effect strength.Source: §4.1.1 · pp. 5–6
  • Toxicity judges were less consistent than Sentiment judges, but their absolute bias was smaller than that of Toxicity classifiers.83% of Toxicity judges exceeded 0.5 MAE, versus 3% of Sentiment judges; absolute bias was 0.16 for judges, compared with 1.03 for Detoxify and 0.82 for Perspective.Source: §4.1.1 · p. 6

4.1.2 DO THE LLM JUDGES DISAGREE WITH EACH OTHER?

  • Judges showed high pairwise agreement, yet arbitrary designs differed more than the impact of any single design choice in isolation.Pairwise agreement was α = 0.97 for Sentiment and 0.96 for Toxicity; mean differences were 0.09 and 0.22 points, respectively.Source: §4.1.2 · pp. 6, 8
  • Scale choice shifted Toxicity bias by up to 0.93 points, compared with an average impact of 0.19 for Toxicity and 0.06 for Sentiment.The centered −3–3 scale largely drove this instability; excluding it, Toxicity judges differed by 0.07 points on average.Source: §4.1.2 · p. 7
  • Model choice shifted rating bias by up to 0.39 points for Sentiment and 0.53 for Toxicity, and was the main source of classification variation.No model consistently ranked best across rating designs; in classification, model choice also changed leniency and accuracy, especially on ambiguous items.Source: §4.1.1 · p. 6, §4.1.2 · p. 7, §4.2 · p. 8

4.2 ACCURACY CLASSIFICATION BENCHMARK

  • 96.5% accuracy was achieved on average across 60 classification designs, although judges differed in leniency and error patterns.Accuracy ranged from 92.7% to 98.3%, and 429 question-answer pairs received full agreement across judges.Source: §4.2 · p. 8
  • Detailed prompts reduced classification leniency from 81.3% with the Base prompt to 52.4% with the Long In-Context prompt, while improving Toxicity agreement with humans.The classification leniency effect was directionally consistent across 9 of 10 models; for Toxicity, agreement rose from α = 0.910 to 0.941.Source: §4.1.1 · p. 6, §4.2 · p. 8
  • Lower reasoning effort did not reduce classification accuracy, and low-effort variants shared prompt sensitivities and accuracy with their default counterparts.Some models showed a slight increase in accuracy at lower reasoning effort.Source: §4.2 · p. 8

5 DISCUSSION

Across both tasks, most design choices have limited impact on performance, but judges can differ in behavior and model identity is the dominant source of variance. The evidence is bounded by the study’s task and design coverage, imperfect ground truth, and interpretive thresholds.

5 DISCUSSION

  • Most design choices have limited performance impact, but equally accurate judges can still differ in behavior, including leniency.The discussion recommends considering leniency or error direction in addition to accuracy.Source: §5 · p. 9
  • Model identity is the dominant source of variance across both tasks and is the primary driver of rating bias direction and classification error type.No single model is consistently best, and model rankings depend on design.Source: §5 · p. 9
  • Robustness decreases as task subjectivity or ambiguity increases, making design choices more consequential in those settings.The discussion contrasts Toxicity with Sentiment and ambiguous items with the full accuracy-classification benchmark.Source: §5 · p. 9

5.1 LIMITATIONS

  • The experiments cover two rating tasks, one accuracy-classification task, limited design choices, and few non-LLM classifiers; ground-truth uncertainty and interpretive thresholds further qualify the findings.Accuracy ground truth partly derives from judge consensus, and thresholds such as |bias| > 0.2 are descriptive aids rather than established reliability standards.Source: §5.1 · p. 9

A BENCHMARK GENERATION

The appendices describe how toxicity and sentiment benchmarks and human survey instructions were prepared, and provide representative prompts for different prompt-detail levels. The highest-toxicity items were excluded from the study because they could be harmful to annotators.

A BENCHMARK GENERATION

  • The toxicity-generation prompt spans five levels, but items at the highest levels were excluded because they were considered potentially harmful to annotators.Table 3 presents the prompt used to generate sentences with different toxicity levels.Source: A BENCHMARK GENERATION · p. 15, Table 3 · p. 15

B TOXICITY AND SENTIMENT SURVEYS: ANNOTATING INSTRUCTIONS

  • Survey instructions defined sentiment and toxicity and asked participants to rate each on a seven-point scale, from very negative to very positive or from harmless to very toxic.Participants were told to judge sentiment or perceived toxicity; toxicity instructions note that some sentences could contain offensive language.Source: B TOXICITY AND SENTIMENT SURVEYS:… · pp. 16–17

C PROMPT DETAIL

  • The appendix presents representative base, short in-context, and long in-context prompts, with the long classification prompt requesting a TRUE or FALSE verdict in a JSON object.Other rating-scale formats follow the same prompt structure, and the full prompts are available in the OSF repository.Source: C PROMPT DETAIL · pp. 18–19

D.1 LLM JUDGE PERFORMANCE

The appendix reports rating-task comparisons against human ground truth across scales and judge designs. Its tables cover effect and error measures, average performance, and an analysis of whether the -3 – 3 scale drives the results.

D.1 LLM JUDGE PERFORMANCE

  • Figure 3 compares mean absolute error and bias against human ground truth by response scale for Sentiment and Toxicity.Error bars indicate 95% confidence intervals.Source: Figure 3 · p. 20
  • Figure 4 ranks judges by Krippendorff’s α against human ground truth within each of twelve prompt-by-scale designs for Sentiment and Toxicity.The caption specifies the ranking scope but does not state a winning model.Source: Figure 4 · p. 21
  • Table 7 reports the share of judge designs exceeding effect-strength, absolute-bias, and MAE thresholds, while Table 8 summarizes average performance and bias ranges.Table 8 notes significance testing for average absolute bias; the supplied passages do not provide the tables’ cell values.Source: Table 7 · p. 22, Table 8 · p. 23
  • Table 9 compares Toxicity results with and without the -3 – 3 scale, including significance, absolute-bias threshold, and mean absolute bias measures.The caption identifies bold values as differences of at least 15 percentage points or 0.05 in mean absolute bias.Source: Table 9 · p. 24

D.2 REMOVING THE -3 – 3 TOXICITY SCALE OUTLIER

Removing the -3–3 Toxicity scale reduces overall bias but does not eliminate Toxicity’s disadvantage relative to Sentiment. Improvements vary across prompt lengths and models.

D.2 REMOVING THE -3 – 3 TOXICITY SCALE OUTLIER

  • 0.16 to 0.11 mean absolute bias after removing the -3–3 scale, while designs exceeding 0.2 bias fall from 32% to 18%; both remain above Sentiment.Sentiment’s corresponding mean absolute bias is 0.07, with 1% of designs exceeding 0.2 bias.Source: §D.2 · p. 24, Table 10 · p. 25
  • Base and Short In-Context improve after removal (45%→27%, 0.21→0.14; 35%→17%, 0.16→0.10), while Long changes little (15%→10%, 0.12→0.11).Long In-Context’s share of significantly biased designs rises slightly, from 72% to 77%.Source: §D.2 · p. 24
  • Most models improve after excluding the outlier scale, but Qwen 3.8 27B is the exception: its share of significantly biased designs rises to 100%, while mean bias remains 0.23.GLM-5.3 improves modestly: designs exceeding 0.2 absolute bias fall from 42% to 33%, and mean bias falls from 0.18 to 0.15.Source: §D.2 · p. 24

D.3 JUDGE AGGREGATION AS A MITIGATION STRATEGY

Three-judge aggregation reduces bias, with the largest and most reliable reductions from the Model (Free) strategy. Gains diminish as ensemble size grows, and model variety accounts for much of the benefit.

D.3 JUDGE AGGREGATION AS A MITIGATION STRATEGY

  • The four strategies vary model, scale, and prompt detail: Model (Free) varies all three, while Model (Fixed), Scale, and Prompt Detail isolate one choice each.Model (Free) allows prompt and scale to differ; the other strategies hold the remaining settings identical.Source: §D.3 · p. 26
  • The unrestricted three-configuration rule, which allowed repeated models, was excluded because it performed worse than Model (Free).Source: §D.3 · p. 26
  • Model (Free) triples cut mean bias 37.0% for Sentiment and 34.6% for Toxicity; Model (Fixed) cuts 27.9% and 18.6%, outperforming Scale and Prompt Detail.Figure 5 reports results for Sentiment, Toxicity, and Toxicity with the -3–3 outlier scale excluded.Source: §D.3 · p. 26, Figure 5 · p. 26
  • Scale variety’s 25.2% Toxicity reduction on the full benchmark falls to 4.6% when the -3–3 scale is excluded.The full-benchmark result is attributed to the known outlier behavior of that scale.Source: §D.3 · p. 26
  • Bias falls as ensemble size grows from two to five, with the largest gain from one to two judges and only small further benefit beyond three.Model (Free) consistently reaches lower bias than Model (Fixed) at the same group size.Source: §D.3 · p. 27, Figure 6 · p. 27

D.4 DESIGN CHOICE IMPACT

The section compares design-choice effects at two levels: varying one choice at a time and comparing the full set of designs. Average and maximum pairwise differences capture typical and extreme shifts.

D.4 DESIGN CHOICE IMPACT

  • Table 11 measures how often judges differ significantly or meaningfully when one design choice changes and others remain fixed.This isolates single-choice comparisons that might otherwise appear impactful only in a few settings.Source: §D.4 · p. 28
  • Table 12 compares every pair among all 120 designs, regardless of how many settings differ; its values are average and maximum absolute differences on the 1–7 scale.The table caption defines n as the number of pairwise comparisons used for each average.Source: §D.4 · p. 28, Table 12 · p. 29
  • Average pairwise difference captures the typical shift, while the maximum shows the most extreme observed difference from a single choice or unconstrained combination.Source: §D.4 · p. 28

E ACCURACY CLASSIFICATION TASK: EXTENDED RESULTS

The extended results organize accuracy-classification performance by benchmark subset and model, then report pairwise robustness scores across design choices.

E.1 LLM JUDGE PERFORMANCE

  • Tables 13–16 report accuracy, leniency, and error counts by benchmark subset and model, with directional error-switch rates for In-Context prompts relative to the base prompt.Table 13 splits results by subset; Tables 14–16 give per-model results for the full, near-agreement, and ambiguous subsets.Source: Table 13 · p. 30, Table 14 · p. 31, Table 15 · p. 32

E.2 DESIGN CHOICE IMPACT

  • Table 17 reports average and maximum pairwise impact for each design choice; for reasoning effort, both values coincide because it has only two settings.Average is the mean pairwise impact, and maximum is the largest observed pairwise impact for that design choice.Source: Table 17 · p. 34
Loading 2610.05094v1…