PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing

Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, Zhao Chen

arXiv:2610.38076v1cs.CLcs.AI

TL;DR

Research has largely optimized LLMaJ prompts without establishing whether semantically equivalent rephrasings yield consistent judgments. The paper studies this Prompt Lottery across evaluation settings and reports that prompt variation can destabilize judgments, while multi-prompt aggregation improves robustness.

  • Problem

    Although LLMaJ research has focused on prompt optimization, less attention has been paid to whether semantically equivalent evaluation instructions yield consistent judgments.

    Source: §1 · p. 2

  • Method

    The paper evaluates paraphrastic prompt variation across judge settings and introduces selective multi-prompt aggregation to improve robustness.

    Source: §1 · p. 2

  • Results

    Prompt rephrasings altered LLMaJ judgments, and selective aggregation achieved 86% of full robustness while reducing inference costs by 60% versus naive aggregation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors argue that LLMaJ evaluation should move beyond single prompts toward systematic multi-prompt assessment.

    Source: §6 · p. 10

    The study tests semantic-preserving variations, not deliberate semantic modifications, and validates equivalence with similarity and human judgments rather than formal logical entailment.

    Source: §6 · p. 9

Abstract

from arXiv · show

LLM-as-a-Judge has become a standard for evaluating large language models, yet most approaches implicitly assume that a single judge prompt functions as a stable evaluation instrument. Prior work has primarily explored deliberate prompt modifications such as scoring rubrics, role descriptions, or chain-of-thought reasoning in order to optimize accuracy. In contrast, we investigate whether validated prompt variants intended to preserve evaluation semantics while differing in surface realization (\textit{paraphrastic rephrasings}) yield consistent judgments under a fixed evaluation task. We introduce \textit{Prompt Lottery}, a phenomenon in which paraphrastic rephrasings produce different LLMaJ judgments. Across a large-scale study spanning diverse evaluation tasks, judge families, and model backbones, we find that these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings. A user study with 10 human annotators shows that LLMaJ judges exhibit 6-9$\times$ larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies. This indicates that single-prompt evaluation is insufficiently robust for model ranking. Finally, we show that multi-prompt aggregation substantially improves robustness by emphasizing judgments that remain consistent across prompt variants.

1 Introduction

LLM-as-a-Judge is widely used, but research has paid less attention to the judge prompt’s initial wording. This paper studies whether meaning-preserving rephrasings change judgments and whether aggregation can improve robustness.

1 Introduction

  • The paper identifies a gap in LLMaJ research: attention has focused on optimizing judgments, with less attention to the formulation of the core evaluation prompt.It distinguishes paraphrastic variation from prompt enhancements that alter task instructions.Source: §1 · pp. 1–2
  • Prompt Lottery names the phenomenon in which prompts intended as semantically equivalent produce systematically different, practitioner-unpredictable evaluation judgments.The introduction illustrates rephrasing with questions about whether a response is understandable or clear and easy to comprehend.Source: §1 · p. 2
  • Prompt sensitivity is presented as a robustness concern because an equally valid rephrasing can reverse a model comparison made using one prompt.The paper calls consistency across meaning-preserving variants paraphrastic robustness.Source: §1 · p. 2
  • The study reports greater judgment variance under prompt changes than in human judgments, and presents selective multi-prompt aggregation as a way to restore reliability.The introduction reports 86% of full robustness with 60% lower inference cost than naive aggregation.Source: §1 · p. 2

2 Related Work

Prior work develops LLMaJ protocols, human alignment, and bias mitigation, while examining prompt modifications and aggregation. This paper focuses on stability under paraphrastic variation rather than optimization or formatting perturbations.

2 Related Work

  • LLMaJ research includes reference-based and reference-free evaluation, pairwise ranking or rubric scoring, and newer agentic or multi-step frameworks.Surveys organize the field around target attributes, evaluation protocols, alignment with human judges, and bias analysis.Source: §2 · pp. 2–3
  • Prior prompt interventions include judge personas, score delimiters, and chain-of-thought reasoning, while aggregation work includes model ensembles and consistency checks.Other approaches use hierarchical tie-breaking or rotate models over a fixed prompt to address bias.Source: §2 · p. 3
  • The paper distinguishes paraphrastic variation from prior format perturbations and synonym or adversarial variations, focusing on prompts intended to preserve semantic equivalence.It also contrasts its system-wide robustness focus with prior work that compares models under paraphrases.Source: §2 · p. 3

3 Experimental Setup

The experiments compare multiple judge families, tasks, datasets, backbones, and semantically equivalent prompt variants. The authors validate the variants and establish human and fixed-prompt agreement baselines for interpreting prompt-related instability.

3 Experimental Setup

  • The setup covers 33 configurations across 12 datasets and six judge families, using five backbones and five prompt variants for 825 experimental conditions.The study uses near-zero temperature (T = 10^-5) to focus on semantic consistency, while noting GPU kernel nondeterminism may remain.Source: §3 · p. 3
  • Judge families span rubric-based and specialized metrics, including binary and Likert scoring across objective and subjective evaluation tasks.The authors selected datasets with accessible code and prompt templates to preserve original conditions and avoid manual reconstruction confounds.Source: §3 · p. 3

3.1 Paraphrastic Prompt Generation and Validation

  • Four generated variants were human-reviewed and syntax-checked; their average sBERT similarity was 0.93 despite lexical overlap of 0.38 ± 0.01.Three LLMs rated prompt similarity at 81%, 84%, and 79%, respectively; human task-equivalence ratings averaged 76/100, with Cohen’s κ = 0.78 and Krippendorff’s α = 0.81.Source: §3.1 · p. 4

3.2 Annotator Alignment Study and Expected Agreement Levels

  • In the user study, 10 annotators rated 650 data points, yielding average human agreement of 0.78 Overlap, 0.71 pairwise Cohen’s κ, 0.80 Fleiss’ κ, and 0.63 Krippendorff’s α.Each example was rated by at least three annotators, and annotators did not rate the same item under different instructions back-to-back.Source: §3.2 · p. 4
  • Fixed-prompt repeated runs established Cohen’s κ = 0.86 and Fleiss’ κ = 0.89, while the paper set κ = 0.80 as the human robustness baseline and κ > 0.61 as the usability threshold.At T = 0.7, fixed-prompt runs had lower Fleiss’ κ = 0.39.Source: §3.2 · p. 4

4 Experiments

The experiments quantify prompt-induced judgment instability, examine its sources and effects on evaluation quality, and test mitigation strategies. They find that instability varies across tasks and models, no prompt variant is consistently best, and rankings can change with prompt wording.

4.1 Agreement Collapses Under Prompt Variation

4.2 Instability Depends on Task Type and Backbone

  • Binary tasks had higher raw overlap than non-binary tasks, but lower chance-corrected reliability; objective judges were more stable than subjective and open-ended judges.Sensitivity varied by backbone, and the reported comparisons also found Prompt Lottery across older and frontier models.Source: §4.2 · pp. 5–6

4.3 No Prompt Wins Consistently—It’s a Lottery

  • Prompt similarity did not predict stability, and no variant consistently won; paraphrases matched or outperformed the default prompt, with small human-correlation differences.The similarity–stability correlation was r = −0.11 (p = 0.157); human-alignment differences were ∆r = 0.00–0.02, with only two of three comparisons significant.Source: §4.3 · p. 6, Table 4 · p. 6

4.4 Do Prompts Reverse Model Rankings?

  • Ranking effects depended on task: SQuAD v2 had τ = 0.176 and 43.3% rank changes, while MTBench had 0% reversals.On SQuAD v2, GPT-4o ranged from 1st to 3rd and Qwen3p6-Plus from 1st to 4th across prompt variants.Source: §4.4 · pp. 6–7, Figure 2 · p. 7, Table 5 · p. 7

5 Mitigating via Aggregation

Prompt aggregation improves agreement with human judgments, and prompt diversity contributes more to reliability than model diversity alone. Selective aggregation uses a classifier to reduce calls while preserving most full-aggregation decisions, with cost estimates based on observed convergence times.

5 Mitigating via Aggregation

  • Plurality aggregation consistently achieves the strongest human alignment among the compared strategies.The comparison uses Pearson correlation with human scores across three datasets with public annotations.Source: §5 · p. 7, Table 6 · p. 8

5.1 What Source of Aggregation Matters?

  • Inter-prompt aggregation achieves stronger reliability than inter-model or hybrid aggregation, suggesting prompt variation contributes more than model diversity alone.The comparison varies models with fixed prompts, prompts with fixed models, or both.Source: §5.1 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

5.2 Selective Aggregation: Task-Dependent Cost Savings

  • The default prompt matches plurality in 94.7% of binary cases but only 44.0% overall; strict plurality is reached in 58.73% of cases.For non-binary tasks, 37.68% reach plurality by the third judge, 21.05% require all five, and 41.27% never converge.Source: §5.2 · p. 8
  • DEBERTA-V3 achieves TPR 85.06% and TNR 69.42%, with TNR stable at 66.22% on a held-out dataset with unseen Prometheus prompts.The classifier was selected to maximize TNR because false negatives can corrupt evaluation quality, whereas false positives incur inference cost.Source: §5.2 · p. 8
  • Approximately 60% fewer LLM calls than naive five-prompt aggregation are achieved by DEBERTA-gating, while recovering 86% of full-aggregation decisions.Performance falls to 65% in difficult no-consensus cases; full aggregation achieves 79% in those worst-case settings.Source: §5.2 · pp. 8–9

5.2.1 Derivation of LLM Call Reduction Under Selective Aggregation

  • 37.68% of non-binary cases reach plurality by three calls, 21.05% by five calls, and 41.27% require more than five calls; the classifier TNR is 69.42%.The derivation compares the gated strategy with full aggregation at 4.247 calls and the naive five-prompt baseline, for which the estimated reduction is approximately 60%.Source: §5.2 · p. 8, §5.2.1 · p. 9

6 Limitations and Final Remarks

The paper concludes that prompt variation can change scores and model rankings, while plurality aggregation improves human alignment and selective aggregation reduces evaluation cost. Its evidence is limited to meaning-preserving paraphrases and lacks budget-matched or orthogonal robustness comparisons.

6 Limitations and Final Remarks

  • On stable instances, DEBERTA reduces LLM calls from 5 to 1.99, while plurality aggregation improves agreement with human judgments over single-prompt evaluation.The authors frame systematic multi-prompt assessment as a methodological direction for robust LLMaJ evaluation and reliable comparative ranking.Source: §6 · p. 9
  • Prompt sensitivity can reverse relative model rankings, and the authors describe it as a departure from semantic invariance that makes evaluation brittle.They recommend treating prompt sensitivity as a first-class concern in evaluation methodology.Source: §6 · pp. 9–10
  • The study tests paraphrases that preserve semantic content, not deliberate semantic changes, and validates equivalence with sBERT similarity and human task-equivalence ratings rather than formal entailment.The reported validation measures are sBERT similarity 0.93 and human task equivalence κ = 0.78.Source: §6 · p. 9
  • The main experiments cover five model families with limited reasoning-augmented models, and mitigation is not compared with budget-matched or orthogonal robustness methods.The authors note that Section 5 gains may partly reflect repeated judging rather than prompt diversity itself.Source: §6 · p. 10

A Mapping of Judge Families to Dataset and Tasks

Table 10 maps 33 experimental configurations by dataset and judge family, with five back-models and five prompt variants per configuration.

A Mapping of Judge Families to Dataset and Tasks

  • Each of 33 configurations pairs a dataset with a judge family and evaluates 25 LLMaJ triples from five back-models and five prompt variants.The table also marks whether a judge is binary or uses a Likert-scale score.Source: Table 10 · p. 14
  • The table uses DQ for Dialogue Quality, GQ for General Quality, QC for QA Correctness, QG for QA Groundedness, and SA for Semantic Alignment.Source: Table 10 · p. 14

B Dimensions of Paraphrastic Variation

The paper identifies recurring paraphrastic dimensions post hoc and tests whether they are associated with cross-model score agreement. None of the coded dimensions consistently identifies wording that raises agreement.

B Dimensions of Paraphrastic Variation

  • Six representative paraphrastic dimensions were identified by post hoc inspection of generated variants rather than imposed as generation constraints.The analysis describes role, syntax, criteria layout, length, verb choice, brevity, and output-format strictness.Source: B Dimensions of Paraphrastic Variation · p. 15
  • Role, syntax, criteria layout, length, verb choice, and brevity matched the permutation null, and none of the six dimensions consistently raised cross-model score agreement.Their eta-squared values of 0.26–0.48 were comparable to the chance baseline for grouping five paraphrases.Source: B Dimensions of Paraphrastic Variation · p. 15, Table 12 · p. 15
  • Output-format strictness had a univariate slope of +0.076 κ per step, but the effect weakened to +0.031 with other features fixed and was not significant (p = 0.17).The MT-Bench family designed to span those phrases had a +0.005 slope.Source: B Dimensions of Paraphrastic Variation · p. 15, Table 12 · p. 15

C Semantic Invariance in Language Understanding: Theoretical Foundations

Semantic invariance motivates expecting equivalent judgments from semantically equivalent prompts, but pragmatic cues can make that expectation fail. The section grounds invariance in three theoretical commitments and describes how implicature, salience, and register may induce prompt sensitivity.

C.1 Theoretical Basis for Semantic Invariance

  • Compositionality holds that complex-expression meaning depends on component meanings and combination rules, so equivalent prompts should yield equivalent results.The section contrasts “Is the response understandable?” with “Is the response clear and easy to comprehend?”Source: §C.1 · p. 16
  • Representation independence holds that semantic content is independent of its encoding, supporting equivalent outputs for different representations of the same content.The account connects this principle to Montague’s formalization, where truth conditions are computed independently of surface form.Source: §C.1 · p. 16
  • Rational agent theory says outputs should depend on beliefs and goals rather than their representation, making representation invariance a sign of rationality and generalization.Source: §C.1 · p. 16

C.2 Why Semantic Invariance Might Fail: Pragmatics

  • Pragmatic cues may undermine invariance: elaboration can imply non-obviousness, explicit scale descriptions increase salience, and register conveys authority or formality.These cues may lead language models to diverge across prompts despite equivalent semantic content.Source: §C.2 · p. 16

C.3 Reconciling Theory and Empirics

The paper reports that paraphrastic prompts sharply reduce judge agreement despite high semantic similarity, with objective tasks showing more invariance than subjective ones. It interprets the pattern through pragmatic sensitivity and proposes aggregation across variants to address it.

C.3 Reconciling Theory and Empirics

  • Agreement falls from κ = 0.89 with a fixed prompt to κ = 0.24 across paraphrastic variants despite semantic similarity above 0.93.Source: §C.3 · p. 16
  • Objective tasks show greater invariance (κ = 0.47–0.69) than subjective tasks (κ = 0.26–0.28), consistent with ambiguity amplifying pragmatic inference.Source: §C.3 · p. 16
  • Multi-prompt aggregation averages pragmatic variants to filter pragmatic noise while preserving semantic signal, assuming rather than denying pragmatic sensitivity.Source: §C.3 · p. 17

D Agreement Across Models and Prompts

E Rank reversal experiments

Rank trajectories across paraphrastic judge prompts show that prompt variation can reorder evaluated systems, though the degree of instability differs by task. SQuAD v2 and SummEval exhibit reversals, while MTBench retains its ordering.

E Rank reversal experiments

F Experimental setup and hyperparameters

The experimental setup evaluates classifiers predicting whether a judge’s score falls in a five-judge plurality cluster, using instance-disjoint splits and specified shared and model-specific configurations. It includes non-fine-tuned embedding baselines and fine-tuned RoBERTa and DeBERTa variants.

F Experimental setup and hyperparameters

F.1 Optuna optimization

This section documents the DeBERTa-v3-large LoRA optimization and Llama 3.1 8B agreement-classifier prompts, alongside judge-prompt groups and templates across evaluation tasks and dimensions. It describes how prompts operationalize evaluation criteria and vary in framing, vocabulary, structure, and answer format.

F.1 Optuna optimization

  • Optuna tuned DeBERTa-v3-large LoRA for validation Matthews correlation coefficient and balanced accuracy over 35 two-epoch trials on one NVIDIA A100 80GB GPU, using an 80/20 stratified holdout split.Gradient accumulation, sequence length, and LoRA target modules were fixed at 8 steps, 512 tokens, and query/key/value projections.Source: §F.1 · pp. 20–21

F.2 Llama 3.1 8B classifier prompts

  • The Llama 3.1 8B classifier predicts whether a judge’s score matches at least three of five semantically similar judges, outputting category 0 or 1.Zero-shot and few-shot prompts share a template; queries are truncated to 3,500 characters and few-shot examples to 1,200 characters each.Source: §F.2 · p. 22

Organization and Navigation

  • The 15 groups contain 4–5 semantically equivalent prompt variants each, varying surface realization, and are organized across evaluation tasks and dimensions.Groups G1–G15 span semantic alignment, QA, response quality, dialogue, summarization, and context-aware or domain-specific evaluation; G15 is used only for gated aggregation.Source: Organization and Navigation · p. 23, Table 16 · p. 23

Quick Reference by Dimension

  • The reference guide notes variation in role framing and instruction syntax across groups, with output-format and contextualization variations in selected groups.It also identifies linguistic-register variation across all groups.Source: Quick Reference by Dimension · p. 23

G.1 Prompts

  • Prompt groups contain five prompts for the same scoring task that vary in framing, vocabulary, or layout, with placeholders identifying judges, groups, and inference-time inputs.The prompts use P1–P5 and G1–G15 labels; conversation, prediction, reference, and context variables are filled at inference time.Source: §G.1 · p. 24

P1 System prompt:

  • The continuation-matching prompts ask judges to rate how much a prediction conveys the reference’s information on a 1–5 scale, with semantic equivalence at the top of the scale.The examples vary descriptions of the dialogue, reference, prediction, score anchors, and required output format.Source: P1 System prompt · p. 24, P4 System prompt · p. 26, P5 System prompt · p. 26
  • The answer-correctness prompts compare a prediction with a ground-truth answer, emphasizing key information, numbers, dates, and contradictions on a 1–5 scale.The variants also permit extra information and instruct judges to treat possibilities as definitive answers.Source: P1 System prompt · p. 27, P2 System prompt · p. 27, P3 System prompt · p. 28
  • The groundedness prompts rate whether prediction details are present in or reasonably inferable from evidence, using a 1–5 scale.The examples variously frame the criterion as evidence support, logical derivation, or traceability.Source: P1 System prompt · p. 29, P2 System prompt · p. 29, P5 System prompt · p. 30
  • Other prompt examples classify response relevance to conversation context or ask whether a response is interesting, using yes/no answers for relevance.The relevance variants frame the decision as appropriate, logically connected, or on-topic; an interest example asks directly whether the response is interesting.Source: P1 System prompt · pp. 33, 35, P3 System prompt · p. 34
  • P1 templates assess conversational responses for engagement, understandability, and context specificity with Yes/No answers, and rate summary coherence, fluency, and context use on 1–5 scales.Source: P1 System prompt · pp. 36, 38, 40, 42, 47, 52

P3 System prompt:

  • P3 templates frame evaluators as precise judges or specialists and provide task-specific criteria and scoring instructions.Examples include binary assessments of conversational engagement or comprehensibility and 1–5 ratings of summary coherence, consistency, fluency, and relevance.Source: P3 System prompt · pp. 36, 38, 45, 48, P2 System prompt · p. 42

P4 System prompt:

  • P4 templates use expert-judge or comparison framing, with prompts for binary conversational judgments and 1–5 summary ratings.Summary prompts refer to a reference summary and ask for coherence, consistency, fluency, or relevance scores.Source: P4 System prompt · pp. 36, 37, 43, 46, 48, 51

P5 System prompt:

  • P5 templates cast evaluation as binary classification for conversational responses or as 1–5 classification of summary quality.The examples specify output labels or rating scales for engagement, understandability, context specificity, coherence, consistency, fluency, and relevance.Source: P5 System prompt · pp. 37, 39, 44, 46, 49

P2 System prompt:

  • P2 includes a prompt that frames context grounding as a fact-checking task.It asks whether answers are grounded in the given context.Source: P2 System prompt · p. 52
  • Prompt examples assess factual grounding and practical applicability by checking support, contradictions, actionable value, and real-world constraints.The examples use a 1–5 scale and request a single-number response.Source: P2 System prompt · pp. 52, 54

P3 System prompt:

  • Other prompts rate context comprehension and completeness, including whether answers use relevant context and address the question’s scope.The examples define 1–5 rating scales and require a single-number response.Source: P3 System prompt · pp. 53, 55

P4 System prompt:

  • A context-completeness prompt checks whether answers use all relevant context, synthesize multiple parts when needed, and avoid omissions.It rates completeness from 1 (misses most relevant context) to 5 (thoroughly uses all relevant context).Source: P4 System prompt · p. 53

P1 System prompt:

  • A technical-accuracy prompt evaluates terminology, scientific accuracy, methodological descriptions, and whether technical detail suits the question.It uses a 1–5 rating scale and requests no explanation or additional text.Source: P1 System prompt · pp. 53–54

Related papers

Loading 2610.38076v1…