Source-linked AI summary
Should I Be Polite to My LLM Relevance Judge? Tone as a Severity Operating-Point Shift
Tian Zhang, Meng Li
TL;DR
Prompt tone has produced contradictory findings in generation tasks, and its effect on LLM relevance judging has not been studied. This paper tests calibrated politeness across judges and finds that agreement changes track shifts in scoring severity, with calibration-based agreement affected more than ranking.
Problem
Prior findings on whether rude prompts help or harm are contradictory and concern generation tasks; tone’s effect on relevance judging remains unstudied.
Source: §1 · p. 1Method
Results
Under query-disjoint cross-fitting, the association between alignment changes and agreement changes is ρ = −0.683 (exact model-block permutation p = 0.019), consistent with the severity account.
Source: §3 · p. 2Takeaways & Limitations
Tone poses a larger threat when LLM judges certify absolute labels or generate training targets; ranking outcomes changed less in this setting but were not immune.
Source: §4 · p. 2The account remains observational, and the study’s limited classifiers, models, paraphrases, ranking-safety evidence, and imperfect tone isolation constrain generalization.
Source: §4 · p. 2
Abstract
from arXiv · showhide
Large language models are increasingly used as relevance judges, yet their labels can shift with prompt surface form. We study one such feature -- tone -- on 3,498 TREC DL19/DL20 query-passage pairs, across eight judge models, five classifier-calibrated politeness levels, and three paraphrases per level. Effects are strongly model-dependent: one judge shows a structured U-shaped response, whereas most show only small changes. Where tone changes agreement, the results are more consistent with a shift in the judge's severity operating point -- its overall scoring leniency -- than with improved judgment. Agreement rises or falls as this shift moves the judge toward or away from human annotators' strictness. A query-disjoint cross-fit retains the expected association (Spearman $\rho = -0.683$; exact model-block permutation $p = 0.019$). Tone affects calibration-based agreement more than ranking outcomes: across 32 model-tone contrasts, the largest absolute mean change in NDCG@10 is 0.011, although Kendall's $\tau$ as low as 0.743 shows that reordering is reduced, not absent. The account reconciles prior contradictory findings and identifies prompt tone as a potential validity threat when absolute relevance labels matter.
1 Introduction
Prior findings on prompt tone are contradictory and confined to generation tasks, leaving its effect on LLM relevance judging unexplored. This paper tests tone across judges and proposes that changes in agreement reflect shifts in scoring severity relative to human judgments.
1 Introduction
- Tone effects on LLM relevance judging are unstudied, while generation-task findings disagree on whether rude prompts help or harm.Prior work also reports model-dependent effects, without explaining the differing directions.Source: §1 · p. 1
- Across eight judges, tone effects are model-dependent: one shows a structured response, whereas most show only small changes.Where agreement shifts, the pattern is more consistent with altered scoring leniency than a uniform gain in discrimination.Source: §1 · p. 1
- The proposed severity operating-point account predicts that tone can help or harm depending on how it moves a model’s leniency relative to a reference.A query-disjoint cross-fit retains the association (ρ = −0.683; exact model-block p = 0.019).Source: §1 · p. 1
2 Method
The study compares LLM relevance scores with human judgments on TREC DL19/DL20 query–passage pairs. It varies calibrated prompt politeness while holding the rubric and output schema fixed, then tests whether severity shifts predict agreement changes.
2 Method
- The dataset contains 3,498 BM25 top-50 query–passage pairs with NIST human judgments from TREC DL19 and DL20.Five cost-efficient models score all pairs twice; three flagship models use a frozen 40% subsample, with one reasoning-native judge reported separately.Source: §2 · p. 1
- Five politeness levels range from rude to deferential, with three paraphrases per level and an unchanged relevance rubric and output schema.Classifier politeness scores increase monotonically across levels, separating level-wide patterns from wording artifacts.Source: §2 · pp. 1–2
- Linear-weighted Cohen’s κ measures agreement between judge scores and qrels, while strictness bias and tonal drift quantify score-level alignment changes.The account predicts whether agreement rises or falls according to whether tonal drift moves the judge toward or away from the human operating point.Source: §2 · p. 2
- A query-disjoint cross-fit estimates alignment changes and agreement changes on separate folds, then combines both directions and permutes whole model blocks for inference.Kendall’s τ between tone-level and neutral rankings provides a complementary ranking-stability check.Source: §2 · p. 2
3 Results
Tone-related agreement changes are model-dependent and align with the proposed severity account. Calibration-based agreement changes more than ranking, while paraphrase controls reveal isolated wording sensitivity.
3 Results
- DeepSeek is the only judge with a structured tone effect: both extremes raise κ, with L1 Δκ = 0.054 and L5 Δκ = 0.053.Paired query-clustered bootstrap intervals exclude zero for both contrasts; most other changes are small or unstructured.Source: §3 · p. 2
- ρ = −0.683 under query-disjoint cross-fitting, with exact model-block permutation p = 0.019, supporting the predicted negative association between alignment and agreement changes.The full-data association remains negative after excluding Gemini L5, DeepSeek, or both.Source: §3 · p. 2
- Across 32 model–tone contrasts, the largest absolute mean ΔNDCG@10 was 0.011, while mean Kendall’s τ against L3 ranged from 0.743 to 0.947.Ranking quality changed less than κ, but reordering was not absent.Source: §3 · p. 2
- One Gemini 3.5 Flash L5 paraphrase reduced κ from 0.45 to 0.27 despite matching sibling politeness scores, indicating idiosyncratic wording sensitivity.The authors distinguish this isolated response from a level-wide tone effect.Source: §3 · p. 2
4 Discussion and Conclusion
A higher κ need not mean better judgment: tone can move a judge toward or away from the human reference, and that alignment could reverse against another reference. In this setting, tone poses a larger threat to absolute labels than to ranking outcomes, though rankings are not immune.
4 Discussion and Conclusion
- A higher κ need not mean better judgment: DeepSeek’s tone extremes make it stricter and raise agreement, consistent with movement toward the stricter qrels.That alignment could reverse against another reference.Source: §4 · p. 2
- For pipelines using LLM judges to certify absolute labels or generate training targets, tone poses a larger threat than for ranking outcomes, which changed far less but were not immune.The authors recommend calibrating tone and varying surface form when absolute values matter, and reporting rank-stability checks when ranking is the target.Source: §4 · p. 2
- The reasoning-native judge used 120% more reasoning tokens on rude prompts than on neutral prompts, yet produced its lowest agreement.This is consistent with evidence that more reasoning need not improve passage reranking [9].Source: §4 · p. 2
- The operating-point account remains observational, and the study’s limited classifiers, models, and paraphrases constrain generalization.The rude wrappers also changed instructions about explanation effort, so tone was not perfectly isolated from instruction content; no equivalence margin established ranking safety.Source: §4 · p. 2