Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms

1 Introduction

• Existing benchmarks provide limited direct evidence about pragmatic competence: standard tests omit it, while toxicity benchmarks capture politeness only indirectly when impoliteness overlaps with abuse.Conventionally polite model outputs also do not establish that their politeness judgments align with human judgments.Source: §1 · p. 1
• The study compares LLM judgments with human annotations in two English-language datasets: one with continuous ratings and one with three-way categorical labels.Because politeness is subjective, disagreement may reflect variation among human evaluators; the study compares crowd labels, expert consensus, and LLM predictions without treating either human reference as definitive.Source: §1 · pp. 1–2
• The paper asks whether models align with humans across models and prompts, what strategies and patterns accompany alignment or misalignment, and whether Neutral tendencies persist against expert labels.The expert comparison focuses on a diagnostic subset of high-disagreement cases.Source: §1 · p. 2
• The contributions include a multi-reference evaluation, evidence of stronger inter-model than model–human agreement, two signatures of miscalibration, and released research materials.The named signatures are neutral compression and surface-cue accumulation; the release includes prompts, metrics, model outputs, and analysis scripts.Source: §1 · p. 2

2 Related Work

• The related work is organized around computational politeness, pragmatic evaluation of LLMs, and human–AI alignment in social contexts.These are the three lines of research the paper says it builds on.Source: §2 · p. 2

2.1 Computational Politeness

• Prior work on computational politeness spans rule-based accounts, supervised and weakly supervised learning, contextualized models, and sociolinguistic applications.Earlier studies mainly addressed politeness detection, generation, or feature extraction rather than whether LLM judgments align with human pragmatic norms.Source: §2.1 · p. 2

2.2 Pragmatic Competence in LLMs

• Pragmatic evaluations report LLM difficulties with implicature, presupposition, reference, deixis, irony, sarcasm, context-dependent impoliteness, and prompt-politeness effects.Production analyses also suggest LLMs may overuse negative politeness strategies relative to humans (Zhao and Hawkins, 2025).Source: §2.2 · p. 3

2.3 Human–AI Alignment in Social Contexts

• Human–AI alignment research examines social and cultural behavior, but pragmatic phenomena such as politeness remain largely absent from standard benchmarks.The paper tests alignment with human annotations rather than relying on surface-level polite behavior.Source: §2.3 · p. 3

3 Datasets

• The Stanford Politeness Corpus contains 10,957 two-sentence requests rated by five crowd annotators on a continuous scale from very impolite to very polite.The study sampled 3,000 requests for evaluation and retained the original continuous scores as reference values.Source: §3 · p. 3
• The three-way corpus contains 16,428 instances, evenly divided among Impolite, Neutral, and Polite labels; the study sampled 3,000 instances, 1,000 per category.Because the repository lacks detailed provenance and label-construction documentation, the categories are treated as repository-provided reference labels.Source: §3 · p. 3

4.1 Evaluated Language Models

• Five contemporary LLMs were evaluated, with two additional open-weight models included for the categorical dataset; models used provider-default sampling without fine-tuning or manual output correction.The evaluated models were queried through OpenRouter between February and May 2026.Source: §4.1 · p. 3

4.2 Prompt Conditions

• Six zero-shot prompts varied role framing, cue access, and output format, including expert criteria, a feature checklist, and scoring across five pragmatic dimensions.The prompt templates were held constant across models.Source: §4.2 · pp. 3–4

4.3 Evaluation Metrics

• Evaluation used correlation, mean absolute error, and Close Rate for continuous scores; categorical measures included accuracy, macro-F1, Cohen’s κ, and inter-model Fleiss’ κ.Directional diagnostics measure excess Neutral predictions, underproduction of Impolite labels, and shifts of reference Polite or Impolite cases to Neutral.Source: §4.3 · p. 4

4.4 Human Baseline Estimation

• The human baseline estimates agreement among five annotators and compares each annotator’s ratings with the mean of the other four.The leave-one-annotator-out comparison approximates how well an individual matches the other annotators’ consensus.Source: §4.4 · p. 4

4.5 Politeness Strategy Detection

• A rule-based detector identified 15 linguistic strategies using lexical lists, regular expressions, and shallow syntactic patterns, with counts normalized for text length.Detected strategies include politeness markers, hedges, modals, interrogatives, deference, gratitude, greetings, and apologies.Source: §4.5 · p. 4
• The strategy contrast defined continuous-score alignment as an absolute model–human difference below 0.5 and strong misalignment as a difference of at least 1.0.Intermediate cases were excluded; categorical alignment and misalignment were agreement and disagreement, and comparisons were treated as exploratory across correlated strategies.Source: §4.5 · pp. 4–5

4.6 Expert Audit Experiment

• The expert audit labeled 318 difficult categorical cases through five trained annotators, using majority vote for consensus and the most frequent prompt label for each model.Because cases were sampled from high-disagreement and strategy-conflict instances, the audit is diagnostic rather than a replacement for crowd annotations or a full-dataset prevalence estimate.Source: §4.6 · p. 5

5.1 Result 1: Politeness Scoring

• Mean inter-model correlation was r = 0.724 versus mean model–human correlation r = 0.385 across six prompts, while even the best scoring models showed weak-to-moderate alignment.The best LLM–human correlation was r = 0.433, compared with a mean human–consensus correlation of r = 0.562.Source: §5.1 · p. 5
• Prompt explained 8.3% of alignment variance versus 23.7% for model identity; expert prompting improved alignment modestly but did not close the gap.The prompt effect was statistically detectable: F(5, 24) = 3.72, p = 0.012.Source: §5.1 · p. 5
• In continuous scoring, direct and indicative constructions aligned better with human judgments, while gratitude, deference, hedging, modals, first-person framing, and positive affect were more frequent in misaligned cases.These strategy patterns were stable across models.Source: §5.1 · p. 5, Figure 3 · p. 7

5.2 Result 2: Politeness Polarity Classification

• Proprietary-model Cohen’s κ ranged from 0.24 to 0.35, while Llama 3 and Llama 3.1 scored κ = 0.171 and κ = 0.161, respectively.Grok performed best among proprietary models (κ = 0.354); the categorical results are shown in Figure 2, Panel C.Source: §5.2 · pp. 5–6, Figure 2 · p. 6
• Across four categorical prompt conditions, proprietary-model Fleiss’ κ ranged from 0.518 to 0.709, compared with mean model-level Cohen’s κ of 0.241 to 0.354 against the human reference.The authors note that these agreement metrics summarize different structures.Source: §5.2 · p. 6
• Across all five proprietary models and four prompts, neutral overproduction and impoliteness suppression were positive in every cell: NCIdiff ranged from +1.0 to +41.2 pp and ISRdiff from +7.4 to +25.1 pp.The means were +21.7 pp for NCIdiff and +16.0 pp for ISRdiff, showing these were not artifacts of the diagnostic subset.Source: §5.2 · p. 6
• Across tasks, explicit low-inference cues were associated with better alignment, but strategy associations differed between datasets and some rapport-building cues were more frequent in misaligned cases.In categorical evaluation, correctly classified cases contained more explicit cues, while deference, apologies, and please were slightly more frequent in mismatches.Source: §5.2 · p. 6, Figure 3 · p. 7

5.3 Expert Audit of High-Disagreement Cases

• Across five model-level comparisons, mean LLM–expert agreement was 63.6% accuracy (κ = 0.395), versus 16.3% accuracy (κ = −0.252) for LLM–crowd; these are diagnostic, not full-dataset estimates.Table 1 values average five model-level results, not a cross-model ensemble.Source: §5.3 · pp. 6–7, Table 1 · p. 7

6.1 Politeness Comprehension in LLMs

• Strong inter-model similarity does not establish human-like pragmatic competence; LLMs overweight explicit markers and compress socially marked judgments toward Neutral.The authors do not claim that RLHF causally produces neutral compression, which may arise from multiple factors.Source: §6.1 · p. 8

6.2 Linear Pragmatic Accumulation

• The authors describe linear pragmatic accumulation: explicit cues align better with human judgments, while context-sensitive strategies such as modalization, first-person framing, positive affect, and face management are associated with divergence.Hard errors include distinguishing appropriate politeness from over-politeness, sincerity from strategic exaggeration, and indirect mitigation from vague or evasive language.Source: §6.2 · pp. 8–9

6.3 Implications for AI Evaluation and Understanding

• Crowd judgments, expert interpretations, and LLM predictions represent partially distinct reference systems, so polite-sounding outputs alone do not establish human-aligned judgment or sociolinguistic competence.The paper cautions against equating aligned behavior with sociolinguistic competence.Source: §6.3 · p. 9
• Neutral compression is a robust behavioral pattern, not a demonstrated causal mechanism, because proprietary training pipelines and other possible contributing factors cannot be observed.The authors list pretraining distributions, supervised instruction tuning, label granularity, prompt framing, and pragmatic uncertainty as possible contributors.Source: §6.3 · p. 9

7 Conclusion

• Across evaluated settings, LLMs show a stable, model-specific politeness norm that is internally consistent and partially expert-like.Their judgments nevertheless do not consistently align with human annotations when contextual pragmatic interpretation is required.Source: §7 · p. 9
• The judgments are biased toward Neutral compression and explicit surface-cue weighting.The paper describes these as behavioral patterns, alongside partial sensitivity to overt politeness cues.Source: §7 · p. 9
• The authors argue that politeness should be a first-class benchmark target, with evaluation attending to pragmatic function, social relations, and communicative context.They frame this as a broader implication for evaluating trustworthy and socially aligned AI.Source: §7 · p. 9

Limitations

• Both datasets are English-centered and drawn from online interactions, limiting generalization to other languages, cultures, and offline settings.Limited conversational context also makes it difficult to distinguish model error from missing information about speaker roles, social distance, or interaction history.Source: Limitations · p. 9
• The 318 expert-annotated cases were sampled from high-disagreement and strategy-conflict instances, so they cannot estimate full-dataset error prevalence.Expert agreement was only fair-to-moderate, reflecting the subjectivity of pragmatic judgment.Source: Limitations · p. 9
• Neutral compression is a behavioral signature, not evidence that RLHF caused the pattern.The authors cannot observe proprietary training data, reward models, system prompts, or safety filters; other factors may also contribute.Source: Limitations · pp. 9–10

Ethics Statement

• The datasets use existing English-language online text, and the expert audit involved five proficient L2 English users with graduate-level linguistics training.The authors caution that these annotations represent particular linguistic and cultural perspectives and should not be generalized without further validation.Source: Ethics Statement · p. 10

A.1 Overview

• The six prompt framings vary evaluator role, access to pragmatic criteria, and output requirements, serving as sensitivity conditions rather than isolated causal tests.For Dataset 1, all prompts requested continuous scores; Dataset 2 used categorical outputs for P2, P3, P4, and P6.Source: §A.1 · p. 12

A.2 Prompt Framing Templates

• The prompt templates range from minimal or objective cue-based judgments to expert or heuristic-guided evaluation and weighted multidimensional scoring.P2 is the minimally scaffolded categorical baseline; P3 adds expert framing, P4 supplies heuristics, and P6 aggregates five weighted dimensions.Source: §A.2 · pp. 12–13

A.3 Cross-Prompt Analysis

• The average inter-prompt correlation was 0.80, indicating fairly stable item rankings, but not prompt invariance.The minimally scaffolded or atheoretical P1, P2, and P5 had mean pairwise r = 0.89; P3 had the lowest mean correlation with other conditions (r = 0.73).Source: §A.3 · p. 14, Table 2 · p. 14

A.4 Statistical Testing of Prompt Effects

• Prompt effects on model–human alignment were detectable but modest: prompt condition accounted for 8.3% of variation, compared with 23.7% for model identity.P3 exceeded P1 by Δr = 0.045 and P2 by Δr = 0.042; P6 exceeded P2 by Δr = 0.038. The authors conclude prompt changes do not close the alignment gap.Source: §A.4 · p. 14

B Prompt-Framing Ablation

• The ablation used 1,000 stratified Dataset 1 requests and compared baseline, expert, and neutral prompts on a common normalized scoring scale.The subset covered impolite, neutral, and polite human-rated cases; normalization was applied to all conditions rather than treated as an ablated factor.Source: B Prompt-Framing Ablation · p. 15
• The evaluation measured score variance, IQR, Pearson correlation with human ratings, and a score-centralization ratio comparing model and human predictions near the center.Ratios above 1.0 indicate greater model concentration in the normalized interval [−0.3, 0.3].Source: B Prompt-Framing Ablation · p. 15, Table 4 · p. 15
• Expert framing increased score variance by approximately 19% and IQR by 19.9% on average, reduced centralization by 17.5%, and raised Pearson correlation with human ratings by 0.040.The same directional ordering held across all three evaluated models; expert-condition centralization ratios remained above 1.0 (1.38–1.48) and correlations remained moderate.Source: Table 4 · p. 15, B Prompt-Framing Ablation · p. 15
• For GPT-4.1, expert framing widened the Polite–Impolite mean-score separation from 1.13 to 1.63 points (44.2%).Neutral-case means changed only from +0.08 to +0.05; the passage presents this category breakdown as an illustration, not a cross-model result.Source: B Prompt-Framing Ablation · p. 15

B.1 Strategy-Level Effects

• The largest strategy-level changes under expert prompting involved multiple hedges, first-person plural framing, strategic negative wording, casual register, and subtle deference.Please, gratitude, overt negative words, and indicative forms changed little.Source: §B.1 · p. 15, Table 5 · p. 15
• Strategy changes were not uniform; the largest correlation increases occurred in categories whose interpretation may depend more on pragmatic function.The analysis does not identify the process producing those changes.Source: §B.1 · p. 15

B.2 Interpretation

• Even under expert framing, scores remained more centralized than human ratings and only moderately correlated with them.The experiment cannot distinguish an elicitation ceiling from a representational or training-related limitation, and it does not directly test Dataset 2’s categorical neutral-compression measures.Source: §B.2 · p. 16

C Human Reference Analysis

• Aggregating five annotators yields ICC_A,k = 0.786, while single-rater measures are approximately 0.42 and within-utterance variation (57.6%) exceeds between-utterance variation (42.4%).The corpus mean is therefore a useful primary reference, but a single annotator is not definitive.Source: C Human Reference Analysis · p. 16, Table 6 · p. 16
• In leave-one-annotator-out comparisons, the held-out annotator averages r = 0.562 against the other four, while three-class agreement remains moderate at 55.7% accuracy, macro-F1 = 0.542, and κ = 0.310.These estimates and the aggregated corpus score are distinct and sensitive to annotator composition and scale construction.Source: C Human Reference Analysis · p. 16, Table 6 · p. 16

D Politeness Strategy Detection System

• The rule-based detector targets 15 surface strategies across five categories, using politeness theory and related work as an inspectable diagnostic inventory rather than a complete account of pragmatic function.Its categories do not mean the datasets were annotated with Brown and Levinson’s framework.Source: D Politeness Strategy Detection System · pp. 16–17
• On 200 Dataset 1 requests, the detector’s 15-strategy macro averages were Cohen’s κ = 0.87, precision = 0.89, and recall = 0.88.Agreement was higher for explicit markers such as please, gratitude, and questions than for deference and factuality; cue validation does not validate every pragmatic interpretation.Source: D Politeness Strategy Detection System · p. 17, Table 7 · p. 17

E.1 Successful Alignment

• Selected diagnostic examples show close model–human agreement when requests contain salient conventional markers or overtly confrontational wording.The examples were selected to illustrate patterns, not estimate their prevalence, and the original ratings provide no annotator rationales.Source: §E.1 · p. 17, E Error Analysis · p. 17, Table 8 · p. 18

E.2 Illustrative Disagreements

• In selected disagreements, multiple hedges and praise coincide with higher model scores, while casual brevity, inclusive framing, and strategic negative wording coincide with lower scores.These examples are compatible with several explanations, including cue accumulation, missing context, and annotation uncertainty, but distinguish none of them.Source: §E.2 · pp. 17–18, Table 9 · p. 18

E.3 Strategy-Conditioned Error Patterns

• Across 20 model–prompt configurations, multiple hedging shows a positive model–human difference in 19, strategic negative wording a negative difference in 19, first-person plural framing a negative difference in 16, and excessive deference a positive difference in 18.The conditional error analysis asks a different question from cue-frequency contrasts: a cue can have relatively small error among its cases yet occur more often in misaligned cases; these patterns do not identify a shared mechanism.Source: §E.3 · pp. 18–19

• The 318 audited cases were deliberately selected for diagnostic difficulty, so these results do not estimate error prevalence in the full dataset.Five linguistically trained experts independently labeled each case; reported means average model-level results rather than representing an ensemble prediction.Source: F Expert Audit Results · p. 19
• Expert agreement on the difficult cases was limited: mean pairwise Cohen’s κ = 0.368, Fleiss’ κ = 0.364, Krippendorff’s α = 0.369, and 53.4% met the strong- or full-consensus criterion.The audit does not identify whether disagreement reflects item ambiguity, individual differences, or the annotation protocol.Source: F Expert Audit Results · p. 19
• 68.4% of mean LLM outputs were Neutral, versus 44.1% for experts and 31.4% for crowd labels; Impolite shares were 5.0%, 24.6%, and 30.5%, respectively.Source: F Expert Audit Results · pp. 19–20, Figure 5 · p. 19
• LLM–expert agreement was 63.6% accuracy, κ = 0.395, and macro-F1 0.557; LLM–crowd was 16.3% accuracy and κ = −0.252; crowd–expert was 31.4% accuracy, κ = −0.029, and macro-F1 0.331.These aggregate comparisons are tabulated in Table 1 in the main text.Source: F Expert Audit Results · p. 19
• Grok had the highest model-level agreement with expert consensus (κ = 0.419), while prompt consistency exceeded model–expert agreement for all five models (κ = 0.701–0.833).Figure 4 presents the model values against mean pairwise expert agreement as a descriptive reference, not a performance ceiling.Source: Figure 4 · p. 19, F Expert Audit Results · p. 20
• Among 118 majority-LLM/expert mismatches, Expert Impolite → LLM Neutral occurred 21 times, Expert Polite → LLM Neutral 11 times, and Expert Neutral → LLM Polite 9 times.The counts describe label shifts, not their linguistic or causal source; the reported percentages use the mismatches as denominator.Source: F Expert Audit Results · p. 20, Table 11 · p. 20
• The audit offers an alternative human reference without resolving which annotation source is correct.Low LLM–crowd agreement is partly expected because high LLM–crowd disagreement was a sampling criterion; higher LLM–expert agreement does not demonstrate general expert-level pragmatic competence.Source: F Expert Audit Results · p. 20