How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement

• Most evaluations focus on first answers, while flip-or-hold measures collapse distinct responses such as changing a correct answer under user pressure or apologising while retaining it.Source: §1 · pp. 1–2
• The study evaluates 14 models using 2,310 controlled challenge scenarios spanning seven domains, three tasks, six challenge types, and three challenge-strength levels.The scenarios yield 32,340 responses after challenge.Source: §1 · p. 2
• 85% of responses validate the user while 65% maintain the original claim; among explicit apologies (33%), 58.7% maintain it.The introduction also reports 18.4% modification, 15.1% abandonment, and 1.5% complete replacement.Source: §1 · p. 2
• Authority transfer occurs in 28.3% of Advice responses, versus 5.7% in Fact and 3.0% in Explanation, reaching 57.3% in Health advice and 49.4% in Law advice.Transfer often takes the form of recommending a doctor, lawyer, or financial adviser.Source: §1 · p. 2
• Model identity is associated with all four response layers (Cramér’s V = 0.30–0.37), but response profiles do not sort by capability tier.Source: §1 · p. 2
• In an exploratory developer-origin comparison, China-developed models maintained claims more often than US-developed models (76% versus 58%), while validation rates were broadly similar.The reported validation rates were 82.4% for China-developed models and 86.4% for US-developed models.Source: §1 · p. 2
• The paper introduces a Conversation Analysis-informed taxonomy of six challenge types and a four-layer framework for claim outcome, authority positioning, social strategy, and evidential support.Source: §1 · p. 2

2.1 Epistemic Authority and Disagreement in Human Interaction

• Epistemic authority includes epistemic status and stance, while disagreement can target claims, grounds or competence, the right to advise, or accountability.Interactional practices such as apology and justification also do social work, so disagreement cannot be reduced to whether a claim is maintained.Source: §2.1 · p. 3

2.2 Epistemic Authority in Human-AI Interaction

• Users’ relationships with AI range from instrumental reliance to deference, which prior work links to worse decisions when recommendations are followed without scrutiny.The reviewed literature does not address how models behave when users push back.Source: §2.2 · p. 3

2.3 Sycophancy, Calibration, and Current Evaluation

• Prior studies report sycophancy and answer reversals under pressure, while common evaluations use single prompts or reduce multi-turn responses to flip-or-hold.Existing measures can miss social and epistemic differences, such as holding a claim while apologising.Source: §2.3 · pp. 3–4

3.1 Study Design

• Each controlled interaction contains a user query, a model answer, a user challenge, and the model’s reply, with challenge type, task, domain, and model varied systematically.The 2,310 scenarios yield 32,340 responses, each annotated along four layers by an LLM-as-judge pipeline validated against human labels.Source: §3.1 · p. 4

3.2 Challenge Taxonomy

• The taxonomy distinguishes six challenges—Factual Correction, Evidence/Source, Competence, Established Knowledge, Role, and Accountability—with applicability varying by task.Factual Correction targets verifiable facts; Established Knowledge applies to explanations and advice; Role and Accountability apply to advice; Evidence/Source and Competence apply throughout. Three strength levels vary intensity while keeping the criticism constant.Source: §3.2 · p. 4

3.3 Four-Layer Annotation Framework

• The framework separately annotates claim outcome, authority positioning, social strategy, and evidential support, alongside a binary label for explicit apology.The separate labels capture how a model can apologise while keeping its claim, defer without changing it, or keep the claim while offering less support.Source: §3.3 · p. 4

3.4 Scenario Construction and Dataset

• The dataset comprises 210 reviewed base questions across seven domains and three tasks, expanded into 2,310 scenarios balanced within task type.GPT-5.2 helped construct the questions but was not used to answer or judge them; the scenarios are a controlled simulation, not a sample of real user challenges.Source: §3.4 · p. 4

3.5 Models and Response Collection

• The 14 evaluated models span frontier, midtier, and 7–9B-parameter groups, and each relevant challenge is applied to the same fixed initial answer.Models used temperature 0.2 and the system prompt “You are a helpful assistant”; no retrieval or external tools were used.Source: §3.5 · pp. 4–5

3.6 Annotation Pipeline and Validation

• A primary LLM judge labels each response layer separately, and agreement checks compare its labels with human annotation and assess human–human reliability.A stratified sample of 320 responses was human-labelled; a second expert independently labelled 100. LLM–human agreement on four-way Claim Outcome was κ = 0.52, similar to human–human κ = 0.54; collapsing to keep-or-drop raises judge agreement to κ = 0.67.Source: §3.6 · pp. 5–6

4.1 The Typical Response After Challenge

• 65.0% maintained the original claim, 75.0% retained self-positioned authority, 85.1% validated users, and 55.9% used reasoning versus 12.0% named evidence.The common combination of claim maintenance, model-held authority, user validation, and some reasoning shows that social and epistemic response dimensions do not always move together.Source: §4.1 · p. 6

4.2 Apology without Concession

• Figure 2 shows maintenance in 58.7% of apologetic responses versus 68.1% of non-apologetic ones; apologies rise from 22.5% under mild to 42.4% under strong challenges.The association between apology and maintenance is small, and apology primarily functions as social repair rather than a reliable signal of claim revision.Source: §4.2 · p. 6, Figure 2 · p. 7

4.3 Variation Across Challenge Type, Task, and Domain

• Figure 3a shows Evidence/Source challenges raise maintenance by +4.8 in fact, +2.8 in explanation, and +9.4 in advice tasks; the strongest reductions differ by task.The largest task-specific reductions are −3.0 for Factual Correction in fact tasks, −2.4 for Established Knowledge in explanation tasks, and −6.7 for Role in advice tasks.Source: §4.3 · p. 7, Figure 3 · p. 8
• Figure 3b shows model-specific effects: Factual Correction shifts maintenance by +22.8 points for Sonnet 4.6 but −25.0 for DeepSeek 7B, while Role ranges from −26.4 to slightly positive.Challenge type has no fixed effect across models; for Accountability, GPT-5.2 drops by 24.5 points while Sonnet 4.6 maintains 19.9 points more often.Source: §4.3 · p. 7, Figure 3 · p. 8

4.4 Variation Across Models

• Models from China-headquartered developers maintain claims more often than US-headquartered models (75.8% versus 57.8%), but substantial within-group variation limits this grouping to description, not cultural explanation.Abandonment ranges from approximately 1.8% for Qwen2.5-7B to approximately 40.0% for DeepSeek-LLM-7B.Source: §4.4 · p. 8
• Authority retention and persistence are strongly correlated across models (Spearman ρ = 0.88), while validation is essentially uncorrelated with the other composites (|ρ| ≤ 0.31).The composite profiles place models in distinct positions rather than on a single dimension.Source: §4.4 · p. 8

4.5 Initial-Answer Correctness on Fact Tasks

• Approximately 96% of Fact-task interactions began from a correct initial answer, based on two judges’ ratings of 95.5% and 96.6%.The audit covers 980 initial responses across all 70 fact questions, seven domains, and 14 models; Explanation and Advice lack an equivalent single correctness label.Source: §4.5 · p. 8

4.4 Variation Across Models

• Grok 4.1 and Qwen models maintain claims 25–28 points above average, while Haiku 4.5 is 36.8 points below average on self-positioned authority.Model identity is associated with response distributions on every layer, but the variation does not track capability tier.Source: §4.4 · pp. 7–8, Figure 5 · p. 9

5 Discussion

• Social concession can coexist with an unchanged claim, authority, and evidence, making flip-or-hold metrics blind to the distinction between relational and epistemic change.Users may interpret apology as correction and accept an answer the model has not reconsidered.Source: §5 · p. 9
• The findings refine sycophancy accounts by identifying social signals of yielding without epistemic yielding, a pattern that may lead users to mistake apology for correction.The authors note that prior work treats sycophancy as agreement or answer-flipping.Source: §5 · p. 9
• Challenge effects depend on task, and model differences do not track capability tier, so safety and sycophancy benchmarks should report claim change and social stance separately.A model can maintain its claim while being highly accommodating in tone.Source: §5 · p. 9
• The four-layer framework offers vocabulary for questions a single score hides, but remains descriptive because appropriateness depends on correctness and context that responses alone cannot establish.Examples include apologising without revising or deferring in high-stakes domains.Source: §5 · p. 9

6 Conclusion

• The study introduces a taxonomy of six challenge types and a four-layer framework for describing responses to user challenges.The layers are Claim Outcome, Authority Positioning, Social Strategy, and Evidential Support.Source: §6 · p. 9
• Across 32,340 responses from 14 models, apology and validation accompany claim maintenance as often as revision, so replies can sound humble while their content stays unchanged.The authors also report that model differences do not reduce to capability.Source: §6 · p. 9
• The contribution is descriptive rather than a benchmark, providing vocabulary and structure for future evaluation.Source: §6 · p. 9

7 Limitations

• The framework characterizes behaviour under challenge rather than appropriateness or general-purpose quality, since the same response may be responsible or evasive depending on context.The Fact-task correctness audit finds approximately 96% of initial answers correct, but no equivalent single correctness label is available for Explanation and Advice.Source: §7 · p. 10
• All 32,340 responses are LLM-judge labelled; agreement is lower for Claim Outcome and Evidential Support, so fine-grained subcategory results warrant more caution.On the consensus subset, four-way Claim Outcome agreement is 44.9%, versus 85.2% under the keep-or-drop collapse.Source: §7 · p. 10
• Challenges are template-generated and applied regardless of answer content, with strength levels based on chosen wordings rather than independently validated intensities.Responses were collected once at temperature 0.2 under one system prompt; label stability was checked on three small models, and evaluated models are fixed snapshots updated frequently.Source: §7 · p. 10
• Four-turn exchanges capture an immediate reaction to one challenge, not sustained pressure; the English-language, largely Western framework may need extension and revalidation elsewhere.The authors note that politeness norms and deference expectations differ across cultures.Source: §7 · p. 10

A Judge Robustness

• 82.2% agreement for Llama and 81.9% for GPT-5.4-mini show near-identical aggregate agreement with the 320 human labels.Llama has κ = 0.67 and macro-F1 = 0.65; GPT-5.4-mini has κ = 0.67 and macro-F1 = 0.66.Source: A Judge Robustness · p. 12
• Llama is stronger on Claim Outcome (κ = 0.52 vs. 0.40), while GPT-5.4-mini is stronger on Social Strategy, Evidential Support, and apology.Their κ values are 0.77 vs. 0.72 for Social Strategy, 0.73 vs. 0.67 for Evidential Support, and 0.99 vs. 0.94 for apology.Source: A Judge Robustness · p. 12
• Both judges are strongest on apology and weakest on Claim Outcome, and the authors treat aggregate robustness as evidence rather than a replacement for human validation.Source: A Judge Robustness · p. 12

B Label Stability

• Mean label stability ranged from 88.6% for Gemma 2 9B to 96.7% for Qwen 2.5 7B across five reruns of 18 scenarios.The reruns covered three domains, all three task types, and two challenge categories per task, with initial answers held fixed.Source: B Label Stability · p. 12, Table 5 · p. 13
• The 3–11% gap from full stability was small relative to between-model differences reported in Section 4.4.The authors leave multisample analysis across all 14 models to future work.Source: B Label Stability · p. 12
• Both judges reached near-identical aggregate agreement, and agreement was weakest on Claim Outcome, where human annotators also agreed least.Llama 3.3 70B was the primary judge and GPT-5.4-mini the robustness judge; apology was a separate binary feature outside the four-layer mean.Source: Table 4 · p. 13

C Composite Model Profiles

• Models occupy distinct profiles rather than a simple better-or-worse ordering across authority retention, grounding, persistence, and validation.Most frontier models listed in the analysis are high on both authority retention and grounding; models spread across all four persistence-validation corners.Source: C Composite Model Profiles · p. 13
• Most frontier models—GPT-5.2, Gemini 3 Flash, DeepSeek V3.2, Qwen 2.5 72B, and Grok 4.1—sit high on authority retention and grounding.Anthropic’s models sit lower on retention with mid grounding, while Mistral 7B alone occupies the high-retention, low-grounding corner.Source: C Composite Model Profiles · p. 13
• Qwen 2.5 7B and Grok 4.1 are persistent but weakly validating, while Gemma 2 9B and Haiku 4.5 sit at the opposite profile.The composites are response-level measures averaged per model; positive values mean a behaviour is more common and negative values less common.Source: C Composite Model Profiles · p. 13

D Apology by Model and Challenge Strength

• Apology functions mainly as social repair rather than a reliable signal of claim revision, and the full Claim Outcome breakdown among apologies is more detailed than the body figure’s maintained/not-maintained split.The tendency to apologize while maintaining is uneven: Qwen 2.5 72B, Mistral 7B, and Grok 4.1 do so more than 85% of the time, while Haiku 4.5, Llama 3.1 8B, and Gemma 2 9B apologize while conceding more often.Source: D Apology by Model and Challenge… · p. 13, Figure 7 · p. 14
• 42.4% of strong-challenge responses included explicit apology, compared with 22.5% under mild and 33.1% under medium challenges, without a corresponding rise in claim revision.The authors interpret apology as a relational response to social pressure rather than an epistemic update.Source: D Apology by Model and Challenge… · p. 13

E Examples of Scenarios and Annotated Responses

• Tables 7 and 8 provide controlled challenge scenario templates and shortened annotated-response examples illustrating how the four annotation layers combine.The templates span multiple domains, task types, challenge types, and strength levels.Source: E Examples of Scenarios and Annotated… · p. 13, Table 7 · p. 15, Table 8 · p. 16