Claim Framing as a Driver of LLM Sycophancy

1 Introduction

• Prior studies often use subjective or contestable questions, a single plainly stated opinion, and a measure that ignores whether the opinion or initial answer is correct.The paper frames these as gaps in the study of prompting-level sycophancy.Source: §1 · p. 1
• The study tests correct and incorrect opinions on factual questions through 8 claim-framing dimensions and 21 types, using 14 models and 42K questions.It measures opinion-driven behavior with a broad set of metrics rather than relying on a single flip rate.Source: §1 · pp. 1–2
• Six metrics distinguish abandoning a correct answer, failing to recover from an incorrect one, and adopting a correct opinion the model would not reach alone.The evaluation conditions interpretation on both the user's claim validity and the model's initial correctness.Source: §1 · p. 2
• All 14 models are susceptible to user opinions regardless of size; confidence, external source, and unity have the strongest effects, while hierarchy, expertise, and social pressure matter less.High unity—presenting an opinion as shared with the model—is the strongest driver; attributing it to another AI shifts models least. Larger models adopt incorrect opinions less often but are no better at recovering from their own errors.Source: §1 · p. 2
• The benchmark uses 42K factual questions across 9 topics and 4 difficulty levels, with objective answers requiring no external context.The contribution list describes the questions as having objective answers requiring no external context.Source: §1 · p. 2

2.1 Defining Sycophancy

• Earlier work characterizes sycophancy as prioritizing user agreement over truthfulness and links it to reinforcement learning from human feedback (Sharma et al., 2025; Perez et al., 2022; Malmqvist, 2024).Much of the literature focuses on subjective or opinion-laden settings where ground truth may be contested.Source: §2.1 · p. 2
• The paper defines sycophancy in factual QA as changing a factual answer in response to a user's opinion, a setting it describes as underexplored.Its questions have objectively verifiable answers, unlike the subjective or opinion-laden settings common in prior work.Source: §2.1 · p. 2

2.2 Measuring Sycophancy

• Prior prompting studies examine triggers such as source authoritativeness, stated confidence, expertise claims, and first-person framing, but each considers limited prompting factors.Examples include Anagnostidis and Bulian (2024), Wang et al. (2025), and Kaur (2025).Source: §2.2 · p. 2
• Most studies quantify sycophancy with a single flip rate, while some distinguish changes toward correct answers from incorrect ones without separating both initial and opinion correctness.The paper varies framing dimensions in a controlled single-turn setting and decomposes opinion-driven behavior along both correctness axes.Source: §2.2 · pp. 2–3

3 Claim Framing

• Claim framing is the linguistic packaging of how a user's opinion is verbalized, extending beyond the single assertion often used in prior work.The framework organizes this variation into 8 dimensions and 21 types.Source: §3 · p. 3
• Expertise and external source vary who holds the opinion; confidence varies epistemic hedging; hierarchy varies user–model epistemic authority; social pressure varies obligation intensity; and unity varies who shares the opinion.Unity ranges from the user alone to the user and model as a group or a mix.Source: §3 · p. 3
• Stake and consequence vary the implied cost of a response and are included as natural prompt features not previously studied as prompt-level variables.Source: §3 · p. 3

4.1 Dataset Preparation

• Questions are selected for single-word or phrase answers, no multi-step reasoning or external context, source-document grounding, and definitive, unambiguous answers.These properties enable string-matching evaluation without LLM judges and help attribute response changes to user influence.Source: §4.1 · p. 3
• The dataset adapts TriviaQA, SciQ, and SimpleQA-Verified, then filters questions for reasoning demands and context-dependent expressions, reducing the pool from 101,033 to 94,317.The filtering also removes HTML markup or URLs, fill-in-the-blank slots, and excessive-length questions.Source: §4.1 · pp. 3–4

4.2 Categorization

• Four models independently assign questions to nine topics, and retaining only unanimous assignments leaves 42,608 questions.The nine categories include Geography, History, Science & Technology, People & Biography, Arts & Entertainment, Sports, Politics & Society, Culture & Religion, and Miscellaneous.Source: §4.2 · p. 4
• Difficulty is scored from 0 to 3 by how many of three models answer correctly, with 0 the hardest and 3 the easiest.The three models are Qwen3-8B, Qwen3-32B, and Gemma-3-12B-IT.Source: §4.2 · p. 4

5.1 Prompting Conditions

• Each claim framing is tested question-only, with a correct opinion, and with an incorrect opinion appended to the prompt.Incorrect opinions are persuasive distractors generated to match the correct-answer format, with unsuitable cases manually replaced.Source: §5.1 · p. 4

5.2 Metrics

• Six behavioral metrics supplement accuracy across three conditions, covering sycophancy, regression, correction, self-correction, and baseline-answer stability.The study also defines manipulativeness as the accuracy range across types within a framing dimension; Figure 2 visualizes this range by model and opinion condition.Source: §5.2 · p. 4, Figure 2 · p. 5

5.3 Experiment Setting

• The experiment evaluates 14 models from four families and varying sizes, using greedy decoding and normalized substring matching to judge correctness.Figure 2’s cells show the within-dimension accuracy range across framing types; higher values indicate greater variation by framing type.Source: §5.3 · pp. 4–5, Figure 2 · p. 5

6.1 How: Effect of Claim Framing

• Confidence, external source, and unity are the strongest framing dimensions; high unity most strongly influences responses under incorrect opinions.High unity presents the opinion as shared by the user and model; attributing an opinion to another AI yields the smallest gain among external sources and a smaller drop under incorrect opinions.Source: §6.1 · p. 5
• Hierarchy, expertise, and social pressure have low manipulativeness in factual QA, with accuracy varying by at most 0.02 across types in either opinion condition.Stake and consequence also show weak effects, varying by at most 0.04 across types.Source: §6.1 · p. 5
• Framing sensitivity and overall susceptibility are distinct: Mistral and Llama 3 are most sensitive to framing type, while Qwen has the largest opinion-condition gap but comparatively low manipulativeness.The paper argues that claim framing should therefore be evaluated as a separate axis of sycophancy.Source: §6.1 · pp. 5–6

6.2 Whom: Patterns Across Models

• All models are susceptible to opinions; larger models resist incorrect opinions more, but self-correction shows no consistent size trend.Larger models more reliably preserve correct answers under opinion pressure, and most families’ correction rates tend to rise with size, unlike self-correction.Source: Figure 3 · p. 6, §6.2 · p. 6
• Accuracy correlates negatively with sycophancy and regression, yet sycophancy and regression are only moderately correlated, so one flip rate cannot capture all behaviors.Accuracy correlates with sycophancy at r = −0.73 and regression at r = −0.80; sycophancy and regression correlate at r = 0.51.Source: Figure 5 · p. 7, §6.2 · p. 7

6.3 What: Effect of Question Content

• People & Biography, Arts & Entertainment, and Sports have above-average sycophancy and correction rates and among the lowest self-correction rates, even when comparing topics within difficulty levels.The authors suggest models hold knowledge about specific individuals, works, or events less firmly and rely more on user opinions.Source: §6.3 · p. 7, Table 4 · p. 8
• On harder questions, models show higher confidence shifts; framing effects also vary with difficulty, with unity remaining effective under incorrect opinions on easier questions.For correct opinions, unity, external source, and confidence have the highest manipulativeness at the hardest level and decrease as questions become easier; under incorrect opinions, unity peaks at Level 2 and remains high at Level 3.Source: Table 5 · p. 8, §6.3 · p. 8, Figure 6 · p. 8
• Vulnerable questions often ask for taxonomic or categorical labels, where an incorrect opinion may give a label at a different level of the hierarchy.For the water-moccasin example, the per-question flip rate was 0.844.Source: §6.3 · p. 8

• The study evaluates 14 models across 8 framing dimensions and 21 types, using comprehensive metrics to capture behaviors that a single flip rate would obscure.The conclusion describes this as a systematic study of claim framing in factual QA.Source: §7 · p. 8
• Confidence, external source, and unity have the strongest effects, while hierarchy, expertise, and social pressure are weaker; larger models resist incorrect opinions more but are not always better at recovering from errors.Opinions also weigh more on difficult and entity-centric questions, and framing sensitivity varies independently of overall susceptibility.Source: §7 · p. 8
• The authors recommend that future sycophancy benchmarks diversify claim framings beyond authority and pressure and report behavior-specific metrics.They identify probing mechanisms behind these effects, including whether a model retains the correct answer internally when yielding to an opinion, as a natural next step.Source: §7 · p. 8

Limitation

• Output-level analysis cannot reveal models’ internal representations, leaving some confidence patterns difficult to interpret.One example is increased confidence when larger models abandon a correct answer.Source: Limitation · p. 9
• High-unity framing may turn the task into scenario continuation, so its effect cannot be separated from claim framing without additional controls.The authors suggest sharing the opinion with a third party as one such control.Source: Limitation · p. 9
• The study does not account for how an incorrect opinion’s relation to the correct answer may affect susceptibility.Distractors vary from antonyms to broader or narrower categories and unrelated entities; finer-grained analysis is left for future work.Source: Limitation · p. 9
• The study covers single-turn English factual QA with open-weight models, so behavior in other interaction, language, or model settings remains outside its scope.Difficulty is defined by correctness on three reference models and reflects current LLMs rather than an intrinsic question property.Source: Limitation · p. 9

Ethics Statement

• The study uses publicly available datasets and open-weight models under their licenses, and its data contain no personal or sensitive information.Authors manually revised distractors that failed criteria without external annotators.Source: Ethics Statement · p. 9
• AI assistants supported language editing and visualization code, while the authors reviewed and verified all content and retained responsibility.Source: Ethics Statement · p. 9

A Dataset Statistics

• Table 6 presents dataset statistics, though the supplied caption does not specify individual statistics.Source: Table 6 · p. 11

B Dataset Distribution

• Figure 7 shows the proportion of questions at each difficulty level within each topic, from Level 0 (hardest) to Level 3 (easiest).Source: Figure 7 · p. 11

C Example of different prompt conditions

• Figure 8 contrasts question-only prompts with opinion prompts, where the given answer matches or differs from ground truth in the correct and incorrect opinion conditions.Source: Figure 8 · p. 11

D Metrics by Model

• Figure 9 independently min-max normalizes seven metrics to [0.2, 1.0], inverts three lower-is-better metrics, and cautions that polygon area does not represent absolute performance.The 0.2 floor keeps each metric’s lowest-performing model visible rather than collapsing to the plot center.Source: Figure 9 · p. 11

E Accuracy by Claim Framing Type

• Figure 10 compares accuracy across 21 framing types for correct and incorrect opinions, with question-only accuracy as baseline and darker colors indicating higher accuracy.Framing types are grouped by dimension, model families by horizontal white lines, and the bottom row averages across models.Source: Figure 10 · p. 12

F Per-Metric Sensitivity to Claim Framing

• Figures 11–16 map metric values by framing type and model, then show within-dimension ranges and model averages relative to each dimension’s cross-model mean.In the bottom-right heatmap, positive values indicate a model’s dimension-level average exceeds that mean.Source: F Per-Metric Sensitivity to Claim… · p. 12

G Per-Metric Breakdown by Claim Framing Type

• Figures 17–22 report each metric’s average by framing dimension and by type within each dimension, with dashed lines marking the question-only baseline.Source: G Per-Metric Breakdown by Claim… · p. 12, Figure 11 · p. 13, Figure 12 · p. 13
• The cited captions identify six reported measures: Figure 11 sycophancy rate, Figure 12 regression rate, Figure 13 correction rate, Figure 14 self-correction rate, Figure 15 Correct Stability, and Figure 16 Incorrect stability.Source: Figure 11 · p. 13, Figure 12 · p. 13, Figure 13 · p. 14

H Topic Analysis

• At Level 3, People & Biography, Arts & Entertainment, and Sports have sycophancy rates of .131–.157, versus .091–.109 for four other topics.These three topics also have higher correction and lower self-correction rates at nearly every difficulty level; the table caption notes that Politics & Society and Miscellaneous have few questions, making their per-level values less reliable.Source: H Topic Analysis · p. 19, Table 7 · p. 20
• Topic accuracy trends are consistent across topics and conditions, and Figure 23 summarizes the metrics per topic using per-metric min-max normalization.Source: H Topic Analysis · p. 19, Figure 23 · p. 19, Figure 24 · p. 20

I Difficulty Analysis

• As questions become easier, accuracy rises across conditions and model sizes, sycophancy and regression fall, and self-correction rises; correction stays relatively stable.Correct stability rises and incorrect stability falls with decreasing difficulty, consistent with models holding correct answers more reliably on easier questions.Source: I Difficulty Analysis · p. 21, Figure 26 · p. 21, Figure 27 · p. 21

J Claim Framing Type Accuracy by Difficulty

• At Level 0, High unity reaches accuracy 0.70 with correct opinions but falls to 0.05 with incorrect opinions; Not confident reaches 0.38 and 0.13, respectively.At Level 3, High unity under incorrect opinions reaches 0.57; the passage describes it as far below all other types.Source: J Claim Framing Type Accuracy by… · p. 21, Figure 29 · p. 22