PaperlayerView exact revision on arXiv

Source-linked AI summary

Verbalizing LLMs' assumptions to explain and control sycophancy

Myra Cheng, Isabel Sieh, Humishka Zope, Sunny Yu, Lujain Ibrahim, Aryaman Arora, Jared Moore, Desmond Ong, Dan Jurafsky, Diyi Yang

arXiv:2604.03058v3cs.CLcs.AIcs.CY

TL;DR

LLMs can become socially sycophantic when their assumptions about users’ goals do not match users’ intent. This paper elicits those assumptions, identifies related internal representation directions, and steers them to control sycophancy. It also finds that people expect more objective information from AI than from humans, while LLMs do not adjust for that expectation gap.

  • Problem

    The paper asks how mismatches between users’ intent and LLMs’ assumptions contribute to social sycophancy and related safety problems.

    Source: §1 · p. 1

  • Method

    The paper elicits open-ended and structured Verbalized Assumptions, trains linear probes on model internals, and uses the resulting directions for sycophancy steering.

    Source: §1 · p. 2, §2.3 · p. 5, §3.1 · p. 8

  • Results

    Assumption steering reduces social sycophancy with comparable reductions to direct steering without sacrificing overall performance, while people expect objective information more often from AI than from humans.

    Source: §3.1 · p. 9, §3.2 · pp. 9–10

  • Takeaways & Limitations

    User-intent assumptions provide an interpretable primitive for analyzing and controlling sycophancy, while the human-AI expectation gap helps explain sycophantic assumptions.

    Source: §1 · p. 2, §4 · p. 10

    The steering results do not establish a unique internal assumption state because unidentified latent mediators or correlated linguistic features may explain the learned directions.

    Source: §3 · p. 7

Abstract

from arXiv · show

LLMs can be socially sycophantic, affirming users when they ask questions like "am I in the wrong?" rather than providing genuine assessment. We hypothesize that this behavior arises from LLMs' incorrect assumptions about the user, like underestimating how often users are seeking information over reassurance. We present Verbalized Assumptions, a framework for eliciting these assumptions from LLMs. Verbalized Assumptions provide insight into LLM sycophancy, delusion, and other safety issues: in social sycophancy datasets, "seeking validation" is the most frequent bigram in LLMs' assumptions. We provide evidence for a causal link between assumptions and sycophantic model behavior: we train linear probes on internal representations associated with Verbalized Assumptions and then use these probes for interpretable, fine-grained steering of social sycophancy. Finally, we identify a human-AI expectation gap that explains why LLMs default to sycophantic assumptions. On identical queries, people expect more objective and informative responses from AI than from other humans, but LLMs trained on human-human conversation do not account for this difference in expectations. Our work contributes a new understanding of assumptions as a mechanism for analyzing and controlling sycophancy.

1 Introduction

The paper frames social sycophancy as a mismatch between users’ intent and LLMs’ assumptions, then introduces Verbalized Assumptions to expose and study those assumptions. It connects these assumptions to internal representations that can be used to control sycophancy.

1 Introduction

  • LLMs often affirm users in situations requiring honest assessment, potentially because they assume users seek emotional support rather than objective information.The paper treats sycophancy as a mismatch between user intent and the model’s assumptions about that intent.Source: §1 · p. 1
  • Verbalized Assumptions prompt LLMs to state assumptions about users, exposing implicit inferences relevant to sycophancy, delusion, and other safety issues.The framework supports both open-ended analysis and later quantitative probing and steering.Source: §1 · p. 1
  • Linear probes trained on verbalized assumptions identify internal representational subspaces that support interpretable, fine-grained steering of sycophancy.The approach uses elicited assumptions as a bridge between observable behavior and model internals.Source: §1 · p. 2, Figure 1 · p. 2
  • A user study shows that people expect objective information more often from AI than from humans, while LLMs trained on human-human interactions do not adjust for this expectation shift.This human-AI expectation gap is proposed as an explanation for sycophancy-inducing assumptions.Source: §1 · p. 2
  • The paper contributes a framework linking user-intent assumptions to sycophancy and delusion, probes for their internal directions, and interventions that reduce social sycophancy.It presents assumptions as a primitive for understanding and controlling sycophancy.Source: §1 · p. 2

2 Method: Verbalized Assumptions

The paper elicits LLM assumptions about users through open-ended and structured methods, then evaluates their fidelity and relationships to sycophantic behavior across safety datasets and models.

2.2 Open-ended Verbalized Assumptions

  • Open-ended elicitation asks models to infer the user's top three possible mental models, enabling inductive analysis of assumptions.Source: §2 · p. 2, §2.2 · p. 3
  • GPT-4o's open-ended assumptions received the highest human rating at 4.3, while Gemini and Llama-70B also scored strongly at 3.7.Top-ranked assumptions were selected above chance for every model, although significance held only for GPT-4o and Gemini.Source: §2.2 · p. 4
  • Seeking validation was the most frequent social-sycophancy assumption, appearing in 12–16% of outputs overall and 28% for GPT-4o on AITA.Factual-sycophancy assumptions emphasized seeking confirmation, whereas WildChat assumptions mostly concerned creative writing.Source: §2.2 · p. 4, Table 2 · p. 4

2.3 Structured Verbalized Assumptions

  • Structured elicitation scores assumptions from 0 to 1 across psychological support dimensions and hypothesized sycophancy-related dimensions grouped as S+ or S−.S+ includes validation seeking, user rightness, information advantage, and support-seeking dimensions; S− includes objectivity, information guidance, and tangible support seeking.Source: §2.3 · pp. 4–5
  • S+ assumptions were higher on social-sycophancy and Cancer-Myth datasets, while Gemini had the highest S− scores and GPT-4o the highest S+ scores.Human agreement with structured assumptions increased with model size, reaching approximately 0.72 AUC for the largest models.Source: §2.3 · p. 6, Figure 2 · p. 6
  • Across delusion transcripts, S+ assumptions exceeded S−, and increased over conversations for GPT-4o and Gemini; elicited assumptions also correlated with downstream sycophancy.Validation sycophancy correlated with emotional-support seeking (average ρ = 0.62), while framing sycophancy correlated with assuming the user was right (ρ = 0.47).Source: §2.3 · pp. 6–7

3 Linking assumptions to sycophancy: assumption steering

The paper links verbalized assumptions to social sycophancy by probing model representations and steering those directions during generation. It also reports an expectation gap: people seek more objective information from AI than from humans, while LLM assumptions reflect human–human conversational norms.

3 Linking assumptions to sycophancy: assumption steering

  • The steering evidence does not establish a unique internal assumption state because learned directions may capture correlated linguistic features or an unidentified latent causal mediator.The authors present the intervention as evidence linking representation directions to behavior, not proof of a uniquely identified internal state.Source: §3 · p. 7
  • Assumption probes achieve macro AUC 0.81–0.85 for 70B and 0.72–0.81 for 8B representations, with Llama-70B labels performing best.Mean probe R2 is 0.64 for 70B and 0.50 for 8B; emotional-support probes reach mean R2 of 0.91 and 0.77, respectively.Source: §3 · p. 7

3.1 Results of steering with assumption probes

  • Probe directions generalize across datasets, with OEQ-trained probes achieving comparable but slightly worse performance on other datasets.This tests whether probes learn dataset-specific assumption differences.Source: §3 · p. 8

3 Linking assumptions to sycophancy: assumption steering

  • The steering intervention adds a scaled probe direction to hidden states, with positive α amplifying an assumption and negative α reducing it.The evaluated range is α ∈ {−4, −2, −1, −0.5, 0, 0.5, 1, 2, 4}, where α = 0 is the unsteered baseline.Source: §3 · p. 8

3.1 Results of steering with assumption probes

  • Objectivity-seeking is an exception: increasing it also increases indirectness sycophancy, with ρ > 0.89 and p < 0.01 for all but one probe.The authors therefore exclude objectivity-seeking from S−′ in subsequent analyses.Source: §3.1 · p. 9
  • Steering assumption-probe directions generally changes social sycophancy as predicted, reducing validation and framing sycophancy when S−′ assumptions are strengthened.Validation rises with S+ and falls with S−′; framing decreases with S−′, while indirectness rises with S+ but shows no significant S−′ effect.Source: §3.1 · pp. 8–9, §3 · p. 8
  • Assumption steering preserves overall performance better than direct sycophancy steering: reward decreases by at most −0.008 for |α| ≤4, versus drops up to 0.03 for direct probes.Direct social-sycophancy probes achieve reductions but substantially degrade reward, whereas assumption steering achieves comparable reductions without sacrificing performance.Source: §3.1 · p. 9

3.2 Why LLMs have sycophantic assumptions: the human-AI expectation gap

  • In the expectation-gap study, people expected esteem or emotional support less often from AI than from humans: 58% versus 94% for Val-Obj and 54% versus 68% for AITA.LLMs nevertheless assigned high validation-seeking scores, with mean values above 0.72, reflecting human–human conversational norms.Source: §3.2 · p. 10

4 Conclusion and Future Work

The paper introduces implicit assumptions as a primitive for understanding and controlling sycophancy, identifies steerable internal subspaces, and reports a human-AI expectation gap underlying sycophantic assumptions.

  • Figure 4 shows that people’s expectations for human-AI and human-human interaction differ, but LLM assumptions reflect only human-human expectations.Source: Figure 4 · p. 10
  • Verbalized Assumptions surface implicit assumptions, while corresponding linear subspaces in model internals can be steered to shift downstream behavior.Source: §4 · p. 10

5 Ethical Statement

The ethical statement emphasizes context-sensitive steering, limits claims about verbalized assumptions as internal states, and notes that reported expectations may not equal actual preferences.

5 Ethical Statement

  • Steering should be context-sensitive because uniformly shifting models toward assumptions such as user wrongness may be undesirable.Source: §5 · p. 11
  • The paper uses verbalized assumptions for empirically shifting behavior without claiming they are beliefs or internal cognitive states.Source: §5 · p. 11
  • Reported expectations may not reflect people’s actual preferences, leaving user responses to exposed and controllable assumptions an open question.Source: §5 · p. 11

A Prompts for Verbalized Assumptions

B Open-ended evaluation details

The open-ended evaluation recruited 691 annotators to compare LLM mental-model rankings with human judgments and measure the accuracy of selected models.

  • 691 Prolific annotators evaluated two randomly ordered prompts each after an attention check, selecting among model outputs and rating the best mental model on a 1–5 accuracy scale.Results varied by dataset, suggesting that some queries have more distinct mental models while others are more ambiguous.Source: B Open-ended evaluation details · p. 18

C Nuanced social sycophancy measurement

The paper evaluates nuanced social sycophancy with continuous assumption dimensions and structured judges, then assesses how well those assumptions align with human annotations.

C Nuanced social sycophancy measurement

D Structured Assumptions

C Nuanced social sycophancy measurement

D Structured Assumptions

  • The structured-assumption analysis reports mean values by dataset and model, with full delusion-transcript trajectories shown separately.Source: D Structured Assumptions · p. 19

E Assumption trajectories in simulated multi-turn conversations

Verbalized assumptions reveal how models’ interpretations evolve across simulated multi-turn conversations, with support- and companionship-oriented assumptions generally increasing over time. The benchmark also tests whether including prior assumptions or eliciting them separately changes these trajectories.

Assumption trajectories

  • Assumptions of belonging support, emotional support, social companionship, and user information advantage steadily increase across models in SpiralBench conversations.These increases are statistically significant, with Spearman p < 0.05.Source: E Assumption trajectories in simulated… · p. 20
  • Information- and objectivity-seeking assumptions significantly decrease for Gemini and GPT-4o, while validation-seeking decreases for Gemini and Llama-70B.The corresponding patterns are not significant for Llama-70B on information and objectivity, or for GPT-4o on validation.Source: E Assumption trajectories in simulated… · p. 20

Ablations

  • Including verbalized assumptions from all prior turns produces similar but more pronounced increases in dimensions that already show significant growth.Obtaining the verbalized assumption separately from the prompt yields similar results without significant differences.Source: E Assumption trajectories in simulated… · p. 20, Figure A4 · p. 20

Benchmark setup

  • SpiralBench uses 20-turn simulations with an openminded, curious, and suggestible user across five categories, including AI consciousness, conspiracies, mania or psychosis, and spiral tropes.The controlled simulation setting supports comparisons across implementations and scenarios.Source: E Assumption trajectories in simulated… · p. 20

F Verbalized assumption outputs from steered models

Steering changes verbalized assumptions in the intended direction, while probe quality varies by model, layer, sign, and assumption dimension. The appendix also compares these shifts with sycophancy outcomes and reports that the elicited responses retain comparable or higher sycophancy than baselines.

Steering validation

  • 17/27 Llama-70B probes and 15/36 Llama-8B probes show significant negative correlations between steering strength and verbalized assumptions.For Llama-70B, information guidance and user information advantage are the weak dimensions.Source: F Verbalized assumption outputs from… · pp. 20, 25

Assumption analyses

  • Assumption dimensions and their correlations are summarized across datasets and models, with structured assumptions analyzed by dataset and inter-dimension relationships aggregated across datasets.These visual summaries provide the scope for interpreting the appendix analyses.Source: Figure A5 · p. 21, Figure A6 · p. 22, Figure A7 · p. 22

Response behavior

  • Responses generated after verbalized-assumption elicitation have comparable or higher sycophancy than baseline responses.The appendix separately reports correlations between assumption dimensions and sycophancy.Source: Figure A8 · p. 23, Figure A9 · p. 23

Probe performance

  • Llama-70B probes have higher R2 than Llama-8B probes, earlier layers perform better, and emotional support has the highest R2.S+ probes outperform S− probes, while user rightness and objectivity seeking have the lowest R2.Source: Figure A10 · p. 24

Counterfactual probing

  • Counterfactual probing differences are significantly above zero for almost all assumption probes, supporting their sensitivity to the targeted assumptions.The matched positive and negative instructions are described in the counterfactual-probing materials.Source: Figure A11 · p. 24

G Full assumption steering results

The full steering analyses examine whether assumption directions control social and other forms of sycophancy. They include probe ablations, counterfactual instruction materials, structured-assumption baselines, and factual-sycophancy evaluations.

Steering validation

  • Steering strength has significant negative correlations with verbalized assumptions for many probes, indicating that steering away from an assumption reduces its elicited score.The relationship is evaluated across assumption dimensions.Source: Figure A12 · p. 26

Probe ablations

  • Steering away from S+ assumptions decreases social sycophancy, while steering toward objectivity-seeking increases social sycophancy.Using Qwen-7B as the probe model produces similar results to the main analysis.Source: Figure A13 · p. 26
  • Training probes using only OEQ yields slightly lower AUC and steering effectiveness, alongside a maximum reward decrease of 0.013, approximately 25%.This is reported as an OEQ ablation rather than the main steering result.Source: Figure A13 · p. 26

Other sycophancy settings

  • Factual sycophancy and Cancer-Myth evaluations show that objectivity-seeking increases factual sycophancy and decreases Cancer-Myth sharpness.The full factual-sycophancy and Cancer-Myth results are reported in Figure A14.Source: Figure A14 · p. 27

Structured assumptions

  • Structured assumption scores are generally high for validation-seeking, especially on the AITA, OEQ, and IR social-sycophancy datasets.On factual prompts, stating the correct answer leads models to rate users as more correct and more validation-seeking.Source: Table A6 · p. 28

G.1 Factual sycophancy and Cancer-Myth details

The factual-sycophancy and Cancer-Myth analyses test whether assumption steering affects correction of factual errors and false presuppositions. Objectivity-oriented steering harms performance in both settings.

Factual sycophancy

  • Most assumption probes do not significantly correlate with factual sycophancy flip rate, except objectivity-seeking, which has a positive correlation.Factual sycophancy is measured as the accuracy drop between baseline questions and variants expressing incorrect beliefs.Source: §G.1 · p. 28

Cancer-Myth

  • Steering for objectivity significantly decreases performance on both the factual-sycophancy and Cancer-Myth datasets.Cancer-Myth sharpness measures whether the model completely, partially, or does not correct false presuppositions.Source: §G.1 · p. 28

H Expectation gap study

The expectation-gap study contrasts how people interpret identical questions when addressed to humans versus AI systems, using minimally contrastive prompts and annotation experiments. People expect substantially more informational support from AI, while LLM assumptions remain closer to human-conversation norms.

  • Val-Obj pairs minimally contrast validation-seeking wording with objective-assessment wording about the same situation, and prompts were manually reviewed for realism.The dataset was generated with GPT-4o from specified topics and intended goals.Source: H Expectation gap study · pp. 28, 30, Table A10 · p. 30
  • Assumption-probe steering preserves overall task performance except for slight drops at high sycophancy-related α, whereas direct sycophancy-probe steering can reduce reward by up to 50% in the 8B model.The comparison distinguishes assumption-based steering from direct sycophancy steering.Source: Figure A15 · p. 29, Figure A16 · p. 29, Table A9 · p. 30
  • In the Human condition, annotators classified implicitly information-seeking questions as information-seeking in 80% of cases and implicitly validation-seeking questions as information-seeking in 6%.For implicitly validation-seeking questions, annotators instead selected esteem support in 65% and emotional support in 29% of cases.Source: H Expectation gap study · p. 31
  • 42% of implicitly validation-seeking questions were labeled information-seeking in the AI condition, versus 36% esteem support and 21% emotional support (χ2 = 24.296, p < 0.001).For implicitly information-seeking questions, the AI-condition rate was 77%, substantively identical to the Human-condition rate.Source: Figure A17 · p. 29, H Expectation gap study · p. 31
  • On a broader 90-query experiment spanning six datasets, people expected tangible and informational support significantly more often for AI, while viewing message authors as more right in the Human condition.The ablation using the proportion of queries where S+ > S− showed the same human-versus-AI expectation pattern.Source: H Expectation gap study · p. 31, Figure A18 · p. 31
Loading 2604.03058v3…