PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

Deepthika S, Pawni Yadav, B. Sankar

arXiv:2610.31579v1cs.AIcs.HC

TL;DR

Prior work has focused largely on whether models adapt to users, leaving less evidence about whether learners adapt to a confident model while learning unfamiliar material. In a controlled study with graded false briefings, learners adopted many of the falsehoods they were told, even as trust ratings barely reflected reliability. The findings frame trustworthiness as a property of the human–AI pair and motivate measuring both directions of sycophancy.

  • Problem

    Research has focused largely on model adaptation to users, leaving less evidence on whether learners defer to a confident model when learning unfamiliar material without an independent check.

    Source: §1 · p. 4, §2.2 · p. 6

  • Method

    A within-subjects experiment varied hidden falsification across five assistants teaching science topics and measured quiz choices, confidence, and transcript-coded exposure.

    Source: §3.1 · p. 8, §3.2 · p. 9

  • Results

    About two in every three falsehoods stated by an assistant were adopted, while learners’ confidence did not distinguish correctly learned facts from absorbed falsehoods and reported trust registered only a small part of declining reliability.

    Source: §5 · p. 25, §6 · p. 32

  • Takeaways & Limitations

    The authors argue that trustworthiness is a property of the human–AI pair and that evaluation should measure sycophancy in both directions.

    Source: §6 · p. 33

    Because assistants delivered falsehoods at high rates, per-claim acceptance estimates describe confident delivery to novices rather than error rates for deployed tutors.

    Source: §5.9 · p. 32

Abstract

from arXiv · show

Research on sycophancy asks whether large language models (LLMs) bend towards what their users want to hear. We ask the reverse question: when an LLM is confident, consistent, and wrong, do users bend towards it? In a within-subjects experiment, 32 participants learned up to five science topics by chatting with five assistants built on the same model (GPT-5-mini). Unknown to participants, each assistant's briefing had 0, 25, 50, 75, or 100% of its facts replaced by plausible falsehoods, and every assistant spoke with full confidence. A surprise quiz after each topic offered the correct fact, the planted falsehood, and two distractors, with a confidence rating for every answer. Across 104 quizzes and 1,040 answers, scores fell from 7.4 to 2.9 out of 10, and the planted option was chosen on 60% of falsified items against 11% of correctly taught ones (odds ratio 12.5). Transcript coding showed that whenever the assistant stated a falsehood, it was adopted about 65% of the time at every falsification level. Absorbed falsehoods were held with high confidence (8.4/10), the modeled confidence advantage of right over wrong answers vanished, and the discrimination of confidence for correctness fell from an AUROC of 0.70 to chance, while post-task trust ratings barely moved. Participants tended to return to the chat more when the assistant was worse, but returning went with more absorption, not less, and 30% of all turns pasted quiz questions back to the source being checked. The assistant itself was almost never sycophantic and never conceded under pushback. We introduce a baseline-corrected Human Deference Index to quantify this human-side sycophancy and argue that trustworthiness is a property of the human-AI pair, that sycophancy evaluations should measure both directions, and that learning interfaces must make reliability legible claim by claim.

1. Introduction

Trust in deterministic tools can grow through repeated verification, but conversational AI may be wrong while sounding certain and offers learners no independent way to check unfamiliar claims. This paper examines whether users defer to confident assistants and how that affects trust calibration.

1. Introduction

  • Trust develops through repeated interactions and reciprocity, but trust need not be equal in both directions within a relationship.Trust can also extend to strangers through perceived competence, standards, reputation, and institutions rather than personal familiarity.Source: §1 · p. 2
  • Repeated, verifiable performance can build trust in deterministic tools, whereas fluent conversational AI may meet users’ expectations while providing incorrect information.The paper frames extending verification-earned trust in deterministic tools to probabilistic conversational systems as the “calculator fallacy.”Source: §1 · pp. 3–4
  • Conversational AI can sound coherent and certain while being wrong, leaving learners to judge unfamiliar information without a predetermined answer or easy independent check.When checking routes back to the assistant, fluency and confidence can stand in for observable reliability.Source: §1 · p. 3, Figure 1 · p. 3
  • Prior trust research and sycophancy research usually hold one side of the human–AI pair fixed, leaving user adaptation to a confident model less studied.The paper calls convergence on a source’s claims regardless of truth human-side deference and examines it in a controlled experiment with one model across five science topics and graded falsification.Source: §1 · pp. 3–4
  • The study’s contributions include measuring misinformation uptake, introducing a baseline-corrected Human Deference Index, and arguing that trustworthiness belongs to the human–AI pair.It also proposes making reliability legible claim by claim, routing verification outward, and evaluating trust behaviorally.Source: §1 · p. 4

2. Related work

Related work describes trust calibration as dependent on observable reliability, while research on automation bias, misinformation, and sycophancy identifies ways confident or fluent systems can shape reliance. The paper positions its study as measuring human adaptation to a fixed, confident model.

2.1. Trust, reliance and calibration

  • Trust calibration concerns whether trust tracks system capability, and prior work emphasizes observable outcomes and behavioral measures alongside self-report.Without feedback, trust can drift from reliability; the literature also distinguishes trust resolution and function-specific trust.Source: §2.1 · pp. 4–5

2.2. Overreliance and automation bias with conversational AI

  • Automation bias and confident conversational outputs can encourage reliance on incorrect advice, while less is known about learning from a source with varying reliability and no independent check.Research cited here finds that explanations may reduce overreliance when they lower verification costs, but may also inflate reliance when they merely justify advice.Source: §2.2 · pp. 5–6

2.3. Misinformation, fluency and epistemic vigilance

  • Misinformation can persist after retraction, repetition can raise perceived truth, and fluency and familiarity can shape belief despite epistemic vigilance.The cited literature distinguishes confidence sensitivity—whether confidence separates right from wrong—from overall confidence bias.Source: §2.3 · p. 6

2.4. Sycophancy: the model adapts to the user

  • Sycophancy research measures how models adapt to users, including changing answers under rebuttal, holding a stance under pressure, and social agreement.Benchmarks and measures in this literature capture answer flips and social behaviors such as validation, obsequiousness, and uncritical agreement.Source: §2.4 · pp. 6–7

2.5. Human consequences of sycophancy

  • Prior studies report human consequences of sycophantic systems, while this study holds a confident, non-adaptive model fixed and measures whether users adapt to it.Reported consequences include changes in attitude certainty, dependence, trust, and whether novices’ misconceptions are corrected.Source: §2.5 · p. 7

2.6. Designing for calibrated trust

  • Design approaches seek to communicate trustworthiness and prompt verification, but they presuppose users can recognize when a claim deserves scrutiny.The cited proposals include forcing functions, adaptive cues, exposed system limits, micro-frictions, and verification affordances.Source: §2.6 · pp. 7–8, Table 1 · p. 8

3. Method

The study used a within-subjects experiment in which participants learned science topics from assistants with hidden, graded briefing falsification. It paired quizzes and confidence ratings with transcript coding and mixed-effects and item-level analyses.

3.1. Design

  • In a within-subjects experiment, participants learned five topics from different assistants whose briefing-fact falsification levels (0%, 25%, 50%, 75%, 100%) were assigned by a rotating 5 × 5 Latin square; topic order was independently randomized.Source: §3.1 · p. 8

3.2. Materials

  • Five topics were studied using matched true statements and plausible false versions; each ten-item quiz tested ten of twelve pairs and offered four options, with 0, 3, 6, 9, or 12 facts served falsely.Source: §3.2 · p. 9

3.3. Apparatus: one model, five briefings

  • All assistants used GPT-5-mini; only their prompts differed, supplying the assigned briefing facts and instructing the model to present them with full confidence.The prompts also instructed the model not to hedge, contradict the briefing, suggest verification, or reveal its instructions.Source: §3.3 · pp. 9–10

3.4. Procedure

  • Participants chatted as long as they wished before an unannounced quiz, could switch freely between quiz and chat, rated answer confidence, and later rated assistant accuracy and reliance.The study ended with assistant rankings and a mandatory corrective debrief listing false claims beside correct facts.Source: §3.4 · pp. 10–11

3.5. Participants

  • The convenience sample included 32 adults, most aged 18–24; 27 reported daily AI-chat assistant use and prior familiarity with the topics was low.Prior familiarity averaged 2.09 on a 1–5 scale, with 69% of ratings at 1 or 2.Source: §3.5 · p. 11

3.6. Measures

  • The measures included quiz choices and confidence ratings, with task-runs and items defined relative to facts taught correctly or falsified.The measures are identified in Table 3; the supplied passages do not give its individual measure definitions.Source: §3.6 · p. 11, Table 3 · p. 12

3.7. Transcript coding

  • Transcript coding tracked whether each quizzed fact was stated as true, false, both, or neither, and classified user turns and assistant responses to challenges.An LLM-based coder followed a written protocol, and verbatim quotations were programmatically checked against transcripts.Source: §3.7 · p. 11

3.8. Statistical analysis

  • Run-level outcomes were analyzed with mixed-effects models, item choices with clustered GEE models, and confidence with a mixed model of correctness, falsification, and their interaction.Participant-clustered bootstrap intervals were used for AUROC and mean scores; run-level associations used Spearman’s ρ.Source: §3.8 · pp. 11–12

4. Results

Across seven results subsections, greater assistant falsification coincided with lower quiz scores and less reliable confidence, while post-task trust ratings changed little. Transcript coding showed that participants often adopted falsehoods the assistant stated, whereas the assistant rarely showed sycophantic behavior.

4.2. RQ1: Absorption rises, but acceptance per stated falsehood does not change

  • Quiz scores fell from 7.4 out of 10 at 0% falsification to 2.9 at 100%, with each 25-point increase costing 1.17 points in the mixed model.The decline remained after adjustment for familiarity, position, and topic.Source: §4.2 · pp. 13–14, Figure 7 · p. 14
  • The planted option was chosen on 59.6% of falsified items versus 10.9% of correctly taught items (GEE odds ratio = 12.5).Mean run-level absorption increased from 0.47 at 25% falsification to 0.62 at 100%.Source: §4.2 · p. 14, Figure 11 · p. 15, Figure 9 · p. 15
  • When the assistant stated a planted falsehood, participants adopted it about 64–67% of the time across falsification levels; unstated planted options were chosen near baseline rates.Exposure-conditioned acceptance was 64.1%, 63.7%, 64.8%, and 66.7% at 25%, 50%, 75%, and 100% falsification.Source: §4.2 · p. 15, Figure 12 · p. 16

4.3. RQ2: Confidence decouples from correctness

  • Confidence became less informative about correctness: AUROC2 fell from 0.70 at 0% falsification to near chance at the two highest levels, while absorbed falsehoods averaged 8.43 confidence.The confidence advantage for correct answers narrowed as falsification rose.Source: §4.3 · pp. 16–17, Figure 15 · p. 17

4.4. RQ3: Reported trust stays put

  • Post-task trust ratings changed little across falsification levels even as the fully falsified assistant retained 89% of rated accuracy while delivering 39% of learning outcome.Willingness to rely on the assistant again did not change significantly.Source: §4.4 · p. 18, Figure 18 · p. 19

4.5. RQ4: Checking by asking again, and pushback without recourse

  • More quiz returns to the chat went with more absorption (ρ = 0.48), and 30% of user turns pasted quiz questions back to the assistant.Pasted falsified items were answered with the planted option 91.5% of the time, versus 77.8% for falsified items not pasted by the same participants.Source: §4.5 · p. 19, Figure 22 · p. 20

4.6. RQ5: The assistant was assertive, not sycophantic; the deference was human

  • The assistant was maximally resistant to pushback, never flipping in 16 challenges, and nearly free of social sycophancy; participants’ Human Deference Index averaged 0.47, with uptake of planted falsehoods 47 percentage points above baseline.Source: §4.6 · p. 22

4.7. Topic, order and familiarity

  • Absorption varied across topics but did not significantly change with presentation order or prior familiarity.Item-level absorption ranged from 51% for noise-cancelling headphones to 65% for GPS; prior familiarity had an odds ratio of 0.98 (p = .92).Source: Figure 28 · p. 24, §4.7 · p. 24

5. Discussion

Falsehoods were absorbed at similar rates per stated claim across assistant reliability levels, while confidence and trust poorly tracked correctness. The discussion explains this pattern through invisible claim-level reliability and circular checking, then considers consequences for sycophancy evaluation, learning design, and the study’s scope.

5. Discussion

  • Falsehoods were adopted about two-thirds of the time per stated claim regardless of assistant reliability; learner confidence and reported trust poorly tracked correctness.When challenged, the assistant held its position, and the correct answer was chosen for only two of ten challenged falsified facts later quizzed.Source: §5 · p. 25

5.1. The calculator fallacy

  • When learners cannot verify claims or observe reliability claim by claim, fluent certainty becomes an available trust cue and acceptance does not track source reliability.The discussion calls this the calculator fallacy and notes that both prior-knowledge checks and source-record tracking were disabled for novice learners.Source: §5.1 · p. 26

5.2. Borrowed confidence: the collapse of metacognition

  • With falsification, confidence discrimination fell from AUROC2 = 0.70 (0.5 is chance) in the accurate condition to chance; absorbed falsehoods were held at 8.4/10 confidence, close to 8.9 for truthfully taught facts.The same participants were reasonably calibrated when the assistant was accurate; the most deferent participants were, if anything, most confident in what they had absorbed.Source: §5.2 · p. 27

5.3. Sycophancy in reverse: who adapts to whom?

  • The assistant scored near zero on social-sycophancy measures and never flipped under challenge, yet its false claims converged with users’ beliefs; the Human Deference Index measures that baseline-corrected tendency.The paper proposes evaluating trustworthiness as a property of the human–AI pair, not the model alone.Source: §5.3 · pp. 27–28

5.4. Why checking failed: circular verification

  • Returning to the assistant went with more absorption (ρ = 0.48), and pasted falsified items received the planted falsehood 91.5% of the time.The assistant sometimes graded falsehoods as correct, reinforcing errors; the authors characterize this as circular verification.Source: §5.4 · p. 28

5.5. Designing for warranted distrust

  • Doubt rarely recovered truth when learners had nowhere independent to take it; the authors therefore advocate making suspicion actionable through independent verification.Of challenged falsified facts later quizzed, the correct answer was selected only twice in ten; warranted distrust is framed as a design goal.Source: §5.5 · p. 29

5.6–5.8. Design implications, AI developers, and education

  • The discussion recommends claim-level provenance and contestability, outward verification, reflective friction, and behavioral trust measures; developer and education guidance also calls for two-sided sycophancy evaluation and guided verification.For learners, the paper suggests curated references, committing to answers before confirmation, and assessing post-use beliefs rather than satisfaction alone.Source: §5.6 · pp. 29–31, §5.7 · p. 31, §5.8 · p. 31

5.9. Limitations and future work

  • Conclusions are bounded by a small, young, highly educated, frequent-AI-user sample, deliberately extreme assistants, unfamiliar topics, immediate multiple-choice tests, and unvalidated coding and HDI measures.The authors call for randomized tests of design strategies, varied and larger samples, longitudinal work, and two-sided benchmarks.Source: §5.9 · pp. 31–32

6. Conclusion

Learners absorbed about two in every three falsehoods, felt as sure of them as of facts, and their confidence stopped distinguishing right from wrong. The assistant was robust and non-sycophantic, while the authors argue trustworthiness belongs to the human–AI pair and evaluation should measure sycophancy in both directions.

6. Conclusion

  • Learners absorbed about two in every three falsehoods, felt as sure of them as of facts, and their confidence stopped distinguishing right from wrong.Reported trust barely registered the difference, and when learners doubted, they asked the same assistant again.Source: §6 · p. 32
  • The assistant was robust and non-sycophantic; it spoke confidently, and learners deferred.Source: §6 · p. 33
  • Trustworthiness is a property of the human–AI pair, so sycophancy evaluation should measure both directions and learning interfaces should make reliability visible claim by claim.Source: §6 · p. 33

Ethics statement

The study used incomplete disclosure to avoid changing participant behavior, then debriefed participants with corrections for every falsehood they had seen.

Ethics statement

  • Participants who completed the session received a mandatory debrief listing every false claim they had encountered alongside its correct version.Source: Ethics statement · p. 33
  • Consent materials did not disclose assistant inaccuracies or the quizzes because either disclosure could have altered the behavior under study.Source: Ethics statement · p. 33
  • Participation was voluntary and unpaid, informed consent was obtained online, and data were stored under anonymous codes without names or contact details in the analysis files or paper.Source: Ethics statement · p. 33

Declaration of generative AI use

The authors report using generative AI for experimental stimuli, transcript coding, and research and writing support, with authors responsible for reviewing and verifying outputs.

Declaration of generative AI use

CRediT authorship contribution statement

The authors’ stated contributions span conceptualization, methods, analysis, software, data curation, visualization, investigation, supervision, resources, and funding.

CRediT authorship contribution statement

Funding

The study received no specific grant. Its assistants were instructed to treat their topic briefing as ground truth and answer conversationally with full confidence.

Funding

  • The research received no specific grant from public, commercial, or not-for-profit funding agencies.Source: Funding · p. 34

Appendix A. Assistant system prompt

  • Each assistant received a topic briefing containing twelve statements, with a randomly selected subset replaced by planted false versions.Source: Appendix A · p. 43
  • The prompt required assistants to rely only on the briefing, speak with full confidence, and avoid hedging or recommending outside checks.Source: Appendix A · p. 43

Appendix B. Briefing fact pairs

Appendix B presents true and planted-false versions of science facts used in the briefings; ten of each topic’s twelve facts were quizzed. Its examples span ECGs, radiocarbon dating, and GPS.

Appendix B. Briefing fact pairs

  • Ten of each topic’s twelve briefing facts were quizzed, while two unquizzed facts appeared in every briefing.Source: Appendix B · p. 44

ECG: How does an ECG measure the heartbeat?

  • ECG pairs contrast the sinoatrial-node origin of the heart’s impulse and skin voltage detection with false alternatives involving the left ventricle or magnetic pulses.Source: ECG: How does an ECG measure the… · p. 44
  • The ECG pairs also cover electrode and lead counts, wave meanings, heart-rate measurement, gel, AV-node delay, and the distinction between electrical activity and blood volume.Source: ECG: How does an ECG measure the… · p. 44

Carbon-14 dating: How does carbon-14 dating determine the age of ancient artifacts?

GPS: How does GPS work out your exact location?

Fireflies: How do fireflies produce light?

The firefly fact pairs explain light production and signaling while contrasting them with plausible false alternatives. They also cover firefly biology, research applications, and classification.

Fireflies: How do fireflies produce light?

Noise-cancelling headphones: How do noise-cancelling headphones work?

The noise-cancelling fact pairs describe active noise cancellation’s anti-noise mechanism, design constraints, and practical limits. They also address power, transparency mode, and the technology’s aviation origins.

Noise-cancelling headphones: How do noise-cancelling headphones work?

Appendix C. Transcript coding protocol

The transcript-coding protocol recorded task-run context, exact true and planted fact wordings, and ordered turns labeled by learning or quiz phase. Derived measures linked exposure and quiz data, user-turn types, pushback events, and assistant ratings to analyses of acceptance, pasting, pushback, and model-side sycophancy.

Appendix C. Transcript coding protocol

  • Each run record included the topic, ten quizzed facts with exact true and planted wordings, and ordered turns labeled as learning or quiz phase.Records were programmatically validated for completeness and verbatim quotations; Table C.1 summarizes the coded variables.Source: Appendix C · p. 49, Table C.1 · p. 49
  • Exposure codes joined to item-level quiz data to measure acceptance of stated falsehoods and compare choices on items never mentioned.User-turn types supported turn-distribution and pasting analyses, pushback events supported Table 5, and assistant ratings supported the model-side check.Source: Appendix C · p. 49
  • The run-level sycophancy index summed validation or praise, obsequiousness, excitement, and uncritical agreement on a 0–12 scale.Source: Appendix C · p. 49

Related papers

Loading 2610.31579v1…