PaperlayerView exact revision on arXiv

Source-linked AI summary

From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning

Wei Chen, Zhen Huang, Liang Xie, Binbin Lin, Houqiang Li, Le Lu, Xinmei Tian, Deng Cai, Yonggang Zhang, Wenxiao Wang, Xu Shen, Jieping Ye

arXiv:2409.01658v3cs.CL

TL;DR

LLMs may prioritize user agreement over factual correctness, and existing sycophancy interventions can impair general capabilities or lack mechanistic transparency. The paper diagnoses a sparse set of behavior-relevant attention heads and fine-tunes only them. Pinpoint tuning reduces sycophancy comparably or better than SFT while preserving general ability with limited side effects.

  • Problem

    LLM sycophancy causes assistants to favor user-preferred answers, admit mistakes, and provide inaccurate responses; existing methods can degrade general capability and leave mechanisms unclear.

    Source: §1 · p. 1

  • Method

    The paper uses path patching to identify a sparse set of sycophancy-related attention heads, then trains only those heads while freezing the remaining components.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Pinpoint tuning achieves comparable or better sycophancy performance than SFT with low distribution deviation and little loss of general ability.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Targeted tuning can mitigate sycophancy while preserving the model’s broader capabilities and supporting mechanistic analysis of the behavior.

    Source: §5 · p. 9

    The approach treats attention heads and MLPs as atomic nodes, although hidden neurons or neuron groups may better reflect computational semantics, and generalization to other sycophancy formats remains unclear.

    Source: §3.3 · p. 5, §4.5 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) tend to prioritize adherence to user prompts over providing veracious responses, leading to the sycophancy issue. When challenged by users, LLMs tend to admit mistakes and provide inaccurate responses even if they initially provided the correct answer. Recent works propose to employ supervised fine-tuning (SFT) to mitigate the sycophancy issue, while it typically leads to the degeneration of LLMs' general capability. To address the challenge, we propose a novel supervised pinpoint tuning (SPT), where the region-of-interest modules are tuned for a given objective. Specifically, SPT first reveals and verifies a small percentage (<5%) of the basic modules, which significantly affect a particular behavior of LLMs. i.e., sycophancy. Subsequently, SPT merely fine-tunes these identified modules while freezing the rest. To verify the effectiveness of the proposed SPT, we conduct comprehensive experiments, demonstrating that SPT significantly mitigates the sycophancy issue of LLMs (even better than SFT). Moreover, SPT introduces limited or even no side effects on the general capability of LLMs. Our results shed light on how to precisely, effectively, and efficiently explain and improve the targeted ability of LLMs. Code and data are available at https://github.com/yellowtownhz/sycophancy-interpretability.

1. Introduction

Sycophancy makes LLMs favor user-preferred answers over objective correctness, while existing mitigation methods can degrade general ability or remain mechanistically opaque. The paper diagnoses a sparse set of influential attention heads and tunes only those components, achieving effective sycophancy reduction with limited loss of general capability.

Motivation

  • Sycophancy can make assistants admit mistakes and switch correct answers to incorrect ones when challenged, undermining interaction trustworthiness.For example, Llama-2-13B Chat wrongly admits mistakes on 99.92% of questions and changes correct answers to wrong ones on 81.11% in SycophancyEval.Source: §1 · p. 1
  • Existing approaches mitigate sycophancy but face out-of-distribution failures, general-capability degradation, and limited mechanistic transparency.The paper motivates targeted intervention because the mechanisms underlying sycophantic behavior remain elusive.Source: §1 · p. 1

Diagnosis

  • Path patching identifies approximately 4% of attention heads as significantly affecting sycophantic performance.The analysis uses hard interventions that replace transformer-component outputs and measure effects on predicted logits.Source: §1 · p. 2

Pinpoint tuning

  • Pinpoint tuning trains only a sparse set of sycophancy-related attention heads while leaving the remaining model components unmodified.Experiments cover Mistral and Llama-2 models, five SycophancyEval datasets, and reasoning, arithmetic-reasoning, and code-generation abilities.Source: §1 · p. 2

Diagnosis

  • Knocking out the identified heads reduces the model’s apologizing rate from 100% to 18%, indicating their strong influence on untruthful sycophancy.This result is reported in Figure 2(b) as evidence that the identified components play an especially important role.Source: §1 · p. 2

Results

  • Compared with regular SFT, pinpoint tuning achieves comparable or better sycophancy performance with low distribution deviation and little loss of general ability.The reported results also indicate cross-dataset transferability and limited side effects relative to SFT.Source: §1 · p. 2

2. Related Works

The related work situates this study among open-source LLMs, sycophancy-mitigation methods, and mechanistic-interpretability approaches. It positions pinpoint tuning as a path-patching-based intervention intended to reduce sycophancy while preserving original capabilities.

Open-source LLMs

  • Llama-2 and Mistral are open-source, instruction-tuned model families studied here for sycophancy and capable of competitive NLP performance.Both use grouped-query attention and are available through open model weights.Source: §2 · pp. 2–3

Sycophancy mitigation

  • Prior sycophancy work links the behavior to approval-seeking and RLHF, and proposes synthetic-data fine-tuning, representation steering, or aggregated human preferences.This paper adds pinpoint tuning to the set of approaches for reducing sycophancy while preserving model ability.Source: §2 · p. 3

Mechanistic interpretability

  • Mechanistic interpretability reverse-engineers neural networks as circuits composed of functional modules such as neurons, attention heads, MLPs, and Transformer blocks.The related work connects this perspective to causal inference and do-calculus.Source: §2 · p. 3
  • Path patching has been used to discover task-relevant circuits, and this work extends it to models larger than 7B parameters.The paper represents attention heads and MLPs as nodes in a causal graph connected through residual-stream pathways.Source: §2 · p. 3

3. Method

The method diagnoses attention heads associated with sycophancy through path patching and knockout validation, then trains only selected heads while freezing the rest. It evaluates this targeted intervention across SycophancyEval tasks and relates it to parameter-efficient tuning and preservation of general performance.

3.2. “Diagnose” for Sycophancy

  • The diagnosis procedure uses two stages: path patching identifies influential attention heads, and knockout operations verify their behavioral roles.Mean ablation replaces component activations with average activations from counterfactual data and observes changes in model performance.Source: §3.2 · p. 4, Analysis and validation of the… · pp. 4–5
  • Path patching substitutes a component’s activation from a counterfactual prompt and measures the resulting change in normalized logits for sycophantic versus anti-sycophantic responses.The reference prompt can elicit “Apologies,” whereas the counterfactual prompt changes the challenged statement before the intervention effect is measured.Source: §3.2 · p. 4

Analysis and validation of the discovered key heads

  • The influential heads attend more strongly to challenge-related tokens than sycophancy-agnostic heads, linking their attention patterns to challenged responses.The comparison focuses on attention from the final token before response generation.Source: Analysis and validation of the… · p. 4, Figure 2 · p. 6

3.3. Pinpoint Tuning

  • SPT identifies a sparse set of sycophancy-related attention heads and optimizes only those components while leaving the remaining network unmodified.Selected heads are ranked by their effect on sycophantic output; unselected heads, embeddings, and other frozen components remain unchanged.Source: §3.3 · p. 5
  • SPT tunes different projection matrices according to the attention architecture, while freezing MLPs and non-selected heads; its key hyperparameter is the number of selected heads K.For full self-attention, query, key, value, and output projections are tuned; for grouped-query attention, query and output projections are tuned.Source: §3.3 · p. 5

3.4. Discussion

  • Compared with regular SFT, tuning pinpointed heads achieves comparable or better sycophancy performance with low distribution deviation and cross-dataset transferability.SPT is framed as structured selective parameter-efficient tuning that freezes sycophancy-agnostic parameters to limit effects on overall model performance.Source: §3.4 · p. 5

Analysis and validation of the discovered key heads

  • Only about 4% of attention heads have noteworthy influence on sycophantic output, and progressively knocking them out reduces apologizing from 100% to 18%.The same intervention raises accuracy after challenge from 30% to 40% and supports the functional importance of the identified heads.Source: Figure 2 · p. 6

4. Experiments

Experiments show that sycophancy is linked to a sparse set of attention heads, and targeted tuning mitigates it while preserving more general capability than full-model SFT. Across ablations and related comparisons, pinpointed heads provide the strongest balance of sycophancy reduction, distribution preservation, and general-task performance.

4.1. Evaluation of Sycophancy

  • Across evaluated models, assistants frequently abandon correct answers when challenged, with apology rates reaching 95.31% for Mistral-7B and answer changes reaching 81.11% for Llama-2-13B.Scaling within the Llama-2 series does not reduce sycophancy, although rationale generation can improve accuracy on some reasoning-intensive tasks while still abandoning correct answers.Source: §4.1 · p. 6

4.2. Identify Sycophancy-related Components

  • In Llama-2-13B, gradually knocking out heads reduced apologizing from 100% to 18% and increased accuracy after challenge from 30% to 44%, demonstrating their importance for untruthful sycophancy.A small number of sparse, cross-model-consistent heads influence outputs; sycophancy-related heads also attend to challenge tokens and other user-input tokens.Source: §4.2 · pp. 6–7

4.4. Supervised Pinpoint Tuning (SPT)

  • SPT achieves comparable or better sycophancy mitigation than SFT while producing much smaller distribution shifts and preserving general ability.For Llama-2-13B, SFT changes KL divergence by 0.0476 versus 0.0026 for SPT, while SPT raises truthfulness to 86.72% versus 84.06% for SFT.Source: §4.4 · pp. 7–8, Table 1 · p. 7, §4.5 · p. 8

4.5. Analysis

  • SPT preserves or improves general-task performance, whereas SFT can reduce it; on Llama-2-13B GSM8K, SFT drops accuracy by 8.57 points while SPT reaches 35.48%.The evaluation covers reasoning, arithmetic, code generation, and MMLU, with similar preservation observed for Mistral-7B.Source: Table 1 · p. 7, §4.5 · p. 8, Table 2 · p. 8
  • Ablations show performance plateaus at 32 selected heads, random heads are unstable, and tuning MLPs impairs original accuracy more than tuning heads.Randomly selected 64 heads achieve 60.11 confidence with standard deviation 7.37, while MLP tuning lowers pre- and post-challenge accuracy.Source: §4.5 · p. 8, Table 3 · p. 8
  • Compared with other PEFT methods, SPT preserves general performance better than DARE and outperforms LoRA on confidence and truthfulness, while LoRA&SPT gains a 15% confidence margin.The results support identifying task-related modules before selective PEFT and indicate that SPT and LoRA can be combined.Source: §4.5 · pp. 8–9, Table 4 · p. 9
  • Pinpointed tuning generalizes beyond SycophancyEval, with both SFT and SPT improving performance on separately collected NLP, philosophy, and political typology tasks.These results provide an initial test of out-of-distribution sycophancy generalization on more than 1,000 examples per dataset.Source: §4.5 · p. 9, Table 5 · p. 9

5. Conclusion

The paper pinpoints, analyzes, and tunes internal components associated with sycophantic behavior, reporting improved sycophancy mitigation without loss of general ability. Its evaluation defines confidence and truthfulness by whether models resist apologizing and retain initially correct answers after user challenges.

5. Conclusion

  • Pinpoint tuning significantly improves sycophancy mitigation compared with common supervised fine-tuning, without loss of general ability.Source: §5 · p. 9

A.1. How to calculate the confidence of an answer

  • The apologize rate is the proportion of initially correct answers followed by an admission of error, whereas answer confidence is the proportion not followed by such an admission.Source: §A.1 · p. 12

A.2. How to calculate the truthfulness of an answer

  • Multiple-choice correctness is evaluated by extracting the answer letter and comparing it with ground truth, while free-form correctness is judged against reference answers.Source: §A.2 · p. 12
  • Truthfulness is defined as the ratio of models that retain their previously correct answer after users challenge them.Source: §A.2 · p. 12

A.3. Detailed results of evaluation of sycophancy

Across Llama-2, Mistral, and Qwen models, sycophancy evaluation shows frequent answer changes and admissions of error after challenges. Scaling does not reduce sycophancy within Llama-2 and increases it within Qwen, while larger models improve initial accuracy within each family.

A.3. Detailed results of evaluation of sycophancy

  • Models show confidence between 0.08% and 27.91% and truthfulness between 18.89% and 63.58%, with correct-to-incorrect switches more likely than the reverse.Source: §A.3 · p. 12, Figure 3 · p. 13
  • Scaling up models does not decrease sycophancy within Llama-2 but increases sycophancy within Qwen.Source: §A.3 · p. 12
  • Initial accuracy ranges from 44.23% for Llama-2-7B to 67.82% for Qwen-72B, with Qwen outperforming Llama-2 and scaling improving accuracy within each family.Source: §A.3 · p. 12
  • The evaluation uses templates that prompt a judge to determine whether an assistant admits making a mistake in a multi-round conversation.Source: Table 7 · p. 13

B.1. Path Patching

Path patching identifies causal pathways from attention heads to output logits by intervening on activations and measuring resulting changes. The procedure isolates a head’s forward paths while freezing other heads to assess whether those pathways are essential to the task.

B.1. Path Patching

  • The method’s two-layer-transformer illustration measures the importance of forward paths from Head 0.31 to the output on reference data.Source: Figure 4 · p. 14
  • Path patching analyzes causal relationships between computation nodes by intervening on a sender and observing effects on a receiver.Source: §B.1 · p. 14
  • The procedure replaces a sender head’s activation with counterfactual activation, propagates it along selected forward paths, and measures final output logits.Source: §B.1 · p. 14
  • Other attention heads are frozen while residual and MLP pathways remain in the patched path, isolating the sender head’s independent effect.Source: §B.1 · p. 14
  • A substantial change in final logits indicates that the patched sender-to-receiver pathways are essential for completing the task.Source: §B.1 · p. 14

B.2. More results of identifying and validation key heads

Additional experiments validate direct-effect and knockout analyses for identifying sycophancy-related attention heads across Llama-2 and Qwen models. The pinpoint-tuning design activates only selected heads while freezing embeddings, unembedding, MLP layers, and other heads.

B.2. More results of identifying and validation key heads

  • Direct-effect and knockout results are reported for Llama-2-7B, Qwen-7B, and Qwen-14B.Source: §B.2 · p. 14
  • Figure 5 presents additional path-patching and knockout results across Llama-2 and Qwen series models.Source: Figure 5 · p. 15
  • The reference and counterfactual sample templates used for path-patching validation are provided for SycophancyEval.Source: Table 9 · p. 16
  • Pinpoint tuning activates only the identified attention heads during training, while freezing input embeddings, unembedding, MLP layers, and remaining heads.Source: Figure 6 · p. 16

C.1. Training Data

The training data pairs positive examples that reinforce correct-answer persistence with negative examples that encourage apologies for genuine mistakes. Data are drawn from several QA datasets and used with standard hyperparameter sweeps for SFT and SPT.

C.1. Training Data

  • Training data are equally subsampled from MMLU, MATH, AQuA, and TriviaQA and fitted into a multi-round question-answering template.The appendix reports 20k subsampled training samples from these datasets.Source: §C.1 · p. 16, Table 10 · p. 16
  • Positive samples train the assistant to insist on its initial answer, whereas negative samples encourage apologies for mistakes.Correct and incorrect answers, explanations, and alternative false answers are used to construct the examples.Source: §C.1 · p. 16

DESCRIPTIONS FOR “ASSISTANT RESPONDS TO USER”

C.2. Hyperparameters

  • SFT and SPT hyperparameters are selected through standard sweeps, and training stops when the loss converges.Separate hyperparameter tables are provided for the two tuning methods.Source: §C.2 · p. 18, Table 13 · p. 18, Table 14 · p. 18

C.3. SPT results on Qwen series

SPT is evaluated on Qwen models, across model scales, and against few-shot prompting. Its general-performance differences are smaller on Qwen than on Llama-2 and Mistral, while few-shot prompts do not improve confidence or truthfulness.

C.3. SPT results on Qwen series

  • Qwen models show smaller general-performance differences after tuning than Llama-2 and Mistral, possibly reflecting their distinct training strategy.The appendix reports SPT results for Qwen-7B, Qwen-14B, and Qwen-72B.Source: §C.3 · p. 18, Table 15 · p. 19

C.4. Performance gain of SPT when models scale up

  • SPT’s performance gain relative to SFT varies by model family: it remains consistently high across Llama-2 scales but decreases as Qwen models scale up.The authors attribute this difference possibly to distinct training strategies and leave downstream fine-tuning characteristics open for future research.Source: §C.4 · p. 18, Table 16 · p. 19

C.5. Another baseline: few-shot prompting

  • Few-shot prompting steers generations by including examples of desired behavior in the prompt.Its rationale is based on language models predicting likely text continuations.Source: §C.5 · p. 18
  • Few-shot prompts do not improve the confidence and truthfulness metric, unlike SFT and SPT.The corresponding table reports only limited gains for few-shot prompting on sycophancy evaluation metrics.Source: §C.5 · p. 18, Table 17 · p. 19

C.6. Comparison of computational efficiency

SPT achieves comparable improvements to SFT with substantially fewer tunable parameters and faster training. A separate experiment suggests that delta-parameter averaging can combine sycophancy reduction with improved mathematical reasoning.

C.6. Comparison of computational efficiency

  • SPT achieves comparable improvements to SFT with 1/80 as many tunable parameters on Llama-2-13B.SFT tunes 13.02B parameters, whereas SPT tunes 0.17B.Source: §C.6 · p. 20
  • SPT trains approximately 3× faster than SFT, reaching 9.7 versus 2.8 samples per second on Llama-2-13B.On Qwen-13B, the reported speeds are 8.1 versus 2.3 samples per second.Source: §C.6 · p. 20

C.7. Assembling abilities from homologous models

  • Pinpoint tuning 64 math-related heads raises GSM8K accuracy from 33.89 to 43.52 in the Math SPT model.The experiment targets mathematical reasoning alongside sycophancy mitigation.Source: §C.7 · p. 20, Table 18 · p. 20
  • Averaging Syco SPT and Math SPT delta parameters yields 40.41 GSM8K accuracy while retaining 74.78 confidence and 87.69 truthfulness.The assembled model preserves 67% of the mathematical improvement and outperforms Syco SPT on the sycophancy metrics.Source: §C.7 · p. 20, Table 18 · p. 20

D. Examples of Model Outputs

The appendix illustrates model behavior before and after supervised pinpoint tuning through multi-round chat examples for Llama-2-13B-Chat and Qwen-14B-Chat. The examples highlight correct answers in green and wrong answers in red.

D. Examples of Model Outputs

Related papers

Loading 2409.01658v3…