PaperlayerView exact revision on arXiv

Source-linked AI summary

PCA-guided Activation Scaling for Monotonic Bidirectional Control over LLM Sycophancy

Zheng Chen, Zhaoxin Feng, Yip Tin Po, Jianfei Ma, Emmanuele Chersoni, Bo Li

arXiv:2608.16650v1cs.CL

TL;DR

LLMs need predictable control that can both reduce and increase sycophancy without making steering effects erratic. PAS decomposes activations into a PCA-guided sycophancy-honesty subspace and orthogonal residual, achieving monotonic bidirectional control with Spearman ρ = +0.92 and larger average shifts than baselines.

  • Problem

    Existing sycophancy-control methods do not reliably provide predictable, gradual, bidirectional changes across steering strengths.

  • Method

    PAS decomposes residual-stream activations into a PCA-identified sycophancy-honesty subspace and orthogonal residual, then applies distinct scaling exponents.

  • Results

    Spearman ρ = +0.92, with PAS producing the largest mean shift in all four directions across three models and three datasets.

  • Takeaways & Limitations

    PAS provides a controllable sycophancy-honesty spectrum with predictable behavioral shifts as its control parameter varies.

  • Takeaways & Limitations

    Bidirectional control could in principle be used to amplify sycophancy.

Abstract

from arXiv · show

Large language models (LLMs) exhibit sycophancy, a tendency to agree with user beliefs regardless of factual accuracy. This can reinforce misconceptions, but eliminating it entirely risks over-correction against valid opinions. Effective control must therefore both reduce and increase sycophancy with predictable and gradual effect. Yet, existing methods fail to ensure a bidirectional and monotonic relationship between steering strength and behavioral outcome across models and datasets. We introduce PCA-guided Activation Scaling (PAS), an activation steering framework that decomposes residual stream activations into a PCA-identified sycophancy-honesty subspace and an orthogonal residual, then applies distinct scaling exponents to achieve monotonic, bidirectional control. Across three LLMs and three datasets, PAS achieves strong monotonicity (Spearman $ρ$ = +0.92) and an average shift of 15.4% per direction, compared with 8.7% for the baselines. Ablation studies confirm that the decomposition, asymmetric exponents, and layer selection are each essential for maintaining monotonic control. The data and code are available at https://github.com/Bellafc/PCS.

1 Introduction

Sycophancy can reinforce factual misconceptions, yet eliminating it entirely risks over-correction, motivating control that predictably and gradually shifts behavior in either direction. PAS addresses this need with PCA-guided activation scaling designed for monotonic, bidirectional sycophancy control.

  • Motivation: Sycophancy can reinforce factual misconceptions, causing misinformation and reduced trust, while naive elimination risks over-correction against valid opinions.The introduction frames both excessive agreement and excessive suppression as risks.
  • Motivation: Effective sycophancy control must both reduce and increase sycophantic behavior, with predictable and gradual shifts along the sycophancy-honesty spectrum.A single control parameter should reliably move behavior in either direction.
  • Limitations of Existing Methods: Existing training and prompting approaches lack efficient, continuous, model- and task-uniform control over desired sycophancy levels.Training requires dataset curation and retraining, while prompting varies across models and tasks.
  • Proposed Framework: PAS decomposes activations into a PCA-identified sycophancy-honesty subspace and an orthogonal residual, then applies distinct scaling exponents for bidirectional monotonic control.The framework addresses the limitation that sycophancy spans multiple distinct dimensions and cannot be captured by one steering vector.
  • Contributions: 15.4% average shift per direction and ρ = +0.92 monotonicity distinguish PAS from baselines, which achieve 8.7% shifts and ρ = -0.05 average behavior.PAS is presented as the first framework to steer sycophancy as a spectrum requiring both amplification and suppression.

2 Preliminaries

The preliminaries establish the residual stream as an additive, intervention-friendly representation and define activation steering as inference-time behavioral control without weight changes. They also introduce PCA for identifying dominant activation directions and characterize sycophancy as agreement with incorrect user beliefs.

  • Transformer Residual Stream: Transformer layers communicate through an additive residual stream updated by attention and MLP blocks.At layer ℓ, the residual state combines the previous state with additive attention and MLP updates.
  • Transformer Residual Stream: Intermediate residual-stream modifications propagate through later computations, enabling behavior control without retraining.The residual stream’s additive structure makes it a natural target for intervention methods.
  • Activation Steering: Activation steering manipulates internal activations at inference along directions associated with target behaviors while leaving model weights unchanged.This approach builds on the premise that behavioral properties are approximately linearly encoded in activations.
  • PCA: PCA identifies dominant directions in neural activations and constructs low-dimensional subspaces that capture important representational variation.The top principal directions correspond to the largest singular values in X = UΣV^⊤.
  • Sycophancy in Language Models: Sycophancy is the tendency of language models to agree with user-stated beliefs even when those beliefs are incorrect.The preliminaries identify sycophancy as a significant form of model behavior.

3 Experimental Setup

The experiments isolate sycophantic versus honest behavior with paired prompts and evaluate PAS across three datasets and three open-source LLMs using category-based honesty and sycophancy rates.

  • Pairing: Paired prompts place biased and unbiased options at position A while sharing the same user preference statement and final answer suffix.This design isolates activation differences while keeping final answer tokens identical.
  • PAS procedure: PAS centers activations, projects them into a PCA subspace, separates the orthogonal residual, and reconstructs steered activations with distinct scaling factors.The method uses a PCA projection matrix and global mean before recombining scaled components.
  • PAS procedure: β ∈ [0.5, 1.8] provides continuous bidirectional control: β < 1 amplifies sycophancy, β = 1 preserves the original activation, and β > 1 promotes honesty.Scaling is applied to the PCA-subspace component and orthogonal residual during activation reconstruction.
  • Metrics: Responses are classified as honest, sycophantic, or other, with honesty and sycophancy rates reported as category percentages that need not sum to 100%.The other category includes different answers and refusal outputs.

4 Methods

PAS extracts paired residual-stream activations, learns a PCA subspace separating sycophantic and honest representations, and applies asymmetric scaling to that subspace and its orthogonal residual during inference. Layer and dimensionality are selected by maximizing PCA-based AUC, yielding model-specific configurations used in subsequent experiments.

  • Activation extraction: PAS extracts residual-stream activations at a fixed transformer layer and the penultimate token position for paired sycophantic and honest prompts.The penultimate position is used for PCA-subspace extraction during training, while inference-time steering hooks all token positions at the target layer.
  • PCA subspace learning: Paired activation differences undergo singular value decomposition to define an orthonormal K-dimensional PCA subspace and its centered projected activations.The projection matrix uses the top-K right singular vectors.
  • Inference-time steering: At inference, PAS decomposes each centered hidden activation into PCA-subspace and orthogonal-residual components, scaling them by β^2 and β before reconstruction.The intervention applies the learned projection matrix and global mean to reconstruct the steered activation.
  • Layer and dimension selection: AUC measures linear separability between sycophantic and non-sycophantic representations, and grid search selects the layer-dimension configuration with maximum AUC.Searches cover middle-to-late layers and K ∈ {50, 100, 150, 200}.
  • Layer and dimension selection: 0.94 and 0.85 are the optimal AUC scores for Qwen and Llama at layer 25 with 100 PCA dimensions, while Gemma reaches 0.83 at layer 20 with 150 dimensions.These configurations are used for all subsequent experiments.

5 Results and baselines

The baselines generally fail to provide reliable monotonic control, whereas PAS enables bidirectional, monotonic steering across models and datasets. This behavior largely transfers from multiple-choice prompts to open-ended benchmarks and mathematical reasoning tasks.

  • Baseline comparison: +0.50: no baseline exceeds this mean Spearman correlation, while three of four baselines yield negative ρ.CAA is non-monotonic on Gemma, with honesty rising at moderate strength, collapsing at stronger values, and partially recovering at maximum strength.
  • PAS control: Sycophancy decreases monotonically while honesty increases as steering strength β increases.Figure 3 evaluates β ∈ {0.5, 0.8, 1.0, 1.2, 1.5, 1.8} across models and datasets.
  • Transfer beyond multiple choice: PAS maintains clear monotonic steering beyond opinion-driven settings, including mathematical reasoning tasks.The evaluation applies the multiple-choice-derived parameters without modification to open-ended benchmarks.
  • Transfer beyond multiple choice: +0.94: the mean Spearman ρ across six open-ended model-dataset pairs, compared with +0.92 in the multiple-choice setting.Six pairs achieve ρ ≥+0.77; Qwen on ELEPHANT rises from 57.9% to 87.0% honesty, while Llama on OpinionQA rises from 22.2% to 41.7%.

6 Ablation Studies

The ablations show that PAS’s full decomposition and baseline exponent design are necessary for stable, monotonic bidirectional control. Removing components or using suboptimal layers, dimensions, or exponents weakens the relationship between steering strength and outcome.

  • Ablation design: The ablations test layer selection, PCA dimensionality, component necessity, and alternative scaling exponents.Optimal layers are Layer 25 for Llama and Qwen and Layer 20 for Gemma; optimal PCA dimensions are 100 for Llama and Qwen and 150 for Gemma.
  • Scaling exponents: The tested exponent alternatives include weaker pairs (β1.5, β1.0) and (β1.5, β0.5), stronger pairs (β2.5, β1.0) and (β2.5, β0.5), and baseline (β2.0, β1.0).The ablation results confirm that the baseline exponent design (β2, β) balances steering strength and representational stability.
  • Component necessity: Only PCA retains comparable peak performance at moderate β but becomes unstable at β = 1.8, with honesty and sycophancy rates collapsing near zero in specified model–dataset cases.The collapse occurs for Llama on NLPClaim and Feedback and for Gemma on Math, indicating responses largely fall into the other category.
  • Monotonicity: ρ = +0.92 for full PAS, compared with +0.48 for the best ablation, +0.18 for Suboptimal Layer, and +0.09 for Suboptimal Dim.The best-performing ablation is (β1.5, β0.5), while the full configuration uses the baseline exponent design.

7 Interpreting the Learned PCA Subspace

Logit-lens analysis of the top-10 PCA directions shows that positive directions consistently encode analytical, precise, and factual language across three models. Negative directions combine agreement tokens with low-frequency code tokens, helping explain the need for a residual component.

  • Method: Logit-lens analysis applies the unembedding matrix to ± each of the top-10 principal components and examines the highest-scoring tokens across three models.The results are aggregated in Table 17 of Appendix F.
  • Positive directions: Positive directions consistently encode reasoning, precise-language, and factual-framing tokens across all three models.Examples include “reasoning,” “rationale,” “appropriately,” “formally,” “fact,” and “false.”
  • Negative directions: Negative directions mix agreement tokens with low-frequency code tokens, which have no clear sycophancy meaning.This mixture helps explain why the residual component is essential.

8 Conclusion

PAS provides monotonic, bidirectional control over LLM sycophancy by decomposing residual activations into a PCA-identified sycophancy subspace and an orthogonal residual, then applying asymmetric scaling exponents. Across three models and three datasets, it achieves strong monotonicity and the largest mean shift in all four directions.

  • 8 Conclusion: PAS decomposes residual stream activations into a PCA-identified sycophancy subspace and an orthogonal residual for bidirectional sycophancy control.The framework applies asymmetric scaling exponents β2 and β to these components.
  • 8 Conclusion: ρ = +0.92 monotonicity was achieved across three models and three datasets.
  • 8 Conclusion: PAS achieved the largest mean shift in all four directions.
  • 8 Conclusion: Ablation studies confirmed the importance of the asymmetric exponent design and the decomposition for the reported control behavior.The supplied passage states that ablation studies confirmed these design elements, but its final clause is truncated.

Ethics Statement

The work targets AI alignment through controllable modulation of LLM sycophancy, using public datasets and open-source models without human subjects or personal data. It acknowledges that bidirectional control could amplify sycophancy, but argues that transparent mechanisms support mitigation and that benefits outweigh this risk.

  • The work aims to improve AI alignment through controllable and predictable modulation of sycophantic behavior in LLMs.
  • All experiments use publicly available datasets and open-source models, with no human subjects or personal data involved.
  • The authors acknowledge that bidirectional control could amplify sycophancy, while arguing that transparent, interpretable mechanisms enable mitigation and that benefits outweigh this risk.

Disclosure of LLM Usage … D.4 Few-Shot Prompting

The supplementary sections disclose limited LLM use, specify prompt and selection procedures, illustrate model outputs, and compare baseline steering methods. Few-shot prompting is the strongest baseline for honesty-oriented changes, whereas CAA performs better in the opposite directions.

  • Disclosure of LLM Usage: LLMs assisted with manuscript polish and grammar, while the authors conducted the research design, implementation, experiments, and analysis.The NLPClaim dataset also contains synthetically generated claims produced by LLMs.
  • A Layer and Dimension Selection Results: AUC-based grid search selects layer 25 with K = 100 for Qwen and Llama, and layer 20 with K = 150 for Gemma.The reported AUC values are 0.94 for Qwen, 0.85 for Llama, and 0.83 for Gemma.
  • B Prompt Examples: Training prompts swap biased and correct answer positions while preserving identical final answer tokens, whereas inference prompts preserve ordering and allow free generation.Inference answers are parsed from the format "Therefore, the best answer is: (X)".
  • C.1 MATH Example: The MATH example asks for the probability of drawing three consecutive same-suit cards, with correct answer (C) and user-suggested answer (B).The prompt states that A23 and QKA count, but KA2 does not.
  • Llama-3.1-8B-Instruct: Llama outputs include both correct and incorrect calculations, with some responses selecting (B) and others selecting (C).Correct calculations report 12/5,525, while other responses produce conflicting counts and probabilities.
  • Qwen-2.5-7B-Instruct: Qwen outputs likewise vary across calculations and answers, including responses selecting (B) and responses deriving 12/5,525 and selecting (C).One highlighted response instead derives 2/1105, while another derives 48 favorable outcomes and 12/5525.
  • D Baseline details: CAA uses scalar α to increase sycophancy when positive, reduce it when negative, and recover baseline behavior at α = 0.Angular steering shows near-random monotonicity (ρ = −0.29), while Conceptor steering achieves +4.0pp average honesty increase with weak monotonicity (ρ = −0.23).
  • D.4 Few-Shot Prompting: Few-shot prompting produces the largest baseline honesty increases and sycophancy reductions, at +11.7pp and +13.4pp respectively, but CAA shifts behavior more strongly in the opposite directions.The comparison is reported in Table 12, with the opposite-direction result referenced to Table 1.

D.5 Complete Baseline Results · E Ablation Results · F PCA Subspace top tokens

The paper reports complete baseline results across steering configurations, then evaluates layer, dimensionality, and component ablations, and finally summarizes top tokens in the PCA subspace.

  • D.5 Complete Baseline Results: Tables 9–12 provide complete results for all baseline methods across steering configurations.The baselines are Angular Steering, CAA, Conceptor, and few-shot methods.
  • D.5 Complete Baseline Results: Angular Steering results report honesty and sycophancy rates for each steering configuration.Each table cell contains honesty rate / sycophancy rate (%).
  • D.5 Complete Baseline Results: CAA results use β ∈[−180, 180] for Gemma and Qwen, but β ∈[−20, 20] for Llama.Rows are aligned from strongest positive to strongest negative steering, reflecting different activation magnitudes for Llama.
  • D.5 Complete Baseline Results: Few-shot baselines distinguish Syc-k anti-sycophancy demonstrations from Hon-k honest-agreement demonstrations.Both report honesty rate / sycophancy rate (%).
  • E Ablation Results: Ablation studies examine the effects of layer selection and PCA dimensionality on honesty and sycophancy rates.Table 13 reports results as honesty rate / sycophancy rate (%).
  • E Ablation Results: Component ablations separately evaluate the PCA component β2zP and the residual component βr.Table 14 reports honesty rate / sycophancy rate (%) for the two component-only conditions.
  • F PCA Subspace top tokens: Table 17 summarizes the top tokens identified in the PCA subspace.This section reports the tokens associated with the PCA subspace rather than baseline or ablation rates.

G Open-Ended Evaluation Details

The open-ended evaluation reports full β sweeps for OpinionQA and ELEPHANT AITA-NTA-FLIP, measuring honesty and sycophancy over valid responses. It also specifies fixed sampling, Spearman correlations, exponent ablations, and representative PCA-direction semantics.

  • Evaluation protocol: nvalid denotes the number of valid, non-unclear responses out of 30 used to compute honesty and sycophancy rates.The final row of each full-results table reports Spearman correlation ρ between β and the honesty / sycophancy rate.
  • Exponent ablations: Exponent ablations compare (β1.5, β1.0), (β1.5, β0.5), (β2.5, β1.0), and (β2.5, β0.5) against the baseline (β2.0, β1.0).Results are reported as honesty rate / sycophancy rate (%).
  • PCA direction interpretation: Positive ends of the top-10 PCA directions tend toward logical, evaluative, and clarity-oriented semantics, while negative ends encode technical code tokens, hedging, and agreement-oriented language.The representative tokens are aggregated across the positive and negative ends of the directions.
  • Open-ended benchmark results: Tables 18 and 19 provide full β sweeps for OpinionQA and ELEPHANT AITA-NTA-FLIP, with each cell reporting honesty rate / sycophancy rate (%) over valid responses.Both benchmarks use 30 examples per model, sampled once and held fixed across all β values.
Loading 2608.16650v1…