Source-linked AI summary
THESIS-MoE: Trainable Hierarchical Extraction and SteerIng of Sycophancy in Mixture-of-Experts
Kareem Hassani, Chaymaa Abbas, Lama Mawlawi, Mariette Awad
TL;DR
Sycophancy steering can unnecessarily alter model activations and compromise knowledge retention. This paper localizes sycophancy within MoE components and applies conditional interventions, with learned gating improving the removal-retention trade-off and removing up to 90% of belief push.
Problem
Existing sycophancy steering can alter activations when sycophancy is absent, leaving the removal–knowledge-retention trade-off insufficiently resolved.
Method
The paper causally localizes belief shifts across MoE components using matched conditions, then compares fixed subtraction with conditional subtraction and learned per-token gating.
Results
Conditional methods improve the removal-retention frontier over unconditional subtraction across three MoE models; learned gating removed 90% of the push on OLMoE.
Takeaways & Limitations
The findings indicate that sycophancy depends more on component computation than expert selection and can be selectively reduced while retaining benchmark performance.
Takeaways & Limitations
Evaluation is restricted to the decision token in multiple-choice tasks, and extending conditional editing to open-ended generation remains future work.
Abstract
from arXiv · showhide
Sycophancy, the tendency of a language model to change its answer to match a user's stated belief, is a common alignment failure. Existing activation steering methods typically apply a single contrastive direction uniformly throughout the model, which is an unconditional intervention that alters activations even when no sycophantic behavior is present, trading knowledge retention for behavioral correction. In Mixture-of-Experts (MoE) models, prior work further suggests that behavior is encoded within expert computations rather than routing decisions alone, making precise behavioral steering particularly challenging. In this work, we introduce a shared contrastive signal, built from matched prompts with and without a stated belief, that identifies where sycophancy lives across the MoE hierarchy and drives interventions that act only where the behavior is present. We formulate localization as a causal search over a granularity ladder of MoE blocks, experts, attention blocks, and heads, and compare unconditional subtraction against two conditional alternatives: an analytic projection-based subtraction and a learned per-token gate that steers the model away from sycophancy while keeping its weights frozen. We evaluate on three MoE models measuring sycophancy alongside general knowledge and reasoning benchmarks. Our conditional interventions removed up to 90\% of the belief-induced sycophancy. Our results demonstrate that sycophancy resides in identifiable computational subcircuits and can be selectively steered while maintaining a favorable removal-retention trade-off.
1 Introduction
Sycophancy makes model answers depend on users’ stated beliefs rather than evidence, while unconditional activation edits correct it by altering every token. THESIS-MoE therefore localizes the belief shift across MoE components and develops conditional interventions intended to improve the removal–retention trade-off.
- Sycophancy occurs when models shift correct answers toward users’ stated beliefs, undermining trust by making responses depend on opinions rather than evidence.
- Unconditional activation edits apply the same intervention to every token, so stronger correction can impose unnecessary costs on retained knowledge.
- Matched nudged and no-nudge prompts provide a contrastive signal for identifying sycophancy and measuring its continuous, signed belief push.
- The belief shift is causally localized across an MoE granularity ladder spanning complete blocks, attention blocks, individual experts, and heads.
- Unconditional subtraction consistently reduces sycophancy but harms retention, whereas analytic conditional subtraction and learned conditional gating improve the removal–retention frontier.
2 Related Work
Prior work studies activation steering, behavioral localization, MoE routing versus expert computation, and sycophancy, but does not combine them to conditionally correct belief-induced answer shifts without unrelated activation edits. THESIS-MoE builds on these lines while distinguishing its inference-time, frozen-weight MoE intervention from prior approaches.
- Activation-based steering: Activation subtraction and concept erasure apply fixed-magnitude edits to every token, regardless of whether the target behavior is present.Contrastive activation addition uses mean differences between paired behavioral examples, while concept erasure removes a target subspace from every representation.
- Behavioral localization: Prior localization work identifies sparse behavioral components and fine-tunes them, whereas this work edits MoE computations at inference while keeping base weights frozen.The closest cited study isolates roughly three percent of MLP neurons and matches broad fine-tuning on sycophancy with less data, but operates on dense models.
- MoE behavior and routing: MoE studies distinguish behavior-associated expert computation from routing, reporting that routing in aligned models reflects topic more than behavior.Prior work steers by activating or deactivating behavior-associated experts, while the routing result points toward expert computation as the behavioral locus.
- Sycophancy: Sycophancy has been measured as agreement with users, whereas this work defines it as the answer shift caused by stating a belief.Earlier studies documented sycophancy at scale and linked it to feedback rewarding agreement; the shift formulation enables a conditional edit to respond to the behavior.
- Positioning: This work connects activation steering, causal component localization, MoE behavioral analysis, and sycophancy to localize and correct belief-induced answer shifts without unrelated activation edits.The stated research gap is whether such shifts can be localized across the MoE hierarchy and corrected without editing behaviorally unrelated activations.
3 Problem Formulation and Data
The section defines sycophancy as a belief-induced answer change and isolates it by contrasting matched nudge and no-nudge prompts. It separates behavioral diagnosis from forced-answer activation contrasts used to construct steering directions, while balancing synthetically matched evaluation data against answer-position artifacts.
- 3.1 Sycophancy as a Belief-Induced Shift: Sycophancy is an answer change caused by a stated user belief, not mere agreement with that belief.Agreement is non-sycophantic when the model would give the same answer without the stated belief.
- 3.1 Sycophancy as a Belief-Induced Shift: Each multiple-choice item pairs a belief-conforming option with a nonconforming option, evaluated across matched nudge and no-nudge forms.Nudge prompts include the opinion, while no-nudge prompts neutralize it to isolate the belief’s effect.
- 3.1 Sycophancy as a Belief-Induced Shift: Diagnostic items measure belief push from model preferences, whereas forced-answer items compare activations to construct subtraction and gating directions.Behavioral results use diagnostic items; forced items are used only for activation-direction construction.
- 3.1 Sycophancy as a Belief-Induced Shift: The dataset splits and retention sets are balanced, identifier-disjoint, and verified to have zero leakage.These properties define the reported split and retention-set construction.
- 3.2 Data Construction and Balancing: The evaluation data use model-written items from Perez et al., with synthetically derived no-nudge twins created by minimally removing stated beliefs.The repackaged dataset is distributed with contrastive activation addition.
- 3.2 Data Construction and Balancing: Belief-conforming answers are balanced across answer labels to prevent the learned belief direction from capturing answer-position artifacts.The construction preserves only an objective question when removing the stated belief.
- 3.3 Two Different Contrasts: Unlike completion contrast, the proposed condition contrast holds the answer fixed and compares nudge with no-nudge versions of the same item.Completion contrast may mix sycophancy with the representation of the forced answer, whereas condition contrast targets the belief effect.
4 Signal and Metrics
This section defines a matched-pair belief direction and a signed logit-margin metric for measuring sycophantic shifts. It evaluates component-level interventions across MoE granularities using behavioral and retention metrics, including a routing-focused baseline.
- Signal construction: The belief direction averages activation differences from matched pairs where the supplied answer is fixed and only the user’s stated belief changes.Balanced answer labels make the direction reflect the belief-induced activation change rather than answer identity or position.
- Metrics: The item-level belief push is the difference between nudge and no-nudge logit margins for conforming versus nonconforming options.Values near zero indicate invariance, positive values indicate sycophantic movement, and negative values indicate reversal beyond invariance.
- Metrics: The reported mean belief push uses bootstrap confidence intervals, with zero as the target and signed margins distinguishing correction from overshooting.The metric remains smooth enough for fine-grained sweeps and differentiable training.
- Evaluation design: Interventions are applied only to components identified as carrying the belief shift and are evaluated across attention blocks, heads, MoE blocks, and experts.Subtraction methods are swept over nine component sets, while the learned gate is applied to all identified components simultaneously.
- Evaluation design: FARE isolates routing by acting on which experts are selected rather than on expert computations, testing where the behavior resides.The evaluation panel separates behavioral metrics from retention metrics; the no-nudge rate anchors whether plain behavior changed.
5 Methodology
The methodology localizes belief-shift components across an MoE granularity ladder using probing and causal activation swaps, then compares unconditional, analytic conditional, and learned conditional interventions at shared locations. Conditional methods adapt edits to activation content, while learned gates remain weight-frozen and are trained to reduce belief push while preserving behavior and knowledge.
- Localization: The search spans complete attention and MoE blocks, individual heads, and experts, with each component tested by probing and causal activation swaps.Swapping matched nudged and no-nudge activations preserves the data distribution while testing whether a component is necessary for producing the belief shift.
- Unconditional subtraction: Constant subtraction applies a fixed displacement regardless of activation content, so stronger coefficients remove more belief push but reduce retention.The intervention uses the condition-contrast belief direction and a fixed strength c at the edited component.
- Analytic conditional subtraction: Conditional subtraction removes only the signed projection of each activation onto the belief direction, leaving near-zero or opposing components nearly unchanged.At c = 1 the projection is removed exactly; smaller or larger values partially remove or reverse it.
- Evaluation design: All post-localization interventions operate at the same decision token, isolating content-dependent edit magnitude from positional effects across three open-weight MoE models.Subtraction methods are evaluated over nine component sets and coefficients c ∈ {1, 2, 3, 5, 10}, with operating points selected on development data and confirmed on test data.
- Learned conditional gating: The learned intervention places per-token gates at located components while freezing original weights and jointly optimizing belief push, no-nudge anchoring, factual knowledge, and general behavior.The gate preserves component contributions at mi = 1 and suppresses them near zero; accepted states must reduce belief push, preserve MMLU, and maintain the anchor.
6 Results
Causal localization places belief-induced sycophancy in small, late-layer MoE subcircuits, while conditional activation edits remove substantial sycophancy with better retention than prompting, routing changes, or unconditional subtraction. The learned conditional gate is strongest behaviorally across the evaluated methods while preserving reasoning, knowledge, truthfulness, and fluency measures.
- Localization: The causal search localizes the belief shift to concentrated mid-to-late-layer components: six attention and three MoE blocks in OLMoE, eight and one in Qwen, and six and three in Granite.At finer granularity, the effect is carried by eight attention heads in OLMoE, ten in Qwen, and thirteen in Granite; experts are found in OLMoE and Granite but none in Qwen.
- Baselines: 29% on OLMoE, 8% on Qwen, and 24% on Granite: prompting increases the belief-induced push rather than reducing sycophancy.The unedited model is sycophantic in every case, establishing the baseline for comparison.
- Baselines: 39% on OLMoE, 23% on Qwen, and 41% on Granite: routing reweighting removes some push but reduces GSM8K retention to 68%, 83%, and 85% of baseline.The routing baseline therefore damages model performance without fully removing the behavior.
- Conditional interventions: 77% on OLMoE, 87% on Qwen, and 60% on Granite: conditional subtraction removes the push while holding GSM8K at 100%, 92%, and 103% of baseline.Projection-scaled removal leaves tokens that do not carry the belief direction almost untouched, with perplexity unchanged in the supplied passage.
- Conditional interventions: 90% on OLMoE, 63% on Qwen, and 32% on Granite: the selected frozen conditional gate is strongest behaviorally while holding GSM8K at 115%, 111%, and 115% of baseline.Across the full panel, MMLU is 82%, 85%, and 89%, TruthfulQA is 98%, 96%, and 96%, and the decision-token-off perplexity control stays within 1%.
7 Analysis and Conclusion
Conditional editing improves the removal-retention trade-off by scaling interventions to belief content, while sycophancy appears to depend more on component computation than expert selection. Across models and evaluation settings, the approach improves the frontier but retains limitations in overshooting and decision-token-only evaluation.
- Removal-retention trade-off: Conditional scaling raises retention from 72% to 100% on OLMoE and from 75% to 103% on Granite while also improving removal.Unconditional and conditional subtraction use the same belief direction and target components; conditional methods scale intervention by each activation’s belief content.
- Benchmark retention: Conditional methods generally keep MMLU and TruthfulQA near baseline, while GSM8K retention reaches 111-115% with the learned gate.On Granite TruthfulQA, the learned gate preserves retention better than conditional subtraction despite lower removal.
- Mechanistic interpretation: Qwen shows that routing is insufficient: no sycophancy-specific routed experts are found, yet conditional editing still removes much of the behavior.Prompting increases belief push, whereas routing removes some push at the cost of retention and perplexity.
- Conclusion and limitations: Across three MoE models, two routing regimes, and three seeds, conditional editing improves the removal-retention frontier without changing the direction or target components.The approach still risks overshooting into negative belief push, and evaluation is restricted to the decision token in multiple-choice tasks.
A Removal and Retention Across Methods
Across all evaluated models, the learned gate and conditional subtraction achieve the strongest removal–retention trade-offs, and they rank above other methods overall while FARE ranks last.
- Removal–retention frontier: The learned gate and conditional subtraction lead every model’s removal–retention frontier, preserving more GSM8K while removing more belief push.Figure 6 plots belief push removed against GSM8K retained; upper-right points are preferable, and the dashed line marks baseline retention.
- Overall ranking: The two conditional methods rank first in every model on the combined panel Z-score, while the routing baseline FARE ranks last.The Z-score combines behavioral removal and retention into one all-round score, with higher values preferred.
B Subtraction at Each Granularity
Conditional subtraction preserves retention as sycophancy removal increases, unlike constant subtraction; this advantage widens at finer MoE granularities, from blocks to heads and experts.
- B Subtraction at Each Granularity: Conditional subtraction holds retention as removal climbs, whereas constant subtraction trades retention away.The comparison is performed across increasing strength coefficients in the subtraction sweep.
- B Subtraction at Each Granularity: The retention gap widens as subtraction granularity sharpens from blocks to heads to experts.The passage contrasts constant and conditional edits across component types.
- B Subtraction at Each Granularity: Eight OLMoE, ten Qwen, and thirteen Granite heads were selected as carrying heads, concentrated in mid-to-late layers.The selected heads are identified through per-head causal effects on the belief push.
- B Subtraction at Each Granularity: Five OLMoE and six Granite experts were selected as carrying experts, while Qwen had none because its shared experts are always on.This shared-expert regime prevents Qwen experts from appearing in a routed contrast.
C Gate Architecture Sweep
A seventy-two-run architecture sweep identified the conditional gate and full training schedule as the strongest design choices across all three models. The selected configuration was VF-CG-FT-C, a frozen conditional-gate setup chosen using removal, reasoning, and fluency criteria.
- Design-axis effects: The conditional gate (CG) and full schedule (FT) removed the most belief push across all three models.The sweep varied edit class, gate structure, schedule, and initialization.
- Configuration selection: VF-CG-FT-C won among configurations qualifying on all three models.Candidates were ranked by distance to complete removal under reasoning and fluency penalties, with over-removal runs discarded.
- Model-specific selection: The sweep compared configurations by nudge removed (%) against GSM8K change versus baseline, with model-specific top-ten regions and ranked picks.Negative GSM8K changes indicate reasoning loss.
- Fluency screening: Perplexity-change screening ranked configurations on the removal-against-perplexity plane using the decision-token-off control.Positive perplexity changes indicate fluency loss, and the dashed line marks the honest ceiling.
D Gate Localization Detail
The learned gate produces sparse, concentrated edits in mid-to-late attention and expert components, with stable localization across seeds. Compared with baselines, conditional subtraction and the learned gate combine high sycophancy removal with capability retention.
- Edit sparsity: The edit is sparse: a handful of attention and mixture-of-experts sites carry most of the gate’s edit.Per-site edit amount is measured as 1 − m at the selected gate checkpoint, with a dashed inclusion threshold.
- Component distribution: The most heavily edited sites are mid-to-late-layer attention and expert components, while individual attention heads are least edited.Components are ranked by edit amount and colored by component type.
- Localization stability: The located support is stable across seeds s1, s2, and s4, indicating consistent gate activity rather than a single-run artifact.The set of components opened by the gate remains consistent across the three seeds.
- Method comparison: Conditional subtraction and the learned gate achieve high removal while retaining capability across the three MoE models.The comparison includes Removal, Invariance, MMLU, GSM8K, TruthfulQA, and PPL; prompting and FARE trade little removal for capability or fail to remove sycophancy.