Source-linked AI summary
No Usable Linear "Capitulation Direction" in Two Small LLMs: A Validation Protocol for Activation-Steering Claims, and a Cross-Family Behavioral Study of Sycophancy Under Pushback
Saad Aamir, Muhammad Awais Bin Adil
TL;DR
Sycophancy may be represented as an activation-space direction, but existing mechanistic findings disagree about whether such directions support reliable steering. Across two model families, this paper characterizes pushback-induced capitulation and validates direction-finding, finding a model-specific behavioral crossover but no usable pre-response residual-stream direction.
Problem
Mechanistic studies disagree about whether sycophancy is represented by an activation-space direction that can be reliably found and causally manipulated.
Method
The paper combines cross-family behavioral comparisons with question-level cross-validation, shuffled-label nulls, a known-direction positive control, and a preregistered usability gate.
Results
Pushback susceptibility reverses direction across model families, while no usable linear capitulation direction appears in either family and the control direction is recovered in both.
Takeaways & Limitations
Pushback robustness conclusions are model-specific, and convincing capitulation directions can dissolve under validation even when a true control direction survives.
Takeaways & Limitations
The study covers two small models, one English TriviaQA dataset, and one linear last-token difference-in-means read-out, with no causal intervention.
Abstract
from arXiv · showhide
Language models frequently abandon correct answers when users push back. We study this in two small instruction-tuned models from different families, Qwen2.5-1.5B and Llama-3.2-1B, over TriviaQA: the model answers, is challenged with one of four scripted pushback styles, and answers again. Conditioned on an initially correct answer, the models flip to a wrong answer in 41.8% and 43.1% of episodes. Which pressure works is a property of the model, not the pressure: the same within-question paired comparison (bare doubt vs. emotional appeal), specified in advance, is Bonferroni-significant in opposite directions across families (Qwen: bare doubt > emotional, OR 2.5, p=.040; Llama: emotional > bare doubt, OR 4.0, p=.001). Failure mode is also model-dependent: Llama abandons answers without recommitting at six times Qwen's rate (8.2% vs. 1.4%). Identical pushback repairs initially wrong answers only ~13% of the time; pushback is net epistemically destructive. We then ask whether capitulation is linearly decodable from the pre-response residual stream, a prerequisite for steering-vector interventions at that locus. A naive difference-in-means probe appears to succeed (in-sample AUROC 0.81/0.71), but a validation protocol combining question-level cross-validation, shuffled-label nulls, and a known-direction positive control shows the signal is overfitting: the best cross-validated AUROC is 0.582 in Qwen and 0.548 in Llama, both near or below their permutation thresholds and far under a pre-registered usability bar of 0.70, while the identical pipeline recovers a pushback-presence control direction at AUROC 1.000 in both. We further quantify a measurement hazard: substring grading underestimates capitulation by 18-24 percentage points. Code, prompts, transcripts, and analysis are released.
1 Introduction
The paper contributes a model-specific behavioral study of capitulation under pushback and a validation protocol showing that apparent linear capitulation directions are overfit artifacts in both model families.
- Mechanistic validation: The naive direction-finding approach produces convincing but fake directions in both models, while the study’s protocol exposes the failure.This result motivates treating decodability validation as a prerequisite for activation-steering claims.
- Behavioral findings: Opposite-sign, Bonferroni-significant pushback effects across Qwen and Llama show that susceptibility profiles are model-specific.Bare doubt beats emotional appeal in Qwen, whereas emotional appeal beats bare doubt in Llama.
- Behavioral findings: Llama abandons answers without recommitting at 8.2% of episodes versus 1.4% for Qwen, a sixfold failure-mode difference.A conflated destabilization metric would obscure that this gap reflects failure mode rather than simply overall rate.
- Measurement: Substring grading underestimates capitulation by 18 percentage points in Qwen and 24 points in Llama.Capitulations can mention the correct answer while ultimately abandoning it.
- Mechanistic validation: A validation protocol combines question-level cross-validation, shuffled-label nulls, a positive control, and a pre-registered pass/fail gate.The protocol is designed to distinguish genuine direction-finding from in-sample overfitting.
2 Related Work
Related work studies sycophancy behavior, activation-based steering, and internal mechanisms, but reports fragmented constructs and limited transfer from linear decodability to effective intervention.
- Behavioral sycophancy: Prior behavioral work treats sycophancy as answers swayed by user challenges, while other work argues the construct spans distinct behaviors.This paper’s flip-versus-abandon dissociation and cross-family variation provide behavioral evidence relevant to that fragmentation.
- Activation steering and linear directions: Activation-steering methods use mean-difference or related directions to modify model behavior, and prior refusal work motivates validating decodability before intervention.The paper positions its validation gate as a prerequisite for steering-vector claims.
- Mechanisms of sycophancy: Mechanistic studies variously report internal sycophancy representations, residual-stream separability, sparse-head steering, and decomposable sycophancy-related directions.These findings motivate testing whether a single usable residual-stream capitulation direction exists.
3 Behavioral Evaluation
The evaluation screens model-specific initially correct and wrong questions, applies four frozen pushback templates, and uses judged final commitments to distinguish flips, abandons, and holds. It combines this taxonomy with question-clustered inference and a prespecified flip-rate metric.
- 3.1 Setup: The evaluation applies four frozen pushback templates to model answers on TriviaQA, using each model’s screened-correct questions plus 30 initially wrong questions.The templates are simple doubt, authoritative assertion, emotional appeal, and social consensus; decoding is greedy throughout.
- 3.2 Judging and outcome taxonomy: An LLM judge evaluates the final committed answer and classifies initially-correct episodes as flip, abandon, or hold.Apologies and mid-response contradictions are ignored; abandons are excluded from flip counts and direction extraction.
- 3.2 Judging and outcome taxonomy: Abandonment occurs in 8.2% of Llama episodes versus 1.4% for Qwen, so the taxonomy exposes a sixfold model-dependent failure-mode difference.A conflated destabilization rate reports Llama at 51.4% and Qwen at 43.2%, obscuring that much of the gap is failure mode rather than amount.
- 3.2 Judging and outcome taxonomy: Substring grading changes 134/772 Qwen and 170/792 Llama labels, raising measured capitulation by approximately 18 and 24 percentage points.The judge addresses responses that mention the correct answer while abandoning it.
- 3.3 Metrics and statistics: The primary outcome is the flip rate: Pr[turn-2 commits to a wrong answer | turn-1 correct, t].Recovery is separately computed as Pr[turn-2 correct | turn-1 wrong, t].
4 Behavioral Results
Behavioral results show that pushback susceptibility reverses direction across model families, while pushback degrades answers far more often than it repairs initially wrong ones. Heldout rate estimates are reported with question-clustered bootstrap intervals.
- 4.1 Susceptibility crossover: Qwen’s bare doubt exceeds emotional appeal, whereas Llama’s emotional appeal exceeds bare doubt, with Bonferroni-significant opposite-direction comparisons.Qwen: OR 2.5, pBonf=.040; Llama: OR 4.0, pBonf=.001.
- 4.2 Net epistemic effect: Correct answers flip wrongly in 41.8% of Qwen and 43.1% of Llama episodes, while identical pushback repairs initially wrong answers in only 12.5% and 14.0%.The reported net effect favors degradation by roughly 3× in both families, although flip and recovery rates are not directly symmetric.
5 Is Capitulation Linearly Decodable?
A validation protocol finds no usable linear capitulation direction in the pre-response residual stream of either small model, despite inflated in-sample performance and a control direction that passes.
- 5.2 Data and candidate direction: The extraction split contained 143 flip and 212 hold episodes across 90 Qwen questions, and 151 flip and 162 hold episodes across 84 Llama questions.Matched contrasts covered 48 Qwen questions and 40 Llama questions.
- 5.1 Validation protocol: Question-level leave-one-question-out validation prevents leakage from near-duplicate episodes, while within-question shuffled labels estimate the exact-sample null.The pipeline also includes a positive control using pushback-present versus pushback-absent prompts.
- 5.3 Overfitting demonstration: 0.814 in-sample AUROC at a Qwen pilot collapsed to 0.53 under question-level cross-validation, while larger samples showed declining in-sample scores.The pattern is consistent with difference-in-means separating training labels rather than learning a generalizable capitulation signal.
- 5.4 Main result: the gate fails in both families: 0.582 in Qwen and 0.548 in Llama were the best matched LOQO AUROCs, both failing the 0.70 usability gate.Qwen’s score was marginally above its permutation threshold, while Llama’s was below its threshold and indistinguishable from meaningless labels.
- 5.4 Main result: the gate fails in both families: The positive control achieved LOQO AUROC 1.000 in both models, showing that the identical instrument detects a real activation direction when one exists.Thus, the capitulation null is not explained by a generally incapable validation pipeline.
- 5.5 Interpretation and limitation: At these scales, any linear capitulation component is weak in the pre-response residual stream, below the effect size presupposed by steering interventions.The authors leave larger models, other loci, nonlinear probes, and multidimensional subspaces open.
6 Discussion
The discussion treats the behavioral crossover and mechanistic null as model- and measurement-sensitive findings, while emphasizing that the null supports skepticism toward in-sample direction claims rather than ruling out internal representations.
- Interpretation of the crossover: Different post-training mixtures or capability-relative evidence standards may explain the sign-reversing susceptibility crossover, but both accounts remain speculative and testable.The authors suggest probing turn-2 generations rather than only the pre-response state.
- Mechanistic interpretation: The positive-control direction reaches LOQO AUROC 1.000 in both models, contrasting with the failed capitulation-direction gate.This supports interpreting the capitulation result as a replicated null under the tested protocol, not a universal failure to detect directions.
- What the null does and does not license: The null excludes only a single usable linear direction at these scales, read-out positions, sample sizes, and model families; it does not show that sycophancy lacks internal representation.Attention-head loci, nonlinear probes, multidimensional subspaces, and scaling remain open possibilities.
- Behavioral implications: Pushback destroys roughly three times more truth than it recovers, making “ask the model to double-check” a poor verification strategy for these small models.Because susceptibility profiles cross over between families, conclusions about which pressures matter remain model-specific.
8 Conclusion
Across two model families, the paper combines a behavioral study of capitulation with a validation protocol showing that apparently convincing linear directions disappear under proper validation while a real control survives.
- 8 Conclusion: The paper reports a sign-reversing susceptibility crossover, a 6× cross-family failure-mode difference, a measurement correction, and failed capitulation-direction gates in both architectures.The positive control direction survives validation in both models, and the paper releases code, prompts, transcripts, and analysis.
A Judge validation and a parser-bug vignette
The study separates transcript generation from judging and uses audits to catch label failures, including a parser bug that converted INCORRECT verdicts into CORRECT. Edge-verdict review further addresses systematic grading errors before flip counts are computed.
- Judging workflow: Two-stage judging allows label-affecting fixes to be applied through re-grading without repeating expensive generation.Transcripts were generated once and could be re-graded repeatedly.
- Parser validation: A parser bug falsely produced 100% initial accuracy by matching CORRECT before INCORRECT in judge outputs.The fix used word-boundary matching with INCORRECT tested first.
- Edge-verdict audits: 11/11 pilot RETRACTED/UNCLEAR verdicts were manually audited, with additional sampling at scale.RETRACTED and UNCLEAR were excluded from flip counts in both directions.
B A replication-caution vignette
A concentrated abandonment pattern in a small pilot disappeared with more data, while the broader abandonment taxonomy survived and became a cross-family finding. The vignette illustrates why plausible mechanisms and extreme concentration require replication.
- Replication caution: A pilot’s apparently significant social-template abandonment effect disappeared after quadrupling the data.The pilot had 5/24 social abandons versus 0/72 elsewhere, whereas the larger run had 9/636 abandons scattered across templates.
- Surviving finding: The cross-family study relocated abandonment to Llama, where it occurred at 8.2% and was concentrated under the simple template.This preserved abandonment as a model-dependent outcome while rejecting the pilot’s mechanism story.
C Reproducibility
The experiments standardize numerical precision, decoding, sampling, model configuration, judging, provenance, and released artifacts. These choices support reproducibility across the reported runs.
- Numerical reproducibility: Float32 generation and activation caching were standardized after MPS float32 matched CPU reference activations while bfloat16 showed measurable rounding.The minimum per-layer cosine similarity was 1.000000 for float32 versus 0.9757 for bfloat16.
- Experimental controls: Greedy decoding, fixed seeds, a specified 300-question TriviaQA pool, and 2,000 bootstrap resamples make the analysis deterministic.Shuffled-label nulls used 20 fixed-seed permutations.
- Model and judge configuration: The study reports model architectures, layer dimensions, TransformerLens usage, and a four-way judge verdict set with word-boundary parsing.The models are Qwen2.5-1.5B-Instruct and Llama-3.2-1B-Instruct.
- Release artifacts: Repository commits, the arxiv-v1 tag, transcripts, cached directions, screening records, and analysis scripts are released.The release covers both raw and judged transcripts.