PaperlayerView exact revision on arXiv

Source-linked AI summary

Probing and Steering for Safety-Relevant Behaviours in Large Language Models

Nimun Bajwa, Massimo Poesio

arXiv:2610.30363v1cs.LGcs.AIcs.CLcs.CY

TL;DR

Activation probes are used to detect and steer safety-relevant behaviours, but prior evidence on design choices comes from mismatched settings. The paper compares estimators, pooling, and layers for three behaviours in one pipeline, finding that the detection/steering pattern holds for two behaviours and reverses for sycophancy.

  • Problem

    Prior studies typically examine one behaviour, estimator, or pooling scheme, leaving their interactions difficult to compare across models, metrics, and token policies.

    Source: §I · p. 1

  • Method

    The paper tests two estimators and three pooling schemes on three behaviours in Llama-3.1-8B-Instruct, evaluating directions for detection and steering.

    Source: §I · p. 1

  • Results

    For deception and high-stakes, logistic regression detects best and mass-mean steers best; for sycophancy, mass-mean detects best and logistic-regression attention steers best.

    Source: §V · p. 7

  • Takeaways & Limitations

    Detection and steering performance can favor different directions, and the dissociation may depend on behaviour or data.

    Source: §V · p. 7

    Sycophancy probes were not tested against an echo shortcut, so their AUROCs may partly measure surface agreement.

    Source: A. Limitations · p. 8

Abstract

from arXiv · show

Activation probes are computed directions in a language model's residual stream, used both to detect safety-relevant behaviours and, by adding their direction back into the stream, to steer them. We explore how the main design choices behind such probes interact, within one pipeline, on one model (Llama-3.1-8B-Instruct), and across three behaviours: deception, high-stakes awareness and sycophancy. The pipeline crosses two estimators (difference-of-means and logistic regression) with three token-pooling schemes (last, mean and learned attention), selects appropriate layers to detect or steer a behaviour, and evaluates every direction twice: as a detector of the behaviour, and as a steering vector to push outputs towards the behaviour. Prior work read together suggests a read/write dissociation, in which logistic regression reads (detects) best but difference-of-means directions write (steer) best. We find that it holds for deception and high-stakes but not for sycophancy, where the pattern reverses. The reversal appears to follow partially from the data rather than from the estimators. The best pooling technique and the layer choice depend on the behaviour and data, and the layer that steers best can sit well before the layer that detects best. The right steering metric to use between changing a model's perception and eliciting behaviour depends on the task - and the two can disagree outright. The exported probes and directions power an interactive interface through which a user can score and steer text directly.

A PREPRINT

Activation probes can both detect safety-relevant behaviours and steer them, but prior evidence on which designs work best comes from different models, behaviours, metrics, and token policies. This paper compares these choices in one pipeline, while related work motivates the estimator and pooling choices it tests.

I. INTRODUCTION

  • The study tests two estimators, three pooling schemes, and three behaviours in one model, evaluating each direction for both detection and steering.Prior work suggested logistic regression detects best while difference-of-means steers best, but comparisons across studies could confound that pattern.Source: §I · p. 1

II. RELATED WORK

  • Linear probes can identify features correlated with truth rather than truth itself, motivating this study’s emphasis on out-of-distribution evaluation (Levinstein and Herrmann, 2025; Marks and Tegmark, 2024, §1).The paper also describes activation probes as a lower-cost alternative to an auxiliary monitor model (Alain and Bengio, 2018; McKenzie et al., 2025).Source: §II · p. 1, §I · p. 1
  • Prior steering studies found mass-mean directions outperform logistic regression directions for eliciting truth and other alignment-relevant behaviours (Li et al., 2023; Marks and Tegmark, 2024; Panickssery et al. 2024).Marks and Tegmark evaluate with a normalised indirect effect on out-of-distribution inputs; this paper adopts their primary causal metric and calibration.Source: §II · p. 2
  • Prior detection work reports logistic regression as a strong deception detector, while learned attention pooling performs well for high-stakes detection (Goldowsky-Dill et al., 2025; McKenzie et al., 2025).Apollo evaluated deception on Llama-3.3-70B-Instruct; McKenzie et al. studied pooling schemes for high-stakes interactions.Source: §II · p. 2

III. METHODOLOGY

The methodology uses a shared, reproducible pipeline on Llama-3.1-8B-Instruct to train probes on contrastive examples and evaluate held-out and out-of-distribution data. The data and detection masks are tailored to deception, high-stakes awareness, and sycophancy.

III. METHODOLOGY

  • All experiments use Llama-3.1-8B-Instruct in 8-bit on one GPU, with fixed seeds, greedy decoding, and one shared pipeline for three behaviours.Each behaviour has a specification covering class labels, mask policy, readout prompt, judge instruction, and coefficient sweep.Source: §III · p. 2

A. Data and Token Masking

  • Probes train on masked tokens from contrastive behaviour-on/off pairs; held-out facts form the ID test set, while evaluation relies more on OOD sets.The detection mask targets where the behaviour lives, and splits are 80/20.Source: A. Data and Token Masking · p. 2, Figure 1 · p. 3

A.1) Deception

  • Deception training pairs frame the same true fact as honest or deceptive, masking only the truncated assistant response to avoid trivial system-prompt cues.OOD evaluation includes context-defined AI Liar pairs and content-defined Roleplaying pairs; only assistant responses are probed.Source: A.1) Deception · pp. 2–3

A.2) High Stakes Awareness

  • High-stakes training uses GPT-4o-generated low/high-stakes scenarios, while Anthropic-HH supplies genuinely OOD multi-turn conversations with all speaker turns masked.Training masks actual content words, excluding template scaffold tokens that were byte-identical across classes and diluted the pooled signal.Source: A.2) High Stakes Awareness · p. 3

A.3) Sycophancy

  • Sycophancy means agreeing with a false user claim; candid negatives include genuine agreements, and the OOD test shifts to a fifth domain, math.Only assistant replies are masked, since sycophancy is present in the reply; each training class has 1,368 examples, with 347 per class held out.Source: A.3) Sycophancy · p. 3

B. Probing

The paper defines probes by crossing an estimator with a pooling strategy, then evaluates detection and steering with separate procedures. It calibrates steering directions and selects layers according to the relevant evaluation metric.

B. Probing

  • A probe crosses an estimator with a pooling strategy; the five variants include mass-mean and logistic-regression directions with last, mean, or learned-attention pooling.Probes are fitted on masked activations at a detection layer selected for the behaviour.Source: B. Probing · p. 4

B.1) Estimator

  • Logistic regression saves the normal to a learned separating boundary, while mass-mean uses the difference between behaviour-on and behaviour-off class means.On paired data, differencing cancels shared pair content, isolating the behaviour; both directions score activations by projection.Source: B.1) Estimator · p. 4

B.2) Pooling Strategies

  • Pooling aggregates activations from tokens in the detection mask: last-token selects one activation, mean averages token scores, and attention learns token weights and a classifier jointly.The attention probe is trained with three seeds, whose scores are averaged; its steering direction is the averaged head weight vector.Source: B.2) Pooling Strategies · p. 4

B.3) Probing Evaluation

  • AUROC with a bootstrap 95% confidence interval is the primary probing metric because it ranks behaviour-on against behaviour-off scores without requiring a threshold.Accuracy requires a fixed threshold that is not comparable across estimators or between ID and OOD data.Source: B.3) Probing Evaluation · p. 4

C. Steering

  • Steering adds calibrated probe directions to residual-stream layers and tokens; coefficient +1 moves an average off-behaviour activation one class gap toward the on-behaviour mean.The steering block is a contiguous five-layer window selected by summed NIE using calibrated mean-pooled mass-mean directions at +1; each layer gets its own direction.Source: C. Steering · p. 4

C.1) Steering Evaluation

  • NIE measures whether steering changes the model’s judgment of a transcript, while judged open-ended generation measures whether it changes behaviour; the instruments can disagree.NIE reads a YES/NO response to a few-shot transcript question; generation steers the prompt and completion, with an LLM judge scoring behaviour and coherence.Source: C.1) Steering Evaluation · pp. 4–5

D. Layer Selection

Detection and steering use separately selected layers, chosen with metrics aligned to their respective goals. The selected probes and calibrated directions are then exported for interactive scoring and steering.

D. Layer Selection

D.1) Probing Layer Selection

  • Detection selects the earliest layer with the highest pair-aware five-fold AUROC for a mass-mean probe across all 32 layers.The logistic-regression curve is a robustness check, but mass-mean decides; selection uses training data and AUROC to match final evaluation.Source: D.1) Probing Layer Selection · p. 5

D.2) Steering Layer Selection

  • Steering uses the contiguous five-layer window with the highest summed NIE, while detection selects one layer by cross-validated AUROC.The steering sweep adds calibrated, mean-pooled mass-mean directions at +1; its block selection reflects causal effects spread across neighbouring layers.Source: D.2) Steering Layer Selection · p. 5

E. Interface

  • The interface lets users score conversation tokens with exported probes and compare unsteered and steered responses at a chosen coefficient.It also supports steering two behaviours simultaneously, each at its own block.Source: E. Interface · p. 5

IV. RESULTS

Across deception and high-stakes awareness, detection and steering vary by probe, evaluation set, and metric. In particular, steering readouts do not always correspond to judged behaviour, especially out of distribution.

A. Deception

  • Deception detection selects layer 12, with cross-validated AUROC at 1.000 on layers 12 and 13; its steering block is layers 11–15.The selected detection layer lies toward the lower end of the steering block.Source: A. Deception · pp. 5–6

A.2) Detection

  • For deception, LR (mean) leads pooled out-of-distribution detection at 0.722, ahead of MM (last) at 0.696; LR (last) scores 0.571.Performance differs sharply by held-out set: MM (mean) reaches 0.992 on AI Liar, while LR (mean) leads Roleplaying at 0.706.Source: A.2) Detection · p. 6
  • For deception, out-of-distribution mass-mean steering leads logistic regression on both held-out sets, while Roleplaying responses are non-monotone.On AI Liar, mass-mean NIE rises from −0.064 at −2 to +0.046 at +2; on Roleplaying it also rises at −2, producing a U-shape.Source: A.2) Detection · p. 6
  • For deception, judged behaviour stays near baseline on both held-out sets despite NIE shifts, while Roleplaying coherence falls at steering extremes.Roleplaying coherence falls to 2.96 and 4.12 at ±2, with higher values near the centre.Source: A.2) Detection · p. 6

B. High-stakes

  • High-stakes detection selects layer 20 of 32, while steering uses layers 6–10, ten layers earlier.The text relates the later detection layer to more complex prompts and a median of 58 probed tokens, versus six for deception.Source: B. High-stakes · p. 6
  • On Anthropic-HH, LR (mean) leads detection at 0.774, while the two last-token probes score 0.687 and 0.686.Every probe drops by 0.20–0.28 OOD; in-distribution scores range from 0.885 to 0.996, led by LR-attention at 0.996.Source: B. High-stakes · pp. 6–7
  • On Anthropic-HH, mass-mean NIE leads at +1.34 at +2, but judged behaviour stays flat across directions out of distribution.In distribution, mass-mean moves judged behaviour from 2.46 at −2 through a 4.08 unsteered baseline to 7.54 at +2; its coherence falls from 7.21 to 3.92.Source: B. High-stakes · p. 7
  • For high-stakes awareness, judged behaviour agrees with the readout in distribution but stays flat for every direction on Anthropic-HH.On Anthropic-HH, mass-mean reaches 2.38 at +2 from a 1.67 baseline; coherence declines under mass-mean from 7.21 at −2 to 3.92 at +2.Source: B. High-stakes · p. 7

C. Sycophancy

Sycophancy reverses the read/write pattern found for deception and high-stakes: last-token probes detect best, while LR-attention steers best. The discussion links this inversion and other probe outcomes to data structure, layer choice, and evaluation metrics.

C. Sycophancy

  • The sycophancy detection layer is 19 of 32 (AUROC 0.940), where genuine agreements among negatives require combining reply stance with claim truth.The authors suggest this relational label may require more layers than a simple lexical contrast.Source: C. Sycophancy · p. 7

V. DISCUSSION

  • MM (last) leads sycophancy detection at 0.976 ID, while LR-attention leads steering on both sets, reversing the deception and high-stakes ordering.MM (last) reaches 0.879 on OOD math; the discussion suggests the inversion may reflect data and pooling rather than estimator differences.Source: C. Sycophancy · p. 7, §V · pp. 7–8

C. Sycophancy

  • Last-token probes lead detection because templated replies differ mainly in the final answer, while mean pooling dilutes that localized signal.LR-attention stays competitive by learning where to look; one example assigns 0.38 of pooling mass to the echoed answer token.Source: C. Sycophancy · p. 7

V. DISCUSSION

  • Coherence often degrades at large steering coefficients, while sycophancy judged behaviour is flat except for an MM spike at +1.At that coefficient, judged behaviour reaches 9.71 ID and 9.42 OOD from baselines near 1.0; coherence falls to 2.88 and 4.17.Source: C. Sycophancy · p. 7, §V · p. 8
  • Detection layers are 12 for deception, 19 for sycophancy, and 20 for high-stakes, following the increasing complexity of the probed behaviour.The steering block can precede detection: causal blocks fall within layers 6–15, and coincide with detection only for deception.Source: C. Sycophancy · p. 7, §V · p. 8
  • Detection AUROC falls from ID to OOD, with smaller gaps when held-out data more closely resembles training data.AI Liar and math retain the training format and barely drop, while Roleplaying and Anthropic-HH differ more and drop most.Source: §V · p. 8
  • The NIE readout and judged behaviour can disagree outright, so the relevant steering metric depends on whether the task concerns perception or elicited behaviour.For sycophancy at +1, the MM readout is negative while the judge reaches its maximum.Source: §V · p. 8
  • Negative steering coefficients have no consistent behaviour here because every steered item starts behaviour-off, so a negative coefficient asks for less of an absent behaviour.The negative-side sweep varies across Anthropic-HH, math, and Roleplaying.Source: §V · p. 8

A. Limitations

The limitations concern sycophancy-label validity, the unvalidated single-judge evaluation, steering-direction availability, and mass-mean-anchored layer choices. Future work proposes suppression tests, sparse-autoencoder directions, larger models, and behaviour composition.

A. Limitations

  • Sycophancy AUROCs may partly measure echo because the probes were not tested on sycophantic versus genuine agreements.The proposed check is to measure AUROC on those two agreement types.Source: A. Limitations · p. 8
  • Behaviour and coherence scores rely on one LLM judge whose agreement with human raters was not measured.The judge is Claude Haiku 4.5.Source: A. Limitations · p. 8
  • No last-token direction was steered, so the sycophancy inversion may partly reflect which directions were available rather than estimator differences.This excludes MM (last), the best sycophancy detector on both sets.Source: A. Limitations · p. 8
  • Detection layers and steering blocks are selected using mass-mean, mildly biasing comparisons of LR probes toward MM-selected locations.The detection layer maximizes MM cross-validated AUROC, and the steering block maximizes MM NIE.Source: A. Limitations · p. 8

VI. FUTURE WORK

  • Suppression remains unevaluated because every steered item is behaviour-off; the authors call for testing behaviour-on items with the NIE denominator reversed.The current sweep measures induction, including at negative coefficients.Source: §VI · p. 8
  • Sparse-autoencoder latents could test whether a sparser basis reads and writes better than dense linear directions.Source: §VI · p. 8
  • A 70B run could directly test whether scale explains gaps to reference papers and whether the findings persist at that scale.Source: §VI · p. 8
  • The interface can add two behaviours’ directions at separate blocks, making their evaluation with both pipeline instruments a proposed extension.The cited injection scheme was found workable where a single summed vector fails.Source: §VI · p. 8

APPENDIX A DECEPTION OUT-OF-DISTRIBUTION PAIRS

The appendices show the out-of-distribution examples and detection masks, probe settings, layer-selection sweeps, and in-distribution detection views. They specify which tokens are masked and how the plotted layer and detection results are presented.

APPENDIX A DECEPTION OUT-OF-DISTRIBUTION PAIRS

  • AI Liar pairs share identical questions and answers, with deception defined by the system-prompt context; Roleplaying pairs share scenarios and questions but differ in answer content.Only assistant-response tokens are eligible for probing in both sets.Source: Appendix A · p. 10, Figure 6 · p. 10

APPENDIX B HIGH-STAKES OUT-OF-DISTRIBUTION EXAMPLES

  • Anthropic-HH examples contrast low- and high-stakes multi-turn conversations, with the detection mask covering every role’s message content.Only scaffold tokens are excluded from the mask.Source: Appendix B · p. 10, Figure 7 · p. 10

APPENDIX C SYCOPHANCY OUT-OF-DISTRIBUTION PAIRS

  • The sycophancy examples pair a matched false-claim case with a genuine-agreement hard negative; the mask covers only the assistant’s reply.Source: Appendix C · p. 11, Figure 8 · p. 11

APPENDIX D PROBE HYPERPARAMETERS

APPENDIX E DETECTION-LAYER SELECTION CURVES

APPENDIX F IN-DISTRIBUTION DETECTION

  • In-distribution detection is shown as AUROC with bootstrap 95% confidence intervals and held-out-test ROC curves for all five probes per behaviour.Exact values are identified as the test (ID) rows of Appendix H.Source: Appendix F · p. 12, Figure 12 · p. 12, Figure 13 · p. 12

APPENDIX G OUT-OF-DISTRIBUTION ROC CURVES

These appendices present out-of-distribution detection curves and full detection metrics, then illustrate token-level scores, attention-pooling weights, and similarities among probe directions.

APPENDIX G OUT-OF-DISTRIBUTION ROC CURVES; APPENDIX H FULL DETECTION TABLES

  • Out-of-distribution ROC curves show all five probes for each behaviour, while the full detection tables report AUROC with bootstrap 95% confidence intervals and TPR at 1% FPR for each evaluation set.TPR at 1% FPR is n/a when a set has fewer than 100 negatives.Source: Appendix G · p. 13, Figure 14 · p. 13, Appendix H · p. 13

APPENDIX I PER-TOKEN PROBE SCORES

  • For one held-out deception pair, masked-token scores project activations onto the LR mean direction; the pair differs only in system persona, so color differences reflect context rather than lexical content.Source: Appendix I · p. 14, Figure 15 · p. 14

APPENDIX J ATTENTION-PROBE WEIGHTS

  • The sycophancy attention example displays LR-attention softmax weights averaged over three seeds; larger values indicate a larger share of the pooled representation.Raw weights sum to one over masked positions; printed values are standardised within the mask.Source: Appendix J · p. 14, Figure 16 · p. 14

APPENDIX K PROBE-DIRECTION COSINES

  • Pairwise cosines compare unit-normalised probe directions at each behaviour’s detection layer; values near 1 indicate directions alike up to scale, while lower values indicate different directions.Signs are aligned toward the behaviour-on class.Source: Appendix K · p. 15, Figure 17 · p. 15

APPENDIX L LINEAR STRUCTURE AT THE DETECTION LAYER

The appendices visualize class separation at detection layers, describe per-layer steering-block selection, and provide the judge prompts used to score behaviour and coherence.

APPENDIX L LINEAR STRUCTURE AT THE DETECTION LAYER

  • For mean-pooled training activations, the appendix contrasts label-blind projections onto the two largest-variance PCA directions with projections onto the label-trained mass-mean direction.Source: Appendix L · p. 15, Figure 18 · p. 16

APPENDIX M PER-LAYER NIE SWEEPS

APPENDIX N JUDGE PROMPTS

  • Each generation receives one judge call for a 1–10 behaviour score and a coherence score, with both outputs bounded server-side to integers from 1 to 10.The prompts define behaviour-specific scoring scales for deception, high-stakes awareness, and sycophancy.Source: Appendix N · pp. 16–17

APPENDIX O FULL STEERING TABLES

The appendices provide full steering-response tables, note an in-distribution scoring exception, and demonstrate the exported probes in an interactive interface. The interface includes exploratory composition of multiple behaviour directions.

APPENDIX O FULL STEERING TABLES

APPENDIX P IN-DISTRIBUTION STEERING

  • For in-distribution deception, judge rows are omitted because greedy decoding makes the generation prompts byte-identical; the appendix otherwise shows in-distribution steering panels for the behaviours.Source: Table V · p. 17, Appendix P · p. 18, Figure 22 · p. 18

APPENDIX Q THE INTERFACE IN USE

  • The interface scores and steers text live, while its exploratory multi-behaviour tab composes directions at separate blocks; examples show high-stakes steering, a deception-plus-sycophancy composition, and prompt-unrelated output at higher combined coefficients.The pipeline does not evaluate composition, and the app marks that mode exploratory.Source: Appendix Q · p. 19, Figure 24 · p. 19, Figure 25 · p. 19

APPENDIX R REPRODUCIBILITY

The appendix documents the setup and controls used to reproduce the experiments, then distinguishes reproducible results from outputs that can vary across runs or hardware.

A. Setup

  • Experiments use Llama-3.1-8B-Instruct in 8-bit on one GPU, with seed 42, greedy decoding, and three behaviours.Detection and steering use behaviour-specific specifications in a shared pipeline.Source: A. Setup · p. 20

B. What is fixed

  • Re-running the three notebooks reproduces reported metrics and figures, except judge scores.Reproducible outputs include AUROC curves and held-out scores, NIE sweeps, steering coefficients, probe-direction cosines, and per-token figures.Source: B. What is fixed · p. 20
  • Fixed seeds, stable fold identifiers, pinned datasets, greedy decoding, and pinned package versions support repeatable reruns.The appendix specifies local random generators for stochastic steps and fixed upstream revisions or commits for the inputs.Source: B. What is fixed · p. 20

C. What is not fixed

  • Judge behaviour and coherence scores can shift on rerun, while deterministic NIE selected the steering blocks.Anthropic does not guarantee identical judge outputs, even at temperature 0.Source: C. What is not fixed · p. 20
  • Across GPUs, AUROCs, NIEs, and cosines remain within bootstrap confidence intervals and preserve layer and block choices, but may differ in final digits.On the same GPU and software, differences are typically zero.Source: C. What is not fixed · p. 20
  • Steered generations are identical on the same hardware, but a changed early token on another GPU can alter the remaining text.This also applies to examples shown in the interface.Source: C. What is not fixed · p. 20

Related papers

Loading 2610.30363v1…