PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating task-specific prompts and sparse autoencoders for activation monitoring

Henk Tillman, Dan Mossing

arXiv:2504.20271v1cs.LG

TL;DR

Monitoring can use model outputs or internal activations, but prompting requires additional inference compute and probing may require labeled data. This paper compares prompted and unprompted activation probes with SAE-based methods, finding prompted probing especially effective in low-data and out-of-distribution settings. It recommends prompted probing when inference compute is available and SAE-based probing when it is limited.

  • Problem

    Prompting avoids labeled training data through zero-shot monitoring but adds inference compute, while activation probing may require substantial labeled data and evidence for proposed probing refinements is mixed.

    Source: §1 · p. 1

  • Method

    The paper systematically compares raw and prompted activation probing with SAE-based probing across moderation, question-answering, and sentiment classification tasks.

    Source: §1 · p. 2

  • Results

    Prompted raw activation probing significantly improves classifier performance over unprompted probing, particularly with limited training data and distribution shift; max-pooled SAE activation probing outperforms raw activation probing.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors recommend prompted probing when inference-time compute is available; with plentiful training data, prompted probing or SAE probing might be optimal depending on whether inference- or train-time compute is available.

    Source: §5 · p. 14

    The paper does not compare its methods with fine-tuning, and anecdotally a much smaller fine-tuned moderation model trained on a different, larger dataset matches or slightly exceeds its best moderation-monitoring approaches.

    Source: §5 · p. 15

Abstract

from arXiv · show

Language models can behave in unexpected and unsafe ways, and so it is valuable to monitor their outputs. Internal activations of language models encode additional information that could be useful for this. The baseline approach for activation monitoring is some variation of linear probing on a particular layer: starting from a labeled dataset, train a logistic regression classifier on that layer's activations. Recent work has proposed several approaches which may improve on naive linear probing, by leveraging additional computation. One class of techniques, which we call "prompted probing," leverages test time computation to improve monitoring by (1) prompting the model with a description of the monitoring task, and (2) applying a learned linear probe to resulting activations. Another class of techniques uses computation at train time: training sparse autoencoders offline to identify an interpretable basis for the activations, and e.g. max-pooling activations across tokens using that basis before applying a linear probe. However, one can also prompt the model with a description of the monitoring task and use its output directly. We develop and test novel refinements of these methods and compare them against each other. We find asking the model zero-shot is a reasonable baseline when inference-time compute is not limited; however, activation probing methods can substantially outperform this baseline given sufficient training data. Specifically, we recommend prompted probing when inference-time compute is available, due to its superior data efficiency and good generalization performance. Alternatively, if inference-time compute is limited, we find SAE-based probing methods outperform raw activation probing.

1 INTRODUCTION

The paper compares prompting and activation probing for monitoring, including combinations that use task-specific prompts or sparse autoencoders. It finds prompted raw activation probing improves performance over unprompted probing, especially with limited training data and distribution shift.

1 INTRODUCTION

  • The work grounds prompting and probing by comparing proposed variants and combinations, with particular attention to probing methods that use sparse autoencoders.Source: §1 · p. 1
  • Prompting can avoid labeled training data through zero-shot monitoring but requires extra inference compute; probing can reuse generation passes but may require substantial labeled data.Source: §1 · p. 1
  • The study compares raw activation probing, prompted probing, and SAE-based probing across moderation, question-answering, and sentiment classification.It also explores max-pooling, few-shot examples, and computational trade-offs.Source: §1 · p. 2
  • Prompted raw activation probing significantly improves classifier performance over unprompted probing, particularly with limited training data and distribution shift.The paper evaluates probing techniques under both limited-data and train/test distribution-shift settings.Source: §1 · p. 2
  • Max-pooled SAE activation probing outperforms raw activation probing, consistent with Bricken et al. (2024).Source: §1 · p. 2

2 RELATED WORK

Related work connects activation probes to analyses of model internals and compares SAE-based probes with conventional linear probes. Prior findings on SAE benefits are mixed, with max-pooling often a key condition.

2 RELATED WORK

  • Structural probing analyzes model internals, while behavioral probing examines outputs; LAT combines aspects of both, and prompted probing is described as a supervised variant of LAT.Source: §2 · p. 2
  • Prior work reports strong deceptive-behavior detection, significant out-of-distribution hallucination detection with shallow neural-network classifiers trained on LLM activations, and entity-knowledge measures from linear classifiers trained on entity-token activations.For truthfulness in multiple-choice and true/false answering, most probe-output disagreements stem from confidence differences, and high-confidence disagreements are relatively rare.Source: §2 · p. 2
  • Prior work finds that activations encode sparse interpretable features, and studies where sentiment and truthfulness information is represented in model responses.Gurnee et al. (2023) introduce a mean-difference heuristic that this paper incorporates as regularization; Orgad et al. (2024) identify exact-answer tokens as salient for truthfulness information.Source: §2 · p. 2
  • Comparisons of SAE-based and conventional probes report mixed results; several studies find little or no SAE benefit unless activations are max-pooled.Bricken et al. (2024) report a minor improvement from SAE features and emphasize max-pooling and task-specific SAE fine-tuning; Kantamneni et al. (2025) and Smith et al. (2025) report little to no benefit in several settings unless max-pooling is applied.Source: §2 · p. 3

3 METHODS

The methods compare prompted and unprompted activation probes, SAE-based probes, and direct prompted model outputs across moderation, question-answering, and sentiment tasks. They also test prompt formats, training-data conditions, and a combined probe-and-model-output classifier.

3.1 DATA AND MODELS

  • Three datasets cover expert-labeled moderation passages, SimpleQA-derived correct-answer classification, and balanced RottenTomatoes sentiment, with fixed 80/20 splits and moderation-only distribution-shift tests.Moderation tests span language and chat-format boundaries; the SimpleQA split keeps questions separate between train and test.Source: §3.1 · p. 3

3.2 PASSAGE PROCESSING

  • Prompted probing adds the monitored category to the passage; suffix-only prompting starts with the passage and shares a forward pass across categories before their category-specific suffixes.This reduces compute for parallel category monitoring relative to rerunning the full templated conversation for each category.Source: §3.2 · pp. 3–5

3.3 SPARSE AUTOENCODERS

  • The SAE is pretrained on GPT-4o base-model activations with TopK k = 64, then fine-tuned on ChatGPT-4o chat data without task-specific classification data.For probing, the TopK activation is replaced post hoc with JumpReLU, using a shared threshold set for an average of k active latents per token.Source: §3.3 · p. 5

3.4 LINEAR CLASSIFIER TRAINING

  • SAE max-pooled probing selects Q latents by class-mean differences; pre-activation variants address information lost when TopK zeros most latent activations.The authors also tested token-wise means of unit-normalized activations, which substantially underperformed last-token probing.Source: §3.4 · pp. 5–6

3.5 ZERO-SHOT PROMPTING BASELINE

  • The zero-shot baseline prompts the model with the prefix-plus-suffix template and scores examples using the difference between its ‘Yes’ and ‘No’ output logits.AUROC is computed from that logit difference.Source: §3.5 · p. 7

3.6 FEW-SHOT PROMPTING BASELINE

  • Few-shot prompting adds 1–32 randomly sampled labeled examples using copies of the same prefix-plus-suffix template, with assistant answers filled in from ground-truth labels..Source: §3.6 · p. 7

3.7 COMBINING LINEAR PROBING AND PROMPTED MODEL OUTPUTS

  • The combined method freezes a trained last-token probe, then trains a second-level classifier on its logit and the model’s ‘Yes’–‘No’ logit difference.Source: §3.7 · p. 7

4 RESULTS

Across moderation, question-answering, and sentiment tasks, prompted probing is strongest in low-data settings, while probing methods can surpass prompted model outputs with enough training data. The section also tests generalization, method combinations, optimization choices, and related baselines.

4.1 KEY RESULTS

  • Prompted last-token probing is broadly the most data-efficient method; with sufficient training data, probing methods can outperform prompted model outputs, especially on hallucination detection.In high-data regimes, prompted probing is comparable to SAE probing; all three probing approaches substantially exceed prompted outputs on SimpleQA hallucinations.Source: §4.1 · p. 8, Figure 2 · p. 8
  • Prompted probing improves or matches naive probes’ out-of-distribution performance, while SAE max-pooling sometimes hurts it; the pattern varies across shifts.For English-to-English generalization, SAE max-pooled probes generalize poorly relative to naive last-token probes; methods are more similar for single-prompt-to-multi-turn shifts. OOD and in-distribution AUROCs can differ because their test sets may differ in difficulty.Source: §4.1 · pp. 8–9, Figure 3 · p. 9
  • On sentiment, SAE max-pooled probes underperform last-token methods in low-data settings, but SAE pre-activation last-token probes are competitive with naive last-token probes.The authors attribute the degradation to max-pooling across tokens, not to SAEs generally.Source: §4.1 · p. 9, Figure 4 · p. 9

4.2 INVESTIGATIONS INTO STACKING MONITORING METHODS

  • Prompted probing does not substantially stack with SAE probing, and adding prompted model output to a probe does not substantially beat the stronger component alone.SAE max-pooled probing roughly matches the strongest prompted-plus-SAE combination; the combined output-and-activation classifier adds no signal beyond its better-performing input.Source: §4.2 · p. 10, Figure 5 · p. 10, Figure 6 · p. 10

4.3 OPTIMIZING THE MONITORING METHODS

  • Probes using activations from about 75% model depth perform roughly best, and suffix-only prompting nearly matches prefix-plus-suffix prompting at lower computational cost.The study uses suffix-only prompting in its experiments; adding a task-specific prefix would slightly improve the earlier results but prevents sharing forward passes across tasks.Source: §4.3 · p. 11, Figure 7 · p. 11, Figure 8 · p. 11
  • Max-pooling helps SAE activation probes but decreases raw-activation probe performance; 1,000 SAE latents are roughly optimal and performance is relatively insensitive to that choice.Harassment monitoring can remain competitive in the highest-data regime with just 10 latents.Source: §4.3 · p. 12, Figure 9 · p. 12, Figure 10 · p. 12

4.4 COMPARISON WITH BASELINES AND RELATED METHODS

  • Few-shot examples do not improve prompted model output in these experiments, and increasing examples tend to hurt moderation classification.The authors note that another dataset or better prompt engineering might change this result.Source: §4.4 · p. 13, Figure 11 · p. 13
  • LAT is competitive in some low-data regimes but does not exceed zero-shot model performance or the authors’ methods in high-data regimes; SAE pre-activation probes show no substantial performance difference from the Bricken et al. method.The authors suggest GPT-4o’s greater capability than the Llama 2 Chat models in Zou et al. (2023) could explain the zero-shot comparison difference; applying an activation function before max-pooling was not important for classifier performance.Source: §4.4 · pp. 13–14, Figure 12 · p. 13, Figure 13 · p. 14

5 DISCUSSION

The discussion recommends choosing monitoring methods according to compute availability and notes that performance varies across datasets. It also identifies limits in the evidence, including an untested collusion risk and no comparison with fine-tuning.

5 DISCUSSION

  • With inference-time compute available, prefer prompted probing; with plentiful data and train-time compute, SAE probing may be optimal; if both are limited, straightforward activation probing is reasonable.The recommendations are scoped to chat-trained LLMs without additional fine-tuning.Source: §5 · p. 14
  • Prior SAE comparisons found mixed performance and weaker generalization; this work finds more favorable SAE performance than Kantamneni et al. (2025) and Smith et al. (2025) on its datasets.Source: §5 · p. 14
  • Prompted probing is competitive in data efficiency and generalization, while unsupervised LAT Scans perform worse but remain worth exploring where labeled data is unavailable.Source: §5 · p. 14
  • Performance trends vary across datasets, potentially reflecting where concept information is concentrated across tokens and affecting which probing method works best.Source: §5 · p. 15
  • Inference-time methods could fail catastrophically if a monitoring prompt triggers collusion; the authors note they lack evidence for performance under strong collusion incentives.Non-prompting side-car methods may avoid acting as such a trigger.Source: §5 · p. 15
  • The study does not compare its methods with fine-tuning; an anecdotal smaller fine-tuned moderation model matched or slightly exceeded the best approaches shown.The authors see greatest relevance in settings where a dedicated fine-tuned monitor is not being considered.Source: §5 · p. 15
  • Zero-shot monitoring estimates may be pessimistic because prompts received little engineering and the monitor used neither a reasoning model nor test-time compute.Source: §5 · p. 15
  • Prompting may elicit concept information implicit in weights, whereas SAE max-pooling aggregates information nearly linearly encoded across tokens; similar performance suggests implicit information may not help monitoring.The authors suggest future work examine whether prompted and unprompted probes use shared representations.Source: §5 · pp. 14–16
Loading 2504.20271v1…