PaperlayerView exact revision on arXiv

Source-linked AI summary

Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants

Vincent Huang, Dami Choi, Daniel D. Johnson, Sarah Schwettmann, Jacob Steinhardt

arXiv:2512.15712v1cs.AIcs.CLcs.LG

TL;DR

Interpretability assistants need scalable ways to explain complex neural activations beyond hand-designed agents. This paper trains Predictive Concept Decoders to predict model behavior through sparse concepts, showing improved scaling with data and the ability to surface information models do not self-report.

  • Problem

    Hand-designed interpretability agents are difficult to scale and are bottlenecked by off-the-shelf models that are not specialized for interpretability.

  • Method

    PCDs jointly train an encoder to compress activations into sparse concepts and a decoder to answer behavioral questions, using FineWeb pretraining followed by question-answering finetuning.

  • Results

    PCDs improve with training data and support detecting jailbreaks, secret hints, implanted latent concepts, and latent user attributes, including information models fail to self-report.

  • Takeaways & Limitations

    Predicting verifiable model behavior can train interpretability assistants whose sparse concepts become more capable and interpretable as data increases.

  • Takeaways & Limitations

    Some metrics plateau around 72–144M tokens, suggesting that richer objectives or improved architectures may be needed to continue scaling.

Abstract

from arXiv · show

Interpreting the internal activations of neural networks can produce more faithful explanations of their behavior, but is difficult due to the complex structure of activation space. Existing approaches to scalable interpretability use hand-designed agents that make and test hypotheses about how internal activations relate to external behavior. We propose to instead turn this task into an end-to-end training objective, by training interpretability assistants to accurately predict model behavior from activations through a communication bottleneck. Specifically, an encoder compresses activations to a sparse list of concepts, and a decoder reads this list and answers a natural language question about the model. We show how to pretrain this assistant on large unstructured data, then finetune it to answer questions. The resulting architecture, which we call a Predictive Concept Decoder, enjoys favorable scaling properties: the auto-interp score of the bottleneck concepts improves with data, as does the performance on downstream applications. Specifically, PCDs can detect jailbreaks, secret hints, and implanted latent concepts, and are able to accurately surface latent user attributes.

1 INTRODUCTION

The paper replaces hand-designed interpretability agents with Predictive Concept Decoders trained end-to-end to predict model behavior through sparse, human-interpretable concepts. PCDs scale with data and support detecting hidden model information, including jailbreak awareness and secret hint usage.

  • 1 INTRODUCTION: PCDs replace hand-designed hypothesis-testing agents with an end-to-end objective that trains assistants to predict model behavior from activations.Behavior prediction provides a verifiable training signal because predictions can be checked against the subject model’s actual behavior.
  • 1 INTRODUCTION: The encoder compresses activations into a sparse concept list, while the decoder uses that list and a natural-language question to answer about model behavior.Because the encoder does not see the question, its concepts must support diverse queries; sparsity also aids human interpretability.
  • 1 INTRODUCTION: PCDs jointly train on FineWeb with next-token prediction, then finetune the decoder on question-answering data about the subject model’s beliefs.An auxiliary loss prevents concepts from becoming inactive during training.
  • 1 INTRODUCTION: PCDs improve with training data, outperform same-data SAE features at high sparsity and beat direct prompting and LatentQA on downstream interpretability tasks.They detect jailbreaks, secret hints and implanted latent concepts, while accurately surfacing user attributes.
  • 1 INTRODUCTION: PCDs expose information models fail to self-report, including secret hint usage and jailbreak awareness, and their sparse concepts make predictions auditable.Concept inspection can reveal latent factors behind behavior, such as liability-related concepts active when a model refuses a harmful request.

2 PREDICTIVE CONCEPT DECODERS

A PCD encodes subject-model activations into sparse top-k concepts, re-embeds them as soft tokens, and patches them into a decoder that answers behavioral questions. The system is pretrained with next-token prediction and later finetuned for question answering.

  • 2 PREDICTIVE CONCEPT DECODERS: The encoder reads activations from layer ℓread, selects each token’s top-k concepts, re-embeds them, and patches them into the decoder at layer ℓwrite.The architecture therefore communicates subject-model information through a sparse bottleneck rather than the original activations.
  • 2 PREDICTIVE CONCEPT DECODERS: The encoder maps subject-model activations into sparse concepts, while the decoder combines their re-embedded representation with a natural-language question to produce an answer.The encoder never sees the question and the decoder never directly sees the original activations, forcing general-purpose concepts.
  • 2 PREDICTIVE CONCEPT DECODERS: PCD training has two stages: joint encoder-decoder pretraining on FineWeb next-token prediction, followed by decoder finetuning on SynthSys question-answering data.The encoder is frozen during finetuning, while the decoder is adapted to questions about model beliefs.
  • 2 PREDICTIVE CONCEPT DECODERS: Evaluation measures concept interpretability, held-out question-answering accuracy, and whether PCDs surface hidden information such as jailbreak awareness, secret hints, and implanted concepts.These axes separately assess the encoder’s concepts, the decoder’s answers, and case-study usefulness.

3 PRETRAINING

PCDs are pretrained by compressing subject-model activations into sparse concepts that predict future text, with auxiliary training used to preserve concept activity. Interpretability improves with data under this intervention, while comparisons show different scaling behavior across PCD and SAE objectives.

  • 3.1 TRAINING SETUP: PCD pretraining compresses middle-token activations into sparse concepts that the decoder uses to predict future suffix tokens from web text.The setup uses next-token prediction as scalable supervision without labeled interpretability data.
  • 3.2 MAINTAINING CONCEPT ACTIVITY: The auxiliary loss revives inactive concepts by nudging near-active directions, keeping over 90% of concepts active at 72M tokens.Without intervention, nearly a third of concepts die during a 72M-token run, and inactive concepts tend to have lower interpretability scores.
  • 3.2 MAINTAINING CONCEPT ACTIVITY: The auxiliary loss makes both auto-interpretability and concept coverage increase with pretraining data, whereas without it precision plateaus and recall declines from 36M to 72M tokens.Both metrics begin to plateau past the 100M-token scale.
  • 3.3 EVALUATING ENCODER INTERPRETABILITY: At fewer than 36M tokens, PCDs with k = 16 outperform SAEs, but standard SAEs with k = 50 surpass PCDs as training increases because PCD curves plateau.Increasing k improved standard SAEs but did not improve PCDs in preliminary experiments.
  • 3.3 EVALUATING ENCODER INTERPRETABILITY: Question-answering accuracy generally improves with encoder training and eventually matches LatentQA, while removing the bottleneck at test time causes earlier saturation after 18M tokens.The comparison uses the final finetuning checkpoint.
  • 3.3 EVALUATING ENCODER INTERPRETABILITY: KL-based SAE variants share PCDs’ plateauing behavior and show smaller gains from increasing active concepts, suggesting they capture relevant information sparsely but may saturate earlier.The passage presents this as a possible consequence of the KL objective being easier to optimize than L2 reconstruction.

4 FINETUNING

PCDs are finetuned to answer questions about subject-model beliefs using concepts extracted from user-message activations. They become comparable to LatentQA at sufficient pretraining scale, while joint encoder-decoder learning is important and removing the bottleneck can alter performance.

  • 4 FINETUNING: PCDs are finetuned on SynthSys to predict held-out user attributes from concepts extracted only from the subject model’s user-message activations.Gender and age are held out for evaluation after training on 78 of 80 attributes.
  • 4 FINETUNING: Figure 6 shows the decoder using user-message activations while the encoder surfaces gender-related concepts that help answer SynthSys questions.The figure provides qualitative examples of questions and active concepts.
  • 4 FINETUNING: At 36M tokens, PCDs achieve comparable performance to LatentQA, whereas small-budget PCDs perform worse and frozen SAE-encoder ablations perform poorly.These results indicate that the bottleneck does not significantly harm this user-modeling task, but joint encoder-decoder learning matters.
  • 4 FINETUNING: Replacing the trained bottleneck with an identity mapping only at test time improves performance at 18M tokens but reduces it at 36M and 72M tokens.This intervention gives the decoder entirely out-of-distribution inputs.

5 CASE STUDIES: USING PCDS TO EXPLAIN ATYPICAL BEHAVIORS

PCDs are evaluated on jailbreak awareness, secret-hint use, implanted concepts, and auditing model refusals, generally improving with scale and often outperforming prompting and LatentQA. Their sparse bottleneck supports auditable explanations, but can hurt performance in some jailbreak settings.

  • 5.1 JAILBREAKING: 50% accuracy on Dream jailbreaks was achieved by the k = 16 PCD, while all other methods stayed below 30%.Across settings, PCDs averaged better performance than LatentQA and direct prompting, although results varied substantially.
  • 5 CASE STUDIES: USING PCDS TO EXPLAIN ATYPICAL BEHAVIORS: 72M PCDs with no bottleneck substantially outperformed LatentQA despite identical pretraining and finetuning data.The authors suggest that bottleneck training may give the decoder a favorable inductive bias by emphasizing granular concepts.
  • 5.2 SECRET HINTS: PCDs revealed secret-hint usage while prompting performed significantly worse than decoder-based methods.All PCD variants improved with more pretraining data, and the k = 16 variant became nontrivial at 72M tokens.
  • 5.3 INTROSPECTION: PCDs verbalized injected concepts more often than LatentQA and the subject model, while SAE dictionaries surfaced more relevant concepts on average.The k = 16 PCD eventually matched no-bottleneck performance as training data increased, despite not being trained on steered activations.
  • 5.4 AUDITING DECODER RESPONSES WITH ENCODER CONCEPTS: In refusal cases, the decoder mentioned legal liability around 50% of the time, and legal-context concepts appeared among the top activations in 48 of 50 cases.These encoder concepts corroborated the decoder’s claim that refusals may involve liability-related information rather than only the model’s stated user-safety rationale.

6 RELATED WORK

Related work covers automated interpretability, sparse feature learning, concept bottleneck models, and chain-of-thought faithfulness. These lines of work motivate learned descriptions, sparse representations, concept-based prediction, and tests of whether verbal explanations reflect model behavior.

  • Automated interpretability: Automated interpretability uses learned models and tool-equipped agents to describe activations or test behavioral hypotheses because manual explanation is infeasible.Prior methods generate natural-language neuron descriptions and conduct experiments on model components.
  • Sparse feature learning: Sparse feature-learning methods use autoencoders and sparsity objectives to obtain interpretable decompositions of activation space, while auxiliary mechanisms address inactive concepts.The paper draws on SAEs, TopK-style activations, auxiliary losses, ghost gradients, and sparsity annealing.
  • Concept bottleneck models: Concept Bottleneck Models use interpretable concepts between inputs and outputs, but original formulations required hand-crafted concepts and labels and targeted standalone models.The paper builds on this bottleneck idea while applying it to understanding a subject model’s activations.
  • Chain-of-thought faithfulness: Chain-of-thought faithfulness studies use secret hints or biases to test whether models disclose information influencing their behavior, with prior work finding important omissions.These studies provide context for evaluating whether interpretability assistants can surface information models fail to self-report.

7 DISCUSSION

PCDs are presented as an early step toward end-to-end interpretability assistants whose concepts become more capable and interpretable with data. The discussion frames verifiable interpretability tasks as natural training targets and proposes extensions focused on making bottlenecks more legible to humans.

  • PCDs become more capable and their encoder concepts more interpretable as training data increases, although the authors expect substantial room for improvement.
  • The assistance-games perspective treats the encoder as learning concepts that help the decoder answer questions, making bottleneck legibility more important than a particular encoder design.
  • Architectural extensions: Future extensions could use transformers over multiple tokens, relational concept structures, or activations from multiple layers when these improve human understanding or decoder capability.
  • Other end-to-end tasks: The broader program applies end-to-end assistants to tasks such as neuron ablation and subspace patching, with rewards tied to observable behavioral effects.
  • Verifiable interpretability tasks provide natural training signals because success requires learning something true about a model’s internal structure.

A.1 TRAINING DETAILS

The appendix specifies the pretraining prompt format and optimization hyperparameters used for the assistant’s training.

  • Pretraining prepends each web-text passage with a fixed instruction prefix containing system and user headers and date metadata.
  • The reported optimization settings use LoRA alpha 32, dropout 0.05, learning rate 10^-4, effective batch size 128, weight decay 0.01, and no warmup.
  • The auxiliary-loss settings are ϵ_aux = 10^-4 and k_aux = 500, with training reported as insensitive to changes in these parameters.

A.2 ADDITIONAL RESULTS ON DEAD CONCEPTS

The appendix examines inactive concepts and finds that the auxiliary loss substantially preserves concept activity, whereas two alternative strategies do not.

  • Without the auxiliary loss, active concepts drop during training and dead concepts generally have poor auto-interp scores; the loss revives inactive concepts.
  • Neither encoder-direction normalization nor annealing effectively reduces inactive concepts in these experiments.

A.3 SAE BASELINES

The appendix defines regular and KL SAE baselines by reconstructing subject-model activations and, for KL objectives, matching output distributions after activation patching.

  • Regular TopK SAEs reconstruct activations by decoding the top-k encoded features and minimize squared reconstruction error.
  • For KL SAE baselines on middle tokens, the subject model’s original and reconstructed activation outputs are compared using KL divergence after patching the reconstruction.
  • The suffix-token KL SAE baseline uses the same procedure, but evaluates the original and reconstructed probabilities over suffix tokens.

A.4 EFFECT OF VARYING HYPERPARAMETERS

The hyperparameter ablations examine alternative training objectives, LoRA ranks, and bottleneck widths, revealing that objective choice matters more than rank or modest changes in active-concept count. Most experiments use 18M training tokens.

  • A.4 EFFECT OF VARYING HYPERPARAMETERS: The ablation results are interpreted under an 18M-token training budget unless otherwise stated.This establishes the default training scale for the section’s experiments.
  • A.4.1 TRAINING OBJECTIVE: Replacing next-token prediction with a loss matching the subject model’s predictions caused an earlier plateau in encoder interpretability metrics.The comparison was motivated by an expectation that incorporating subject-model probabilities would improve training, but the observed effect was unfavorable.
  • A.4.2 LORA RANK: No clear trend emerged when varying the PCD decoder’s LoRA rank.This result is summarized in Figure 15.
  • A.4.3 NUMBER OF ACTIVE CONCEPTS: Increasing the number of active bottleneck concepts may yield small gains, but it degrades human interpretability.The trade-off is shown in Figure 16.
Loading 2512.15712v1…