PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla

Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, Vladimir Mikulik

arXiv:2307.09458v3cs.LG

TL;DR

Circuit analysis has mostly been demonstrated in small models, leaving its scalability and ability to reveal processed information uncertain. This paper studies multiple-choice label selection in Chinchilla 70B using attribution, attention visualization, patching, and subspace analysis, finding scalable circuit identification but only a partial semantic explanation of correct-letter heads.

  • Problem

    Existing circuit analyses focus on relatively small models and often identify information flow without explaining what information the circuit processes.

  • Method

    The paper analyzes Chinchilla 70B’s multiple-choice label-selection circuit using logit attribution, attention visualization, activation patching, and SVD-based query, key, and value subspaces.

  • Results

    Existing techniques identify output nodes in Chinchilla 70B, while correct-letter-head subspaces suggest an ‘n-th item in an enumeration’ feature but explain randomized-label behavior only partially.

  • Takeaways & Limitations

    Circuit analysis can yield useful internal mechanisms in very large language models, but feature semantics may remain noisy, messy, and incomplete.

  • Takeaways & Limitations

    The authors note that understanding the semantics of processed features remains difficult, and causal interventions can be affected by downstream compensation or backup behavior.

Abstract

from arXiv · show

\emph{Circuit analysis} is a promising technique for understanding the internal mechanisms of language models. However, existing analyses are done in small models far from the state of the art. To address this, we present a case study of circuit analysis in the 70B Chinchilla model, aiming to test the scalability of circuit analysis. In particular, we study multiple-choice question answering, and investigate Chinchilla's capability to identify the correct answer \emph{label} given knowledge of the correct answer \emph{text}. We find that the existing techniques of logit attribution, attention pattern visualization, and activation patching naturally scale to Chinchilla, allowing us to identify and categorize a small set of `output nodes' (attention heads and MLPs). We further study the `correct letter' category of attention heads aiming to understand the semantics of their features, with mixed results. For normal multiple-choice question answers, we significantly compress the query, key and value subspaces of the head without loss of performance when operating on the answer labels for multiple-choice questions, and we show that the query and key subspaces represent an `Nth item in an enumeration' feature to at least some extent. However, when we attempt to use this explanation to understand the heads' behaviour on a more general distribution including randomized answer labels, we find that it is only a partial explanation, suggesting there is more to learn about the operation of `correct letter' heads on multiple choice question answering.

1. Introduction

The paper tests whether circuit analysis scales to Chinchilla 70B by studying how it converts correct answer text into multiple-choice labels. Existing techniques identify output nodes and support a partial, low-dimensional account of correct-letter heads, whose feature semantics remain messy.

  • Prior circuit analyses mostly studied small models and mapped relevant components without sufficiently characterizing the information flowing through them.
  • The study examines the algorithmic challenge of mapping a correct answer’s text to its label in MMLU using the 70B-parameter Chinchilla model.
  • Logit attribution, attention visualization, and activation patching identify correct-letter heads and other output nodes involved in the multiple-choice circuit.
  • SVD-based analyses suggest query and key subspaces encode an ‘n-th item in an enumeration’ feature, while the value subspace encodes token identity.
  • The proposed feature account only partially explains correct-letter-head behavior on broader distributions, especially when answer labels are randomized.

2. Background

The background introduces Chinchilla, the MMLU label-versus-text task, and activation patching as a causal intervention for tracing model components. It frames the model as a decoder-only transformer whose components contribute directly to logits under a fixed scaling factor.

  • 2.1. Chinchilla: Chinchilla 70B is a compute-optimally trained decoder-only transformer with 80 layers and 64 attention heads per layer.
  • 2.1. Chinchilla: RMSNorm normalizes inputs by their root mean square, while its learned gain is folded into the following weight matrix for analysis.
  • 2.1. Chinchilla: With fixed RMS scaling, the model’s logits can be expressed as contributions from the unembedding matrix, MLP outputs, and attention-head outputs.
  • 2.2. Massive Multitask Language Understanding (MMLU): MMLU prompts require models to identify the correct answer and output its label, with letter tokens A, B, C, and D tokenized separately.
  • 2.2. Massive Multitask Language Understanding (MMLU): Only Chinchilla 70B performs well in standard label-scored MMLU and exceeds random performance on the synthetic label-selection task among the three tested sizes.
  • 2.3. Activation Patching: Activation patching intervenes on model nodes by replacing activations from an original prompt with activations from another prompt whose correct answer differs.

3. Identifying the circuit using existing techniques

Logit attribution, attention-pattern visualization, and activation patching identify a compact set of final nodes that directly support Chinchilla’s multiple-choice answer-label output. The analysis finds 45 relevant nodes, validates their combined effect, and characterizes specialized MLPs and attention-head groups while noting important interpretive limits.

  • Identifying final nodes: 45 nodes explain 80% of the summed positive direct effect, comprising 32 attention heads and 13 MLPs.The analysis finds a few moderately influential nodes followed by a long tail of smaller direct effects.
  • Validating final nodes: Activation patching shows that direct effect does not always predict total effect, with the top two direct-effect nodes having significantly lower total effect.The authors note that total effects include additional pathways and may be affected by prompt mixing or fixed-RMS assumptions.
  • Validating final nodes: L24 H18 has a large total effect because it moves information from correct content tokens to the final token, influencing the queries of correct-letter heads indirectly.This distinguishes an important upstream contributor from nodes with large direct effects.
  • Validating final nodes: Patching the 45 identified nodes together recovers most of the model’s loss and accuracy on the chosen MMLU subset.The combined patch is evaluated across different correct-answer targets.
  • MLP behaviour: MLPs are often highly specialized, strongly contributing to one or two answer letters while contributing little or harming performance on others.This specialization is especially apparent when logits are evaluated relative to the other possible answer letters.
  • MLP behaviour: Treating whole MLP layers as atomic nodes hides the individual neurons or neuron groups that may encode features in superposition.The paper leaves analysis of these finer-grained units for future work.
  • Analysing attention patterns: Attention patterns cluster heads into correct-letter, uniform, single-letter, and amplification groups, but most single-letter heads cannot generalize alone.Single-letter heads generally do not vary with the correct answer, although they could contribute to a generalizing algorithm in aggregate.
  • Analysing attention patterns: Heads attend to answer labels rather than answer contents, suggesting they identify the label of an already-determined correct answer, although they may also identify both answer and label.This behavior is constrained by causal masking, which prevents information from answer contents flowing into corresponding labels.

4. Understanding the Semantics of the Correct Letter Heads

The analysis finds that Correct Letter heads operate in low-dimensional subspaces combining general enumeration-position features with label-specific behavior. These features partially generalize beyond ABCD, but the resulting semantic explanation remains incomplete.

  • 4. Understanding the Semantics of the Correct Letter Heads: Correct Letter heads can be losslessly reduced to a 3-dimensional QK subspace after accounting for query and key means.The low-rank approximations retain performance while revealing lower-rank computation than the original spaces.
  • 4.1. Distilling Heads: SVD on centered queries and keys shows that only the query-delta/key-delta interaction provides substantial information for distinguishing correct labels.The mean key is shared across answers, while the informative term is q_δ · k_δ.
  • 4.1. Distilling Heads: Three principal components capture roughly 65-80% of key and query variance and 80-90% of value variance across Correct Letter heads.The three-component choice is supported by a knee in the cumulative explained-variance curves.
  • 4.2. QK Subspace Semantics: Query and key projections suggest a general enumeration-position feature, while significant performance loss for random letters indicates additional label-identity dependence.Different separators and removing the prelude preserve performance, whereas random capital letters recover only one third to one half of the loss.
  • 4.2. QK Subspace Semantics: Correct Letter heads perform poorly on numeric labels, suggesting that their learned behavior is not uniformly invariant across label formats.Chinchilla fails the task even in the base numeric setting, and these heads seemingly do not contribute to performance there.
  • 4.3. Correct Letter Head Pseudocode: The proposed pseudocode labels keys as item_nums, queries as correct_item_num, and values as token_identities, but these names are only first approximations.Representations are more invariant to random letters than numbers, and magnitude and direction changes complicate off-distribution behavior.

5. Discussion

The discussion qualifies the circuit and semantic analyses: causal interventions can expose only a cross-section of relevant nodes, and feature interpretations are distribution-dependent. The authors also identify substantial automation and generalization gaps.

  • 5. Discussion: Total-effect analyses identify a cross-section separating ancestors and descendants rather than the full set of relevant circuit nodes.Downstream compensation, denoising interventions, and the chosen base distribution can distort causal conclusions.
  • 5. Discussion: The reported meanings of nodes apply only to the tested distribution, although matching results on a synthetic multiple-choice dataset suggest some broader generalization.The analysis also omits circuit components that identify the possible answer-token subset independently of correctness.
  • 5. Discussion: Pseudocode descriptions trade faithfulness for readability because feature invariances and input-mutation details are difficult to encode.The authors emphasize that even approximate semantic hypotheses may omit behaviorally relevant feature details.
  • 5. Discussion: Whether other models implement similar algorithms remains open, because the study focuses on the final label-manipulation stages and leaves much of the upstream circuit undiscovered.The authors call for further causal-intervention comparisons and more automated analyses that preserve faithfulness and completeness.
  • 5. Discussion: The study does not provide evidence about the discussed nodes’ function on general text prediction beyond the narrow MMLU distribution.Such behavior could clarify whether the nodes reflect superposition or more general computation.

6. Related work

The related work positions this study within mechanistic circuit analysis, causal intervention, activation interpretation, circuit validation, and low-rank representation methods.

  • 6. Related work: Prior circuit studies examine where facts are stored, grammatical identification, modular addition, and factual-recall pathways.These works establish a broader program of tracing internal mechanisms underlying model behavior.
  • 6. Related work: Circuit-node relevance is commonly established through interventions that differ in direction, affected pathways, and evaluation metrics.The present work uses a denoising-style intervention approach related to prior causal methods.
  • 6. Related work: The paper interprets intermediate activations through logit attribution and attention patterns, while related work also optimizes directions for proposed high-level features.Its approach uses SVD to identify subspaces explaining activation variation in a dataset.
  • 6. Related work: Dimensionality reduction has been used in interpretability to identify activation-space directions and outside interpretability for model compression and efficient fine-tuning.This places the paper’s low-rank approximation technique within an established methodological family.

7. Conclusion

The study applies circuit-analysis tools to Chinchilla 70B multiple-choice answering and identifies output components whose low-rank representations preserve MMLU performance. Its semantic analysis finds both general and specific features, while remaining noisy and incomplete.

  • 7. Conclusion: Chinchilla 70B’s multiple-choice circuit includes attention heads and MLPs that directly contribute to producing the correct answer label.The components form seemingly independent groups associated with the model’s successful behavior.
  • 7. Conclusion: Low-rank representations compress the identified heads without harming performance on MMLU.The representations indicate that the heads use both token-identity and list-position features.
  • 7. Conclusion: Interpretability techniques scale to very large models, but their outputs can remain noisy or contradictory and require improved tools.The authors highlight automation as important because this research is labor intensive.

A. Results on synthetic multiple choice

On a synthetic multiple-choice task requiring symbol manipulation but no factual knowledge, only Chinchilla 70B performs well, and its circuit largely matches the one found on MMLU.

  • A. Results on synthetic multiple choice: Only Chinchilla 70B performs well on the synthetic multiple-choice task, while smaller models fail to identify the correct label.The task asks which option equals a specified token and does not require factual knowledge.
  • A. Results on synthetic multiple choice: For smaller models, loss equals the entropy of a uniform four-member distribution, indicating uncertainty about the correct label rather than the answer set.
  • A. Results on synthetic multiple choice: The synthetic-task circuit largely contains the same responsible nodes as MMLU, including correct letter and amplification heads.

B. Preliminary analysis on the rest of the circuit

The analysis traces correct-letter behavior through content-gathering heads, diffuse MLP-mediated label processing, and a low-dimensional but incomplete feature interpretation.

  • B.1. Content Gatherers: Path patching identifies L19 H51, L24 H18, L26 H20, L26 H32, L32 H26, and L35 H56 as the strongest-effect heads.
  • B.1. Content Gatherers: The first three heads attend to final content tokens of the correct answer, while the latter three attend mostly to the final two tokens and may amplify that signal.The amplification interpretation is explicitly described as speculative.
  • B.2. Label Processing: A diffuse, inconsistent set of MLPs mediates how correct-letter heads upvote the corresponding label after attending to it.
  • B.3. Open Questions: The analysis leaves open how content is aggregated, how queries and keys are formed, and how symbol-binding MLPs reinforce token identity.

C. Nuances in identifying output nodes

Identifying output nodes from direct effects is useful but can produce false positives because later nodes may cancel earlier contributions, and total effects vary substantially across prompts.

  • C. Nuances in identifying output nodes: Direct-effect attribution can falsely identify output nodes because later nodes may cancel the effects of earlier nodes.
  • C. Nuances in identifying output nodes: Additional filtering by high total effect is proposed to reduce these false positives, but the authors do not apply it further.
  • C. Nuances in identifying output nodes: Total effects show high variance across prompts, making it difficult to rule out nodes with low median total effect.Even L30 H26, with median total effect zero, has substantial total effect on some prompts.
  • C. Nuances in identifying output nodes: Almost all nodes have negative total effects when B is correct, a behavior the authors do not explain.

D. Net Direct Effect By Letter

Net direct effects vary substantially across correct-letter settings, while attention visualizations identify only a small number of heads that meaningfully use other token positions.

  • D. Net Direct Effect By Letter: Aggregate net direct effects show large variance between settings defined by the correct letter.The results are therefore reported separately by setting.
  • D. Net Direct Effect By Letter: The attention analysis uses value-weighted attention, reporting attention probability multiplied by the L2 norm of the value vector.Prelude, label tokens, final tokens, and the maximum remaining position are reported separately.
  • D. Net Direct Effect By Letter: Only L24 H18 and L28 H23 make meaningful contributions from token positions beyond the usual label-focused positions.

F. Do the keys alone encode the correct answer?

The analysis decomposes each head’s attention dot product into four components and finds that only centred query–key interactions identify the correct answer letter.

  • The full dot product contains four components, including interactions involving the key and query means and centred keys and queries.The section focuses on centred keys and queries while assessing the other three terms.
  • The comparison averages each dot-product component over 1,024 prompts, with 256 prompts for each correct letter A–D.
  • Only the dot products between centred queries and centred keys identify which letter is correct across all heads.The remaining three components are uninformative about the correct answer.

G. More Low Rank Results

Further analyses examine low-rank approximations and the geometry of correct-letter heads, including their performance effects and query–key centroid structure.

  • Low-rank queries, keys, and values are evaluated through changes in direct effect and loss under prompt mutations.Figure 33 examines direct effects, while Figure 34 compares full-rank and low-rank attention.
  • Accuracy over the answer set is compared for full-rank and low-rank attention under true-target and random-target prompt mutations.True targets evaluate on the intervention prompt, whereas random targets run on the intervention prompt but evaluate on the original prompt.
  • Cosine similarities and projections quantify how query and key deltas relate to key centroids across heads.Figures 36 and 37 report these relationships for key and query deltas.
  • The centroids of key and query deltas show significant overlap for at least two letters in every head.
Loading 2307.09458v3…