Source-linked AI summary
What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2
Nabeelah Faumi, Deshan Sumanathilaka
TL;DR
How depth-reduction distillation changes the internal algorithms of language models remains largely uncharacterized despite its preservation of aggregate metrics. This study compares GPT-2 and DistilGPT-2 using representational and causal probes, finding a contrast between IOI circuit collapse and GT circuit remapping.
Problem
How depth-reduction distillation affects learned circuits and representations remains an open question, even when aggregate metrics are preserved.
Source: §1 · p. 1Method
Results
The IOI circuit collapses while the GT circuit is actively remapped into the student’s shallower stack.
Source: §1 · p. 2Takeaways & Limitations
The contrast shows that aggregate metrics alone can miss circuit-level failure, including IOI’s negative mean logit difference of −0.467.
Source: §1 · p. 2The analysis covers only two circuits and one teacher–student pair, and full GT survival beyond the comparison-head match remains unverified.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
Knowledge distillation compresses large language models into compact students that preserve aggregate metrics, yet the mechanistic consequences of depth-reduction distillation remain largely uncharacterized. This work applies linear CKA, logit lens analysis, attention pattern decomposition, sparse autoencoder feature statistics, and activation patching to compare GPT-2 and its six-layer student DistilGPT-2 across two canonical circuits: the Indirect Object Identification (IOI) circuit and the Greater-Than (GT) circuit. The results reveal a compression-collapse dichotomy: the IOI circuit suffers complete mechanistic collapse (DistilGPT-2: 35.2% accuracy vs. 94.8%, zero name-mover preservation), while the GT circuit survives with 87.7% task performance recovery through active remapping of the teacher's layer-9 comparison head into student layer~4 (mean Pearson r = 0.945). These findings demonstrate that circuit compressibility governs mechanistic survival under distillation, and that aggregate metrics are insufficient diagnostics for circuit integrity.
1 Introduction
The paper asks whether halving GPT-2’s depth preserves its internal representations and reasoning circuits, and how the student compensates for missing late-layer components. It reports a contrast between circuit collapse and active remapping.
1 Introduction
- The study asks whether DistilGPT-2 preserves GPT-2’s internal representations and circuits despite halving depth, and how it compensates for absent upper-layer components.The comparison focuses on GPT-2’s 12 layers and its six-layer DistilGPT-2 student, including IOI and GT circuits that rely on layers 9–11.Source: §1 · p. 1
- The analysis combines representational and causal probes to track circuit preservation, focusing on circuits established through convergent causal evidence.The probes include representational similarity analysis, direct logit attribution, and activation patching.Source: §1 · p. 1
- −0.467 is DistilGPT-2’s mean IOI logit difference, indicating active misdirection rather than mere uncertainty.The passage identifies this failure mode as invisible to aggregate metrics.Source: §1 · p. 2
- The GT circuit is actively remapped into the student’s shallower stack, while the IOI circuit collapses.The contrast is presented as evidence about how different computational mechanisms survive depth halving.Source: §1 · p. 2
- The work identifies the failure of multi-stage IOI pipelines when critical late-layer components are removed, contrasting it with GT remapping.The GT circuit is described as a cognitive algorithm the student can compress and remap into available upper layers.Source: §1 · p. 2
2 Related Work
Related work situates the study in mechanistic circuit analysis, distillation, and research on compression-related representational change. It distinguishes this work from prior studies centered on pruning, quantization, or broader model-family effects.
2 Related Work
- IOI is a multi-stage circuit of name-mover, S-inhibition, and induction heads, while GT year-encoding heads feed comparison heads in layers 9–11.Prior causal work uses activation and path patching, complemented by direct logit attribution.Source: §2 · p. 2
- DistilGPT-2 halves GPT-2’s depth by initializing from every other teacher layer and combining language-modeling with distillation objectives.It retains GPT-2’s embedding dimensionality.Source: §2 · p. 2
- Prior compression studies report circuit degradation and feature changes, but mainly examine pruning and quantization rather than distillation-induced depth reduction.SAE analyses have also found representational changes that standard metrics may miss.Source: §2 · p. 2
- Related distillation studies report simplicity bias, component reorganisation, and unique feature directions across model families.The cited work includes Wu et al. (2024); Haskins and Adams (2025); Baek and Tegmark (2025).Source: §2 · p. 2
3 Background and Models
The paper frames mechanistic analysis around causally sufficient circuits and compares teacher and student representations using shared feature and layer references. GPT-2’s depth reduction leaves late-layer circuit components without direct student counterparts.
3 Background and Models
- A circuit is a minimal subgraph causally sufficient for a behavior, analyzed through attention heads and residual-stream components.The residual stream is described as a running sum of contributions from heads and MLPs, enabling direct logit attribution.Source: §3 · p. 2
- The IOI and GT circuits depend critically on upper-layer interactions structurally absent in the six-layer student; GPT-2 IOI heads in layers 6–11 and GT heads in layers 9–11 have no direct counterpart.Source: §3 · pp. 2–3
- Pre-trained GPT-2 sparse autoencoders provide a fixed reference basis for testing whether DistilGPT-2 activations occupy the same sparse feature subspace.The analysis applies the reference SAEs to both models at matched layers.Source: §3 · p. 3
- GPT-2 has 12 layers and about 117M parameters; DistilGPT-2 has six layers and about 82M parameters, initialized from every other teacher layer.The student retains the same embedding dimensionality and is fine-tuned with language-modeling and soft-label distillation objectives.Source: §3 · p. 3
4 Methods
The methods combine representational comparisons with circuit-level causal analysis of IOI and GT. They compare matched layers and use task-specific patching and attribution procedures to examine preserved functions and candidate head matches.
4 Methods
- The study uses CKA, logit-lens and attention-pattern analyses for representations, then activation patching and direct logit attribution for IOI and GT circuits.Experiments use 1,000 TinyStories validation samples filtered to 64–256 tokens.Source: §4 · p. 3
4.1 Representational Similarity via CKA
- CKA compares residual-stream activations at matched layers after mean-pooling non-padding positions; same-index and proportional schemes define layer correspondences.The proportional mapping pairs student layers D0–D5 with teacher layers G0, G2, G4, G7, G9, and G11.Source: §4.1 · p. 3
4.2 Logit Lens Analysis
- The tuned logit lens projects each intermediate hidden state through the final layer norm and unembedding matrix, computing scores under both layer-mapping schemes.The provided passage describes the projection and states that the scores are computed under both mappings.Source: §4.2 · p. 3
4.3 Attention Pattern Analysis
- Attention analysis measures head entropy, top-k attention mass, structural attention weights, and cosine similarity between matched attention matrices.The structural weights include BOS, previous-token, and EOS attention.Source: §4.3 · p. 3
4.4 SAE-Based Feature Analysis
- Pre-trained GPT-2 SAEs serve as a fixed reference basis for comparing sparse feature representations across matched layers.The described metrics include activation rate, L0 sparsity and change, and feature survival.Source: §4.4 · p. 3
4.5 Indirect Object Identification Circuit Analysis
- IOI analysis restricts causal patching to seven student-available heads and three inter-head paths, then tests whether student heads assume the name-mover role.Name-mover matching compares attention patterns on discovery and held-out evaluation sets.Source: §4.5 · p. 4
4.6 Greater-Than Circuit Analysis
- GT analysis tests whether any student head replicates GPT-2’s comparison head (9,1), measuring patching scores relative to a noise ceiling and contributions to GT probability mass.The passage notes that only the two year-encoding heads are present in DistilGPT-2; comparison and output-computation heads in layers 9–11 are absent.Source: §4.6 · p. 4
5 Results
DistilGPT-2 preserves the GT circuit through remapping but loses the IOI circuit, while representation and attention comparisons show that layer correspondence affects measured similarity. Aggregate perplexity does not reveal these circuit-level differences.
5.3 Logit Lens
- Under proportional mapping, student relative accuracy exceeds its matched GPT-2 counterpart at every layer except the final, while same-index mapping reverses this pattern.Under same-index mapping, the student underperforms at every layer, with relative accuracy reaching 3.95 at L4.Source: §5.3 · pp. 4–5, Table 3 · p. 5
5.1 Perplexity Baseline
- Perplexity gaps were 6.01 on TinyStories and 16.31 on WikiText-2, yet they did not signal the student’s IOI collapse or its 87.7% GT performance recovery.The student has 52% higher perplexity while retaining GT performance and losing IOI performance.Source: §5.1 · p. 4, Table 2 · p. 5
5.2 Representational Similarity via CKA
- Proportional mapping recovers layer-5 CKA from 0.314 under same-index matching to 0.790, with proportional overall mean CKA = 0.921 ± 0.109.Early layers are near-perfectly aligned (CKA = 0.9998), while similarity degrades with depth.Source: §5.2 · p. 4, Figure 1 · p. 5
5.4 Attention Pattern Analysis
- Attention patterns have mean cosine similarity 0.894 ± 0.086 under proportional mapping, versus 0.631 ± 0.255 under same-index matching.The student’s previous-token attention peaks earlier, at layer 1, and its mean entropy is 2.094 bits versus GPT-2’s 2.001 bits.Source: Figure 2 · p. 5, §5.4 · p. 5
5.5 SAE Feature Analysis
- Under same-index matching, layer-5 feature survival is 0.086 and reconstruction error is 34.9× the teacher’s; proportional matching recovers mid- and late-pipeline survival to approximately 0.53.The proportional early pair (G1/D0) has near-zero survival (0.0002), indicating divergence in that layer pair.Source: §5.5 · pp. 5–6
5.6 Indirect Object Identification Circuit Preservation
- On IOI, DistilGPT-2 scores 35.2% accuracy versus GPT-2’s 94.8%, with a negative mean logit difference of −0.467 ± 1.335 and at most 0% name-mover preservation.Available-head patching yields negligible recovery; the critical PT→IND pathway is non-significant (d = 0.031, p = 0.329).Source: §5.6 · p. 6
5.7 Greater-Than Circuit Preservation and Compression
- DistilGPT-2 recovers 87.7% of teacher GT performance, shifting peak attribution from GPT-2 layer 9 to student layer 4, where L4H1 matches L9H1 (mean Pearson r = 0.945 ± 0.079).Both heads attend to the YY token in all five representative examples.Source: §5.7 · p. 6, Figure 5 · p. 7
5.7.1 GT Circuit: Head-Level Activation Patching
- DistilGPT-2 preserves causal contributions from both available GT year-encoding heads, with normalized scores 0.517 and 0.465; path patching was not performed because no inter-head paths had both endpoints within layers 0–5.The overall effect size is Cohen’s d = 0.681.Source: §5.7.1 · pp. 6–7
6 Discussion
Representations remain similar under proportional layer mapping but diverge under same-index matching, while circuit outcomes split: IOI collapses and GT survives through active remapping. The authors offer circuit compressibility as a candidate explanation, while noting that broader testing is needed.
6.1 RQ1: Representational Similarity After Depth Halving
- At the first layer pair, proportional-mapping CKA and attention cosine similarity are near 1.0, but same-index CKA falls to 0.314 by layer 5.SAE feature survival falls to 0.086, with reconstruction error 34.9× the teacher’s; same-index logit-lens predictiveness also declines with depth.Source: §6.1 · p. 7
- Proportional mapping largely recovers the layer-5 same-index collapse, and functional SAE survival reaches approximately 0.53 at mid- and late-pipeline pairs.Relative logit-lens accuracy exceeds 1 at every layer except the last; magnitude drift is δ ≈ 0.51–0.55.Source: §6.1 · p. 7
6.2 RQ2: Circuit Preservation Under Depth Halving
- IOI collapses completely, with negative mean logit difference and negligible recovery, while GT survives as peak contribution shifts from teacher layer 9 to student layer 4.Student L4H1 reproduces GPT-2 L9H1 attention patterns with mean Pearson r = 0.945; no matched IOI head reproduces IO-selective attention.Source: §6.2 · pp. 7–8
- The authors propose that circuits survive when their critical computations fit within student depth, whereas longer pipelines collapse, and call for testing intermediate-depth circuits across more model pairs.They describe this compressibility account as a candidate explanation, not a confirmed general principle.Source: §6.2 · pp. 7–8
7 Conclusion
The study compares GPT-2 and DistilGPT-2 mechanistically and finds a compression-collapse dichotomy across circuits. It argues that perplexity alone is insufficient to assess circuit preservation and suggests direct circuit evaluation and functionally targeted intermediate-representation alignment.
7 Conclusion
- The comparison of GPT-2 and DistilGPT-2 finds a compression-collapse dichotomy: circuits fitting the student’s available depth are preserved, while longer pipelines are destroyed.The study characterizes this through representational and causal probes.Source: §7 · p. 8
- Aggregate metrics such as perplexity are insufficient for assessing mechanistic preservation, so the authors suggest evaluating circuit integrity directly.They also suggest distillation objectives that align intermediate representations at functionally critical layers.Source: §7 · p. 8
- The suggested objective targets intermediate representations at functionally critical layers.This is a proposed practice, not an intervention evaluated in the study.Source: §7 · p. 8
Limitations
The study’s scope and methods leave several boundaries on its conclusions: it examines two circuits and one teacher–student pair, uses a fixed SAE reference basis, and does not fully verify GT survival.
Limitations
- The analysis covers two circuits and one teacher–student pair, leaving generalization to other circuits, distillation methods, and model families open.The authors identify testing intermediate-depth circuits across additional model pairs as an important extension.Source: Limitations · p. 8
- The IOI null result does not rule out DistilGPT-2 solving a residual version of the task through an alternative mechanism.The authors suggest causal scrubbing or automated circuit discovery to search for such subgraphs.Source: Limitations · p. 8
- Using pre-trained GPT-2 SAEs as a fixed reference basis creates a systematic blind spot to student-only features.The authors state that student-specific SAEs would resolve this limitation.Source: Limitations · p. 8
- GT survival is not fully verified beyond the L4H1–L9H1 match; the year-encoding heads and causal paths through to output remain to be tested.The authors specify activation patching and path patching for that verification.Source: Limitations · p. 8
A.1 Distillation Objective Interventions
The analysis characterizes the mechanistic effects of soft-label distillation but does not intervene in distillation itself. Alternative objectives and their effects on circuit preservation remain open questions.
A.1 Distillation Objective Interventions
- The study characterizes soft-label distillation’s mechanistic consequences but does not intervene in the distillation process.Whether alternative objectives preserve circuit integrity during depth reduction remains open.Source: §A.1 · p. 10
- Intermediate-layer matching, attention transfer, and circuit-aware losses are among the alternative objectives left for future investigation.The passage frames their potential to preserve circuit integrity as an open question.Source: §A.1 · p. 10
- The GT result is preliminary evidence that standard distillation can preserve circuits compatible with student depth; targeted objectives might extend this to deeper circuits such as IOI.The passage presents this as a possibility, not an established effect.Source: §A.1 · p. 10
A.2 SAE Streaming Metrics: Implementation Details
The appendix describes a streaming procedure for SAE statistics and identifies tables summarizing CKA, cosine similarity, attention-pattern metrics, and SAE analyses.
A.2 SAE Streaming Metrics: Implementation Details
- SAE statistics use one streaming pass over batches of 64 activation vectors, avoiding materialization of the full sparse feature matrix.Feature survival is computed from online sufficient statistics; activations are stored to disk, and each SAE is moved from CPU to GPU only for batch encode/decode operations.Source: §A.2 · p. 10
A.3 CKA Full Results by Depth and Seed
- Table 7 reports CKA and cosine similarity by depth quartile under each mapping scheme, averaged across seeds 42, 123, and 7.Source: §A.3 · p. 10, Table 7 · p. 11
A.4 Attention Pattern Analysis: Numerical Summary
- Table 8 summarizes attention-pattern metrics, with head similarity and entropy change reported as mean ± standard deviation across 1,000 TinyStories samples.Source: §A.4 · p. 10, Table 8 · p. 11
A.5 SAE Feature Analysis Results
- Table 9 provides an SAE analysis summary across layer pairs.Source: Table 9 · p. 11