PaperlayerView exact revision on arXiv

Source-linked AI summary

Mechanistic Failure Modes of Weight Pruning in Transformer Language Models

Nabeelah Faumi, Deshan Sumanathilaka

arXiv:2610.37126v1cs.LGcs.CL

TL;DR

The paper asks whether preserved accuracy and perplexity show that pruning preserves internal computation. It tests pruning choices against GPT-2’s IOI circuit and finds distinct mechanistic failure modes that output metrics and structural checks can miss.

  • Problem

    Pruning is judged largely by accuracy and perplexity, leaving open whether those measures establish that a model’s internal computation is preserved.

    Source: §1 · p. 1

  • Method

    The study factorially varies pruning criterion, component type, and sparsity in GPT-2, then evaluates IOI behaviour and circuit integrity.

    Source: §3.1 · p. 2, §3.3 · p. 3

  • Results

    Pruning produces distinct failure modes driven primarily by component type: magnitude head pruning eliminates output-facing heads, while magnitude neuron pruning corrupts the MLP context and misdirects attention.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Behavioural metrics and component-presence checks can miss mechanistic damage, so circuit-level causal analysis is a complementary evaluation of compressed models.

    Source: §1 · pp. 1–2

    GPT-2 small has no safety fine-tuning, so remarks about refusal or harm-avoidance mechanisms are extrapolations rather than measured results.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Pruned language models are judged almost entirely by accuracy and perplexity, which assumes that preserved output behaviour implies preserved internal computation. We test this assumption with a factorial pruning study on GPT-2, crossing two importance criteria (magnitude, random), two component types (attention heads, MLP neurons) and three sparsity levels (3%, 50%, 70%). Using the fully characterised indirect object identification (IOI) circuit as ground truth, we measure circuit integrity with activation patching, path patching and direct logit attribution. We find two qualitatively distinct failure modes. Magnitude head pruning >= zeroes the circuit: output-facing heads are removed, and the model falls to near-chance. Magnitude neuron pruning at 30% inverts it: all 26 circuit heads remain intact, but Name Mover contributions flip sign, and the model systematically prefers the wrong name (17.7% accuracy). Perplexity ranks it as less damaged than random head pruning (74.6 vs 140.1). A replication of the Greater-Than circuit shows that the critical component type depends on the circuit's architecture. These results show that output metrics and structural checks can both miss mechanistic damage, motivating circuit-level causal analysis as a complementary evaluation of compressed language models.

1 Introduction

The paper asks whether benchmark performance is enough to establish that pruning preserves a model’s internal computation. Using a well-characterised circuit, it examines how pruning choices affect circuit integrity beyond accuracy.

1 Introduction

  • Pruning is commonly judged by accuracy and perplexity, implicitly treating preserved output behaviour as evidence of preserved internal computation.The paper questions whether that judgment is safe.Source: §1 · p. 1
  • The study varies which components are removed and how they are selected to test for distinct effects on a well-characterised circuit.It compares attention heads with MLP neurons, and magnitude-based with random selection, across sparsities.Source: §1 · p. 1
  • The paper identifies circuit zeroing and inversion, and shows that behavioural and structural checks can miss mechanistic failure.An inverted circuit can remain structurally intact while producing systematic errors, and perplexity can rank it as less damaged.Source: §1 · p. 1
  • The authors motivate circuit-level causal analysis as a complementary evaluation of compressed language models.The introduction closes by outlining the paper’s subsequent review, experimental setup, results, and discussion.Source: §1 · pp. 1–2

2 Background and Related Work

The background presents circuit analysis as a way to test component roles, distinguishes structured pruning choices, and identifies a gap in understanding their mechanistic effects.

2.1 Mechanistic Interpretability and Circuit Analysis

2.2 Weight Pruning

  • Magnitude pruning removes low-magnitude weights, while random pruning provides a controlled baseline; structured pruning can remove whole heads or neurons (Han et al., 2015).Heads perform relational computation across token positions, whereas MLP neurons store features and representations that downstream heads depend on (Rai et al., 2024).Source: §2.2 · p. 2

2.3 Pruning and Circuit Integrity: Prior Work

  • Prior work studied attribution-guided pruning and circuit survival, but had not characterised distinct circuit failures across pruning strategies or their invisibility to accuracy evaluation.The paper states that it addresses both gaps.Source: §2.3 · p. 2

3 Experimental Setup

The experiments use GPT-2 and a factorial pruning design, evaluating both IOI behaviour and circuit contributions under controlled, matched starting conditions.

3.1 Base Model

  • All variants derive from one pretrained GPT-2 checkpoint with no post-pruning weight updates, enabling comparisons at matched starting conditions.The model has 117M parameters, 12 layers, 12 heads per layer, and hidden dimension 768.Source: §3.1 · p. 2

3.2 Pruning Variants

  • The study crosses magnitude and random selection, head and neuron pruning, and 30%, 50%, and 70% sparsity in 12 variants.Magnitude pruning removes globally low-L2-norm head Q-weight slices or neuron Win columns; random pruning samples a fixed fraction per layer.Source: §3.2 · pp. 2–3

3.3 Evaluation Datasets and Tasks

  • IOI evaluation uses the mean logit difference between indirect-object and subject names; circuit analysis applies activation patching and class-level DLA.The evaluation uses 15 templates, 14 objects, 28 single-token names, and corrupted counterparts; TinyStories perplexity measures language-modelling performance.Source: §3.3 · p. 3

3.4 Implementation Details

  • Implementation disables LayerNorm folding and verifies sparsity after conversion.DLA uses a linear approximation of final LayerNorm centred on the full residual stream; experiments use a fixed seed of 42 on an NVIDIA L4/T4 GPU.Source: §3.4 · p. 3

4 Results

Pruning produced both circuit zeroing and inversion in IOI, while conventional metrics could rank the strongly inverted model as less damaged. The Greater-Than replication shows that which component type matters depends on circuit architecture.

4.1 Behavioural Baseline

  • The unpruned GPT-2 baseline reaches 94.8% IOI accuracy and mean IOI logit difference 2.942 ± 1.797 on the 300-sentence evaluation set.TinyStories perplexity is 11.54 and serves as the reference baseline.Source: §4.1 · p. 3

4.2 Failure Mode I: Zeroing

  • At 50% and 70% sparsity, magnitude head pruning physically eliminates output-facing IOI heads, yet residual non-circuit computation leaves 45.7%–47.9% accuracy.At 50%, DLA scores for all four major circuit classes are exactly zero, while remaining model computations incidentally favor the IO token on most sentences.Source: §4.2 · p. 3

4.4 High-Sparsity Limit of Inversion

  • At 70% random neuron pruning, NM, NNM, and BNM DLA retention is approximately zero, and accuracy is 47.9%, marking a noise-dominated limit of inversion.The logit difference moves from −0.896 at 30% to −0.502 at 50% and −0.038 at 70%.Source: §4.4 · p. 4

4.3 Failure Mode II: Inversion

  • At 30% sparsity, magnitude neuron pruning leaves all 26 IOI circuit heads intact but flips major contributions, yielding a −2.475 mean logit difference and 17.7% accuracy.Name Mover and Backup Name Mover heads point toward the subject; removing MLP representations corrupts the context those heads read.Source: §4.3 · p. 4, Table 2 · p. 5

4.5 Perplexity and Representational Fidelity

  • At 30% sparsity, magnitude neuron pruning has TinyStories perplexity 74.61 versus 140.10 for random attention-head pruning, despite a worse IOI logit difference (−2.475 versus −0.706).At 70%, early-layer CKA remains at least 0.900 across strategies, while late-layer CKA ranges down to 0.260; magnitude neuron pruning has CKA 0.500.Source: Table 3 · p. 5, §4.5 · p. 5

4.6 Attention Diffusion and Sharpening

  • At 70% head pruning, attention entropy rises from 2.013 nats to 3.561 for magnitude and 3.536 for random pruning, while neuron pruning can sharpen attention.Aggregate entropy can miss inversion: the 30% magnitude-neuron model looks comparatively intact by this measure even as Name Mover heads attend to the subject.Source: §4.6 · p. 6

4.7 Greater-Than Circuit Replication

  • Magnitude neuron pruning inverts Greater-Than logit difference at every sparsity, while both neuron-pruning criteria reach 0.0% accuracy by 50%.The GT circuit is MLP-dominant, so surviving attention heads cannot compensate for the removed computation.Source: §4.7 · p. 6, Table 4 · p. 7

5 Discussion

Pruning can zero a circuit or invert its behavior, and the failure signature depends on both component type and selection criterion. Structural integrity checks and perplexity can miss inversion, while some weak pathways recur across conditions.

5.1 Component Type and Selection Criterion Jointly Determine Failure Mode

  • At 30% sparsity, magnitude head pruning shows a zeroing profile, while random head pruning flips NM, NNM, and especially BNM contributions.Magnitude selection mainly removes late-layer output heads; random selection also damages upstream paths feeding surviving Name Movers, yielding a partial-inversion signature.Source: §5.1 · p. 7

5.2 Why Inversion Is the Most Dangerous Failure Mode

  • At 30% magnitude neuron pruning, all 26 circuit heads remain intact, yet accuracy is 17.7% and perplexity is 74.61 versus 140.10 for random attention pruning.Name Mover and Backup Name Mover contributions turn negative, producing a mean logit difference of −2.475 and systematic preference for the subject.Source: §5.2 · pp. 7–8

5.3 Structural Weak Points

  • The S-INH→NM pathway degrades in every reported condition, while NNM heads show negative patching effects across all twelve conditions.The proposed explanation—that NNM heads need greater precision to target the subject token—is a hypothesis the authors do not test directly.Source: §5.3 · p. 8

6 Conclusion

Weight pruning disrupts Transformer circuits through distinct failure modes driven primarily by component type rather than sparsity. Head pruning can cause transparent failure, whereas neuron pruning can produce confident misdirection.

6 Conclusion

  • Magnitude attention head pruning eliminates output-facing heads, producing undirected near-chance failure, while magnitude neuron pruning corrupts MLP context and produces confident misdirection.At high sparsity, random neuron pruning instead approaches a terminal, noise-dominated regime.Source: §6 · p. 8
  • Circuit failure modes are driven primarily by component type rather than sparsity level.Magnitude attention pruning produces circuit zeroing; magnitude neuron pruning produces circuit inversion.Source: §6 · p. 8
  • At high sparsity, random neuron pruning's correct- and incorrect-sign contributions fade together into a terminal, noise-dominated regime, not a stable third mechanism.Source: §6 · p. 8

Limitations

The study’s conclusions are bounded by its model, circuit, data, pruning protocol, and feature-analysis setup; safety implications are extrapolations rather than measured results.

Model and scale

  • Experiments use only GPT-2 small (117M parameters), so the reported failure modes may not transfer directly to larger models.The authors identify replication at larger scales as the immediate extension.Source: Limitations · p. 8

Circuit coverage and data

  • Only IOI and Greater-Than are studied on synthetic prompts, leaving natural-language effects and generality across circuits untested.The authors frame the taxonomy and component-type asymmetry as hypotheses for other circuits, not general principles.Source: Limitations · p. 9

Pruning protocol

  • The protocol prunes whole heads or neurons using simple magnitude or random criteria, without weight updates afterward.Unstructured pruning, layer or block removal, gradient- or attribution-based criteria, and postpruning fine-tuning are not studied and may change or repair the observed failure modes.Source: Limitations · p. 9

Safety implications

  • Safety implications are extrapolated from S-Inhibition and Negative Name Mover circuits because GPT-2 small has no safety fine-tuning.Refusal- or harm-avoidance effects are not measured.Source: Limitations · p. 9

Feature-level analysis

  • The SAE was trained on the base model, so high reconstruction error indicates activations leave its feature manifold, not which features the pruned model has.Layer 11 is excluded because the base model’s own reconstruction error exceeds the validity threshold.Source: Limitations · p. 9

A.1 Logit-Lens Divergence

Logit-lens tracking finds that divergence begins at layer 6 in most pruned models, with magnitude attention pruning diverging later and magnitude neuron pruning showing an early drop.

A.1 Logit-Lens Divergence

  • Eight of twelve pruned models first diverge at L6, and none diverges earlier, using a significant accuracy drop exceeding 5 percentage points.Divergence is assessed with a Bonferroni-corrected paired t-test at p < 0.05.Source: §A.1 · p. 10
  • Magnitude attention pruning diverges latest: at L9 for 30% and 50% sparsity, and at L7 for 70% sparsity.Relative accuracy is above 0.89 at L9 and 0.883 at L7.Source: §A.1 · p. 10
  • All magnitude neuron models diverge at L6, with mean relative accuracy 0.670 at 30%, while random neuron pruning reaches 0.881 at 30%.This criterion reversal matches the CKA result reported in Table 3.Source: §A.1 · p. 10

A.2 Base Model DLA Reference Profile

The base GPT-2 DLA profile establishes the reference contributions used for retention percentages and distinguishes direct output contributors from upstream signal processors.

A.2 Base Model DLA Reference Profile

  • Table 6’s base GPT-2 DLA profile supplies the reference for all retention percentages reported in Table 2.NM heads dominate positive contribution, while NNM heads make a large correct-sign negative contribution as a self-correcting brake.Source: §A.2 · p. 10
  • Negative retention indicates a sign flip relative to the base that actively harms correct IO prediction; for NNM, positive compressed DLA yields negative retention because its base DLA is −2.094.Source: §A.2 · p. 10

A.3 Residual Path Patching: Full Four-Pathway Results

Across the appendix analyses, the S-INH→NM pathway is a recurring weak point, while pruning type shapes routing, causal contributions, and residual-stream feature integrity.

A.4 Attention Pattern Analysis

  • Head pruning diffuses attention under both criteria, while neuron pruning leaves routing comparatively intact; random head pruning lowers mean cosine similarity more than neuron pruning at matched sparsities.At 30%, mean cosine similarity is 0.704 for random head pruning versus 0.894 for neuron pruning; at 70%, it is 0.482 versus 0.819.Source: §A.4 · p. 10, Table 8 · p. 12

A.3 Residual Path Patching: Full Four-Pathway Results

  • The S-INH→NM pathway is the only pathway that degrades significantly under every pruning condition, including random neuron pruning at 30%.At that condition, all other pathways show reorganisation rather than degradation.Source: §A.3 · p. 10, Table 7 · p. 11

A.6 SAE Reconstruction Errors: Corroborating Evidence for Residual Stream Collapse

  • At 30% magnitude neuron pruning, Layer 1 reconstruction error reaches 155.87 (>350× base) despite 26 intact heads; random neuron errors are 16.70 and 33.11 at Layer 6, 30%–50%.The magnitude-pruned residual stream is described as collapsing into a low-rank subspace that corrupts token-identity encodings read by Name Movers.Source: §A.6 · p. 11

A.5 Activation Patching Results

  • S-Inhibition is the most degraded activation-patching class in all twelve conditions (d = 0.141–0.414), while NNM has negative d in every condition.Negative d indicates a paradoxically larger patching effect, consistent with disinhibition rather than simple reduction.Source: §A.5 · p. 10, Table 9 · p. 12

Related papers

Loading 2610.37126v1…