PaperlayerView exact revision on arXiv

Source-linked AI summary

What Does Layer-Importance Reveal About Transformers and State-Space Models?

Istabrak Abbes, Nizar Islah, Irina Rish, Sarath Chandar

arXiv:2609.16537v1cs.LGcs.AI

TL;DR

The paper asks whether transformer-derived layer-importance analysis transfers to state-space models and addresses this by separating pretrained reliance from adaptation-driven change. Its NPA diagnostic reveals negative alignment in evaluated transformers up to 14B but overlapping, positive alignment in evaluated Mamba-style SSMs, with corresponding differences in selective-fine-tuning forgetting.

  • Problem

    Transformer layer-importance tools may not transfer reliably to SSMs because pretrained reliance and adaptation dynamics can differ by architecture.

  • Method

    The paper defines Necessity from ablation-induced loss increase, Plasticity from task-specific update magnitude, and NPA from their rank correlation.

  • Results

    Every evaluated residual transformer up to 14B shows negative NPA, whereas evaluated Mamba-style SSMs show positive alignment with overlapping regions.

  • Takeaways & Limitations

    Highly plastic-layer placement increases transformer forgetting, while the tier-dependent effect disappears in evaluated Mamba-style SSMs.

  • Takeaways & Limitations

    Conclusions are scoped to evaluated checkpoints and scales; other adaptation procedures and broader architecture coverage were not tested.

Abstract

from arXiv · show

Transformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the analytical knowledge built for transformers transfers to SSMs. We address this through the lens of layer importance which underpins compression, selective fine-tuning, and interpretability across both families. We decompose layer importance into two distinct notions. \emph{Necessity} captures how much the pretrained model depends on a layer's existing contribution, measured by the loss increase from bypassing it. \emph{Plasticity} captures where the model absorbs new information during fine-tuning, measured by the magnitude of task-specific weight updates. Our analysis reveals that the two families behave fundamentally differently: in every evaluated residual transformer up to $14$B parameters, Necessity and Plasticity anti-align across depth, whereas in the evaluated Mamba-style SSMs they point to overlapping regions. The sign of this alignment also predicts downstream adaptation behavior. In the evaluated transformers, concentrating updates in the most plastic layers increases catastrophic forgetting, while this tier-dependent effect disappears in the evaluated Mamba-style SSMs.

1 Introduction

The paper asks whether transformer-based layer-importance analysis transfers to SSMs, separating pretrained dependence from adaptation-driven change. NPA reveals opposite architecture-specific patterns and links them to selective fine-tuning stability.

  • 1 Introduction: Necessity measures loss increase after bypassing a layer, whereas Plasticity measures task-specific update magnitude during fine-tuning.NPA is the mean Spearman correlation between the resulting layer rankings.
  • 1 Introduction: NPA is negative in every evaluated residual transformer up to 14B but positive in every evaluated Mamba-style SSM, separating the families.Transformers concentrate Necessity early and Plasticity late, whereas SSMs place them in overlapping regions.
  • 1 Introduction: The architecture split survives removal of the first layer, adapter-free full fine-tuning, and evaluation beyond Mamba to RWKV and hybrid checkpoints.The largest transformers instead move toward near-zero alignment rather than entering the SSM regime.
  • 1 Introduction: In a controlled two-task probe, concentrating updates in highly plastic transformer layers increases forgetting, while this tier-dependent effect disappears in evaluated Mamba-style SSMs.This connects NPA sign with downstream selective-adaptation behavior.

2 Related Work

Prior work studies layer importance for pruning, task-aware elimination, adaptation, and attribution, mainly within transformers. This paper extends the disagreement question across transformer and SSM architectures.

  • 2 Related Work: Transformer research uses layer-importance estimates for pruning, selective fine-tuning, interpretability, and task-aware layer elimination.Examples include resnorm diagnostics, Shapley-style sensitivity, and learned significance masks.
  • 2 Related Work: Earlier studies document stability or disagreement within individual estimators, whereas this work compares layer rankings across architectures and importance properties.The contribution shifts method disagreement from feature attributions to architecture-conditional layer rankings.

3 Layer-Importance Diagnostic Protocol

The protocol separates layer importance into Necessity and Plasticity, then measures their agreement with NPA rather than assuming one universal depth ranking.

  • 3 Layer-Importance Diagnostic Protocol: Necessity is the loss increase caused by bypassing a layer, so larger values indicate stronger reliance on its pretrained computation.It operationalizes dependence relevant to pruning and interpretability.
  • 3 Layer-Importance Diagnostic Protocol: Plasticity is the magnitude of task-specific LoRA updates, so larger values identify layers receiving more adaptation.The score uses the Frobenius norm of each layer’s learned update.
  • 3 Layer-Importance Diagnostic Protocol: NPA is the Spearman correlation between Necessity and Plasticity rankings: positive values indicate overlap, negative values indicate opposing depth concentration, and near-zero values indicate no stable ordering.The protocol compares the relationship independently of any single proxy estimator.

4 Transformers and SSMs Disagree on Layer Importance

Transformers and SSMs organize pretrained reliance and adaptation differently across depth, and this architecture-dependent alignment changes how selective fine-tuning affects forgetting.

  • 4 Transformers and SSMs Disagree on Layer Importance: Every evaluated transformer up to 14B has negative NPA, while evaluated Mamba-style SSMs have positive NPA; larger transformers trend toward zero instead.The per-task pattern is consistent within families and does not arise from a few outlier tasks.
  • 4 Transformers and SSMs Disagree on Layer Importance: The split persists under full-parameter fine-tuning and weight-normalized updates, so it is not explained by LoRA rank, scaling, initialization, or target-module selection.The sign is preserved across retrained checkpoints.
  • 4 Transformers and SSMs Disagree on Layer Importance: Transformers place Necessity near the input and Plasticity near the output, whereas Mamba profiles overlap through mid-network and boundary regions.Large transformers gain early-layer Plasticity mass, weakening anti-alignment without producing SSM-like positive agreement.
  • 4 Transformers and SSMs Disagree on Layer Importance: Removing layer 0 makes transformer alignment less negative but does not reverse it, ruling out an embedding bottleneck as the sole explanation.Mamba alignment changes minimally.
  • 4 Transformers and SSMs Disagree on Layer Importance: EWC achieves the lowest forgetting in Mamba-790M, whereas bottom-k placement matches or outperforms EWC in the evaluated transformers.This supports architecture-specific choices between global regularization and avoiding high-plasticity layers.

5 Mechanistic Account

Residual transformers separate pretrained dependence from adaptation across depth, whereas Mamba-style SSMs lack this systematic separation. Curvature, residual propagation, layer-0 controls, and scale analyses support this architecture-dependent account.

  • 5.1 Residual geometry biases Necessity toward early layers in transformers: Residual propagation makes earlier transformer contributions structurally more consequential, producing early Necessity concentration and a bias toward layer 0.Earlier perturbations pass through more downstream blocks, although Jacobian products can amplify, suppress, rotate, or cancel them.
  • 5.2 Curvature and adaptation favor late transformer layers: Late transformer layers provide favorable low-cost sites for task-specific updates, creating spatial separation between early Necessity and late Plasticity.Their direct influence on final outputs allows adaptation with less disruption to earlier computations, predicting negative alignment.
  • 5.2 SSM dynamics avoid systematic separation: Mamba’s continuous selective input injection and recurrent dynamics remove the architectural pressure forcing Necessity and Plasticity toward opposite depth ends.Its layers read the current token through learned gates, so no single layer has the transformer-like privileged initialization role.
  • Every evaluated residual transformer up to 14B shows negative alignment, while Mamba-style SSMs show non-negative alignment with overlapping Necessity and Plasticity regions.The contrast persists across estimators, survives removal of the first layer, and extends beyond Mamba to RWKV and a hybrid architecture.
  • 5.3 Layer-0 controls rule out a pure embedding-bottleneck explanation: Removing layer 0 weakens transformer anti-alignment but leaves it negative, while Mamba checkpoints remain positive, ruling out a single embedding-bottleneck explanation.The contrast therefore reflects broader early-side Necessity and late-side Plasticity separation rather than one anomalous layer.
  • 5.4 Curvature supports the Necessity–Plasticity split: Fisher curvature co-varies with Necessity and anti-correlates with Plasticity in transformers, but both Fisher relationships are non-negative in SSMs.This supports curvature as a complementary explanation for why pretrained dependence and adaptation separate in transformers but not Mamba-style SSMs.
  • 5.5 Scale and estimator structure: At larger transformer scales, added early Plasticity modes and profile flattening collapse alignment toward zero rather than producing the positive alignment observed in SSMs.The measured checkpoints support the trend, but the mechanism does not determine where alignment terminates or justify extrapolating its sign.
  • 5.5 Scale and estimator structure: Layer-importance estimators partition into Necessity- or Plasticity-aligned groups in transformers, while clustering in a mixed-positive region for Mamba-style SSMs.Thus estimator meaning depends on architecture and on which property—pretrained dependence or adaptation dynamics—it measures.

6 Conclusion

The paper concludes that layer importance is not a model-independent depth ordering. Separating Necessity from Plasticity reveals an architecture-dependent split that also informs selective adaptation stability.

  • Residual transformers up to 14B have negative alignment, with early layers most necessary and later layers most plastic, whereas evaluated Mamba-style SSMs overlap these roles.The split survives adapter-free controls and reappears within a hybrid checkpoint.
  • Concentrating updates in the most plastic transformer layers increases forgetting, while this tier-dependent stability cost largely disappears in evaluated Mamba-style SSMs.Layer-0 controls and curvature measurements support a broader separation between pretrained dependence and adaptation rather than a single anomalous layer.
  • Layer-importance methods should be treated as measurements of specific properties, because transformer-calibrated tools may fail silently on recurrent or hybrid architectures.The geometry of stored computation and new adaptation can differ fundamentally across architectures.

7 Limitations

The theory separates rigorous architectural claims from empirical observations and states important scope boundaries. Its conclusions depend on operational importance measures, explicit assumptions, and evaluated checkpoints rather than all possible architectures or adaptation procedures.

  • Plasticity is adaptation-dependent rather than intrinsic, because it is operationalized through task-specific weight updates; full fine-tuning and weight-norm normalization preserve NPA’s sign but do not remove all dependence.Other adaptation procedures, optimizers, and budgets were not tested.
  • The negative-transformer result is established only through 14B parameters; 32B and 70B models move toward near-zero alignment, with no evidence beyond 70B.Hybrid-model analysis also relies on a single checkpoint with one attention/SSM ratio.
  • The framework covers transformers, Mamba-style SSMs, RWKV, and hybrids, but remains limited relative to the broader architecture space.Hybrid and very large transformer results support interpreting NPA as continuous rather than a strict family label.
  • Ablation measures coarse layer-level functional reliance, while activation patching, causal tracing, and optimizer-state analyses could reveal finer mechanisms.The mechanistic account is therefore partial.
  • Theorems A.3 and A.6 provide rigorous mass-displacement statements, while the Spearman bound additionally requires an explicit profile-shape assumption verified empirically.This separation exposes the premises supporting the theoretical claims.

A.4 Theorem 3: Failure of Mass Displacement on Selective SSMs

The selective-SSM theorem argues that the mass-displacement mechanism forcing negative Necessity–Plasticity correlation in residual transformers fails under symmetric expected impulse response and distributed input injection. Consequently, the bound forcing negative correlation vanishes with depth, although positive empirical correlation requires additional structure.

  • The expected impulse-response condition bounds influence by distance after averaging over data, accommodating input-dependent selective recurrence rather than requiring pointwise kernel symmetry.This is a weaker assumption tailored to selective Mamba-like dynamics.
  • Selective SSMs violate privileged input injection because information enters through per-token selective gates at every layer rather than a unique residual-stream initialization.This removes the first structural ingredient of the residual-transformer mass-displacement argument.
  • Under selective-SSM assumptions, the mass-disagreement coefficient δ vanishes as L/K grows, so the transformer mechanism forcing negative rank correlation no longer applies.The theorem assumes expected symmetric impulse-response decay and no privileged input injection.
  • The theorem does not prove positive SSM correlation: empirical NPA may be positive, zero, or weakly negative depending on additional profile structure such as shared mid-network unimodality.The observed positive Mamba correlation is therefore an empirical fact consistent with, but not entailed by, the theory.
  • The conclusion applies beyond selective Mamba to architectures satisfying the same assumptions, including full SSMs, RWKV-style models, and linear attention with symmetric expected kernels.Architectures satisfying only the impulse-response condition inherit only partial vanishing of δ.

B.4 Necessity–Plasticity projection of additional estimators

Additional importance estimators occupy distinct Necessity–Plasticity positions on transformers, separating pretrained-checkpoint diagnostics from methods incorporating adaptation information. Causal validation is method-relative and generally shows small or unstable gains rather than a universal advantage.

  • B.4 Necessity–Plasticity projection of additional estimators: Gradient and Resnorm load toward Necessity, whereas Activation norm and TELL-TALE load toward Plasticity; ShapLoRA lies near the boundary.Transformer loadings are Gradient (ρP = −0.66, ρN = +0.47), Resnorm (ρP = −0.89, ρN = +0.46), Activation norm (ρP = +0.84, ρN = −0.35), TELL-TALE (ρP = +0.65, ρN = −0.32), and ShapLoRA (ρP = +0.22, ρN = −0.08).
  • B.4 Necessity–Plasticity projection of additional estimators: LongBench is supported by LoRA delta in all three seeds, SQuAD by ablation in all three, and PIQA fails under LoRA delta in all three.These binary labels indicate robustness of a proxy direction, not uniformly strong practical gains.
  • B.4 Necessity–Plasticity projection of additional estimators: LoRA delta is the most consistently interpretable high-performing method across Qwen3-8B and Llama-3.1-8B, while gradient can achieve competitive aggregate support despite near-flat rankings.Resnorm falls at or below random on both models.
  • B.4 Necessity–Plasticity projection of additional estimators: Most supported task–model–method pairs have small primary-metric gaps, typically 0.01–0.05, while random placement and several extensions remain near zero.Metric gaps should be inspected directly because support labels encode direction but not magnitude.
  • B.4 Necessity–Plasticity projection of additional estimators: Across k-sweeps, the qualitative method-relative verdict is budget-insensitive: mean support is 0.23 with mean gap −0.025 for Llama and 0.29 with mean gap +0.005 for Qwen.The fixed k = ⌈L/6⌉ choice lies within the swept envelope.

C.5 Task-A adaptation quality: top-k versus bottom-k

Top-k placement can improve Task-A adaptation modestly in transformers, but its forgetting penalty is substantially larger than the learning-rate difference. In SSMs, top-k and bottom-k achieve similar adaptation and forgetting outcomes.

  • The forgetting analysis directly measures post-Task-A perplexity before Task-B training alongside the resulting forgetting ratios for both placement strategies.This design tests whether lower forgetting merely reflects weaker initial Task-A learning.
  • In transformers, top-k yields PPLbefore 1.71 versus 2.00 for bottom-k, yet forgetting ratios are 3.24 versus 2.33, respectively, across 75 checkpoint pairs.The 17% Task-A adaptation advantage is smaller and opposite in direction to the 39% forgetting disadvantage.
  • The 17% starting-perplexity advantage cannot explain the 39% forgetting-ratio disadvantage, ruling out differential Task-A learning as the sole account.This supports a bottleneck effect beyond the confound.
  • In SSMs, top-k and bottom-k have nearly identical PPLbefore values, 6.53 versus 6.73, and statistically indistinguishable forgetting ratios, 2.11 versus 2.00.The comparison covers 54 model pairs.

D.1 Head-to-head comparison with published placement methods

Matched causal validation shows that LoRA-delta placement remains informative for Mamba but weakens with transformer scale, while estimator and submodule choices materially affect placement conclusions.

  • Published placement methods: Competitor placement methods underperform random on both transformer checkpoints and Mamba-790m, with near-zero or negative mean primary-metric gaps.The matched protocol evaluates ShapLoRA and TELL-TALE on common task subsets.
  • Scale and architecture extension: Mamba-790m shows positive LoRA-delta top-k versus bottom-k primary gaps at both seeds, whereas Qwen3-14B favors random top-k over LoRA delta.The extension uses the same k = 6, LoRA rank 8, and 300 training steps at seeds 43 and 44.
  • Mamba placement patterns: LoRA-delta placement shifts Mamba top-k layers toward the mid-network and retains high support rates, unlike the systematic failures seen for transformers on several tasks.Under ablation, Mamba top-k layers instead follow the boundary pattern at all tested scales.
  • Submodule-scope control: Negative cross-method correlations persist within attention, MLP, and full-block scopes, ruling out mixed-submodule comparisons as the source of transformer anti-correlation.For Llama-3.1-8B, the correlations are −0.581, −0.266, and −0.298, respectively.
  • Submodule comparisons: Within LoRA delta, attention and MLP rankings converge on the same terminal layers, whereas ablation identifies more model-dependent submodule-critical sets.LoRA submodule correlations reach +0.839 in Llama and +0.659 in Qwen; ablation overlap differs sharply across models.
  • Adapter-free control: Full-parameter fine-tuning preserves the sign of LoRA-based NPA across all retrained checkpoints, although update-profile correlation declines with model size.The full-versus-LoRA correlation falls from 0.96 on Qwen3-0.6B to 0.39 on Qwen3-14B, while sign agreement remains intact.

E.4 Component-level analysis of a hybrid checkpoint

The hybrid Zamba2 checkpoint contains opposing component-level importance profiles: attention absorbs adaptation mass, while Mamba blocks carry most Necessity.

  • Component-level dissociation: Attention comprises 16.7% of Zamba2’s layers but receives 36.3% of LoRA and 31.6% of full-fine-tuning update mass.These shares are roughly twice the attention component’s layer share.
  • Component-level dissociation: Attention carries only 10.4% of ablation Necessity mass, about 0.6× its layer share, while Necessity concentrates in Mamba blocks.The same checkpoint therefore separates adaptation concentration from pretrained computational dependence.
  • Whole-model interpretation: Zamba2’s near-zero whole-model NPA of +0.05 averages transformer-like attention behavior with SSM-like Mamba behavior rather than showing no underlying effect.The component decomposition reveals opposing profiles within a single model.
  • Scope: The component-level conclusion is limited to one checkpoint with one attention-to-SSM ratio and should not be extrapolated to other mixtures.The authors explicitly treat claims about other ratios as beyond the measured setting.
  • Validation scope: Causal-validation labels use a shared likelihood proxy, so they should not be interpreted as universal end-task performance claims.The proxy provides a controlled cross-task comparison but is not interchangeable with task-native metrics.

H Forgetting

Forgetting differs by architecture and plasticity placement: transformer top-k placement increases forgetting relative to bottom-k, whereas this tier effect disappears in evaluated SSMs.

  • Metric dependence: Task-native MCQ interpretation remains metric-relative: top-k versus bottom-k advantages depend on both the estimator and model family.Shared-proxy rankings do not guarantee the same downstream conclusion under task-native metrics.
  • Forgetting by architecture: In transformers, top-k placement produces higher forgetting than bottom-k across model scales, while SSM placement strategies show no consistent ordering.Figure 20 encodes top-k as the most-plastic tier and reports median forgetting ratios across seeds and task pairs.
  • SSM behavior: Mamba-style SSMs show statistically indistinguishable top-k and bottom-k forgetting ratios, with p = 0.45.This is consistent with Plasticity being distributed rather than concentrated in one vulnerable tier.
  • Scale-dependent plasticity: Qwen3-32B develops an early-layer LoRA-delta peak alongside its terminal cluster, producing bimodal plasticity absent from smaller Qwen3 variants.The bimodality is identified as the proximate clue for scale-induced collapse of transformer alignment.
Loading 2609.16537v1…