PaperlayerView exact revision on arXiv

Source-linked AI summary

Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models

Sangyoon Bae, Sk Miraj Ahmed, Shinjae Yoo, Jiook Cha

arXiv:2610.09709v1cs.LG

TL;DR

Existing OOD diagnostics require source or target data, while weight-only approaches apply one statistic across architectures. This paper conditions spectral weight statistics on architecture–strategy cells; acting on spectral concentration narrows the OOD gap by 24% at 87.5% ID retention.

  • Problem

    Data-dependent diagnostics require labeled source examples or target access, while weight-space methods apply one statistic across architectures and rank models at correlations around 0.4–0.5.

    Source: §1 · p. 1

  • Method

    The paper formalizes representation geometry within architecture–strategy cells and derives weight-only spectral proxies, using each cell's distinct geometric fingerprint.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    24% narrower OOD gap at 87.5% ID retention; metric-guided layer selection beats random targeting in 9 of 10 models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The diagnostic transferred to EEG, genomic, and protein model families, with predicted directions holding after the cell, metric, and sign were logged before evaluation.

    Source: §1 · p. 3

    The framework ranks models within a cell rather than predicting absolute target accuracy; divergent architectural subtypes need separate calibration, and anchoring direction in a domain with few public models requires at least four reference models.

    Source: U LIMITATIONS · p. 37

Abstract

from arXiv · show

Can we determine whether a foundation model will generalize out-of-distribution (OOD) before any target data is available? Existing diagnostics require source or target data, which rules them out before a target domain exists. Those that use the weights alone apply one statistic to every architecture, and do not separate robust models from fragile ones. We show the answer is encoded in the spectral structure of pretrained weights. Two forces shape that structure. Architecture determines how information is stored in weight matrices. Pretraining strategy determines what is rewarded. Together they set a spectral geometry that governs OOD robustness. We prove that the OOD accuracy gap is bounded by how tightly the source representations concentrate. A statistic computed from the pretrained weights alone serves as a proxy for that concentration. The direction of that proxy reverses between architecture families. We operationalize it: the direction is stable within one (architecture X strategy) combination, the finest grouping we test, which we call a cell. Pooled over 116 models spanning 7 modalities, a single statistic ranks OOD robustness weakly, because cells of opposite direction cancel. Within a cell, the statistic selected for it orders 92% of model pairs by OOD robustness in-sample. The selection does not leak the target: for each model family outside the matrix we logged the cell, metric and sign before running its OOD evaluation, and the predicted direction held in every case: EEG, genomic and protein. Acting on spectral concentration narrows the OOD gap by 24% at 87.5% ID retention. The diagnostic operates on released weights alone, so OOD robustness becomes checkable at model-selection time, before data or compute is committed to a target domain.

1 INTRODUCTION

OOD diagnostics either require source or target data or apply one statistic across architectures, while architecture and pretraining strategy jointly shape weight spectra that encode OOD behavior. Conditioning on those fingerprints recovers predictive signal and supports weight-only model selection.

1 INTRODUCTION

  • Data-dependent diagnostics require labeled source examples or target access, while weight-space methods use one statistic across architectures and rank models at correlations around 0.4–0.5.The paper argues that pooling models shaped by different forces, rather than noisy spectral signals alone, flattens the relationship.Source: §1 · p. 1
  • Architecture and pretraining strategy jointly shape distinct spectral fingerprints; pooling incompatible fingerprints cancels OOD-predictive signal, while conditioning restores it.Architecture controls how information is stored, while strategy controls what is rewarded.Source: §1 · p. 2, Figure 1 · p. 2
  • The sign of ρ depends on the architecture–strategy cell and its calibrated statistic, not on the domain.AR-CLM cells are positive, whereas BiDi-MLM/Protein, CNN, and SSM-v2/RWKV are negative under their respective metrics.Source: Figure 1 · p. 2
  • 24% narrower OOD gap at 87.5% ID retention; metric-guided layer selection beats random targeting in 9 of 10 models.The diagnostic's predicted direction also held in EEG, genomic, and protein families evaluated after the cell, metric, and sign were logged.Source: §1 · p. 3

2 RELATED WORK

Prior weight-space methods connect spectral statistics to model quality, but correlations can reverse across architectural subgroups. Related work also links representation geometry to OOD behavior and describes how architecture and pretraining objectives shape that geometry.

2 RELATED WORK

  • Weight-space approaches tie spectral statistics to model quality, and prior work reports sign reversals between architectural subgroups; this paper operationalizes that pattern with 15 architecture–strategy cells and a statistic selected for each.The framework extends the relationship from ID generalization to OOD robustness.Source: §2 · p. 3
  • Prior studies link representation rank and geometry to OOD behavior but require source-domain access; architecture and pretraining objectives also impose characteristic pressures on representation geometry.The paper reads one empirically selected spectral statistic per architecture–strategy cell as a weight-space proxy for those quantities.Source: §2 · p. 3

3 THEORY: SPECTRAL CONCENTRATION PREDICTS OOD ROBUSTNESS

This section develops the link between feature-level spectral concentration and OOD degradation, explaining how architecture and pretraining strategy shape the weight-space statistics and polarity that track it. It also reports a genomic SSM cell’s directional but nonsignificant association and outlines how the appendix distinguishes proved claims from assumptions in connecting residual weights to feature geometry.

2. Under T1–T6 (Appendix C.1),

  • Under T1–T6, feature log stable rank orders OOD degradation: higher log-sr(F) implies a larger OOD accuracy gap within the specified regime.The theorem concerns source features from two checkpoints in the same architecture–strategy cell.Source: §2 · p. 3, §3 · p. 3

4.1 WITHIN EVERY CELL, A SPECTRAL STATISTIC ORDERS MODELS BY OOD ROBUSTNESS

  • Every tested architecture–strategy cell reached |ρLOO| ≥0.700, with a mean of 0.824 across the 15 cells.Nine cells had two-sided p ≤0.05, four more had one-sided significance in the prespecified direction, and two were marginal.Source: §4.1 · p. 5, Table 1 · p. 6
  • Mean |ρLOO| rose from 0.245 pooled to 0.824 cell-conditioned, and cell-specific metrics correctly ranked 92% of 454 within-cell model pairs.The cell-specific metric achieved pairwise ranking accuracy 0.924, with a median of 0.958.Source: §4.1 · p. 5, Table 3 · p. 6

4.3 TRAINING DATA DIVERSITY CAN CHANGE THE PROXY POLARITY

  • Training-data diversity can reverse weight-metric polarity without changing the feature-level relation; BiDi-MLM/Protein maps PR-mean inversely (ρLOO = −0.891).Across broad- and narrow-domain pretraining, the cell-specific metric’s |ρLOO| ranged from 0.714 to 1.000 across all 15 cells.Source: §2 · p. 5, §4.3 · p. 7

5 ACTING ON SPECTRAL CONCENTRATION NARROWS THE OOD GAP

  • 24% less OOD drop at r=64 came with 87.5% in-distribution accuracy retained across 22 NLP models and 1,232 SVD bottleneck interventions.The rank-r intervention acts directly on source features and requires a source-domain forward pass; OOD drop rose monotonically with r (ρ = +0.395, p < 0.0001).Source: §5 · p. 7, Figure 3 · p. 9
  • Metric-guided repair of the K=4 most-flagged layers beat random targeting in 9/10 NLP models (p = 0.011); EEG CBraMod showed −0.038 targeted versus +0.026 anti-targeted.Repair effects were architecture-conditional: DICE achieved ∆=−0.224 in BiDi-MLM but zero in six AR-CLM models.Source: §5 · p. 7, Figure 2 · p. 8

6 CROSS-DOMAIN VALIDATION

  • All 3 fully evaluated prospective cases confirmed the logged metric-direction predictions, while AMPLIFY’s rank-percentile verdict failed and BioNeMo ESM-2 remained pending.Cell, metric, and sign were locked before OOD outcomes were observed for EEG, AMPLIFY protein, and Evo2 genomic evaluations.Source: §6 · p. 8, Table 5 · p. 10

B MODEL LISTS BY CELL

  • In the HyenaDNA genomic SSM cell, two models at chance cross-mark accuracy were retained; CV(log -Frob) had ρLOO = +0.700 (p = 0.188, n.s.).The direction was consistent, but the result did not meet the p ≤0.05 threshold; Caduceus was excluded for insufficient replication in its own cell.Source: B MODEL LISTS BY CELL · p. 15

C DETAILED THEORY ASSUMPTIONS AND PROOFS

  • The appendix distinguishes proved claims from explicit within-cell or training-family assumptions, proving Theorem 1 before connecting residual weights to feature geometry and deriving architecture-conditional statistics.Its stated sequence is Theorem 1, Lemma 2, then Propositions 3–5.Source: C DETAILED THEORY ASSUMPTIONS AND PROOFS · p. 16

C.1 THEOREM 1: STRUCTURED SPECTRAL SPREADING

The theorem links structured spectral spreading to lower stable rank and weaker normalized margins, yielding an OOD-gap ordering under explicit assumptions. Its proxy arguments and polarity conditions remain partly empirical rather than universal.

PROOF: STABLE-RANK BRANCH

  • The stable-rank branch establishes that the perturbed checkpoint has higher log stable rank than the original.The derivation uses core-energy control for the perturbed spectrum.Source: PROOF: STABLE-RANK BRANCH · p. 17

PROOF: NORMALIZED-MARGIN BRANCH

  • The normalized-margin branch uses the spectrum-dependent surrogate G(F) to order checkpoints, then bounds their target-error difference under the local-shift assumptions.G(F) is not presented as a universal feature statistic.Source: §C.1 · p. 16, PROOF: NORMALIZED-MARGIN BRANCH · p. 18

C.1 THEOREM 1: STRUCTURED SPECTRAL SPREADING

  • Under the stated structured-spreading regime, increasing spectral energy across sufficiently many weak directions raises stable rank and increases OOD fragility.The result does not claim that arbitrary stable-rank changes alone cause OOD degradation.Source: §C.1 · p. 16, PROOF: NORMALIZED-MARGIN BRANCH · p. 18

C.2 PROOF OF LEMMA 2

  • The residual-block singular-value argument is exact only for a simplified linear residual recurrence, not for a general Transformer’s raw-weight product.For nonlinear residual blocks, the result uses a local analogue.Source: §C.2 · pp. 18–19

C.3 PROOF OF PROPOSITION 3: AR-CLM / HT-SR CHAIN

  • The AR-CLM proxy chain relies on training-family regularity and effective-block order preservation, which the cited HT-SR literature motivates but does not prove.The training-dynamics relation is an assumption rather than a matrix identity.Source: §C.3 · p. 19

C.4 PROOF OF PROPOSITION 4: BIDI-MLM / PR POLARITY

  • For BiDi-MLM polarity, stable aggregate alignment and concentration support a positive sign; sufficiently strong changes in either can instead produce a negative derivative.The polarity argument depends on assumptions about layerwise participation ratios and cross-layer alignment.Source: §C.4 · p. 20

C.6 WHAT REMAINS EMPIRICAL

  • The theory leaves normalized-margin concentration, training relations, proxy order preservation, covariance stability, and some polarity assignments as empirical conditions.These are calibration conditions, not results supplied by citations.Source: §C.6 · p. 21

D MECHANISTIC HYPOTHESES: WHY EACH ARCHITECTURE SELECTS ITS METRIC

  • The final cell-specific proxy is calibrated on reference models, while leave-one-out correlations assess ranking conditional on that calibration.The reported out-of-sample test fixes the metric and sign before observing the new family’s OOD outcome.Source: D MECHANISTIC HYPOTHESES: WHY EACH… · p. 21

D.1 CV-TYPE METRICS FOR TRANSFORMER-BASED MODELS

The selected spectral statistic and its direction vary across architecture–strategy cells, with multiple metrics showing strong within-cell OOD-ranking correlations. The section also describes data-scarcity and architectural regimes, a failure-mode taxonomy, and the evaluation protocol.

D.1 CV-TYPE METRICS FOR TRANSFORMER-BASED MODELS

  • For Transformer cells, log ∥W∥F ranks broad-domain NLP CLMs (|ρLOO|=0.714, n=16), while CV(log -Frob) ranks narrow-domain protein CLMs (|ρLOO|=0.817, n=10).BiDi-MLM NLP instead uses PR-mean (ρLOO=+0.808, n=10), whereas protein BiDi-MLM reverses direction (ρLOO=−0.891, n=11); supervised ViT uses MP-energy (ρLOO=+0.714, n=7).Source: §D.1 · pp. 21–22

D.2 PR-TYPE METRICS FOR NON-TRANSFORMER ARCHITECTURES

  • For non-Transformer vision, MP-energy correlates positively with OOD fragility in DINOv2 (ρLOO=+0.800, n=5) and Swin (ρLOO=+0.893, n=8).The Swin value is reported as the highest among vision cells; lower MP-energy indicates a more structured signal subspace.Source: §D.2 · p. 22

D.3 MP-ENERGY FOR CLIP/SIGLIP

  • For CLIP/SigLIP, lower MP-energy indicates sharper semantic compression in the signal subspace and is linked to greater robustness against domain shift in the noise subspace.MP-energy is the fraction of spectral mass inside the Marchenko–Pastur bulk.Source: §D.3 · p. 23

D.4 WHY INVERTED DIRECTION IN DATA-SCARCE TRAINING

  • In data-scarce settings, higher PR-mean predicts robustness for Chronos (ρLOO=+0.714), while higher CV(log -Frob) does so for protein CLMs (ρLOO=+0.817).The section also reports positive correlations for genomic BiDi-MLM log ∥W∥F (ρLOO=+0.829) and EEG log-NFR (ρLOO=+0.822); LUNA-base is an exception to the EEG direction, with scale escape offered as a conjecture.Source: §D.4 · p. 23

D.5 SSM SPECTRAL SIGNATURE: CONCENTRATED SPECTRA AND SCALE-DRIVEN FRAGILITY

  • SSMs have lower per-layer spectral entropy than genomic Transformers (1.6–4.1 versus 5.6–6.7; p=0.004), while Mamba scale-series entropy and OOD drop rise together (ρLOO=+0.800, n=5).HyenaDNA’s five-model cell has CV(log -Frob) ρLOO=+0.700 (p=0.188), and its small and large checkpoints show chance-level cross-mark accuracy.Source: §D.5 · p. 24

D.6 FAILURE MODE TAXONOMY

  • The six-archetype failure-mode taxonomy organizes each cell by spectral signature, predictive metric, and recommended repair strategy; repairs target the failure mode rather than architecture alone.It provides a shared vocabulary for comparing cells across modalities.Source: §D.6 · p. 24

E OOD EVALUATION PROTOCOL AND PER-CELL TASK DETAILS

  • Each cell uses one canonical modality-matched OOD shift, with frozen encoders and a source-trained linear probe evaluated on the target domain.The reported OOD drop is ∆acc = accsrc − acctgt; per-cell variation is non-trivial.Source: E OOD EVALUATION PROTOCOL AND PER-CELL… · p. 24

G BOOTSTRAP CONFIDENCE INTERVALS FOR ALL CELLS

The section reports how cell-level evidence is assessed, including uncertainty and the distinct evidentiary role of small cells. Sensitivity analyses show the main result is not driven by the two marginal cells.

G BOOTSTRAP CONFIDENCE INTERVALS FOR ALL CELLS

  • ρLOO is the median of the n leave-one-out Spearman correlations, each computed on the remaining n−1 models.The section reports Fisher-z 95% confidence intervals and uses exact permutation p-values for n=5; n=4 uses a pre-specified one-sided exact test.Source: G BOOTSTRAP CONFIDENCE INTERVALS FOR… · p. 24, Table 7 · p. 26
  • 9 of 15 cells have two-sided confidence intervals excluding zero, 4 more reach one-sided significance, and 2 are marginal.The 15 groups were pre-specified, but cell metric and sign selection use the same outcomes and are not corrected for in these p-values.Source: G BOOTSTRAP CONFIDENCE INTERVALS FOR… · p. 27
  • Dropping the two marginal cells leaves mean |ρLOO| at 0.828, versus 0.824 across all 15 cells.The LOCO range [0.811, 0.832] brackets 0.828; the authors retain all 15 pre-specified cells because both marginal cells clear |ρLOO| ≥0.700.Source: G BOOTSTRAP CONFIDENCE INTERVALS FOR… · p. 27
  • The n=5 cells provide transfer and consistency evidence, while the main claim rests on the 10+ confirmatory cells.SSM Mamba-2 and RWKV-CLM reach two-sided significance at n=5, but the small sample alone is not treated as establishing generality.Source: G BOOTSTRAP CONFIDENCE INTERVALS FOR… · p. 27

H PARAMETER COUNT VS. WEIGHT GEOMETRY

The section compares cell-specific spectral predictors with parameter count and other weight-space or data-dependent alternatives. Cell-specific spectral metrics generally match or outperform size-based prediction, though the comparison has cell-level exceptions.

H PARAMETER COUNT VS. WEIGHT GEOMETRY

  • The comparison evaluates each cell’s selected spectral metric against log10(nparams) as a size-only predictor.The per-cell comparison reports absolute LOO correlations, while the signed log nparams correlation indicates whether its prediction direction agrees with the expected direction.Source: H PARAMETER COUNT VS. WEIGHT GEOMETRY · p. 28
  • Across 15 primary cells, the spectral champion matches or outperforms log nparams in 13, counting cells where log n has lower |ρLOO| or predicts in the wrong pre-specified direction; mean |ρLOO| is 0.824 versus 0.686 (Δ=+0.138).Source: H PARAMETER COUNT VS. WEIGHT GEOMETRY · p. 28
  • The champion also exceeds the best fixed weight-space alternative, PR applied uniformly, whose mean |ρLOO| is 0.724.The cell-specific champion mean is 0.824; the WeightWatcher flagship αmean has mean |ρLOO| 0.448 and is nonsignificant in 10 of 15 cells.Source: H PARAMETER COUNT VS. WEIGHT GEOMETRY · p. 28
  • For cells where runtime baselines were available, the data-free selected metric matches or outperforms them in 12 of 16 cell–baseline pairs.Doctor score and MC entropy were tested in six cells, Mahalanobis distance in four; these baselines require source-domain test data and inference.Source: H PARAMETER COUNT VS. WEIGHT GEOMETRY · pp. 28–29

I LEAVE-ONE-CELL-OUT ROBUSTNESS ANALYSIS

Leave-one-cell-out analysis tests whether any single cell drives the aggregate result. The mean correlation and the finding that cells prefer distinct metrics remain stable across all 15 rounds.

I LEAVE-ONE-CELL-OUT ROBUSTNESS ANALYSIS

  • Across 15 LOCO rounds, mean |ρLOO| over the remaining cells ranges from 0.811 to 0.832, against a baseline of 0.824.The maximum variation is ±0.013, and all rounds retain distinct selected metrics across cells.Source: I LEAVE-ONE-CELL-OUT ROBUSTNESS ANALYSIS · p. 29
  • The LOCO range reflects the variation in the 14-cell mean after dropping one cell, while 0.824 remains the fixed-champion baseline.The reported LOCO stability is ±0.006 s.d.; the main-table value uses the pre-specified metric without fold-level re-tuning.Source: I LEAVE-ONE-CELL-OUT ROBUSTNESS ANALYSIS · p. 29
  • Near-tied metrics can change labels when the held-out cell changes, without materially changing the achieved |ρLOO|.The authors interpret this as within-cell metric redundancy rather than instability of the main finding.Source: I LEAVE-ONE-CELL-OUT ROBUSTNESS ANALYSIS · p. 29

J CELL BOUNDARY VALIDATION VIA EXPANSION EXPERIMENTS

Expansion experiments show that pooling related but distinct pretraining objectives can weaken cell-level prediction. A four-level comparison finds that conditioning on both architecture and strategy gives the strongest signal among the tested groupings.

J CELL BOUNDARY VALIDATION VIA EXPANSION EXPERIMENTS

  • Adding MSN models to ViT×DINO lowers |ρLOO| from 0.800 at n=5 to 0.732 at n=7, and shifts the selected metric.The passage attributes the mismatch to MSN’s distinct inductive bias and reports that its spectral structure does not align with the DINO scaling curve.Source: J CELL BOUNDARY VALIDATION VIA… · p. 30
  • Pooling audio SSL models across objectives lowers |ρLOO| to 0.494 for 13 models, with αmean selected as the best metric.The narrower wav2vec2×Contrastive group reaches |ρLOO|=0.880 on speaker-OOD evaluation; adding WavLM-base first reduces it to 0.750.Source: J CELL BOUNDARY VALIDATION VIA… · p. 30
  • Both expansion experiments degraded prediction, supporting the architecture × strategy boundary as the coarsest tested grouping that retains signal.The authors attribute the loss to distinct objectives leaving spectral fingerprints that do not combine into a consistent OOD-predictive signal.Source: J CELL BOUNDARY VALIDATION VIA… · p. 30

K FOUR-LEVEL CONDITIONING COMPARISON

  • Mean |ρLOO| rises monotonically across conditioning levels: 0.245, 0.478, 0.745, and 0.824, on the same 116 models.The increments are +0.233 for strategy, +0.267 for architecture, and +0.079 for the architecture × strategy interaction, totaling +0.579.Source: K FOUR-LEVEL CONDITIONING COMPARISON · p. 30, Table 10 · p. 31
  • Neither axis alone is sufficient: the same strategy can require different metrics across domains, and one architecture can require different metrics across strategies.The architecture × strategy combination is the minimal tested unit where within-cell metric–OOD relationships are consistent.Source: K FOUR-LEVEL CONDITIONING COMPARISON · pp. 30–31
  • Per-domain selected-metric correlations are +0.714 for NLP, +0.714 for time series, and +0.700 for genomics.The genomic correlation is marked nonsignificant; the selected metrics are log ∥W∥F for NLP and genomics, and PR-mean for time series.Source: K FOUR-LEVEL CONDITIONING COMPARISON · p. 31

M WITHIN-VISION DATA-DEPENDENT PREDICTOR COMPARISON

Cell-specific spectral prediction outperforms uniformly applied weight metrics and approaches runtime baselines while requiring only pretrained weights. The comparison uses the same per-cell leave-one-out framework across checkpoints.

M WITHIN-VISION DATA-DEPENDENT PREDICTOR COMPARISON

  • The comparison aggregates per-cell leave-one-out Spearman correlations over 116 models and 15 primary cells, evaluating predictors on the same checkpoints and holdout framework.Table 11 contrasts uniformly applied weight-space methods with runtime baselines evaluated only where source-domain data are available.Source: M WITHIN-VISION DATA-DEPENDENT… · p. 31, Table 11 · p. 32
  • The cell-specific spectral approach matches or exceeds runtime baselines in 12 of 16 cell–baseline pairs, while the best globally applied weight metric, PR, reaches mean |ρLOO| = 0.724.The cell-specific mean is 0.824; log nparams reaches 0.686. Runtime baselines require source-domain data and are available only in some cells.Source: M WITHIN-VISION DATA-DEPENDENT… · p. 31
  • Doctor score averages |ρLOO| = 0.694 across six cells and reaches 1.000 in ViT×Supervised, but requires inference on held-out source data.Mahalanobis distance averages |ρLOO| = 0.562 across four cells with source labels; it also requires source data.Source: M WITHIN-VISION DATA-DEPENDENT… · p. 31

N DATA SCARCITY VS. ARCHITECTURAL INVERSION

The selected metric changes with architecture, strategy, and domain, while its direction is cell-specific rather than universal. Across the reported cells, calibration yields strong within-cell OOD correlations despite these changes.

N DATA SCARCITY VS. ARCHITECTURAL INVERSION

  • Across the five cells in Table 12, higher selected-metric values predict higher OOD drop, but BiDi-MLM/Protein, CNN, SSM-v2, and RWKV cells have negative directions.The matrix therefore does not have one universal metric direction.Source: Table 12 · p. 32, N DATA SCARCITY VS. ARCHITECTURAL… · pp. 32–33
  • With architecture and strategy fixed, changing training domain can alter metric selection; the five listed cells all show positive ρLOO, meaning higher metric values correspond to higher OOD drop.Source: Table 12 · p. 32
  • For AR-CLM, selected metrics yield positive ρLOO values of +0.714, +0.714, and +0.817; BiDi-MLM NLP and Genomic are also positive, while Protein is −0.891.The Protein result is linked to the anti-collapse pretraining objective, with higher PR-mean tracking greater robustness.Source: N DATA SCARCITY VS. ARCHITECTURAL… · p. 32
  • SSM/RWKV cells use distinct metrics and directions: Mamba uses Hent at +0.800, Mamba-2 uses ¯α at −1.000, and RWKV uses CV(log -Frob) at −1.000.These contrasts show that metric selection differs even within the SSM/linear-RNN group.Source: N DATA SCARCITY VS. ARCHITECTURAL… · p. 33, Figure 9 · p. 33

O SSM ARCHITECTURE ANALYSIS

The SSM analysis applies the feature-geometry result to standard projection layers and finds distinct predictive metrics across SSM and linear-RNN variants. Some cells share a composite with bidirectional masked LMs, while others show variant-specific directions.

O SSM ARCHITECTURE ANALYSIS

  • Theorem 1 applies directly to standard linear projection layers surrounding SSM recurrent cores; the structured state-transition matrix ¯A is excluded from metric computation.The listed projections include input, output, B, C, and dt projections.Source: O SSM ARCHITECTURE ANALYSIS · p. 33
  • HyenaDNA and Caduceus share the Sssm composite with bidirectional masked LMs, with ρLOO = −0.900 (p < 0.05).The ESM-family comparison is ρLOO = −0.952 (p = 0.0003, n = 8).Source: O SSM ARCHITECTURE ANALYSIS · p. 33
  • The SSM/CLM/Genomic cell has CV(log -Frob) as its best single metric, but ρLOO = +0.700 does not meet the primary-cell inclusion threshold (p = 0.188).Two HyenaDNA models have degenerate target accuracy of 0.500 on the cross-mark shift; the cell is retained but not counted among the 15 primary cells.Source: O SSM ARCHITECTURE ANALYSIS · p. 33
  • For SSM/CLM/NLP, Mamba selects Hent at ρLOO = +0.800, while Mamba-2 selects ¯α at −1.000; the Mamba result is marginal, whereas Mamba-2 has exact p = 0.008.The proposed interpretations associate higher mean spectral entropy with greater fragility in Mamba and higher mean PL exponent with greater robustness in Mamba-2.Source: O SSM ARCHITECTURE ANALYSIS · p. 33
  • RWKV selects CV(log -Frob) with ρLOO = −1.000 (n = 5, exact permutation p = 0.008), unlike Mamba’s selected Hent metric.The passage relates this difference to distinct spectral footprints for time-mixing and diagonal state-space recurrence.Source: O SSM ARCHITECTURE ANALYSIS · pp. 33–34

P FULL REPAIR ABLATIONS

The repair and validation sections compare architecture-conditional interventions, prospective prediction, and a practical diagnostic workflow. Their results include a strong targeted-versus-random trade-off, while the paper identifies calibration and mediation limits.

P FULL REPAIR ABLATIONS

  • DICE yields ∆=−0.224 in BiDi/MLM but zero across six AR-CLM models, while SpNorm gives the largest reported gains in large AR-CLM models.SpNorm yields ∆=−0.348 for GPT-2-large and −0.301 for GPT2-medium; DICE and SpNorm also show zero results in the opposite architecture groups in the reported tests.Source: P FULL REPAIR ABLATIONS · p. 34
  • LoRA rank adaptation and γ-surgery show no significant association with ∆acc, consistent with feature rank rather than weight rank carrying the effect.The reported correlations are +0.008 (n = 464) and +0.050 (135 interventions), respectively; both leave log-srfeat unchanged.Source: P FULL REPAIR ABLATIONS · p. 34
  • Copying four vulnerable layers from robust checkpoints reduces OOD drop by ∆=−0.190 in the large pair and −0.216 in the base pair; the symmetric swaps also reduce drop.The symmetric-direction changes are −0.129 and −0.231, respectively.Source: P FULL REPAIR ABLATIONS · p. 34

R ID–OOD TRADE-OFF: SPECTRAL TARGETING VS. RANDOM TARGETING

S AMPLIFY PROSPECTIVE VALIDATION: FULL CASE STUDY

  • The prospective PR-mean prediction ranks AMPLIFY-120M as MID and AMPLIFY-350M as FRAGILE, but their actual OOD drops are 0.267 and 0.132, respectively.The metric orders the two models correctly, but the ESM-2 size–fragility calibration direction does not transfer to AMPLIFY’s architecture subtype.Source: S AMPLIFY PROSPECTIVE VALIDATION: FULL… · p. 36
  • Within the BiDi-MLM × Protein cell, ESM-2’s size–OOD-drop slope is +0.07 per log10-parameter unit, while AMPLIFY’s is −0.57.The sign change is detectable from within-cell data; the ESM-2 calibration predicts AMPLIFY-350M’s drop at about 0.53, versus its actual 0.132.Source: S AMPLIFY PROSPECTIVE VALIDATION: FULL… · p. 36, Figure 12 · p. 37

T DIAGNOSTIC PROTOCOL

  • The diagnostic identifies architecture, strategy, and domain, retrieves the selected weight metric and direction, then computes it on pretrained weights; unknown directions require at least four same-cell models for calibration.The protocol specifies branching on architecture, pretraining strategy, and domain before computing the metric.Source: T DIAGNOSTIC PROTOCOL · pp. 36–37

U LIMITATIONS

  • The framework ranks models within a cell rather than predicting absolute target accuracy, and subtypes with different spectral scaling need matching calibration sets.The interventions have not yet separated OOD-gap narrowing from lower ID accuracy in the two cases where both were measured.Source: U LIMITATIONS · p. 37
Loading 2610.09709v1…