PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

Chenyu Zhu, Ruoyu Zhao, Zhichao Lu

arXiv:2609.23087v1cs.LGcs.AI

TL;DR

Architecture budgets do not distinguish structurally different designs, although those choices matter for trained or deployed behavior. The paper proposes a specification-only spectral score and an exact dynamic-programming optimizer, reporting stronger ranking and architecture-selection results across its tested settings.

  • Problem

    #Params and #FLOPs capture size and compute but give identical scores to structurally different architectures under the same budget, while training-free proxies depend on instantiated networks, inputs, and black-box search.

    Source: §1 · pp. 1–2

  • Method

    NSC computes a closed-form score from matrix singular spectra and architectural specifications under standard initialization, while NSC-DP exactly maximizes its additive score under resource constraints.

    Source: §1 · p. 2, §3.4 · p. 8

  • Results

    Across seven Transformer and CNN families NSC ranks architectures more accurately than #Params, #FLOPs, and training-free proxies; NSC-DP finds a better-than-baseline Transformer-XL architecture in 2 seconds and the best 5.7B LLaMA pruning model across eight tasks without calibration data.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    NSC provides a specification-based way to distinguish structural allocations, and its additive form supports globally optimal search with respect to the NSC objective.

    Source: §1 · p. 2

    The channel model defines ψ but is not claimed to represent a network’s forward pass, and architecture comparisons require a fixed initialization protocol across candidates.

    Source: §3.1 · p. 4, §3.2 · p. 6

Abstract

from arXiv · show

Modern Transformer design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute but not architectural structure: two architectures with identical parameter budgets but different depth-width, head, or FFN allocations receive identical scores yet behave differently. We propose Neural Spectral Capacity (NSC), a closed-form scalar grounded in the singular-value spectrum of each weight matrix. Under standard random initialization, the Marchenko-Pastur law renders NSC computable from the architectural specification alone, with no model instantiation, data, or gradients. Its layer-wise additive structure admits NSC-DP, an exact dynamic-programming solver returning the architecture globally maximizing NSC under resource constraints in seconds on a CPU -- a guarantee that black-box search over existing training-free proxies cannot provide. Empirically, NSC outperforms #Params, #FLOPs, and representative training-free proxies in ranking across seven Transformer and CNN families (on FlexiBERT, $τ= 0.505$ on pairs differing in #Params by less than 10%, where #Params collapses to 0.082); NSC-DP discovers a Transformer-XL architecture on WikiText-103 that beats the human-designed baseline in 2 seconds; and prunes LLaMA-7B to the best 5.7B model across eight commonsense reasoning tasks without any calibration data, about 5900x faster than the strongest training-free proxy baseline.

1. Introduction

The paper frames architecture design and compression as allocating capacity under a budget, a decision that #Params and #FLOPs cannot resolve when structures differ. It introduces NSC and NSC-DP, then evaluates them in ranking, architecture search, and pruning.

Motivation

  • At fixed parameter or compute budgets, #Params and #FLOPs assign the same scores to structurally different architectures, while design and compression require choosing how capacity is distributed.Existing training-free proxies depend on an instantiated network and chosen input, and their black-box scores require heuristic search that returns only the best architecture visited.Source: §1 · pp. 1–2

Neural Spectral Capacity

  • NSC is a closed-form scalar of the architectural specification, grounded in weight-matrix singular spectra and computable under standard random initialization without model instantiation, data, or gradients.The Marchenko–Pastur law reduces expected per-matrix capacity to a function of matrix dimensions and initialization variance.Source: §1 · p. 2

NSC-DP

  • NSC’s layer-wise additivity enables NSC-DP to return the architecture globally maximizing NSC under resource constraints, a guarantee black-box search over non-decomposable proxies cannot provide.The guarantee is optimality with respect to the NSC proxy objective.Source: §1 · p. 2

Empirical validation

  • Across seven Transformer and CNN families, NSC ranks architectures more accurately than #Params, #FLOPs, and training-free proxies; on FlexiBERT, τ=0.505 for pairs within 10% of #Params, versus 0.082.On Transformer-XL, NSC-DP beats the human-designed baseline in 2 seconds, over 400× faster than training-free proxies with heuristic search. For LLaMA-7B pruned to 5.7 B, it produces the best model across eight commonsense tasks without calibration data, ∼5900× faster than the strongest baseline.Source: §1 · pp. 2–3

2. Related work

Related work studies network spectra to understand or repair existing models, scores architectures at initialization for search, or prunes pretrained models under deployment budgets. The paper distinguishes NSC by computing a score from specifications and optimizing it exactly.

Random matrix theory in deep learning

  • Prior random-matrix work uses trained-weight spectra to diagnose training quality, derives activation spectra using nonlinearities and input distributions, or prunes singular values within the Marchenko–Pastur bulk.NSC instead treats the Marchenko–Pastur law as a deterministic equivalent for computing capacity from a specification, not as a null model, noise criterion, or baseline to extend.Source: §2 · p. 3

Training-free proxies and architecture search

  • Training-free proxies score initialized networks using activations, gradients, or attention on sampled inputs, then typically rely on black-box search that returns the best architecture visited.NSC differs by using a closed-form, data-agnostic score and layer-wise additivity for exact dynamic-programming optimization.Source: §2 · p. 3

Structured pruning of large language models

  • Structured LLM pruning methods use calibration data, pretrained-weight importance, continued pretraining, or supernet selection, whereas NSC scores a pruned subnetwork from its specification alone.The paper positions its LoNAS-supernet application as requiring neither pretrained weights nor calibration data.Source: §2 · pp. 3–4

3. Neural Spectral Capacity (NSC)

NSC converts weight-matrix spectral capacity into a specification-based architecture score, then uses its layer-wise additivity for constrained optimization. The section develops the score, its closed form, its aggregation, and the NSC-DP solver.

3. Neural Spectral Capacity (NSC)

  • NSC builds on the connection between a weight matrix’s singular-value spectrum and the information capacity of a linear Gaussian channel, then aggregates matrix capacities into a layer-wise additive architecture score.This additive form enables NSC-DP for resource-constrained architecture search.Source: §3 · p. 4

3.1. Spectral capacity of a weight matrix

  • For a linear Gaussian channel, the matrix score sums contributions from its singular values; matrices with identical dimensions can still have very different spectral capacities.A LLaMA-7B FFN matrix has ψ≈3174 at full rank versus ψ≈2.5 after rank-1 truncation, with parameter count unchanged.Source: §3.1 · p. 4
  • Across 10 GPT-2 architectures, initialization-time ψ rankings were preserved through training (ρ(ψ0, ψT)=1.00), and converged ψ ranked final PPL perfectly; the passage cautions that the sample is small.The figure tracks ψt during training and indicates final perplexity by color and right-axis label.Source: §3.1 · p. 5, Figure 1 · p. 5

3.2. Spectral capacity in closed form

  • Under the Marchenko–Pastur law, the limiting capacity per sub-channel yields a deterministic ψ function of matrix dimensions and initialization variance, independent of a particular weight realization.The definition uses the law’s limit functional at the matrix’s aspect ratio and signal-to-noise parameter.Source: §3.2 · p. 5
  • The closed-form estimate’s relative error falls below 0.4% at min(m, n)=128 and to ~10^-4 by min(m, n)=1024.On the LoNAS-LLaMA-7B supernet, ψMP matches ψW at τ=ρ=1.0000 across 129 Pareto-optimal subnets, with 0.039% mean SVD-oracle regret and 720× compute saving.Source: §3.2 · p. 6

3.3. From single matrices to layers and networks

  • NSC aggregates per-head attention projections, attention output projections, and FFN matrices within layers, then sums layer scores; this is a modeling choice, not a claim about end-to-end mutual information.The authors report that summation ranks best on average, while the bottleneck rule assigns FlexiBERT only 22 distinct scores and turns anti-correlated on AutoFormer-Tiny.Source: §3.3 · pp. 6–7

3.4. Architecture design via NSC-DP

  • Given additive layer scores and additive resource costs, NSC-DP enumerates network-level choices and exactly solves the resulting layer-level knapsack, returning the global optimum for the NSC objective.One run also provides the proxy-optimum at every reachable discretized budget, and searches spaces of up to 10^32 architectures in seconds on a single CPU core.Source: §3.4 · pp. 7–8
  • NSC-DP applies when both the NSC score and resource constraint decompose into independent per-layer sums; the paper cites AutoFormer and LoNAS-LLaMA as examples.For these spaces, layer-level choices include MLP ratio and head count, or FFN width and LoRA rank, respectively.Source: §3.4 · p. 8

4. Experiments

Across ranking, architecture search, and LLaMA-7B pruning, the experiments evaluate NSC and NSC-DP against architectural quantities and training-free proxies. NSC ranks architectures beyond parameter count, while NSC-DP finds strong resource-constrained designs and pruned models.

4.1. Ranking quality across architecture families via NSC

  • On FlexiBERT pairs differing in #Params by less than 10%, NSC reaches τ=0.505 while #Params falls to τ=0.082; NSC also outperforms #Params, #FLOPs, and proxies across families.NSC leads W-PCA by +0.060 τ on FlexiBERT and is competitive with leading proxies on GPT-2 at orders-of-magnitude lower wall-clock cost.Source: Figure 2 · p. 9, §4.1 · pp. 9–10

4.2. Architecture search via NSC-DP

  • On WikiText-103, NSC-DP finds a Transformer-XL with test PPL 23.087, the lowest among methods, in 2.0 CPU seconds.On AutoFormer-Tiny, it matches the day-scale oracle within 0.03 accuracy points and outperforms heuristic alternatives on Acc@1.Source: §4.2 · p. 10, Table 4 · p. 11

4.3. Structured pruning of LLaMA-7B via NSC-DP

  • For LLaMA-7B pruning, NSC-DP selects configurations from the pruned subnetwork specification alone, without pretrained weights or calibration data.The baselines use pretrained weights and a WikiText-103 calibration mini-batch.Source: §4.3 · pp. 10–11
  • At a 5.7 B budget, NSC-DP produces the highest-quality pruned LLaMA-7B, beating W-PCA by +1.64 pp on Avg8 and every baseline on every task.The comparison covers eight commonsense reasoning tasks; Table 5 also includes the un-pruned 6.7 B SuperNet reference.Source: Table 5 · p. 11, §4.3 · p. 12
  • NSC-DP completes LLaMA-7B pruning in 0.46 seconds, about 5,900× faster than W-PCA; the same pass yields seven operating points spanning 1.29–1.41 TFLOPs.Figure 3 plots Avg8 against TFLOPs at T=256 for those operating points, read from the same backward pass.Source: §4.3 · p. 12, Figure 3 · p. 12

5. Conclusion

The conclusion presents NSC as a specification-based measure of Transformer capacity and NSC-DP as an exact solver for resource-constrained architecture choice. It reports empirical results across ranking, search, and pruning, and notes extensions to attention-head grouping and MoE expert allocation.

5. Conclusion

  • NSC is a closed-form architectural scalar computed from a Transformer’s specification alone, while NSC-DP exactly maximizes it under resource constraints.Across seven Transformer and CNN families, NSC ranks architectures more accurately than #Params, #FLOPs, and representative training-free proxies.Source: §5 · p. 12
  • NSC-DP finds a Transformer-XL architecture that beats the human-designed baseline in 2 seconds on one CPU core.It also produces the best 5.7 B LLaMA-7B pruning result across eight commonsense reasoning tasks without calibration data, about 5900× faster than the strongest baseline.Source: §5 · p. 12
  • Per-head decomposition extends NSC to MHA, GQA, and MQA grouping, while its layer-wise additive sum scores per-layer expert allocation in MoEs.The paper frames NSC as a potential quantitative tool for resource-constrained Transformer architecture choice.Source: §5 · p. 12

A. RMT in deep learning: lifecycle comparison

The comparison organizes methods by network lifecycle, architectural inputs, use of the Marchenko–Pastur law, and outputs. Examples show that spectral capacity can distinguish matrices, operations, and compressed subnetworks even when parameter count cannot.

A. RMT in deep learning: lifecycle comparison

  • The comparison table groups methods by lifecycle stage, required inputs beyond architecture, use of the Marchenko–Pastur law, and output.Its rows are ordered by where each method applies in the network lifecycle.Source: Table 6 · p. 16, A. RMT in deep learning: lifecycle… · p. 16
  • ψMP ranks a 2000 × 2000 matrix (≈1,161) above a 4000 × 1000 matrix (≈909), despite identical mn.In low-rank truncation of a 4096 × 11008 LLaMA-7B matrix, rank 1000 retains only 42.5% of full capacity while parameter count stays approximately 45M across ranks.Source: B. 𝜓is not parameter count: extended… · p. 17, Figure 4 · p. 17
  • On FlexiBERT, NSC achieves Spearman ρ=0.884 versus ρ=0.676 for #Params, with operation-type structure identified as the source of the ranking gap.Different operation types contribute different sets of weight matrices, so equal dimensions and parameter counts need not yield equal spectral capacity.Source: B. 𝜓is not parameter count: extended… · pp. 17–18
  • At a fixed 5.7B compression budget, parameter count is constant across candidate LLaMA-7B subnets, while ψ ranks how much spectral capacity their retained FFN dimensions preserve.The section reports that NSC-guided selection achieves the highest average across seven commonsense reasoning benchmarks at that budget.Source: B. 𝜓is not parameter count: extended… · p. 18

C. Choice of aggregation: protocol and analysis

The analysis argues that summing layer capacities avoids the bottleneck rule’s collapse and the product rule’s depth confounding, while providing the additivity NSC-DP requires. The appendices relate spectral capacity to Gaussian-channel mutual information, characterize its gradient responsiveness, and derive its Marchenko–Pastur limit.

C. Choice of aggregation: protocol and analysis

  • On GPT-2, summation outranks the bottleneck (0.968 vs. 0.813); summation avoids bottleneck collapse and the product rule’s depth confounding.The authors identify summation as the only additive rule, which is the structural property required by NSC-DP.Source: C. Choice of aggregation: protocol and… · pp. 18–19
  • On AutoFormer-Tiny, 895 of 1,001 architectures share one bottleneck score, and only 3 of 8 layer configurations serve as the minimum.The minimum concentrates on weak configurations; adding layers can only lower it, despite deeper architectures tending to be more accurate in that search space.Source: C. Choice of aggregation: protocol and… · p. 18
  • The product rule scores 0.917 on FlexiBERT but drops to 0.712 on GPT-2, where freely varying depth overwhelms within-layer differences.With layer capacities above one, the product grows geometrically with depth.Source: C. Choice of aggregation: protocol and… · pp. 18–19

D.1. Proof of the mutual information identity

  • For the stated Gaussian linear channel, mutual information equals half the spectral score ψ(W), connecting that score to the singular values of W.The derivation uses determinant identities and the fact that WᵀW’s eigenvalues are squared singular values.Source: §D.1 · p. 19

D.2. Gradient Responsiveness

  • The gradient of ψ with respect to W weights singular values by g(σ) = 2σ/(1 + σ²), maximized uniquely at σ = 1.This is not a task-loss gradient; the link between ψ-dynamics and task-loss dynamics is empirical, not derived from this analysis.Source: §D.2 · p. 19

D.3. Proof of Theorem 1 (Marchenko-Pastur Spectral Capacity)

  • Under the stated large-matrix assumptions, the empirical spectrum converges to the Marchenko–Pastur law, yielding the claimed limit for ψ_MP.The paper defines ψ_MP as the architecture proxy, not as an approximation of a finite-sample quantity.Source: §D.3 · p. 20
  • At fixed matrix size product mn, ψ_MP is maximized at aspect ratio γ = 1.The paper describes this as the spectral-theoretic counterpart of maximizing mutual information across parallel Gaussian channels at fixed total power through uniform power allocation.Source: §D.3 · p. 20

D.4. Strict Concavity of 𝜓MP and Optimality of Uniform FFN Allocation

The appendix proves strict concavity of ψMP and derives uniform FFN allocation under a fixed budget. Additive capacity and integer resource costs also let one DP pass encode choices across budgets.

D.4. Strict Concavity of 𝜓MP and Optimality of Uniform FFN Allocation

  • ψMP is strictly concave in either matrix dimension for fixed other dimension and s > 0, without requiring any relationship between the dimensions.In Transformer-XL, n corresponds to dff and m to dmodel, with the result holding across the full positive range.Source: §D.4 · p. 21
  • All seven marginal-gain curves decrease, and all 189 discrete second differences are negative in the Transformer-XL search space, matching Proposition 2.The accompanying table reports that marginal gain attenuates by roughly 2× across the FFN search range for each embedding dimension.Source: Figure 5 · p. 23, §D.4 · p. 23, Table 7 · p. 24
  • Under a fixed total FFN parameter budget, the unique continuous NSC-maximizing allocation is uniform across layers; the discrete-grid optimum is its rounding.Strict concavity and Jensen’s inequality imply that replacing a nonuniform allocation with its average strictly increases the objective.Source: §D.4 · p. 24

E.1. Motivation

  • Because resource costs are integer sums and ψMP is additive by layer, the final DP table encodes the Pareto front across the discretized budget grid.Changing the deployment budget means reading a different table cell, not running another search.Source: §E.1 · p. 24

E.2. Empirical demonstration: seven subnets from a single DP run

  • Seven LoNAS operating points are read from the same 0.46 s DP backward pass, with no per-budget search or proxy recomputation.The main-text 5.7B result is another point on that Pareto front.Source: §E.2 · p. 24, Table 8 · p. 25
  • Figure 3 visualizes an accuracy–TFLOPs Pareto front whose monotone shape follows from ψMP’s per-layer additivity.The authors state that adjacent budgets produce adjacent subnets, removing run-to-run variance from stochastic searches.Source: §E.2 · p. 25

E.3. Discussion

The spec-faithful FlexiBERT protocol restores declared architecture dimensions and improves NSC’s ranking association with GLUE scores. NSC-DP also yields a smooth accuracy–compute Pareto front from one pass, with accuracy decreasing as compute falls.

E.3. Discussion

  • The seven NSC-DP operating points are read-outs of one 0.46 s backward pass; each additional budget costs O(B) table look-ups rather than a new search.The paper describes NSC-DP as the only evaluated method to return an entire Pareto front in one sub-second invocation without labels or GPU forward passes.Source: §E.3 · p. 25
  • Avg8 falls from 65.20% at 1.41 TFLOPs to 60.73% at 1.29 TFLOPs, a 4.47 pp range across 0.12 TFLOPs of compute reduction.The authors attribute the front’s smoothness to the closed-form additive structure, which makes budget changes produce small, predictable subnet changes.Source: §E.3 · p. 25

F.1. FlexiBERT: original (Serianni) vs. spec-faithful setting

  • The original FlexiBERT evaluation fixed hidden size at 256 and intermediate size at 1024, collapsing two of the four declared search dimensions.Consequently, proxies were evaluated on a scrambled specification rather than the architecture whose GLUE score was being predicted.Source: §F.1 · p. 26
  • The spec-faithful setting scores all 500 architectures using their declared hidden and per-layer FFN dimensions, leaving the architecture set, GLUE ground truth, and proxy implementations unchanged.Results under the original protocol are retained for direct comparison with prior tables.Source: §F.1 · p. 26
  • NSC’s FlexiBERT Kendall τ rises from 0.544 to 0.695 under the spec-faithful protocol, while #Params rises from 0.454 to 0.485.The original protocol hard-coded hidden size and FFN dimensions, so many architectures were scored on a specification different from the one trained.Source: §F.1 · p. 26

F.2. Controlled-correlation analysis across architecture families

NSC retains ranking signal after controlling for parameter count and compute across Transformer and CNN families. NSC-DP also searches AutoFormer and NATS-Bench-SSS spaces quickly, with strong accuracy results in both.

F.2. Controlled-correlation analysis across architecture families

  • The controlled-correlation analyses use windowed Kendall τ on pairs within 10% of a control value and a uniform Conover–Iman rank-OLS estimator for partial τ.The GPT-2 search space makes #Params and #FLOPs rank-equivalent, while ZeroLM is not applicable to the CNN benchmarks.Source: §F.2 · pp. 26–27
  • On FlexiBERT, NSC retains τ=0.505 under #Params-windowed control while #Params falls to τ=0.082; under joint control, NSC reaches τ̃=0.477.Under joint control, this exceeds W-PCA (0.444); GPT-2’s joint-control result is lower, with NSC at τ̃=0.329 behind W-PCA (0.382) and ZeroLM (0.377).Source: §F.2 · p. 27, Table 10 · p. 28, Table 11 · p. 28
  • NSC ranks first in every controlled column on AutoFormer-Tiny and AutoFormer-Small, retaining τ=0.503 under #Params-windowed control and τ̃=0.439 under joint (#Params, #FLOPs) control on Tiny.On Small, its τ̃=0.253 under joint partial control, versus 0.159 for ZeroLM and ≤0.086 for #Params, #FLOPs, and W-PCA.Source: §F.2 · pp. 28–29
  • On MobileNetV3, NSC ranks first in five of six columns, with τ=0.535 under #Params-windowed control and τ̃=0.278 under joint control.On AutoFormer-Small, NSC also ranks first across all controlled columns.Source: §F.2 · p. 29, Table 13 · p. 29, Table 15 · p. 30

F.4. NATS-Bench-SSS end-to-end search

  • NSC applies to the CNN size space because it scores weight-matrix spectral capacity independently of how each matrix is used.The reported NATS-Bench-SSS space varies the channel counts of five convolutional layers across 32,768 architectures.Source: §F.4 · p. 30

F.3. AutoFormer-Small and AutoFormer-Base end-to-end search

  • Under matched parameter budgets, NSC-DP takes 0.345 s on AutoFormer-Small and 0.600 s on Base; it ranks second on Small Acc@1 (81.392) and third on Base (82.086).On Acc@5, it ranks second on Base (95.688) and third on Small (95.644).Source: §F.3 · p. 30, Table 16 · p. 31

F.4. NATS-Bench-SSS end-to-end search

  • 93.44% on CIFAR-10 puts NSC-DP 0.04 points ahead of TAS; it is second on CIFAR-100 (70.68%) and ImageNet16-120 (46.87%).It exhaustively scores all 32,768 architectures in 0.54 s, 4,000–14,000× faster than TAS; reported accuracies use the best of its top 10 candidates.Source: §F.4 · p. 31
  • On CIFAR-10/100/ImageNet16-120, NSC-DP reaches within 0.21/0.66/0.53 points of the oracle, respectively, using the best of its top 10 proxy-ranked candidates.The strict top-1 result is only marginally lower on each dataset; the comparison therefore uses more oracle queries than the one-query baseline protocol.Source: §F.4 · p. 31

F.5. Structural properties of NSC-DP-discovered architectures

NSC-DP’s discovered architectures reflect the proxy’s additive structure and allocation properties. The closed-form score also closely tracks empirical SVD scores across practical matrix sizes.

F.5. Structural properties of NSC-DP-discovered architectures

  • At the 38.4 M budget, NSC-DP selects an 18-layer, d_model=512 Transformer-XL with uniform d_ff,l=1,152 across layers.Strict concavity in FFN width makes uniform allocation optimal under the additive parameter constraint; NSC-DP then selects the network-level (d_model, L) configuration.Source: §F.5 · pp. 31–32
  • On AutoFormer-Tiny, NSC-DP tends to select deeper, narrower configurations and conservative head counts under the 5.7 M budget.The passage attributes this pattern to decreasing marginal returns for layer-level resource choices and NSC’s discounting of redundant near-orthogonal projections.Source: §F.5 · p. 32
  • At the 5.7 B LoNAS-LLaMA-7B budget, NSC-DP selects LoRA rank 32 at every block and nearly uniform FFN widths across 32 blocks.The stated proxy rationale is that maximal LoRA rank maximizes its projection contribution, while FFN-width concavity favors uniform allocation.Source: §F.5 · p. 32

G.1. Finite-size Convergence of ψMP

  • For matrix dimensions from 128 to 4096, the closed-form ψMP has relative error below 1% and at least an order of magnitude less than score differences between candidate architectures.At min(m,n)=128, all nine tested aspect-ratio and initialization combinations have relative error below 4×10^-3; error decays approximately as 1/min(m,n).Source: §G.1 · pp. 33–34, Figure 6 · p. 33
  • The approximation is already below 1% at the smallest benchmark matrix dimension, 128, and is reported to track empirical SVD scores throughout the tested range.Aspect-ratio differences disappear beyond min(m,n)=512, and the three initialization conventions produce overlapping error curves.Source: §G.1 · p. 34, Table 18 · p. 34

G.2. Closed-form 𝜓MP vs. Trained-Weight 𝜓𝑊: A Consistency Ablation

The paper tests whether the architectural-shape-based closed form agrees with scores from trained-weight SVD on LoNAS-LLaMA-7B. Across 129 Pareto-optimal subnets, the two scoring back ends closely agree and the closed form is cheaper to build.

G.2. Closed-form ψMP vs. Trained-Weight ψW: A Consistency Ablation

  • ψMP and ψW give the same ordering of all 129 Pareto-optimal subnets, with Kendall τ and Spearman ρ both 1.0000.The table reports metrics across the jointly selected 129 subnets.Source: Table 19 · p. 35, §G.2 · p. 36
  • Both back ends select LoRA rank 32 at every block and return identical FFN sums and parameter counts at every operating point.Any remaining disagreement is limited to assigning FFN widths across layers, where ψMP is layer-symmetric.Source: §G.2 · p. 36
  • Mean SVD-oracle regret is 0.039%, with a maximum of 0.104% across the 129 subnets.The paper characterizes this regret as below the noise floor at the resolution of downstream zero-shot benchmarks.Source: §G.2 · p. 36
  • Building the ψMP cache takes 0.5 s of CPU time without a pretrained checkpoint, versus about 6 minutes on an RTX 5090 for ψW, a reported ∼720× overhead.The conclusion reports the same Pareto ranking and regret below 0.11% as empirical justification for retaining ψMP as canonical.Source: §G.2 · p. 36

H. Robustness to Initialization Convention

NSC rankings are tested under Xavier, Kaiming, and TruncNorm-0.02 initialization conventions. Rankings remain similar across conventions, while matching the model’s actual initialization is reported to benefit GPT-2 ranking quality.

H. Robustness to Initialization Convention

  • The robustness check recomputes NSC rankings on FlexiBERT under two protocols and on GPT-2 using three initialization conventions.The conventions are Xavier, Kaiming, and TruncNorm-0.02.Source: H. Robustness to Initialization… · p. 36
  • Xavier–Kaiming pairwise Spearman ρ is at least 0.995 on every benchmark, while TruncNorm-0.02 preserves ρ≥0.88.The table describes τ and ρ as Kendall and Spearman correlations against ground truth, and ρpair as agreement between initialization settings.Source: Table 20 · p. 37, H. Robustness to Initialization… · p. 37
  • Absolute ranking quality remains stable across initialization conventions, with the largest reported drop Δρ=0.10 on FlexiBERT under the spec-faithful protocol.GPT-2 ranking quality under its matching TruncNorm-0.02 convention rises from τ=0.814 to τ=0.849.Source: H. Robustness to Initialization… · p. 37
Loading 2609.23087v1…