Neural Spectral Capacity with Residual Context for Heterogeneous Resource Allocation within Transformers

1. Introduction

• NSC predicts relative trained performance from projection dimensions and initialization statistics without data or instantiated weights, but ignores projection position and scores compatible reorderings identically.This leaves structural differences among equally sized networks unresolved.Source: §1 · p. 1
• ResNSC uses a specification-derived reference for accumulated residual updates to set projection input and residual-state statistics while retaining additive, data-free evaluation.For output projections, the input model includes earlier within-branch transformations.Source: §1 · p. 2
• Because variable residual widths make later scores depend on earlier choices, ResNSC-DP adds residual history to NSC-DP and jointly allocates residual and FFN widths under a resource budget.The method reports bounds on the best score within a specified finite candidate set.Source: §1 · p. 2
• 3.28–5.46 percentage points: ResNSC-DP improves eight-task average accuracy over NSC across eight matched LLaMA-7B pruning budgets.At 121M and 374M pretraining scales, it also achieves lower mean perplexity than uniform, VWT, and TLM allocations at comparable parameter budgets.Source: §1 · p. 2
• Reversing selected widths preserves projection shapes and NSC scores yet worsens performance in pretraining and inherited-weight pruning.Figure 1 illustrates the 374M FineWeb-Edu reversal.Source: Figure 1 · p. 2, §1 · p. 3

Training-free architecture evaluation

• Training-free methods use activation diversity, gradient statistics, or parameter and activation information, whereas NSC scores projections from dimensions and initialization statistics without data or instantiated weights.ResNSC retains specification-only evaluation while conditioning projection scores on residual context.Source: §2 · p. 3

Residual context in Transformers

• Transformer Circuits describes branches updating a shared residual representation, while prior work studies normalization and modifies residual mixing through learned connections.ResNSC uses this structural perspective to construct a specification-derived projection reference; its statistical assumptions are modeling choices.Source: §2 · p. 3

Layer-wise heterogeneous Transformer architectures

• Layer-wise search, prescribed VWT and TLM profiles, geometry-guided allocation, and structured pruning provide approaches to heterogeneous architectures and subnetworks.ResNSC supplies a specification-derived width-selection objective; in LoNAS, it selects FFN widths while the supernet supplies weights, and ResNSC-DP retains residual history when widths vary.Source: §2 · p. 3

3. Residual-Aware Neural Spectral Capacity

• ResNSC extends NSC’s local capacity framework with residual history and within-branch propagation, then allocates widths using history-aware ResNSC-DP under a resource budget.The score is computed from reference covariances derived from architecture specifications.Source: §3 · p. 3

3.1. Preliminaries: local capacity and NSC

• NSC uses isotropic inputs and unit noise to score projections deterministically from dimensions and initialization conventions, without data or instantiated candidate weights.Its reference conditions do not encode projection position or residual history; ResNSC retains the local-capacity expression while changing the contextual covariances.Source: §3.1 · pp. 3–4

3.2. Residual reference from architectural specifications

• The residual reference assumes centered, uncorrelated unit-variance embedding coordinates and centered branch updates with identity covariance on active coordinates, uncorrelated with other updates.The ledger tracks coordinate variances before each branch; each block contributes attention then FFN updates on its active width, so preceding residual widths determine later context.Source: §3.2 · p. 4, Figure 2 · p. 4
• Mean-field normalization preserves relative coordinate variances, while inverse-background weighting discounts read signals and supplies the reference noise variance for write scoring.The background is coordinate-wise 1 + p_b,i, retaining sensitivity to accumulated residual variance scale.Source: §3.2 · pp. 4–5

3.3. Projection evaluation and architecture scoring

• Query, key, value, gate, and up projections use read covariances; attention-output and FFN-down projections incorporate prescribed within-branch propagation into their input covariances.The prescribed output-projection references use random propagation matrices independent of the output matrix being scored.Source: §3.3 · p. 5
• Deterministic spectral formulas evaluate read and write capacity expectations as scalar fixed-point equations using dimensions, residual history, and reference conventions, without data or instantiated weights.Write expectations include both the output matrix and upstream reference.Source: §3.3 · p. 5
• ResNSC sums seven projection terms using each block’s intermediate residual context, so it can distinguish reorderings even when their final ledgers agree.Embedding, normalization, and readout count toward resource cost but not the seven projection terms.Source: §3.3 · pp. 5–6, §3.4 · p. 6

3.4. Budget-constrained allocation with residual history

• ResNSC-DP records counts of preceding blocks at each residual width because layer and budget alone cannot distinguish prefixes with different ledgers and future scores.The counts reconstruct the ledger; nondominated-prefix search uses completion bounds, and a gap at most ε certifies ε-optimality for the numerical objective, while exhausting the queue establishes optimality.Source: §3.4 · pp. 6–7

4.1. Architecture search and pretraining

• The pretraining experiments jointly vary residual and FFN widths at 121M and 374M under the Uniform model’s parameter ceiling, with Uniform, VWT, and TLM as comparators.Models are trained from scratch on the same FineWeb-Edu subset; the 121M main comparisons use three independent runs, while 374M results use one run plus two final-decay continuations.Source: §4.1 · pp. 7–8
• Reversing the ResNSC-DP allocation raises mean FineWeb-Edu perplexity from 19.39 to 20.27 at 121M and from 13.60 to 14.48 at 374M, despite unchanged NSC scores.WikiText-2 and LAMBADA perplexities also rise at both scales, showing that placement of the same projection shapes affects trained performance.Source: §4.1 · p. 8
• ResNSC-DP architectures achieve the lowest mean perplexities across three datasets and both scales; at 374M, they reach 44.60% Avg8 versus 44.26% for NSC and VWT.Against NSC-DP, FineWeb-Edu perplexity falls from 20.00 to 19.39 at 121M and from 13.82 to 13.60 at 374M.Source: §4.1 · pp. 8–9, Table 2 · p. 8
• At 121M, full ResNSC achieves the lowest mean FineWeb-Edu and WikiText-2 perplexity among the tested scoring variants, while depth-only and no-propagation variants achieve lower LAMBADA perplexity.The ablations test removing residual context, retaining branch position or mean reference variance, and removing the propagated spectrum under the same menus and parameter ceiling.Source: Table 3 · p. 8, §4.1 · p. 9

4.2. Structured pruning of LoNAS-LLaMA-7B

• The LoNAS-LLaMA-7B pruning comparison uses matched parameter counts and shared menus, while ranking experiments compare scores with trained-performance labels across language and vision search spaces.The pruning setup uses inherited supernet weights; ranking is assessed on FlexiBERT, GPT-2, and AutoFormer-Small candidates.Source: §4.2 · p. 9, §4.3 · p. 10
• Across eight matched pruning budgets, ResNSC-DP improves eight-task average accuracy over NSC by 3.28–5.46 points.At 5.690B parameters, ResNSC exceeds NSC by 3.40 Avg8 points and W-PCA by 5.04; the allocation retains 98.8% of the unpruned supernet’s Avg8 with approximately 16% fewer parameters.Source: §4.2 · p. 9, Figure 3 · p. 9

4.3. Ranking across architecture search spaces

• ResNSC retains ranking correlations comparable to NSC on FlexiBERT and GPT-2 and higher on AutoFormer-Small, while assigning more distinct scores on FlexiBERT and AutoFormer-Small.Distinct-score counts rise from 297 to 421 on FlexiBERT and from 952 to 1,001 on AutoFormer-Small; both methods distinguish all 200 GPT-2 candidates.Source: §4.3 · p. 10, Table 5 · p. 10

5. Conclusion

• ResNSC scores projections using architecture-implied residual context; ResNSC-DP adds residual history to NSC-DP and bounds the best score within a specified finite candidate set.The evaluation retains NSC’s additive, data-free character.Source: §5 · pp. 10–11
• ResNSC achieves lower perplexity than uniform, VWT, TLM, and same-menu NSC allocations at 121M and 374M, and higher eight-task average accuracy than NSC at every matched LoNAS budget.Across broader search spaces, ResNSC retains strong global ranking correlations while distinguishing more candidate architectures.Source: §5 · pp. 10–11
• Paired reversals show that identical projection shapes and NSC scores can perform differently after from-scratch training and under weight inheritance.Source: §5 · p. 10

B.1. Residual windows and a process realizing the ledger

• Residual windows select the active prefix for each branch; earlier residual-write windows determine the ledger, while FFN intermediate width affects projection capacities and parameter costs, not variance increments.Coordinates outside the write window are preserved.Source: §B.1 · p. 15

B.2. Source normalization and inverse-background evaluation

• Inverse-background weighting evaluates projections relative to accumulated residual variance, while source normalization removes overall scale but preserves relative coordinate variances.The unit term retains NSC’s unit-noise reference; when source power is zero, the normalized input covariance and read metric are set to zero.Source: §B.2 · p. 15

B.3. Depth normalization and the scalar specialization

• For full-width references, depth normalization gives attention and FFN precisions Z_L/(2ℓ) and Z_L/(2ℓ+1), reflecting the intervening attention write.For fixed active widths below the full width, the ledger is likewise depth-determined.Source: §B.3 · p. 16

B.4. Source support and the mean propagated covariance

• A wider output can distribute features over a larger destination without increasing the current input rank, which remains at most the embedding width E.Subsequent reference writes update the ledger.Source: §B.4 · p. 16

C. Projection-specific reference ensembles; C.1. Complete reads and inherited entry variances

• The read evaluation uses complete projection ensembles; for attention, stacked Q, K, or V matrices are evaluated as one observation, and under matched ensembles joint multi-head capacity is no greater than separate-head scoring.Source: §C.1 · p. 17

C.2. Normalized propagation and the gated FFN reference; C.3. The complete block sum

• Write references propagate normalized input covariance through upstream factors and, for gated FFNs, an independent SiLU gate; block scoring updates the ledger after attention before recomputing FFN metrics.The block sum counts three attention reads and two FFN reads, with remaining terms scoring O and down projections.Source: §C.2 · p. 17, §C.3 · p. 18

D. Deterministic evaluation of local capacity

• Deterministic equivalents approximate the finite-dimensional ensemble expectations and define the numerical objective optimized by ResNSC-DP.The large-system interpretation scales dimensions proportionally while retaining aspect ratios and controlled covariance spectra.Source: D. Deterministic evaluation of local… · p. 18

D.1. Single-projection evaluation

• For nonuniform histories, the single-projection read evaluation uses the entire reference-metric eigenvalue list; with an isotropic metric, it reduces to the Marchenko–Pastur Shannon transform used by NSC.The fixed-point equation is solved by bisection, and a zero metric gives zero capacity.Source: §D.1 · p. 19

D.2. Propagated write evaluation

• The propagated-write evaluation solves a coupled fixed-point system using a positive solution and a finite-support gate law, with numerical settings specified for deterministic evaluation.Attention uses a point mass at 1; gated FFNs use 32-point Gauss–Hermite quadrature, and the coupled equations use damped fixed-point iteration.Source: §D.2 · p. 19
• Finite-matrix validation compares deterministic values with means of 1,200 independent reference draws; maximum relative differences are 0.337% for reads and 0.686% for writes.Source: §D.2 · p. 20, Table 8 · p. 20

E.1. Search domain and parameter costs

• The search domain is a finite product of per-block residual- and FFN-width choices, with block costs plus fixed costs defining the resource budget.The parameter convention includes norms and untied embedding and readout matrices.Source: §E.1 · p. 20

E.2. Order dependence and sufficient history

• Residual-width count vectors reconstruct the history used for future scores, and prefixes with no greater cost and at least as much score dominate other prefixes.Prefixes sharing a count vector have identical additional scores for every common suffix.Source: §E.2 · p. 21

E.3. Resource-price suffix bounds

• Resource-price suffix potentials relax the remaining budget into a linear cost penalty and provide upper bounds on feasible completion scores.Taking the minimum across prices preserves the upper-bound property; the state graph permits backward potential computation.Source: §E.3 · p. 21

E.4. ResNSC-DP and the global objective gap

• ResNSC-DP maintains lower and upper bounds on the finite-menu objective, returning an ε-optimal architecture when their gap is at most ε or an optimum when the valid queue is exhausted.The theorem assumes exact arithmetic for the fixed objective and admissible suffix bounds; numerical reporting uses float64 and an absolute tolerance of 10^-6.Source: §E.4 · pp. 22–23

E.5. Fixed-residual-width dynamic programming

• With fixed residual width, the ledger is determined by depth, so an exact multiple-choice knapsack recurrence can optimize FFN choices under the stated costs.This specialization retains residual-context weighting and propagated write sources while each layer’s value is independent of earlier FFN choices.Source: §E.5 · p. 23

E.6. Joint-width search results

• Both complete joint residual- and FFN-width searches return the reported allocations with zero final numerical objective gap after exhausting the valid queue.Table 9 reports the final objectives and numerical gaps for both searches.Source: §E.6 · p. 23, Table 9 · p. 23

F. Training and evaluation statistics

• Pretraining uses fixed FineWeb-Edu subsets with approximately 8.34B available tokens at 121M and 37.45B at 374M, while total training uses approximately 22 tokens per parameter.All architectures at a scale share the same subset, split, and token budget.Source: F. Training and evaluation statistics · pp. 23–24
• At 121M, each arm has three independent runs; at 374M, reported variation comes from two final-decay continuations of one full run, and perplexity is averaged after exponentiating each run’s loss.Evaluation includes FineWeb-Edu held-out cross-entropy, WikiText-2 word perplexity, LAMBADA final-word perplexity, and eight-task Avg8 at 374M.Source: F. Training and evaluation statistics · pp. 24–25

F.1. Scoring ablation protocol

• The 121M scoring ablations compare removing residual context, replacing width history with branch depth, averaging history across coordinates, and removing propagated write spectra.The variants select architectures from the same width menus under the same parameter ceiling; the propagated-spectrum variant retains read scoring and residual-dependent receiver covariance.Source: §F.1 · p. 25

G. Layer-wise architecture configurations

• Each block uses pre-RMSNorm causal attention followed by a pre-RMSNorm SwiGLU FFN, with branch outputs added to the active residual coordinates and inactive coordinates preserved by identity.Attention has 16 query and 16 key/value heads, and the FFN projects from active width to its selected width and back.Source: G. Layer-wise architecture… · p. 26
• Six allocation rules are compared: Uniform, VWT, TLM, ResNSC, NSC, and Reverse, which reverses ResNSC’s sequence while preserving its projection-shape inventory.ResNSC jointly selects active residual and FFN widths; NSC optimizes its original score under the same menu and parameter ceiling.Source: G. Layer-wise architecture… · p. 26
• The search independently chooses each layer’s active width and FFN ratio from finite menus under the uniform model’s parameter ceiling, without a monotonicity or taper constraint.Parameter counts include embeddings, readout, and normalization; the 121M VWT configuration exceeds the Uniform search ceiling by 12,960 parameters.Source: G. Layer-wise architecture… · p. 26

• Both methods select rank 32 throughout, with each block choosing an FFN width from a five-value menu and a Q/K/V LoRA rank of 28 or 32.Source: H. LoNAS-LLaMA protocol and budget… · p. 26, Table 16 · p. 28
• Tables 14 and 15 give exact per-layer residual and FFN intermediate widths in input-to-output order for the 121M and 374M architectures.Source: Table 14 · p. 27, Table 15 · p. 27
• The archived proxy searches use 50 individuals over 20 generations, or 1,000 evaluations, under a 5.7B ceiling; Reverse flips FFN-width assignments across pretrained layers.Source: H. LoNAS-LLaMA protocol and budget… · p. 28
• LoRA scoring adds contextual scores for three Q/K/V adapter paths to the seven-projection score; at fixed residual width, those terms are independent of FFN width.Rank 32 scores higher at every layer on the stated menu.Source: H. LoNAS-LLaMA protocol and budget… · p. 28
• Search takes a median 0.28 seconds across five fresh-process runs on one CPU thread, and matched ResNSC/NSC configurations plus the highest-budget Reverse control have identical FLOPs.The timing covers spectral scoring and joint FFN/rank selection at all eight budgets.Source: H. LoNAS-LLaMA protocol and budget… · p. 28
• Table 16 reports eight matched parameter budgets with identical ResNSC and NSC parameter counts, and defines Δ as the Avg8 difference from the corresponding NSC model.Source: Table 16 · p. 28

• The benchmark inventory comprises 500 FlexiBERT entries, 200 GPT-2 entries, and 1,001 entries each in AutoFormer-Tiny and AutoFormer-Small, scored against their specified task labels.Source: §I · p. 28
• All scores are computed analytically without candidate training or forward passes, and the section reports detailed ranking correlations alongside the parameter-controlled pairwise results.Source: §I · p. 29, Table 18 · p. 30
• The archived benchmark scoring uses distinct projection inventories and variance conventions, including GPT-2's entry standard deviation of 0.02 and AutoFormer's attention width of 64H.FlexiBERT retains its archived operator-specific inventory; AutoFormer also uses head dimension 64 and a rectangular output projection.Source: §I · p. 29
• For these archived sets, ResNSC applies the residual reference even to post-normalized models and uses a Gaussian hidden reference and unit gate for ordinary FFNs.Its propagation model substitutes a Gaussian closure for a GELU covariance kernel; GPT-2 and AutoFormer use complete Q/K/V projections in ResNSC rather than NSC's headwise decomposition.Source: §I · p. 29
• Parameter-controlled ranking retains unordered pairs with less than 10% relative parameter difference, excludes score or performance ties, and reports accuracy and τPW on the restricted pairs.The table reports pair counts before tie exclusion; τPW uses the restricted pair set.Source: §I · pp. 29–30, Table 19 · p. 30
• ResNSC accuracy differences were −0.09, +0.99, −2.72, and −1.25 percentage points across FlexiBERT, GPT-2, AutoFormer-Tiny, and AutoFormer-Small, respectively, under the common-pair protocol.Source: §I · pp. 28, 30, Table 19 · p. 30