PaperlayerView exact revision on arXiv

Source-linked AI summary

Effective Does Not Mean Useful: Conditional Functional Substitutability for Redundancy and Scaling in Transformers

Jiaheng Chen, Jiaxing Li, Yucheng Xiao, Xinyong Cai, Juncheng Bu, Lan Yu, Tinghe Zhang

arXiv:2609.39259v1cs.AIcs.LG

TL;DR

Importance and similarity provide only indirect proxies for neural redundancy, leaving its functional, input-dependent character and relation to scaling unclear. The paper introduces CFS to measure downstream substitution and study Transformer scaling and adaptive computation. It reports functional reorganization with scale and a better performance–computation trade-off from predicted-CFS selection than importance-based selection.

  • Problem

    Neural redundancy is commonly characterized by importance or representational similarity, both indirect proxies, while the mechanisms underlying predictable scaling behavior remain poorly understood.

    Source: §1 · p. 1

  • Method

    The paper introduces CFS, an input-conditioned measure of whether one intermediate state can reproduce another’s downstream response, and uses predicted CFS for component selection.

    Source: §1 · pp. 1–2

  • Results

    Across Transformer vision and language models, CFS reveals systematic functional reorganization with scale, and CFS-guided selection achieves a better performance–computation trade-off than importance-based selection.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The results suggest functional organization as a design variable beyond parameter count, while the fixed-capacity study does not establish causal benefits from optimizing CFS structure.

    Source: §7 · p. 9, §8 · p. 10

    The fixed-capacity study changes architectural decomposition and does not establish causal benefits from optimizing CFS structure.

    Source: §7 · p. 9

Abstract

from arXiv · show

Modern neural networks scale predictably, yet the mechanisms behind these regularities remain unclear. Neural redundancy is typically characterized by component importance or representational similarity, both indirect proxies. We view redundancy as an input-conditioned, dynamic relation: intermediate computational states are functionally redundant when they induce similar downstream responses. We introduce Conditional Functional Substitutability (CFS) to directly characterize such functional substitution. CFS exposes functional relations and reduction potential missed by conventional importance- and similarity-based measures. Across modalities and Transformer families, CFS reveals systematic functional reorganization with scale. Controlled scaling further shows that performance gains need not track growth in substitutability, while fixed-capacity models with more independent functional structure perform better, providing a functional account of diminishing returns. Predicted CFS further enables dynamic computation with a better performance--computation trade-off than importance-based component selection, suggesting new directions for redundancy-aware computation and more efficient model scaling.

1 INTRODUCTION

The paper introduces Conditional Functional Substitutability (CFS) as a direct, input-conditioned measure of functional redundancy, then uses it to study scaling and adaptive computation. Its results connect functional organization with scaling behavior and show that CFS-guided selection improves the performance–computation trade-off over importance-based selection.

1 INTRODUCTION

  • Importance measures removal or perturbation effects and similarity compares representations, but both are indirect proxies for redundancy.The paper frames redundancy instead as a dynamic relation between intermediate states and their downstream responses.Source: §1 · p. 1
  • CFS measures whether one intermediate state can reproduce another’s downstream response for a given input, exposing functional reduction potential missed by importance and similarity proxies.The relation is input-conditioned and directional; a computation can be locally effective without being independently indispensable.Source: §1 · pp. 1–2
  • Across Transformer families, scaling reorganizes functional relations; controlled results link more independent functional structure at fixed capacity with better language-modeling performance.Performance gains can accompany markedly different growth in substitutability, providing a functional account of diminishing returns.Source: §1 · pp. 1–2
  • CFS-guided component selection achieves a better performance–computation trade-off than importance-based selection.The method predicts CFS from forward model states and recovers part of the oracle advantage without intervention-time information.Source: §1 · p. 2

2 RELATED WORK

The related work situates CFS among approaches to redundancy, scaling mechanisms, and adaptive computation. It distinguishes CFS by directly modeling replacement through input-conditioned downstream effects.

Neural redundancy and functional substitutability

  • Prior redundancy work uses component importance, similarity, stitching, or interventions; CFS instead models whether one component’s downstream effect can be recovered by another for a given input [1, 3, 7].The cited literature includes pruning and similarity-based reuse as well as intervention-based studies of compensation and dormant backups.Source: §2 · pp. 2–3

Neural scaling and its mechanisms

  • Proposed explanations of scaling laws include data geometry, skill-frequency structure, training dynamics, statistical and approximation theory, and representational superposition [1, 3].CFS complements these views by tracking functional substitutability and independent structure as scale changes.Source: §2 · p. 3

Adaptive computation

  • Adaptive Transformer methods use dynamic allocation, pruning, conditional blocks, or adaptive depth; CFS-guided routing selects components from predicted pairwise substitutability under a fixed budget [1, 2].The passage contrasts this with methods based on learned policies, importance, or compute objectives.Source: §2 · p. 3

3 METHOD

The method decomposes Transformer computation into intervenable units, defines input-conditioned substitution by downstream behavior, and summarizes the resulting functional organization with complementary measures.

3.1 FUNCTIONAL UNIT DECOMPOSITION

  • Attention heads are treated as functional units because their projected outputs form parallel additive paths into the residual stream; a gate vector enables interventions on selected units while the all-ones setting recovers the original model.The formulation can extend to other decomposable components.Source: §3.1 · p. 3

3.2 CONDITIONAL FUNCTIONAL SUBSTITUTABILITY

  • CFS tests whether unit j compensates for unit i by removing i, scaling j, and measuring the fraction of deletion-induced functional damage recovered for that input.The measure is functional, input-conditioned, and directional, so it describes a dynamic relation rather than a static component score.Source: §3.2 · pp. 3–4

3.3 QUANTIFYING FUNCTIONAL ORGANIZATION

  • Pairwise CFS forms an input-conditioned graph whose Raw Oracle records each source’s strongest available substitution, while Matched Oracle controls for candidate-set size.Raw Oracle is the primary measure of available substitutability; Matched Oracle is a candidate-count control, not its replacement.Source: §3.3 · p. 4
  • Functional Coverage measures how small a representative set can cover all components at threshold τ, while effective functional rank measures spectral compactness without a substitution threshold.Smaller coverage indicates a more compact functional organization; the measures capture complementary aspects of organization.Source: §3.3 · p. 4

4 FUNCTIONAL SUBSTITUTABILITY AS A LENS FOR NEURAL REDUNDANCY

CFS measures input-conditioned functional substitution between components, revealing redundancy that importance and similarity proxies miss. Joint-subset evaluation shows substantial reduction potential beyond importance-based selection.

4 FUNCTIONAL SUBSTITUTABILITY AS A LENS FOR NEURAL REDUNDANCY

  • CFS measures whether one intermediate state can reproduce another’s downstream response for a given input, providing a pairwise, directional relation for functional redundancy.The section first contrasts CFS with importance and similarity, then evaluates joint reduction under complete functional information.Source: §4 · p. 4

4.1 BEYOND IMPORTANCE AND SIMILARITY

  • Input-conditioned representative selection raises recovery from 0.0015 to 0.1213 without adaptive calibration; functional selection reaches 0.3701 versus 0.1988–0.2592 for alternatives under oracle compensation.For a fixed representative, input-conditioned compensation raises recovery from 0.0015 to 0.2186.Source: §4.1 · p. 5
  • Parameter-, contribution-, and attention-based similarities are essentially uncorrelated with CFS, while Taylor importance reaches only 0.2386 Spearman.Current-layer proximity does not reliably predict downstream functional substitutability.Source: §4.1 · p. 5, §4 · p. 5
  • For a fixed layer and source, the modal substitute covers only 31.25% of inputs, and each source encounters an average of 4.97 distinct best substitutes out of five.CFS also varies with input perturbation and is directional, though strongly one-sided relations are uncommon.Source: §4.1 · p. 5

4.2 ORACLE FUNCTIONAL REDUCTION POTENTIAL

  • The joint-subset oracle reduces KL by 61.6–81.3% relative to Taylor while improving dense-prediction fidelity at every evaluated budget.It is an upper bound within the evaluated intervention space, not a deployable pruning algorithm.Source: §4.2 · pp. 5–6, Table 2 · p. 6

5 FUNCTIONAL ORGANIZATION UNDER NEURAL SCALING

Across natural and controlled scaling, CFS reveals changes in functional organization that do not uniformly track performance gains. Fixed-capacity comparisons associate more independent functional structure with better performance, while training traces show heterogeneous consolidation.

5.1 FUNCTIONAL ORGANIZATION ACROSS MODEL SCALE

  • Raw Oracle rises while Cover/H and Rank/H fall for ViT and same-granularity Qwen2.5, with a weaker trend in BERT.Raw Oracle reflects both substitute quality and candidate availability; Matched-2 is not universally monotonic.Source: §5.1 · p. 6

5.2 CONTROLLED CAPACITY SCALING

  • Validation loss improves with depth while Raw Oracle changes little, from 0.2125 at D12 to 0.2154 at D18.Widening to W1152 raises Raw Oracle by 0.0397, versus 0.0029 for deepening to D18; D18 also yields a larger validation-loss reduction per added parameter.Source: §5.2 · p. 7, Table 4 · p. 7

5.3 FUNCTIONAL ORGANIZATION AT FIXED CAPACITY

  • At identical parameter budgets, validation loss rises from 3.5707 to 3.5780 as Raw Oracle increases from 0.2012 to 0.2583 and Rank/H falls from 0.9507 to 0.8725.The association is not causal because head granularity also changes architectural inductive bias.Source: §5.3 · p. 7, Table 5 · p. 7

5.4 FORMATION AND STABILIZATION DURING TRAINING

  • Layer-11 similarity to final substitute rankings is 0.612/0.242/0.787 at epochs 25/50/75; epoch-75 similarities across layers 3/7/11 are 0.839/0.429/0.787.Consolidation is gradual but non-monotonic and heterogeneous across layers; epoch-1 deletion effects of O(10^-5) mean early low CFS does not imply stronger independence.Source: §5.4 · p. 7, Figure 2 · p. 8

6 FROM FUNCTIONAL ANALYSIS TO ADAPTIVE COMPUTATION

The paper predicts CFS from forward states and uses it to select components per input. This routing preserves dense-model behavior better than Taylor selection under equal logical budgets, though current execution does not deliver wall-clock savings.

6.1 OBSERVABILITY OF FUNCTIONAL SUBSTITUTABILITY

  • Forward features yield 0.220–0.225 recovery, while downstream sensitivity reaches 0.2643 recovery and 0.4701 Spearman; forward states nevertheless provide the routing signal.The gap is consistent with substitutability depending on downstream propagation rather than local geometry alone, and appears across additional vision and language evaluations.Source: §6.1 · p. 8

6.2 CFS-GUIDED DYNAMIC FUNCTIONAL ROUTING

  • CFS reduces KL by 43.5–49.2% relative to Taylor across all three budgets, with all paired 95% confidence intervals below zero.Accuracy trends higher but is not statistically resolved; the evidence supports output fidelity as the primary advantage.Source: §6.2 · pp. 8–9, Table 7 · p. 9
  • Joint-subset supervision improves on pairwise CFS, reducing KL from 0.1217/0.0315/0.00874 to 0.1190/0.0281/0.00699 at K = 3/6/9.Gains are resolved at K = 6 and 9 and borderline at K = 3; current dense kernels make these logical, not wall-clock, budgets.Source: §6.2 · p. 9

7 DISCUSSION

The discussion frames functional organization as a possible design variable while identifying limits in causal evidence and deployment gains.

7 DISCUSSION

  • The fixed-capacity comparison changes architectural decomposition, so it does not establish causal benefits from optimizing CFS structure.Testing that possibility requires regularizing functional organization while holding architecture and parameter count fixed.Source: §7 · p. 9
  • Routing improves preservation under logical budgets; wall-clock gains require backends that exploit input-dependent structured sparsity.CFS is evaluated with attention heads as functional units; broader units and higher-order interactions remain avenues to test.Source: §7 · p. 9

8 CONCLUSION

The conclusion presents CFS as a functional view of redundancy that connects Transformer scaling, performance at fixed capacity, and adaptive computation.

8 CONCLUSION

  • CFS characterizes input-conditioned, directional functional redundancy by testing whether one component can reproduce another’s downstream role.Source: §8 · p. 10
  • Across vision and language Transformers, CFS reveals systematic functional reorganization with scale, while more independent structure is linked to better performance at fixed capacity.Source: §8 · p. 10
  • Predicted CFS supports dynamic component selection that better preserves dense-model behavior than importance-based routing under equal logical budgets.Source: §8 · p. 10

A ADDITIONAL TRAINING-DYNAMICS ANALYSIS

Supplementary analyses track how functional organization changes during training and examine its input dependence, directionality, and spectral structure. They also clarify how CFS is aggregated and how conventional proxies relate to functional substitution.

A ADDITIONAL TRAINING-DYNAMICS ANALYSIS

  • Performance improves while substitution, coverage, and effective rank remain in motion, with functional consolidation progressing at different rates across layers.The trajectory comes from one ViT-S AugReg run over 100 epochs; epochs 1–3 are warm-up behavior because deletion effects approach the validity threshold. Graph snapshots show each source head’s strongest positive outgoing edge, not necessarily the edge selected for every input.Source: A ADDITIONAL TRAINING-DYNAMICS ANALYSIS · p. 13, Figure 3 · p. 14, Figure 4 · p. 14

B.1 AGGREGATION, VALIDITY, AND FUNCTIONAL OUTCOMES

  • 10^-5 deletion-effect filtering avoids unstable normalization, while CFS is computed per input and discrepancy is measured in each model’s native predictive output space.Normalized coverage and effective rank retain common semantics across model families, but absolute KL values are not compared across them.Source: §B.1 · p. 16

B.2 DIRECTED SPECTRAL EFFECTIVE RANK

  • Lower Rank/H indicates substitution concentrated into smaller effective spectral support, while lower Cover/H indicates compression at the chosen recovery threshold, not better task performance.The singular-value statistic preserves the directionality of the CFS matrix rather than symmetrizing it; Rank/H describes spectral compactness, not exact intrinsic functional dimension.Source: §B.2 · p. 16

C.1 CONVENTIONAL PROXY DEFINITIONS

  • Geometric proximity, parameter similarity, and scalar source importance need not identify the alternative path that best reproduces downstream behavior.Taylor measures first-order source importance, whereas CFS measures a directional relation between a source and a candidate substitute.Source: §C.1 · p. 17

C.2 ADDITIONAL INPUT-CONDITIONING STATISTICS

  • 31.25% median modal-substitute coverage and 4.97 distinct best substitutes on average show that a source’s preferred substitute varies across inputs.The variation is structured: median agreement is 83.3% under mild color perturbations, 50.0% under horizontal flips, and 16.7% across images; corresponding matrix correlations are 0.955, 0.870, and 0.334.Source: §C.2 · pp. 17–18

C.3 DIRECTIONALITY

  • 0.0785 is the median absolute directional gap across 5,760 unordered head pairs; strongly one-sided relations are uncommon, but CFS cannot generally be represented as undirected similarity.The 75th-percentile gap is 0.1767; 3.94% of pairs exceed S > 0.10 in one direction with S ≤ 0 in reverse, and 1.62% meet the stronger S > 0.25 condition.Source: §C.3 · p. 18

D EXACT JOINT-SUBSET ORACLE: ADDITIONAL DETAILS

These analyses bound the gains available from joint-subset removal, show that scaling paths can reorganize substitutability differently, and test CFS-guided routing against importance-based selection.

D EXACT JOINT-SUBSET ORACLE: ADDITIONAL DETAILS

  • At K = 3/6/9, oracle-minus-Taylor KL 95% confidence intervals were [−0.1549, −0.1088], [−0.0477, −0.0306], and [−0.0154, −0.0079], while fidelity improved at every budget.The oracle is an upper bound within the measured intervention space, not a deployable pruning procedure.Source: D EXACT JOINT-SUBSET ORACLE:… · p. 18

E.1 NATURAL-FAMILY COMPARABILITY

  • Matched-2 is not universally monotonic: ViT follows 0.3379, 0.2852, 0.2889, 0.3737, while Qwen2.5 rises from 0.1620 to 0.2745 between 1.5B and 14B.The natural-family comparisons are cross-sectional, and Matched-2 fixes candidate count while Raw Oracle includes available alternatives.Source: §E.1 · pp. 18–19

E.2 CONTROLLED WIDTH AND DEPTH SCALING

  • From shared W768/D12, widening to W1152 raises Raw Oracle by 0.0397 versus 0.0029 for deepening to D18, although D18 reduces validation loss more per added parameter.This is a limited comparison of two capacity increases, not a general claim that depth is preferable to width.Source: §E.2 · p. 19

E.3 FIXED-CAPACITY HEAD DECOMPOSITION

  • At fixed 124.44M parameters, validation loss is 3.5707/3.5752/3.5780 as Raw Oracle rises 0.2012/0.2125/0.2583 and normalized effective rank falls 0.9507/0.9335/0.8725.Changing head granularity also changes architectural inductive bias, so this establishes association rather than a causal effect of optimizing CFS.Source: §E.3 · p. 19

F.1 R1 AND R2 SUPERVISION

  • R1 combines pairwise CFS ranking and regression with related supervision; R2 adds direct joint-subset targets because pairwise substitutions may not compose.The joint teacher evaluates candidate masks by simultaneous hard gating and records their KL divergence from the dense model.Source: §F.1 · p. 20

F.2 ROUTING CONFIDENCE INTERVALS

  • With shared calibration, CFS KL advantages are 0.0696/0.0178/0.0068; with oracle pairwise compensation, they are 0.0604/0.0154/0.0049, with all six confidence intervals excluding zero.Accuracy trends higher under CFS, but its confidence intervals overlap zero; dense-output fidelity is the primary routing evidence.Source: §F.2 · p. 20

F.3 JOINT-SUBSET SUPERVISION ABLATION

Direct joint-subset supervision improves KL over pairwise-only supervision, with gains statistically resolved at two of three budgets. The online implementation reproduces offline masks, but its allocation is logical rather than wall-clock acceleration.

F.3 JOINT-SUBSET SUPERVISION ABLATION

  • R2 lowers KL versus R1 from 0.1217/0.0315/0.00874 to 0.1190/0.0281/0.00699 at K = 3/6/9.Gains are resolved at K = 6 and 9 but borderline at K = 3, supporting joint supervision for higher-order interactions.Source: §F.3 · p. 20

F.4 ONLINE EXECUTION CHECK AND SYSTEMS BOUNDARY

  • Online and offline masks agree on 447/448 images (99.78%) at each budget, but dense fused kernels make these allocations logical rather than wall-clock savings.Real systems gains would require a backend that exploits input-dependent structured sparsity.Source: §F.4 · pp. 20–21

Related papers

Loading 2609.39259v1…