PaperlayerView exact revision on arXiv

Source-linked AI summary

Common-Mode Collapse and Recovery in Direct Feedback Alignment

Varun Reddy, Bernardo L. Sabatini, Houman Safaai

arXiv:2609.31589v1cs.LGcs.NE

TL;DR

DFA can stall when shared output error drives tanh hidden units toward saturation, leaving activity differences suppressed. The paper decomposes the update and models this common-mode drive; its controls and tests show that collapse and learning costs depend on the setting.

  • Problem

    The paper asks how DFA’s broadcast teaching signal can stall learning, particularly when a mean prediction–target mismatch creates shared error and saturates tanh units.

    Source: §1 · p. 1

  • Method

    The paper uses an exact mean–covariance decomposition to isolate the common-mode update, then tests a reduced model and interventions across learning conditions.

    Source: §3 · p. 3, §3.1 · p. 4

  • Results

    Prior-calibrated initialization suppresses collapse and speeds learning in the MNIST baseline at no extra training cost, but is insufficient on CIFAR-10.

    Source: §9 · p. 9

  • Takeaways & Limitations

    Class information can remain decodable during collapse even as reduced feature variation weakens readout gradients and learning resumes as covariance grows.

    Source: §9 · p. 9

    Evidence comes mainly from small networks trained with plain SGD at fixed learning rates; the reduced model captures early collapse but not participation rebound.

    Source: §9 · p. 9

Abstract

from arXiv · show

Direct feedback alignment (DFA) trains hidden layers through fixed random projections of output error. With tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near the loss of a constant predictor of class frequencies. We trace this stall to the error's common mode, the component shared across inputs. An exact mean-covariance decomposition separates a rank-one update formed by the mean teaching signal and mean presynaptic activity. Its leading component drives tanh units toward saturation. At initialization, random feedback provides no systematic correction of the shared error on average; readout learning limits its duration. A reduced model initialized from the network, without fitted parameters, predicts the concentration of activation sensitivity across 48 settings. On MNIST, class decodability largely survives collapse, but readout learning remains slow at a fixed learning rate. Adam learns faster despite deeper collapse. Calibrating the baseline readout to the class prior suppresses collapse and speeds learning; weaker feedback trades less collapse for slower learning. Replacing errors by their signs sustains collapse; subtracting the signal's batch mean prevents sustained collapse and improves learning in the tested setting. Related effects occur in deeper and convolutional networks and on CIFAR-10, with severity and cost depending on the readout, optimizer and input statistics.

1 INTRODUCTION

The introduction frames learning with broadcast error signals as a problem in neural learning theory and asks how DFA’s teaching signal can stall learning. It previews a reduced model, tests across learning conditions, and studies decodability and intervention costs.

1 INTRODUCTION

  • DFA broadcasts output error through fixed random matrices, but its performance varies across tasks and architectures and learning can plateau.The introduction links potential failure modes to saturating tanh derivatives and previously observed plateaus or low-rank updates.Source: §1 · p. 1
  • A mean prediction–target mismatch creates shared error that can push tanh units toward saturation, suppressing activity differences across inputs.The paper calls this representation collapse; class information can remain decodable while input-dependent learning stays slow.Source: §1 · p. 1
  • The paper derives a reduced model, tests its predictions across learning conditions, separates decodability from learning speed, and assesses intervention costs across settings.The planned scope includes learning rules, architectures, and readouts.Source: §1 · p. 2

2 SETTING AND THE PHENOMENON

The paper studies DFA with fixed random feedback and compares its training dynamics with matched backpropagation. In the MNIST baseline, DFA stalls near the class-prior loss as hidden activity collapses, then recovers.

2 SETTING AND THE PHENOMENON

  • DFA projects logit error through fixed random feedback matrices and multiplies by activation derivatives to form hidden teaching signals and local updates.The baseline is an MNIST MLP with three tanh layers of width 300, zero biases, and SGD at η = ηout = 10−3.Source: §2 · p. 2
  • The paper measures representation cosine, mean-activity energy, gate energy and participation, and activity–target covariance to track collapse and learning.Gate participation approaches 1/dℓ when one unit dominates, but equally weak gates can also produce high participation, so it is not a measure of feedback strength.Source: §2 · p. 2
  • DFA rapidly reaches the class-prior loss and plateaus for several hundred steps before improving; accuracy stays near chance during the plateau.Figure 1 compares probe loss with matched BP using the same initialization and minibatches, and tracks hidden cosine, gate participation, and error components.Source: §2 · p. 2, Figure 1 · p. 3
  • Along the DFA trajectory, raw DFA weight-gradient estimates are several times larger than BP gradients measured at the same network state.This comparison is reported at the tested learning rate, where matched BP passes through the prior loss without pausing but learns more slowly later.Source: §2 · p. 3
  • During the plateau, top-layer vectors become nearly parallel, while gate participation falls across hidden layers, most strongly in deeper layers.The shared-error norm falls from 1.278 to 0.10 within 100 updates, while input-dependent error remains between 0.90 and 0.95 throughout the plateau.Source: §2 · p. 3

3 THEORY: THE COMMON-MODE TERM

The theory decomposes DFA updates into mean and input-varying components to explain how shared error drives early collapse. Its reduced model predicts early concentration across settings, while recovery involves dynamics the model does not capture.

3.1 EXACT DECOMPOSITION AND THE COMMON-MODE CLOSURE

  • The exact mean–covariance decomposition isolates a rank-at-most-one mean update; the common-mode closure approximates mean gated error by mean gate times mean broadcast error.The closure neglects the gate–error residual, which is only 0.01 of the leading term at initialization but exceeds it by step 300.Source: §3.1 · p. 4
  • The mean update shifts tanh preactivation means toward saturation, while saturation slows fixed-direction drift and the bias update contributes to that drift.Holding presynaptic activity fixed, exactly centering it and freezing postsynaptic biases removes local mean drift.Source: §3.1 · p. 4

3.2 WHY THE DRIVE PERSISTS, AND THE COLLAPSE NUMBER

  • At initialization, random feedback has no systematic restoring effect on shared error on average; readout learning supplies the systematic restoring term in the approximation.The collapse number is unchanged by scaling both learning rates, but ignores evolving activity energies and input-dependent learning.Source: §3.2 · p. 4

3.3 COUPLED DYNAMICS ACROSS LAYERS

  • Across 48 settings, the network-initialized reduced model captures baseline plateau loss, onset, and the layer ordering of participation minima without fitted parameters.The model predicts early concentration and approximate exit times, but not participation recovery.Source: §3.3 · p. 5, Figure 2 · p. 5

3.4 RECOVERY AND THE DEEPER-BUT-SHORTER EFFECT

  • During recovery, label covariance grows again, beginning in earlier layers; a phenomenological covariance model predicts when this growth lowers loss.The reduced model does not reproduce participation rebound, turnover in participating units, or growth in within-unit spread.Source: §3.4 · p. 5
  • Increasing feedback gain deepens initial collapse while shortening the plateau over the tested range.The figure relates collapse scaling and transient dose to learning cost and shows that input centering with frozen hidden biases jointly prevents collapse.Source: §3.4 · p. 5, Figure 3 · p. 6

4 PREDICTIONS AND THEIR TESTS

The tests show that collapse varies with the shared-drive dose and input/readout means, while reducing collapse can carry a learning-speed cost.

Learning rate, gain and initial predictions

  • Across a tenfold learning-rate range, collapse severity stays within seed noise while exit time scales as 1/η; stronger g∥ē0∥ produces more parallel hidden activities.The gain and initial-prediction sweeps follow one empirical power-law trend, although other sweeps at similar drive scatter widely.Source: §4 · p. 6

Readout and input means

  • Balanced softmax with standardized inputs yields peak cosine 0.271, but class-skewed sampling restores it to 0.938, against 0.386 for BP.Per-pixel centering prevents layer-1 collapse but only reduces deeper collapse; combining it with frozen hidden biases prevents collapse throughout the network.Source: §4 · p. 6

Learning cost

  • Input centering with frozen biases or output-error centering reaches 50% accuracy after 370 rather than 690 updates, but improves time to 80% accuracy little.At step 300, hidden learning signal is 6.0 versus 182 under error centering, slowing the readout’s fit to the prior.Source: §4 · p. 7
  • At g = 0.03, minimum participation is 0.98 but learning time rises to 1897 rather than 537 updates.The delay persists at η = 0.1: plain DFA reaches 80% accuracy in 130 updates, versus 28 with calibrated readout and 20 for BP.Source: §4 · p. 7

Unit drift and feedback orientation

  • Pre-aligned feedback delivers dose 0.12 rather than 4.6 and learns quickly with little collapse, but reorientation also changes Bℓē, so the control does not isolate restoration.Source: §4 · p. 7

5 DECODABLE FEATURES CAN SUPPORT SLOW LEARNING

Collapsed features largely retain class decodability, yet fixed-rate readout learning is slow; recovery also separates activation sensitivity from update alignment.

Decodability

  • At collapse, ridge decoding retains 88.6% held-out accuracy, close to 89.3% at initialization.Source: §5 · p. 7

Readout learning

  • After 3,000 updates, a fixed-rate sigmoid readout reaches 20.0% held-out accuracy on collapsed features versus 63.4% on initial features.Collapsed features have 5.9-fold smaller RMS variation, lowering effective readout learning rate about 35-fold; rescaling raises accuracy to 80.1%, close to 82.9% for rescaled initial features.Source: §5 · p. 7
  • Adam learns faster despite deeper gate concentration: learning time is 117 rather than 537 updates, with minimum participation 0.046 rather than 0.214.Source: §5 · p. 8

Recovery

  • During recovery, participation rebounds while gradient cosine remains near zero, then alignment rises at nearly fixed participation.Tracking activation sensitivity alone therefore does not determine update direction; task-dependent covariance and alignment also matter.Source: §5 · p. 8

6 GENERALITY AND LIMITS

Related collapse signatures appear across feedback rules, network architectures, objectives, and datasets, but their severity depends on the readout and input statistics.

Rules, architectures and objectives

  • Layerwise FA, a six-layer MLP, and a small CNN show the same signature, while objective and activation choices alter the transient.The six-layer MLP’s layers 4–6 reach cosine 1.000; the CNN’s deepest block reaches minimum participation 0.021 versus 0.84 for a matched BP run.Source: §6 · p. 8

CIFAR-10 and the size of the input mean

  • On CIFAR-10, the tanh CNN reaches minimum participation 0.009 under DFA versus 0.756 under BP, while standardized inputs with balanced softmax largely avoid collapse and imbalanced sampling restores it.The larger input mean also drives layer-1 collapse; output-prior calibration reduces mean error but does not prevent collapse.Source: §6 · p. 8

A common mode that does not decay

  • With sign-error feedback, MNIST remains collapsed through 3,000 updates; subtracting the signal’s batch mean prevents sustained collapse and improves accuracy from 0.75 to 0.90.The sign signal’s population mean does not shrink as the readout learns, and prior-bias initialization leaves hidden updates unchanged.Source: §6 · p. 8

7 INTERVENTIONS AND THEIR COSTS

The tested interventions target error means, activity means, or update geometry, with effectiveness and learning costs varying by setting.

Interventions

  • The proposed interventions are to subtract mean error, center presynaptic activity, or shorten the error’s duration.Figure 10a–c compares them in three networks.Source: §7 · p. 8

Centering the broadcast error

  • Subtracting mean error before projection prevents strong convergence of hidden activities in the tanh MNIST baseline, while centering the teaching signal after the derivative gate is less effective.With ReLU, GELU, or linear units, error centering can drive loss far above the constant-predictor baseline as mean activity grows; input centering suppresses this extreme growth.Source: §7 · p. 8

Calibration and activity centering

  • Prior-bias initialization suppresses MNIST collapse and shortens learning; per-pixel centering protects the first layer, while batch normalization reduces collapse but does not isolate mean subtraction.Neither prior-bias initialization nor activity centering alone suffices in the CIFAR-10 CNN.Source: §7 · p. 9

Changing update geometry

  • Muon-style orthogonalization of hidden updates reduces collapse, whereas an error-side conditioner retains high cosine across tested ridge strengths and lowers step-3000 accuracy to 0.58 at default strength.Source: §7 · p. 9

8 RELATED WORK

The paper situates its account of early saturation within prior work on plateaus and alignment, and distinguishes its focus on first-moment shared signals from other explanations of learning dynamics.

8 RELATED WORK

  • The account explains a mean-driven contribution to early saturation, a recognized source of plateaus in sigmoid networks.Prior work also reports sigmoid-unit saturation while readout biases learn under BP, and weakened tanh DFA gradients from saturation (Launay et al., 2019).Source: §8 · p. 9
  • Unlike align-then-memorise dynamics, this stall precedes substantial alignment; whether common mode contributes to layerwise FA remains open (Boeshertz et al., 2026).Other distinctions include pretrained-weight feedback in Lee & Lee (2026) and second-moment conditioning in Safaai et al. (2026), whereas this work studies first moments before alignment.Source: §8 · p. 9
  • The discussion connects error centering to prior work on centered activities, errors, activation slopes, and modulatory signals, while distinguishing sensitivity from useful learning.It also relates the sensitivity distinction to dormant units and plasticity loss.Source: §8 · p. 9

9 DISCUSSION

The discussion frames the early DFA stall as competition between shared drive and input-dependent learning, then considers interventions, scope, and an open biological question.

9 DISCUSSION

  • Random-feedback mean error can push units toward saturation before readout learning captures the class prior, while class information remains decodable and reduced feature variation weakens readout gradients.Learning resumes as covariance grows, changing units regain sensitivity, and updates align with the loss gradient.Source: §9 · p. 9
  • Prior-calibrated initialization speeds learning in the MNIST baseline at no extra training cost, but is insufficient on CIFAR-10; input centering protects early layers.For the tested sign-error rule, centering the signal batch mean prevents sustained collapse and improves learning.Source: §9 · p. 9
  • Evidence comes mainly from small networks trained with plain SGD at fixed learning rates, and the reduced model captures early collapse but not participation rebound.Adaptive optimizers can shorten the transient, and the recovery extension is phenomenological; the findings support an early-dynamics mechanism, not a general account of DFA performance.Source: §9 · p. 9
  • Whether biological circuits regulate a similar interaction between shared teaching signals and input-dependent plasticity remains open.Source: §9 · p. 10

SUPPLEMENTARY MATERIAL

The supplement documents the experimental protocol, measurements, and plateau criteria used to evaluate collapse and recovery.

SUPPLEMENTARY MATERIAL

  • The supplement organizes materials into protocol and measurements, a mathematical model, mechanism tests, decodability and recovery analyses, and intervention scope and costs.Source: SUPPLEMENTARY MATERIAL · p. 13

A.1 DATA, NETWORKS AND RANDOMNESS

  • Experiments use specified validation splits and fixed probes across MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, with validation diagnostics rather than test-set benchmarks.The supplement also specifies preprocessing, network variants, output/loss choices, randomization, and run counts.Source: §A.1 · p. 13

A.2 DIAGNOSTICS AND NOTATION

  • Diagnostics distinguish activity–label covariance, scale-normalized linear accessibility, centered dimensionality, gate participation, and virtual-update behavior.The scale-normalized statistic is not held-out decoder accuracy or an information-theoretic quantity; virtual updates measure a full-probe local update, not the actual stochastic step.Source: §A.2 · pp. 15–16

A.3 PLATEAU DETECTION AND REPORTING

  • A plateau requires smoothed probe loss near the constant-predictor loss for at least 50 steps and accuracy near the class-prior baseline; learning time is measured separately.Participation minima use the first 300 steps, and reported curve bands are standard deviations rather than confidence intervals.Source: §A.3 · p. 16

A.4 INTERVENTION IMPLEMENTATIONS AND COST

The supplement details intervention implementations and costs, then derives the common-mode model and states its approximations and scope.

A.4 INTERVENTION IMPLEMENTATIONS AND COST

  • Interventions include error and teaching-signal centering, prior-bias and mean-logit calibration, batch normalization, second-moment preconditioning, and gradient orthogonalization.The orthogonalization control applies five Newton–Schulz iterations and restores gradient norm; it is not the full momentum-based optimizer.Source: §A.4 · p. 16
  • Table 3 reports additional per-step costs and descriptive runtime relative to plain DFA; small timing differences near unity indicate comparable measured cost, not established speed improvements.Intervention costs depend on implementation and hardware.Source: §A.4 · p. 16, Table 3 · p. 17, §A.5 · p. 17

A.5 CODE AND COMPUTATION

  • The supplement provides configurations, analysis scripts, tests, reference outputs, and records mapping printed comparisons to their sources.Source: §A.5 · p. 17

B DERIVATIONS AND REDUCED-MODEL ASSUMPTIONS

  • The derivation proceeds from an exact update decomposition to approximate mean drift, accumulated drive, gate participation, and a phenomenological plateau-exit extension.Source: B DERIVATIONS AND REDUCED-MODEL… · p. 17

B.1 EXACT MEAN–COVARIANCE DECOMPOSITION

  • The exact batch update separates a covariance term from the outer product of mean teaching signal and mean presynaptic activity; at initialization, the residual has zero expectation over feedback draws.Zero expectation does not bound the residual norm in an individual network or during training.Source: §B.1 · p. 17

B.2 GATED DRIFT

  • The gated-drift approximation holds presynaptic activity fixed and omits upstream, covariance, and residual contributions; accumulated drive can grow faster than the saturated mean.The omitted terms are not restored by evolving energies in the reduced model.Source: §B.2 · pp. 17–18

B.3 READOUT DYNAMICS AND THE DOSE

  • The readout model tracks mean error as logits approach the class-prior fixed point, while general readouts use numerical integration from their actual initial mean logits and target mean.The product scaling observed when varying initial prediction q is empirical, not an exact consequence of the fixed-activity calculation.Source: §B.3 · p. 18

B.4 PARTICIPATION QUADRATURE

  • The large-dose participation asymptote belongs to the stated closure and cannot be extrapolated indefinitely at fixed width, where participation is at least 1/dℓ.Source: §B.4 · pp. 18–19

B.6 APPROXIMATE COVARIANCE DYNAMICS AND EXIT PREDICTION

The reduced covariance model adds evolving label-covariance amplitudes to estimate when loss exits the prior-predictor plateau, while mean-drive controls test the proposed collapse mechanism. Centering interventions prevent or suppress collapse, and feedback orientation changes its severity.

B.6 APPROXIMATE COVARIANCE DYNAMICS AND EXIT PREDICTION

  • The reduced model augments collapse dynamics with scalar covariance amplitudes and an accumulated loss-reduction variable to estimate the first exit below 0.9 of prior-predictor loss.Its covariance closure adds upstream transmission and local learning contributions as a phenomenological approximation.Source: §B.6 · p. 19
  • The model omits prediction-dependent covariance and higher-order terms away from a constant predictor, and its variance cascade cannot reproduce the observed participation rebound.It can predict an exit while gates remain concentrated.Source: §B.6 · p. 19

C.1 MEAN-DRIVE AND ALIGNMENT CONTROLS

  • Combined input centering and frozen hidden biases prevent collapse throughout the network, while either intervention alone is insufficient.Centering the output error before projection also suppresses baseline collapse.Source: §C.1 · p. 20, Table 4 · p. 21
  • Pre-aligned DFA and BP with twelvefold-scaled readout weights do not collapse; random feedback has a higher shared-to-input-dependent error ratio than either control.The ratios ten updates into training are 1.28 for random feedback, 0.23 for pre-aligned DFA, and 0.07 for scaled-readout BP.Source: §C.1 · pp. 20–21, Figure 5 · p. 20
  • Centered top-layer activity variance falls to 0.028 of initialization under baseline DFA, versus 0.83 under BP and 0.97 with prior-bias initialization.Effective rank also falls under DFA when strong collapse is absent, so low rank alone does not identify the common-mode transient; centering can slow prior fitting as activity energy falls.Source: §C.1 · p. 21, Table 5 · p. 21

C.2 SCALING WITH DRIVE STRENGTH AND LEARNING RATE

The section tests how drive strength, omitted terms, and recovery approximations affect collapse and model predictions. The reduced model tracks participation across most tested settings, but prediction limits and scope boundaries remain.

C.2 SCALING WITH DRIVE STRENGTH AND LEARNING RATE

  • When SGD rates vary together, peak cosine and participation minima change little while rescaled exit time remains 0.52–0.55.Peak cosine is 0.998–0.999 and participation minima are 0.21–0.23 across the tested sweep.Source: §C.2 · pp. 21–22
  • Greater gain shortens learning despite deeper collapse, while at η = 0.1 prior bias cuts 80%-accuracy time from 130 to 28 updates, versus 20 for BP.Across the gain sweep, learning time ranges from 1897 updates at g = 0.03 to 260 at g = 10 as minimum participation falls from 0.98 to 0.021.Source: §C.2 · p. 22

C.3 MEASURED LIMITS OF THE MEAN-DRIVE APPROXIMATION

  • At step 50, the leading mean-route fraction is 0.94; upstream motion dominates initially, while the gate-error residual becomes comparable later.The residual is relatively larger for ReLU, and omitted covariance contributions mean error centering does not isolate the rank-one term.Source: §C.3 · pp. 23, 25

C.4 REDUCED-MODEL ACCURACY AND DETECTION LIMITS

  • Participation predictions correlate 0.98–0.99 across five sweeps (class-count r = 0.10), while width-depth tests yield r = 0.96 and mean absolute error 0.07.The width-depth comparison retains detection errors and overlapping settings, so it is not wholly independent validation.Source: §C.4 · pp. 25–26, Table 13 · p. 26, Figure 7 · p. 27
  • Across the width-depth grid, the model misses 4 detected network plateaus and predicts 3 extra ones; median relative exit-time error is 0.24 among 39 paired exits.Numerical checks support the default resolution in selected settings, not a convergence guarantee across every condition.Source: §C.4 · p. 26, Figure 7 · p. 27

C.5 ALTERNATIVE RECOVERY APPROXIMATIONS

  • The full approximation predicts more observed exits than the version without local growth; removing participation dependence delays many predictions, though covariance magnitudes remain imperfect.The alternatives use the same collapse trajectories and exit criterion.Source: §C.5 · p. 27, Table 16 · p. 28
  • With zero population input–label covariance, the scalar closure remains at zero and cannot represent nonlinear-feature recovery, which was not tested here.The fitted rate exponents are empirical, and gain and hidden-rate sweeps are not independent confirmations.Source: §C.5 · p. 28

D DECODABILITY AND GATE RECOVERY

  • The next section separates retained class information from readout-learning speed and examines how sensitivity and useful updates change during recovery.The fixed first-300-update participation window is not a common dynamical phase across learning rates.Source: D DECODABILITY AND GATE RECOVERY · p. 28, §C.5 · p. 28

D.1 FROZEN-FEATURE DECODING AND READOUT CONTINUATION

The analyses separate representational capacity from readout learning, reconstruct how unit means and spreads relate to gate concentration, and compare descent, interventions, and protocol changes. Across these tests, collapse and recovery depend on the learning signal, optimizer, and network setting.

D.1 FROZEN-FEATURE DECODING AND READOUT CONTINUATION

  • Centered-feature SGD is equivalent to using weight rate ηout/s^2 with an unchanged intercept rate, separating an amplitude bottleneck from lost decodability without correcting covariance anisotropy.The frozen-feature controls begin with identical checkpoint logits, but ordinary SGD learns differently over the fixed budget because it depends on feature coordinates.Source: §D.1 · p. 29

D.2 RECONSTRUCTING PARTICIPATION FROM UNIT MEANS AND SPREADS

  • At step 100, shared-spread reconstruction reached a median reconstructed-to-measured participation-loss ratio of 0.994, while zero-mean reconstruction reached 0.0022.At step 3000, the ratios were 0.515 and 0.401, versus 0.84 for the full reconstruction; the moment substitutions are conditional comparisons, not causal partitions.Source: §D.2 · p. 30, Figure 8 · p. 30, Figure 8a

D.3 GATE TURNOVER AND USEFUL DESCENT

  • The step-120 top-fifth gate-energy set loses overlap with the current top-fifth toward chance, so recovery does not rely on one persistent selected channel.This turnover does not rule out predictive value in early gate statistics.Source: §D.3 · p. 30, Figure 9 · p. 31
  • Norm-matched virtual updates show that the fixed early set retains useful descent direction and its complement also supports descent later; gate participation alone determines neither direction nor magnitude.Pre-aligned feedback's larger projected descent reflects both near-BP-aligned direction and a larger gradient-norm ratio.Source: §D.3 · p. 31

E.1 INTERVENTION COMPARISONS

  • At step 300, error centering reduced top-layer mean-activity energy from 182 to 6.0, but step-1500 mean-error norm and loss were 0.61 and 3.35, versus baseline 0.05 and 1.31.The conditioner retained high cosine at its selected ridge, but selection used the same validation accuracy shown and it cost more than centering; finite-horizon growth with non-saturating activations does not establish divergence.Source: §E.1 · pp. 31–32, Figure 10 · p. 32

E.2 OPTIMIZER DEPENDENCE

  • 117 updates for larger-rate Adam, 480 for smaller-rate Adam, and 537 for SGD were the mean times to sustained loss recovery; larger-rate Adam also reached minimum participation 0.046 versus 0.214 for SGD.Every run reached the criterion; the larger Adam rate produced deeper gate concentration but faster learning. Its update became positively aligned with negative true gradient during early recovery, unlike SGD and smaller-rate Adam for longer.Source: §E.2 · pp. 33–34, Table 21 · p. 34

E.3 RECONSTRUCTING THE ORIGINAL DFA PROTOCOL

  • At the larger rate, at least 99.9% of top-layer units saturated within ten updates in all three reconstructed-protocol arms, including BP.At the smaller rate, few units crossed |h| > 0.9; in fully saturated runs, participation stayed high because all gate energies were equally small, making the saturated fraction more informative.Source: §E.3 · p. 34, Table 22 · p. 35

E.4 SIGN-ERROR FEEDBACK

  • Subtracting the batch mean of sign(e) prevented sustained collapse and improved learning to error-centered DFA levels, while uncentered sign-error feedback sustained collapse and slow learning.Because sign(e) does not depend on the readout, prior-bias initialization leaves hidden updates unchanged; the comparison supports persistent mean drive, rather than quantization alone, as a cause of sustained collapse.Source: §E.4 · p. 35

E.5 CHANGES IN ARCHITECTURE, ACTIVATION AND OBJECTIVE

The paper tests the transient across architectures, objectives, readouts, input statistics, and class-prior conditions. The comparisons show that cosine alone is not sufficient to identify collapse, and that collapse and learning costs vary with the setting.

E.5 CHANGES IN ARCHITECTURE, ACTIVATION AND OBJECTIVE

  • Across deeper MLPs, CNNs, and layerwise FA, strong activity convergence occurs; in the six-layer tanh MLP, layers 4–6 reach cosine 1.000 and a 520-step plateau.Non-saturating activations require gate and loss measurements, because positive activity bias can raise cosine without suppressing input dependence; error centering can destabilize them.Source: §E.5 · pp. 35–36, Figure 12 · p. 36

E.6 CIFAR-10 CONTROLS AND A LEARNED MNIST PRIOR SHIFT

  • On CIFAR-10, prior-bias initialization or mean-logit calibration alone leaves substantial CNN activity convergence, while combining input and error centering reduces it; balanced softmax avoids much of it only with standardized inputs and balanced priors.On imbalanced MNIST, calibration reduced initial mean error to 0.03 and peak cosine from 0.96 to 0.36, yet balanced accuracy after 450 updates was 0.21 rather than 0.30; both reached 0.89 by update 3,000.Source: Figure 13 · p. 37, §E.6 · p. 38, Table 25 · p. 38

E.7 BATCH FLUCTUATIONS AND THE NUMBER OF CLASSES

  • At fixed initialization, centered batch-error RMS exponents ranged from -0.503 to -0.493, but calibrated CIFAR-10 stayed near unit peak cosine across tested batch sizes, with nonmonotonic dependence.CIFAR-100 also showed high-cosine transients under both readouts, with milder gate concentration under softmax; matched BP had high cosine too, while DFA concentrated gate energy more and both rules remained early in training after 3,000 updates.Source: §E.7 · p. 39, Figure 14 · p. 40, Table 29 · p. 40

E.8 COMPARISON WITH ALIGN-THEN-MEMORISE DYNAMICS

  • In the teacher–student ladder, collapse first appears when trainable hidden biases are added; nonzero-mean targets alone do not produce it, and collapse does not set plateau length in the synthetic task.BP also stays near the constant predictor in several rungs without full collapse, and distinct cosine and alignment patterns distinguish pre-alignment from post-alignment plateaus.Source: Figure 15 · p. 41, §E.8 · p. 41
Loading 2609.31589v1…