PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning

Yatin Dandi, Matteo Vilucchio, Hugo Tabanelli, Florent Krzakala

arXiv:2610.32849v1cs.LGcond-mat.dis-nnstat.ML

TL;DR

Forward-only spectral learning cannot use features learned above to refine earlier feature selection, leaving a central question about how upper-layer learning changes lower-layer discovery. This paper derives and implements backward-coupled spectral learning, then reports hidden-feature recovery in a hierarchical model and improved accuracy over forward Neural LoFi on binary CIFAR-10 at matched widths.

  • Problem

    Forward Neural LoFi selects features layer by layer, so features learned above cannot refine the lower directions selected earlier.

    Source: §1 · pp. 1–2

  • Method

    The paper couples a neuron's incoming row and next-layer column, then uses upper sensitivities as effective labels for backward spectral corrections.

    Source: §1 · p. 2, §2 · p. 4

  • Results

    Backward coupling recovers hidden lower directions in a hierarchical model and improves accuracy over forward Neural LoFi on binary CIFAR-10 at matched widths.

    Source: §1 · pp. 2–3, §5 · p. 13

  • Takeaways & Limitations

    The results illustrate that upper-layer information can revise earlier representations in the hierarchical model and image experiments reported.

    Source: §1 · pp. 2–3

    The local coupling analysis assumes smooth activations and fixed surrounding layers, while the three-layer evolution assumes fixed filtered dimensions and Gaussian lifts.

    Source: §2 · p. 4, §3 · p. 8

Abstract

from arXiv · show

During joint training, backpropagation allows later layers to reshape earlier representations, a feedback mechanism absent from purely layer-wise learning. We extend Neural Low-Degree Filtering (Neural LoFi), a recently proposed layer-wise spectral surrogate for feature learning, to capture this interaction. Starting from the exact backpropagation gradients, we derive a local coupling between a neuron's incoming weights and its corresponding next-layer column. In a controlled small-initialization regime with smooth activations and fixed surroundings, these dynamics amplify singular modes of a cross-layer correlation operator. Each growing mode pairs an earlier feature with an upper-layer response through shared neuron coefficients. This motivates a backward-coupling algorithm in which upper-layer sensitivities generate sample-dependent effective labels for earlier spectral steps, while corrected features refine upward transmission. A function-space variational principle characterizes the selected feature-response pairs by maximizing the label correlation of their product under kernel-norm constraints. For three-layer networks, we derive self-consistent continuous-time dynamics for the filtered hidden-layer features, the activation kernels, and the network output under the proposed backward coupling. In a hierarchical multi-index model, we establish a feature-recovery separation: backward coupling recovers lower-layer directions invisible to the original forward label-weighted statistic. The linear specialization recovers residual-dependent singular-mode dynamics, while paired modes provide a dynamical mechanism for linked low-rank covariance components consistent with the rainbow-network picture. Experiments with fully connected and convolutional networks on binary CIFAR-10 show improved accuracy over forward Neural LoFi at matched widths.

1 Introduction

Forward Neural LoFi selects features layer by layer, so an upper representation cannot refine a lower feature choice that could benefit from it. This paper extends that spectral approach with backward feedback, analyzes paired feature–response learning, and reports hidden-feature recovery and improved binary CIFAR-10 accuracy over forward Neural LoFi at matched widths.

1 Introduction

  • Upper-layer responses can change which lower features are selected, addressing the feedback absent from forward-only Neural LoFi.The approach couples a neuron's incoming row to the next-layer column that reads its activation.Source: §1 · p. 2
  • BackLoFi uses upper-layer sensitivities as sample-dependent effective labels for earlier spectral steps, then uses corrected activations to guide upward transmission.Its feature–response selection maximizes the label correlation of their product under kernel-norm constraints.Source: §1 · p. 2
  • In a hierarchical multi-index model, backward coupling reveals hidden lower directions absent from the original label-weighted second moment.Modulating by a sensitivity of the upper conditional predictor creates signal in the hidden block.Source: §1 · p. 2
  • For three-layer networks, the paper proves convergence to self-consistent feature and kernel dynamics that retain earlier column corrections as lower features evolve.The account generalizes the cited linear-network analysis of Saxe et al. (2014) to nonlinear networks.Source: §1 · pp. 2–3
  • Shared neuron coefficients link low-rank covariance components in adjacent weight matrices, providing a local dynamical mechanism for the rainbow-network structure.The paper relates this mechanism to Guth et al. (2024).Source: §1 · p. 3
  • Fully connected and convolutional networks on binary CIFAR-10 show improved accuracy over forward Neural LoFi at matched widths.The experiments test whether using upper-layer information to revise earlier representations is useful beyond the controlled theoretical setting.Source: §1 · p. 3

2 Neural Low-Degree Filtering with backward coupling

The section derives how joint training couples each neuron's incoming row and outgoing column, then uses that mechanism to motivate BackLoFi's top-down spectral corrections. It also relates paired modes to structured adjacent-layer covariances and to singular-mode dynamics in linear networks.

2 Neural Low-Degree Filtering with backward coupling

  • The incoming row and outgoing column update each other: the column sets the sample-dependent gain for the row, while the row's activation determines the column update.In layer-wise Neural LoFi, the upper readout is effectively sample-independent and rows evolve independently under label-weighted moments.Source: §2 · pp. 3–4
  • With vanishing initialization, fixed surroundings, and smooth activations, adjacent-layer updates amplify singular modes of a cross-layer correlation operator at small times.Each growing mode pairs a lower feature with an upper response, with initialization-dependent coefficients shared across neurons in both weight factors.Source: §2 · p. 4
  • BackLoFi sweeps layer boundaries from top to bottom, using upper sensitivities as effective labels and corrected activations to estimate upward transmission.The algorithm's use of labels instead of residuals is a design choice justified near initialization.Source: §2 · pp. 4–5
  • Shared neuron coefficients link the covariances of consecutive layers, providing a local mechanism for the rainbow-network structure described by Guth et al. (2024).The selected mode matrices are linked through the neurons shared by adjacent layers.Source: §2 · pp. 6–7
  • For aligned singular directions in deeper linear networks, end-to-end mode strength follows a dimension-dependent amplification law; the nonlinear local counterpart uses cross-layer correlation modes.For D = 2, growth is exponential at small mode strength before saturation; for D > 2, additional powers change early amplification.Source: §2 · p. 7

3 Lessons from learning with backward coupling

Backward coupling selects lower-feature and upper-response pairs by their product's label correlation, then follows the resulting representation changes. In a continuous three-layer limit, the paper derives a self-consistent evolution in which features, kernels, coupling, response, and output evolve together.

3 Lessons from learning with backward coupling

  • Backward coupling selects a lower feature and an upper response whose product is maximally correlated with the label under their respective kernel-norm constraints.Neither feature needs to be predictive on its own; the paired updates let the lower cue modulate where the upper response is expressed.Source: §3 · p. 7
  • For a continuous three-layer variant, discrete updates converge in sequential infinite-width limits to self-consistent dynamics for filtered features, coupling, response, and output.The selection pair is determined by the current sensitivity and input kernels.Source: §3 · pp. 8–9
  • As representations and sensitivities evolve, the selected lower feature changes the upper kernel and response, so the feature-selection geometry evolves during training.Earlier column updates are retained in the accumulated coupling function.Source: §3 · p. 9
  • An upper response can make a lower feature detectable through correlation with the modified label even when its correlation with the label alone is weak.The passage describes reliable selection when the leading population singular-pair gap dominates empirical operator error.Source: §3 · p. 9

4 Mechanistic Numerical Experiments

The experiments examine whether backward information unlocks features hidden from forward layer-wise learning and whether backward correction improves matched-width image classification.

Synthetic setting

  • Forward LoFi recovers visible U, then Q and Z from its quadratic representation, but cannot access hidden hV and hW; an upper-predictor sensitivity supplies the missing signal.The informative backward score has a nonzero linear component along Z, making the hidden interaction visible to a corrected lower-level pass.Source: §4 · pp. 10–11
  • Both spectral methods recover the visible anchor, while backward correction also recovers hidden blocks V and W, as do joint-training dynamics relative to layer-wise counterparts.Figure 2 plots test MSE and overlaps with the visible and hidden blocks against sample exponent α.Source: Figure 2 · p. 11, §4 · p. 11
  • For ϵ = 1/2, predicted characteristic scales are αU = 5/2 and αBack = 3, with the d^3 prediction conditional on a d^-1/2 informative cross-layer gap.Conditional on an informative backward score, hidden-subspace recovery requires d^(2+ϵ) samples.Source: §4 · p. 11

Image classification

  • BackLoFi improves on forward Neural LoFi at every training-set size in both architectures; on the convolutional network, 5,000 images comes within 0.15 points of forward Neural LoFi trained on 50,000.It matches gradient descent through n = 5,000 on the fully connected network and leads by two points at n = 500 on the convolutional network; gradient descent pulls ahead only on the largest training sets.Source: §4 · p. 12, Figure 3 · p. 12
  • The exact and first-order backward signals produce nearly the same curves; the first-order signal retains one score per selected mode.Those sample-wise scores summarize the sensitivity passed down from upper layers.Source: §4 · p. 12
  • Over most of the sample range, a few low-rank couplings between layer rows and next-layer columns reproduce the joint-training effect without gradient steps, using upper-layer scores.The paper identifies these paired modes as what joint training adds to layer-wise learning.Source: §4 · pp. 12–13

5 Conclusion

Backward Neural LoFi couples lower features to upper responses, with local paired-mode amplification and a three-layer self-consistent dynamics. The conclusion reports hidden-feature recovery and matched-width image-accuracy improvements, while leaving repeated corrections and finite-sample guarantees open.

5 Conclusion

  • The conclusion reports improved accuracy over forward learning at matched widths and hidden-feature discovery in the hierarchical experiment, while identifying open guarantees for repeated corrections and finite samples.Finite-sample guarantees for the implemented algorithm with learned upper scores remain open.Source: §5 · p. 13

A Further related works

  • Backward coupling is related to upper-layer feature correction, but specifically uses the upper response as a modified label to alter which lower spectral directions are visible.The discussion also places this transformation among multi-pass learning approaches.Source: A Further related works · p. 20
  • The cross-layer matrix Aℓ correlates two representations through labels, and its leading singular pairs select the coupled features.This contrasts with label-weighted second-order spectral operators discussed in the related-work survey.Source: A Further related works · pp. 20–21
  • The related-work discussion distinguishes backward coupling by linking consecutive-layer covariances and tying adjacent-layer rows and columns through shared neurons.The rainbow model instead assumes layer covariances are independent up to rotation.Source: A Further related works · p. 21

B Coupled mode amplification: proof of Theorem 1

  • Theorem 1's local analysis shows paired incoming rows and outgoing columns share a growing singular-mode component under vanishing initialization and fixed surrounding layers.The conclusion of the formal statement bounds the column's growth from its smaller initialization scale during the stated interval.Source: B Coupled mode amplification: proof of… · p. 21, §B.1 · p. 23

B.1 Assumptions and formal statement

  • The formal result assumes joint equal-rate training of one row-column pair with all other weights frozen, smooth activation, and layer-wise vanishing initialization.The theorem also assumes the leading correlation does not cancel.Source: §B.1 · p. 22

B.2 Proof

The proof controls the replacement of exact gradients by a fixed linear row-column system and bounds its singular-mode evolution. It also specifies the continuous-time three-layer updates used for the convergence analysis.

B.2 Proof

  • The proof replaces activation terms, upper sensitivity, and residuals by leading approximations, with separate remainder bounds controlling the resulting linear system.The error bounds cover Taylor remainders, frozen sensitivity, and replacing the residual by the label.Source: §B.2 · pp. 24–25
  • The fixed linear system amplifies paired singular modes: row and column projections share a growing component and have opposite-sign decaying components.For a zero singular value, both projections remain constant.Source: §B.2 · p. 25
  • Theorem B.1 controls the nonlinear trajectory's deviation from the linear approximation over the stated interval, with tighter column bounds on a shorter interval.The proof integrates the remainder using the linear growth bound.Source: §B.2 · p. 25

C Proof of Theorem 2

  • The continuous three-layer variant applies the same step size to row correction, column correction, and readout updates before taking the time-step limit.A common positive rate rescales time; unequal rates change the relative dynamics.Source: C Proof of Theorem 2 · p. 26
  • At each step, the algorithm selects a leading singular pair from the label-weighted upper sensitivity and input-feature correlation operator.It computes the upper sensitivity as g2 = w3σ′(h2), then forms the paired scores ψ and χ.Source: C Proof of Theorem 2 · p. 26

C.1 Full description of the Theorem and the underlying order parameters

The theorem describes a continuous three-layer system in which a variationally selected feature-response pair updates lower features and accumulated upward coupling. Under spectral-gap and width assumptions, the algorithmic interpolants and sequential width limits converge to the stated evolution.

C.1 Full description of the Theorem and the underlying order parameters

  • The state tracks filtered features, a coupling function for accumulated columns, readout sensitivity, and output to determine evolving feature and sensitivity kernels.The coupling acts on current lower activations, while output alone does not determine sensitivity when hidden features change.Source: §C.1 · pp. 26–27
  • At each time, the selected upper response and lower feature maximize label correlation of their product under unit kernel-norm constraints.The coupled singular-vector equations express each selected function through the current upper and input kernels.Source: §C.1 · p. 27
  • The lower filter follows the selected feature, while accumulated column corrections and readout gradient flow update upward transmission and output.The forward filter refreshes upper-layer features as the lower kernel changes.Source: §C.1 · p. 27
  • The forward filter selects a first feature by first-order correlation and later coordinates by successive second-order correlation, with orthogonality measured in the current kernel's RKHS.The stated activation assumption requires bounded σ, σ′, and σ′′.Source: §C.1 · pp. 27–28
  • With spectral gaps, the piecewise-linear algorithm converges uniformly on [0,T] as dt tends to zero, and the continuous flows converge almost surely under the stated sequential width limits.The limits take p1 to infinity first and p2 second.Source: §C.1 · p. 28

C.2 Proof of Theorem C.1

The proof closes the three-layer dynamics by tracking accumulated column coupling, selected feature pairs, readout sensitivity, and output. Regularity and stability arguments establish the continuous-time and sequential width convergence claims.

C.2 Proof of Theorem C.1

  • Each column update adds a rank-one contribution to the coupling kernel, whose action on moving lower features retains both new columns and earlier corrections.The product derivative accounts for the changing lower-feature curve.Source: §C.2 · pp. 28–29
  • A separated leading singular value makes the maximizing feature pair locally Lipschitz in the evolving state, supporting local existence and uniqueness of the closed dynamics.The proof uses the spectral gap to control dependence of the singular vectors on the state.Source: §C.2 · pp. 28, 29, 32
  • The readout response changes through both readout learning and movement of the upper preactivation, and these changes update the next selected feature pair.The response retains the product form of readout weights and activation derivative.Source: §C.2 · pp. 30–31
  • The algorithm's one-step updates approximate the continuous equations with O(dt^2) local error and O_T(dt) uniform error over [0,T].Convergence transfers to filtered features, output, kernels, and selected scores.Source: §C.2 · pp. 32–33
  • Coupling finite-width and Gaussian lifts on shared probability spaces yields almost-sure convergence, first as p1 grows with p2 fixed and then as p2 grows.The limit requires the stated admissibility and spectral-gap conditions.Source: §C.2 · pp. 33–34

D The exact linear reduction

The linear reduction shows how an incoming row and its outgoing column are coupled by the residual input–output correlation, while the staircase analysis studies how sensitivity-weighted labels recover hidden directions missed by forward selection.

D The exact linear reduction

  • Near zero, the exactly coupled incoming-row and outgoing-column updates amplify shared singular modes; prediction supplies saturation as the network fits the target.The linear result assumes linear activations, unit-rate squared-loss gradient flow, and whitened inputs.Source: D The exact linear reduction · p. 34
  • For more than two aligned layers, early mode growth follows a higher-power law, so the two-layer constant-rate exponential approximation does not describe simultaneous deep-layer growth.For D > 2, the early growth law is ˙m_i ≃ D s_i m_i^(2−2/D).Source: D The exact linear reduction · p. 35

E The Deep Staircase: A Theoretical Setting

  • The staircase analysis derives population mechanisms and sample-complexity scalings, but does not provide full finite-sample proofs.The authors say rigorous recovery guarantees could combine these population calculations with concentration and spectral-perturbation arguments.Source: E The Deep Staircase: A Theoretical… · p. 35

E.1 Teacher and high-dimensional regime

  • The hidden-subspace sample-complexity setting assumes a full-rank hidden coupling with balanced singular values, excluding directions whose interaction strength vanishes asymptotically.This assumption fixes the coupling’s Frobenius norm at Θ(1).Source: §E.1 · p. 36

E.2 BackLoFi in the deep-staircase setting

  • In the staircase procedure, upper-layer scores become effective labels for a second lower covariance pass; the recovered directions correct the first-level representation before upper features and the predictor are refit.This setting retains only BackLoFi’s effective-label channel, not its paired column correction.Source: §E.2 · p. 37

E.3 The staircase mechanism in population

In the population staircase model, ordinary forward statistics recover visible directions but miss the hidden block. Sensitivity-weighted labels create a hidden covariance signal, allowing the corrected representation to carry an additional prediction channel.

E.3 The staircase mechanism in population

  • The first forward operator has signal on hU, while its entire hV ⊕ hW block vanishes.The ordinary upper forward step then identifies Q and Z from the visible representation.Source: §E.3 · p. 39
  • The upper sensitivity’s odd-in-Z response is the component that can produce a nonzero hidden coupling; the even response cannot.The hidden coefficient b(Q, Z) is itself odd in Z.Source: §E.3 · p. 40
  • A retained score with a nonzero odd component creates a covariance spike on the hidden V, W subspace that is absent from the ordinary forward operator.The lower covariance is rerun with the effective label y s_i.Source: §E.3 · p. 41
  • With full-rank C, the leading pV + pW hidden eigendirections recover the joint hV ⊕ hW subspace, though not separate V and W bases.The recovered hidden directions are appended to the visible representation.Source: §E.3 · p. 41
  • After the visible Q direction is retained, the corrected upper span can include Z and T, whose interaction ZT supplies a nonzero low-degree prediction channel for the hidden branch.This result assumes a spectral gap for the corrected upper step.Source: §E.3 · pp. 41–42

E.4 Finite-sample scaling and the learned-score bottleneck

The section estimates stagewise sample scales for quadratic-feature recovery and identifies learning the backward score as the additional bottleneck. Its implemented algorithm replaces low-importance columns with read and paired directions, while the scaling analysis leaves several concentration and spectral-isolation conditions unresolved.

E.4 Finite-sample scaling and the learned-score bottleneck

  • The finite-sample analysis identifies effective dimensions and spectral gaps but does not prove sharp concentration for the complete learned-score pipeline.The Gaussian approximation also does not bound the replacement error when all latent coordinates enter the cross-layer row.Source: §E.4 · pp. 42–43
  • For fixed representations and scores, eigenspace recovery follows an effective-dimension and population-gap condition; reusing samples that fitted them needs separate dependence control.The concentration argument assumes independent samples and negligible truncation bias.Source: §E.4 · p. 42
  • Hidden-subspace recovery requires d^(2+ε) samples once an informative backward score is available, while learning that score adds a bottleneck.The score-selection prediction uses a d^-1/2 cross-layer contribution and requires it to remain spectrally isolated in the complete matrix.Source: Table 1 · p. 45, §E.4 · p. 45

F The implemented algorithm

  • At fixed widths, each pass replaces the lowest-importance columns in both adjacent layers, treating forward, read, and paired columns alike.Importance is computed on the current network; the backward signal contributes its mean and sample dependence along retained boundary modes.Source: F The implemented algorithm · p. 47

F.1 Variants tested and not retained

The retained algorithm uses a set of implementation choices that differed from tested variants. At fixed width, moderate rank fractions reduce test error until about half the columns have been replaced; successive passes largely select the same read subspace.

F.1 Variants tested and not retained

  • Gate-only signals and common column scales do not improve over forward Neural LoFi; mean-field and exact signals match the first-order signal within seed noise.The first-order signal was kept because it requires no propagation through installed blocks; only per-column scaling improved over a common scale.Source: §F.1 · p. 48
  • Replacing the previous pass’s slots yields no gain beyond the first, while whole-block refitting and dense interpolation match test error within 0.1 points at every training size.These alternatives were tested on the fully connected network and not retained.Source: §F.1 · p. 48

G.1 Number of passes at fixed width

  • At rank fractions 5–12%, test error reaches its minimum after about half the filter columns are replaced, then rises quickly with further passes.At 3% error is still decreasing slowly after ten passes; at 2% it stays at the forward level.Source: §G.1 · p. 48, Figure 4 · p. 49

G.2 What successive passes select

  • More than 0.97 of each pass’s read directions lie in the earlier read span, versus at most 0.02 for a random subspace.Thus successive passes resample random features within the same read subspace, and the error depends on the total fraction replaced rather than the per-pass step.Source: §G.2 · p. 49, Figure 5 · p. 49

G.3 Damped updates

The damped update reaches essentially the same minimum test error as the fixed-width swap while applying repeated, non-replacing corrections. Its descent is slower at the smaller rank fraction, and its error rises gradually after the minimum.

G.3 Damped updates

  • 18.7%, 16.9%, 13.6%, 12.6%, and 11.0% are damped-rule minima at n = 500, 1,000, 5,000, 10,000, and 50,000, respectively.These values use rℓ = 25% of kℓ; the error decreases monotonically for about 25 passes.Source: §G.3 · p. 50, Figure 6 · p. 50
  • The damped minima are within 0.2 points of the swap minima at every listed training-set size and are never significantly below them.The swap minima are 18.7%, 16.8%, 13.4%, 12.4%, and 11.1% at n = 500, 1,000, 5,000, 10,000, and 50,000.Source: §G.3 · p. 50
  • At rℓ = 10% of kℓ, descent is slower and stops 0.2 points higher, with minima at passes 30–48 for n ≥ 5,000.The 25% rank-fraction curve rises by 0.15–0.25 points by pass 50.Source: §G.3 · p. 50
  • The damped rule applies the same direction-selection rule at every pass, with its scale equivalent to replacing β by βα/(1 − α).Positive homogeneity and the ridge readout absorbing the factor 1 − α explain this equivalence.Source: §G.3 · p. 50

G.4 Gradient-descent training curves

The section compares full-batch gradient-descent test-error curves with BackLoFi on fully connected and convolutional networks. The best learning rate depends on architecture and sample size, and the large-sample curves remain step-budget limited.

G.4 Gradient-descent training curves

  • Learning rate 0.1 is best at every n for the fully connected network; for the convolutional network, 0.003 is best through n = 5,000 and 0.005 from n = 10,000 upward.The convolutional rate 0.001 trails at every n.Source: §G.4 · p. 50, Figure 7 · p. 51, Figure 8 · p. 51
  • At large n, test error is still decreasing at the final step on both networks, so the plotted gradient-descent curves are step-budget limited.The budgets are 3,000 steps for the fully connected network and 5,000 for the convolutional network.Source: §G.4 · p. 50
  • Figure 7 plots fully connected test error against full-batch gradient-descent steps for three learning rates, with BackLoFi shown as a dotted curve.The curves report means and standard errors over five seeds.Source: Figure 7 · p. 51
  • Figure 8 plots convolutional test error against steps for three learning rates at each n, with BackLoFi shown as a dotted curve.The 0.003 and 0.005 curves average three seeds; at n ≥ 5,000 the 0.001 curve uses one seed.Source: Figure 8 · p. 51

H Hyperparameters of the figures

The section specifies protocols and hyperparameters for the paper’s experiments, including binary CIFAR-10 evaluation, model configurations, and baseline training settings.

Image-experiment protocol

  • Image experiments use binary CIFAR-10 labels, training subsamples of size n, 104 test images, and test error as the fraction of wrong signs.A ridge readout uses leave-one-out selection among 50 log-spaced penalties in [10^-6, 10^6], with selections made on seed means.Source: H Hyperparameters of the figures · p. 51

Figure 2 settings

  • The Figure 2 hierarchical model uses d = 100, sixteen orthonormalized first-level quadratic directions, and no additive label noise.Its finite-dimensional realization is (pU, pV, pW) = (6, 5, 5), corresponding to ε = 1/2 scaling.Source: Figure 2 · p. 51

Gradient-descent baseline

  • The gradient-descent baseline uses two trainable ReLU hidden layers of width 512 after a frozen lower random-feature map.End-to-end training updates both hidden layers jointly; layer-wise training splits 4,000 updates between the two layers and freezes the first during the second stage.Source: H Hyperparameters of the figures · p. 52

Figure hyperparameters and reference values

  • At n = 50,000, BackLoFi test error is 11.1% for the fully connected network and 6.7% for the convolutional network.The corresponding FwdLoFi values are 12.3% and 7.8%, respectively.Source: H Hyperparameters of the figures · p. 52

Damped-update settings

  • The experiment used P = 1–50 passes, β = 0.25, filter widths kℓ ∈ {32, …, 2,048} selected best per point, n ∈ {500, 1k, 5k, 10k, 50k}, and 10 seeds.The forward scale τ entering Γ was measured with forward-phase filters on current features. The swap reference was the 4-pass arm of Figure 3 at its best rℓ for each n.Source: H Hyperparameters of the figures · p. 53

Related papers

Loading 2610.32849v1…