PaperlayerView exact revision on arXiv

Source-linked AI summary

A Fine-Grained Spectral Perspective on Neural Networks

Greg Yang, Hadi Salman

arXiv:1907.10599v4cs.LGcs.NEstat.ML

TL;DR

The paper asks whether neural networks universally favor simple functions and how depth, training scope, and learning-rate tuning affect learning. It analyzes CK and NTK spectra and tests spectrum-based predictions. It finds simplicity bias is not universal, while depth and training choices have feature-dependent effects.

  • Problem

    The paper asks whether the simplicity bias observed for ReLU networks holds across architectures, and how network hyperparameters affect learning and generalization.

    Source: §1 · pp. 1–2

  • Method

    The authors analyze CK and NTK spectra, use fractional variance as a heuristic for kernel effects, and test predictions with neural-network experiments.

    Source: §1 · p. 2

  • Results

    Simplicity bias is not universal, and deeper networks can help with complex features while excess depth can hurt; training all layers favors complex features while last-layer-only training favors simpler ones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The spectral perspective offers a first step toward understanding how hyperparameters affect neural-network initialization and generalization, though fractional variance remains a rough indicator.

    Source: §8 · p. 8

    The predicted maximum learning rate concerns infinite-width networks in the NTK regime and is not necessarily the finite-width network's maximum learning rate.

    Source: §8 · p. 8

Abstract

from arXiv · show

Are neural networks biased toward simple functions? Does depth always help learn more complex features? Is training the last layer of a network as good as training all layers? How to set the range for learning rate tuning? These questions seem unrelated at face value, but in this work we give all of them a common treatment from the spectral perspective. We will study the spectra of the *Conjugate Kernel, CK,* (also called the *Neural Network-Gaussian Process Kernel*), and the *Neural Tangent Kernel, NTK*. Roughly, the CK and the NTK tell us respectively "what a network looks like at initialization" and "what a network looks like during and after training." Their spectra then encode valuable information about the initial distribution and the training and generalization properties of neural networks. By analyzing the eigenvalues, we lend novel insights into the questions put forth at the beginning, and we verify these insights by extensive experiments of neural networks. We derive fast algorithms for computing the spectra of CK and NTK when the data is uniformly distributed over the boolean cube, and show this spectra is the same in high dimensions when data is drawn from isotropic Gaussian or uniformly over the sphere. Code replicating our results is available at github.com/thegregyang/NNspectra.

1. Introduction

The paper uses the CK and NTK spectra to study neural-network initialization and training, addressing simplicity bias and how hyperparameters affect generalization. It reports that simplicity bias is not universal and that depth and training choices affect which features networks learn.

1. Introduction

  • For wide networks, the CK describes initialization and last-layer training, while the NTK describes training all parameters and the network after training.The paper uses these kernels to examine hyperparameter effects on generalization.Source: §1 · p. 1
  • The paper develops fast CK and NTK spectrum algorithms on the Boolean cube and reports similar spectra in high dimensions for Boolean-cube, spherical, and standard-Gaussian data.Source: §1 · p. 2
  • Simplicity bias is not universal: it appears more or less for ReLU networks but can be made arbitrarily weak in sigmoidal networks by changing weight variance and depth.In the extreme, a deep erf network with large weights produces random functions distributed like white noise; the remaining weak ordering is that more complex eigenspaces cannot have larger eigenvalues than simpler ones.Source: §1 · p. 2
  • The paper proposes fractional variance—the share of a kernel's trace contributed by an eigenspace—as a heuristic for predicting how kernel changes affect learning performance.An eigenspace with large fractional variance is expected to make a ground-truth function from that space easier to learn.Source: §1 · p. 2
  • Deeper networks can learn more complex features, but excess depth can be detrimental; training all layers favors complex features, while last-layer-only training favors simpler ones.These predictions are motivated by fractional-variance differences in the NTK and CK and are verified experimentally.Source: §1 · p. 2
  • The authors use CK and NTK spectra to predict SGD's maximal nondiverging learning rate.Source: §1 · p. 2

2. Kernels Associated to Neural Networks

The section presents CK and NTK as kernels associated with wide neural networks and explains how their spectra relate to initialization, training, and generalization. It frames kernel alignment with target functions through eigenvalues and eigenfunctions.

2. Kernels Associated to Neural Networks

  • The CK is the kernel associated with the random network's last-hidden-layer embedding, and its Gaussian-process limit describes the network at initialization.Training a linear model on that embedding corresponds to the CK.Source: §2 · pp. 2–3
  • The NTK corresponds to training all parameters and, in the infinite-width limit, describes the linearized model formed from gradients at initialization.The CK and NTK therefore correspond to different training settings.Source: §2 · p. 3
  • CK spectra inform the initialization prior, while NTK spectra inform the implicit bias of gradient-descent training.Source: §2 · p. 3
  • Kernel suitability for a target function can be assessed through alignment between kernel eigenvalues and the squared coefficients of the target's expansion in the eigenfunction basis.The paper studies the cleaner case where the target is an eigenfunction and seeks a kernel with a large corresponding eigenvalue.Source: §2 · p. 3
  • The paper calls kernels whose values depend on the two inputs through a function of their pairwise kernel quantities “Neural Kernels.”Source: §2 · p. 4

3. The Spectra of Neural Kernels

The section characterizes top unique eigenvalues of neural kernels across several high-dimensional input distributions and develops the Boolean-cube spectral description. On the Boolean cube, the Fourier basis diagonalizes these kernels, with eigenvalues determined by the kernel function.

3. The Spectra of Neural Kernels

  • For fixed k and sufficiently large dimension, the kth largest unique eigenvalue corresponds to an eigenspace of dimension d^k/k! + o(d^k), with the stated high-dimensional limit applying across Boolean-cube, sphere, and standard-Gaussian data.The paper focuses on these top unique eigenvalues, noting that their eigenspaces can be very large relative to available dataset sizes.Source: §3 · p. 4

3.1. Boolean Cube

  • On the Boolean cube, every Fourier character χS is an eigenfunction of a neural kernel, with an eigenvalue depending only on the size of S.These characters form the complete eigenfunction set by dimensionality considerations.Source: §3.1 · p. 5
  • The Boolean-cube eigenvalue is defined by averaging the kernel against the corresponding parity character, yielding a quantity that depends only on the subset size.A shift-operator reformulation is noted as useful for computation.Source: §3.1 · p. 5

4. Clarifying the “Simplicity Bias” of Random Neural Networks

The spectral account shows that simplicity bias depends on network nonlinearity, sampling variances, and depth, rather than holding uniformly. Although eigenvalues obey a weak ordering within even and odd degrees, their variance shares can still be nearly uniform.

4. Clarifying the “Simplicity Bias” of Random Neural Networks

  • Erf networks’ simplicity bias weakens as weight variance rises, while depth and sampling variances also affect whether random networks favor simple functions.The experiments report that for erf with σ2 w = 4, no sampled function repeats.Source: §4 · pp. 5–6
  • A 32-layer erf network with σ2 w = 4 has equal nonzero odd-degree eigenvalues, yielding white noise over odd functions and no simplicity bias beyond oddness.By contrast, for relu with σ2 b = 2, the constant-function eigenspace accounts for more than 80% of variance.Source: §4 · p. 6
  • CK and NTK eigenvalues are nonincreasing separately across even and odd degrees, but this ordering does not require degree-wise variance shares to decrease.The weak ordering does not rule out white-noise behavior, as in deep erf CKs with large σ2 w.Source: §4 · p. 6

5. Deeper Networks Learn More Complex Features

The paper studies how network depth and kernel spectra relate to learning functions of different degrees. Optimal depth tends to rise with degree, but excessive depth can hurt performance; spectral theory also predicts maximal learning rates across several data distributions.

5. Deeper Networks Learn More Complex Features

  • Degree-k fractional variance is inversely related to validation loss when learning degree-k polynomials, but it does not precisely predict that loss.The paper leaves finding a better generalization measure for future work.Source: §5 · p. 6
  • Higher-degree eigenspaces generally attain their greatest fractional variance at greater depths, for relu and for erf when comparing degrees of the same parity.For erf, the depth effect is stronger than for relu; at higher degrees, the gap between maximal and nearby fractional variances narrows.Source: §5 · p. 6
  • Across nonlinearities and hyperparameters, NTK fractional variance tends to exceed CK at larger degrees, while CK tends to be higher at degree 0 or 1.The fraction of hyperparameters with greater NTK than CK fractional variance rises with degree.Source: Figure 4 · p. 7
  • For last-layer MLP training, theory predicts empirical maximum learning rates highly accurately using the MLP CK and 1/Φ(0); for all-layer training, NTK-based predictions are less accurate but highly correlated and generally underpredict.Source: §5 · p. 7
  • The optimal depth for learning a polynomial increases with its degree, and past that depth, greater depth can harm performance, including on MNIST and CIFAR10.The increasing optimal-depth pattern is also observed with standard Gaussian and uniform-sphere inputs.Source: §5 · p. 8

6. NTK Favors More Complex Features Than CK

The paper compares learning complex features by training all network parameters with NTK dynamics versus training only the last layer with CK dynamics. Experiments support an advantage for all-layer training on more complex targets, while the spectral measure remains an imperfect performance predictor.

6. NTK Favors More Complex Features Than CK

  • NTK fractional variance is generally higher at larger degrees and CK fractional variance at smaller degrees, suggesting all-layer training favors complex features while last-layer training favors simpler ones.Experiments find last-layer training meaningfully better only for constant targets.Source: §6 · p. 8
  • The comparison persists for standard Gaussian and uniform-sphere inputs, but fractional variance does not perfectly predict performance.For erf networks with k ≥1, NTK fractional variance is not always greater than CK’s, yet last-layer training is not observed to outperform all-layer training.Source: §6 · p. 8
  • Training a large set of networks on homogeneous polynomial targets is used to test the predicted difference between last-layer and all-layer training.The comparison is reported in Fig. 3(c).Source: §6 · p. 8

7. Predicting the Maximum Learning Rate

The paper uses kernel spectra to predict SGD’s maximal nondiverging learning rate, addressing the practical challenge of setting the upper limit for learning-rate searches. Its prediction is accurate across toy distributions and real datasets.

7. Predicting the Maximum Learning Rate

  • The spectral prediction is intended to set the correct upper limit for a learning-rate search, a practical challenge in deep-learning benchmarks.The paper studies the maximal nondiverging learning rate over real datasets and toy input distributions.Source: §7 · p. 8
  • For full-batch gradient descent, the kernel dynamics converge only when α < (max_i λ_i)^-1.K is the CK for last-layer-only training and the NTK for training all parameters.Source: §7 · p. 8
  • The theoretical prediction 1/Φ(0) is accurate for the maximum learning rate on the Boolean cube, sphere, Gaussian, MNIST, and CIFAR10.On the Boolean cube, the maximum is max(µ0, µ1), with µ0 ≈ Φ(0) and µ1 ≪ µ0 in large dimensions.Source: §7 · p. 8

8. Conclusion

The paper presents spectral analysis of neural kernels as an initial step toward understanding how hyperparameters affect neural-network initialization and generalization, while treating fractional variance as only a rough predictor of test loss. It situates this work alongside Gaussian-process, signal-propagation, and other spectral perspectives.

8. Conclusion

  • The authors treat fractional variance as a rough indicator and call for more precise predictions of test loss.They frame their study of how hyperparameters affect initialization and generalization through neural-kernel spectra as a first step.Source: §8 · p. 8

A. Related Works

  • Gaussian-process and Neural Tangent Kernel work established infinite-width descriptions of neural networks and also informed new models and algorithms.The paper situates its work within this line of research, including Neal (1995), Lee et al. (2018), and Jacot et al. (2018).Source: A. Related Works · p. 13
  • Signal-propagation studies correspond to infinite-depth limits of CK for forward signals and NTK for backward gradients.Related results include training a 10,000-layer CNN and batch normalization causing gradient explosion (Yang et al., 2019).Source: A. Related Works · p. 13
  • Random-matrix studies use free probability to compute singular-value distributions, while Tensor Programs unify GP, NTK, signal-propagation, and random-matrix perspectives.Tensor Programs also permit mechanical computation of many infinite-width limits, including for recurrent neural networks (Yang, 2019a;b).Source: A. Related Works · p. 13
  • Unlike prior real-frequency studies of ReLU networks, this work computes all NTK eigenvalues and eigenfunctions and thereby complements their low-frequency findings.It also contrasts with Fisher-spectrum work that computes summary statistics; the nonzero Fisher eigenvalues are shared with the NTK.Source: A. Related Works · p. 13
  • Work on one-hidden-layer networks over the sphere uses spherical harmonics, whereas the Boolean cube focus enables fine-grained study of hyperparameter effects on spectra.The paper relates this focus to Smola et al. (2001) and concurrent studies by Xie et al. (2016) and others.Source: A. Related Works · p. 13

B. Universality of Our Boolean Cube Observations in Other Input Distributions

The paper examines whether its Boolean-cube observations extend to Gaussian, spherical, and real-data inputs, finding optimal depths across these settings. It also tests maximum-learning-rate predictions under different image preprocessing choices.

B. Universality of Our Boolean Cube Observations in Other Input Distributions

  • The study does not test simplicity bias on uncountable-support inputs, and feature complexity is unclear for MNIST and CIFAR10.The authors therefore do not test whether optimal depth increases with feature degree on those datasets.Source: B. Universality of Our Boolean Cube… · pp. 14–15
  • For sphere and Gaussian inputs, optimal depth exists for each polynomial degree and increases with degree, matching the Boolean-cube pattern.Validation-loss curves for degrees 0–2 are very close across distributions; degree 3 agrees less precisely but follows the same overall trend.Source: B. Universality of Our Boolean Cube… · pp. 13–14, Figure 6 · p. 14
  • Training only the last layer works better for degree-0 polynomials, while training all layers fares better for higher-degree polynomials on sphere and Gaussian inputs.This repeats the Boolean-cube pattern.Source: B. Universality of Our Boolean Cube… · pp. 13–14
  • On MNIST and CIFAR10, test error is best around depth 5 when training all layers, while last-layer-only training is best at depth 1.For both training regimes, performance worsens for depths shallower or deeper than the optimum.Source: B. Universality of Our Boolean Cube… · pp. 14–15, Figure 7 · p. 15

C. Theoretical vs Empirical Max Learning Rates under Different Preprocessing for MNIST and CIFAR10

  • With MNIST and CIFAR10 centered and normalized to the sphere, the theory accurately predicts the maximum learning rate in practice.The reported experiment compares this preprocessing with alternatives.Source: C. Theoretical vs Empirical Max… · p. 15
  • With PCA, ZCA, or no preprocessing, the theoretical maximum learning rate is generally less accurate but remains correlated; ReLU on PCA- or ZCA-preprocessed CIFAR10 is an apparent exception.The theoretical prediction is 1/Φ(0).Source: C. Theoretical vs Empirical Max… · p. 15

D. Visualizing the Spectral Effects of σ2

The contour plots map how CK and NTK fractional variances change with depth and weight or bias variance. Across the sphere and Gaussian, last-layer training favors low-degree polynomials, while all-layer training favors higher-degree ones.

ReLU contours

  • For ReLU CK, fractional-variance contours vary with depth and bias variance while fixing σ2_w = 2; ReLU NTK uses the same fixed σ2_w for consistency.The CK fractional variances depend only on the ratio of weight and bias variances, unlike those for ReLU NTK.Source: D. Visualizing the Spectral Effects of… · pp. 15, 17

Erf contours

Figure 8

  • Training only the last layer works better for degree-0 polynomials, while training all layers fares better for higher degrees on the sphere and standard Gaussian.This repeats the pattern observed over the Boolean cube, using the same experimental setup with changed input distributions.Source: Figure 8 · p. 16

Figure 9

  • Theoretical maximum learning rates are compared with empirical maxima across preprocessing schemes, with weaker prediction accuracy for ReLU networks on PCA- or ZCA-processed CIFAR10.The prediction remains generally correlated with empirical maxima outside that exception.Source: D. Visualizing the Spectral Effects of… · p. 16, Figure 9 · p. 16

3D contours

  • For higher-degree features, the highest fractional-variance contours occur at an intermediate depth, indicating that neither especially shallow nor especially deep networks are optimal.The plots show level surfaces in depth and σ2_b; higher-degree contours can peak away from the depth extremes.Source: D. Visualizing the Spectral Effects of… · p. 20, Figure 13 · p. 20

E. Experimental Details

The experimental appendix specifies the polynomial-learning and maximum-learning-rate setups, then gives the infinite-width convergence results underpinning CK and NTK. It tests the theories on synthetic distributions and image datasets, with the setup details varying by experiment.

E.1. Fig. 3

  • In the Fig. 3 experiments, ReLU MLPs learn randomly generated degree-k multilinear polynomials on 128-dimensional Boolean inputs, using fresh SGD batches and best validation loss across 100 checks.Each target combines 10 sampled degree-k monomials with normalized random coefficients; training runs for 100,000 iterations with batch size 1,000.Source: §E.1 · p. 21

Hyperparameters for Fig. 3(a)

  • Fig. 3(a) compares last-layer CK training with all-layer NTK training using 10 seeds per hyperparameter combination and half the theoretical maximum learning rate.The maximum-rate convention depends on the stated L2-loss scaling.Source: Hyperparameters for Fig. 3(a) · p. 21

Hyperparameters for Fig. 3(b)

  • Fig. 3(b) uses 100 seeds per combination and trains only the last-layer weights and bias; its empirical maximum rate is found by binary search.The search starts at 16 times the theoretical maximum rate and halves the interval until its width is within 1% of that maximum.Source: Hyperparameters for Fig. 3(b) · pp. 21–22

Hyperparameters for Fig. 3(c)

  • Fig. 3(c) trains all layers with NTK parametrization on ground-truth polynomials of degrees 0 through 6, using one seed per combination.The experiment compares learning functions of different polynomial degrees.Source: Hyperparameters for Fig. 3(c) · p. 22

E.2. Max Learning Rate Experiments

  • The maximum-learning-rate experiments compare theoretical rates from CK for last-layer training or NTK for all-layer training with empirical rates found by binary search.For n-dimensional outputs, the theoretical maximum is nΦ(0)^−1; MNIST and CIFAR10 rates are 10 times those for the Boolean cube, sphere, and Gaussian under the stated loss scaling.Source: §E.2 · p. 22

Hyperparameters

  • The image experiments use MNIST and CIFAR10 with no preprocessing, PCA128, or ZCA128; synthetic-distribution targets are degree-1 polynomials, while image targets are one-hot dataset labels.PCA128 retains 128 dimensions; ZCA128 retains those components but rotates back to the original image dimension.Source: §E.2 · p. 23, Hyperparameters · p. 23

F.1. Convergence of Infinite-Width Kernels at Initialization

  • At infinite width, hidden-layer activations converge to Gaussian processes with CK kernels, and the corresponding NTK converges to its infinite-width kernel.The CK convergence applies to any finite input set as hidden-layer widths tend to infinity; the NTK statement is given for inputs x and y.Source: §F.1 · p. 23

F.2. Fast Evaluations of CK and NTK

This section presents fast evaluation bounds for CK and NTK and examines their spectra across the boolean cube, sphere, and isotropic-Gaussian domains. For sphere and isotropic-Gaussian inputs, it uses spherical harmonics to analyze the spectra and describes how eigenvalues simplify in high dimensions.

F.2. Fast Evaluations of CK and NTK

  • CK and NTK can both be computed in O(|X|^2 L) time, where X is the set of input points and L is the number of layers.Source: §F.2 · p. 23

F.3. Linear Evolution of Neural Network under GD

  • In the infinite-width limit, the NTK stays constant during gradient flow, while training only the last layer yields a corresponding linear evolution with the CK.The passages describe the resulting dynamics as kernel gradient descent; the constant-NTK result is attributed to Jacot et al. (2018).Source: §F.3 · p. 24

F.4. Relationship to Gaussian Process Inference.

  • With infinite training, last-layer training yields the mean GP posterior for prior N(0, Σ), while all-layer training yields the mean for prior N(0, Θ), but its final covariance is not that posterior’s covariance.The posterior relationships are attributed to Lee et al. (2019).Source: §F.4 · p. 25

G. A Brief Review of Hilbert-Schmidt Operators and Their Spectral Theory

  • A symmetric positive-semidefinite Hilbert-Schmidt kernel has eigenvalues and eigenfunctions, and boundedness suffices for the paper’s kernels to be Hilbert-Schmidt.The spectral theorem represents the kernel through these eigenpairs, with convergence in L2(X × X).Source: G. A Brief Review of Hilbert-Schmidt… · p. 25

H. Eigendecomposition of Neural Kernel on Different Domains

  • For CK and NTK of an MLP, the kth largest unique eigenvalue has an eigenspace of dimension d^k/k! + o(d^k) for sufficiently large d, on the boolean cube, sphere, or standard Gaussian domain.The result assumes a polynomially bounded activation; the threshold dimension may depend on k.Source: H. Eigendecomposition of Neural Kernel… · p. 26

H.1. Boolean Cube

  • On the boolean cube, every parity character χS is an eigenfunction of K, and its eigenvalue depends only on the cardinality of S.For fixed k, the passages also state convergence of the boolean eigenvalues as d grows, with a derivative-based expression for the limit.Source: §H.1 · pp. 26–27

H.1.1. COMPUTING THE BOOLEAN EIGENVALUES

  • Boolean-cube eigenvalues can be computed from Φ on a grid spaced 2/d apart; isolating the alternating part gives a numerically stable evaluation for CK and NTK.The passages contrast this approach with sphere and Gaussian eigenvalues, whose Gegenbauer-polynomial integrals are difficult to estimate stably and efficiently in high dimensions.Source: §H.1.1 · p. 29

H.2. Sphere

  • On the sphere, the kernel K is diagonalized by spherical harmonics.The passage introduces this established result as a basis for deriving spectral information for isotropic Gaussian distributions.Source: §H.2 · p. 29
  • As sphere dimension grows, each fixed-degree eigenvalue of an MLP’s CK or NTK simplifies to a derivative of its defining function.The theorem applies to CK and NTK kernels on the sphere.Source: §H.2 · p. 31

H.3. Isotropic Gaussian

  • For degree-s positive-homogeneous kernels, including MLP CKs with ReLU, Gaussian eigenspaces combine a radial factor with degree-l spherical harmonics and have eigenvalue λ²a_l.The radial factor is R(∥x∥²/d), and the ReLU CK is a stated example of the theorem’s scope.Source: §H.3 · pp. 32–33
  • For isotropic-Gaussian inputs, CK and NTK eigenvalues at spherical-harmonic degrees converge to derivatives of the defining function as dimension grows.The Gaussian result states convergence of the top eigenvalues in the large-dimension limit.Source: §H.3 · pp. 33–34

H.4. In High Dimension, Boolean Cube ≈Sphere ≈Standard Gaussian

In high dimensions, the studied CK eigenvalues over the boolean cube, sphere, and standard Gaussian are close for the low degrees examined. This closeness matters practically because finite training samples primarily capture the top eigenvalues, whose eigenspaces account for nearly all fractional variance.

H.4. In High Dimension, Boolean Cube ≈Sphere ≈Standard Gaussian

  • The section numerically verifies the high-dimensional eigenvalue closeness predicted by Theorem 3.1 while keeping degree k fixed as input dimension grows.The reported comparison is limited to the top, low-degree eigenvalues.Source: §H.4 · p. 34
  • By dimension 128, the erf CK eigenvalues shown for degrees k ≤ 5 are very close across the boolean cube, sphere, and standard Gaussian comparison.The figure uses σ2 b = 0.001 and depth 2; its Gaussian comparison is the sphere kernel induced by ˆΦd.Source: §H.4 · p. 34, Figure 18 · p. 35
  • For iid training samples, the finite kernel submatrix approximates the largest N eigenvalues, so smaller eigenvalues are difficult to detect and may have little effect on learning.The section notes that an N-sample dataset supplies only an N × N kernel submatrix.Source: §H.4 · pp. 34–35
  • 275,584,033 samples would be needed to detect µ6 at d = 128 and assess its possible difference from the sphere eigenvalue a6.The corresponding eigenspace dimensions grow as Θ(d^k), requiring increasing sample counts to detect differences at higher degrees.Source: §H.4 · p. 35
  • For any ϵ > 0, sufficiently large d lets some fixed degree cutoff k capture more than 1 − ϵ of total fractional variance across the cube, sphere, and standard Gaussian.The asymptotic fractional variance is determined by derivatives of Φ at t = 0.Source: §H.4 · p. 35

I. Basic Properties of Neural Kernel Functions

This appendix establishes regularity and series properties of the functions associated with neural CKs and NTKs. These properties support the paper’s spectral arguments by giving conditions for continuity, analyticity, and convergence.

I. Basic Properties of Neural Kernel Functions

  • The CK and NTK Φ functions have Taylor expansions with nonnegative coefficients that converge absolutely on [−1, 1].The proof uses induction over depth; for the NTK, products and sums of such series preserve nonnegative coefficients.Source: §I · pp. 35–36
  • For nontrivial activations, the CK and NTK function Φ is jointly continuous, analytic in t, and has derivatives continuous in its arguments.If the activation is polynomially bounded, the CK’s Φ is polynomially bounded; for the NTK, this follows when the activation derivative is polynomially bounded.Source: §I · pp. 35–36
  • For a nontrivial activation square-integrable against N(0, σ2), the normalized Gaussian activation correlation has a Taylor series whose coefficients sum to 1 and converge absolutely on [−1, 1].The expansion is obtained from the activation’s Hermite coefficients.Source: §I · p. 37
  • The normalized activation correlation ˆφ is continuous, analytic in t, and has continuous t-derivatives; polynomially bounded activations also yield polynomially bounded ˆφ.These properties are stated for activations square-integrable against the relevant Gaussian distributions.Source: §I · p. 37

J.1. Boolean Cube Spectral Theory

This section develops network-kernel spectral analysis on the Boolean cube and examines how its observations extend to Gaussian and spherical inputs. It covers cube eigenfunctions and eigenvalue ordering, fixed-degree spectral limits, and how network bias varies with activation, weight variance, and depth.

J.1. Boolean Cube Spectral Theory

  • Every Boolean character χ_S is an eigenfunction, and these form all eigenfunctions; its eigenvalue depends only on |S|.The passages give the eigenvalue definition but its displayed formula is incomplete in the supplied text.Source: §J.1 · p. 38

J.1.1. WEAK SPECTRAL SIMPLICITY BIAS

  • For MLP CK and NTK kernels on the Boolean cube, eigenvalues are nonincreasing separately across even degrees and across odd degrees.The stated orderings are μ0 ≥ μ2 ≥ … and μ1 ≥ μ3 ≥ … .Source: §J.1.1 · p. 39

We furthermore have the identity

  • For fixed k as d→∞, Theorem H.2 states a Boolean-cube eigenvalue limit involving the kth derivative of Φ.The supplied theorem statement omits the limit’s displayed formula; Φ^(k) denotes the kth derivative.Source: We furthermore have the identity · p. 40
  • The fixed-k Boolean-cube limit argument must control endpoint behavior because Φ^(k) may fail to exist at 1 or −1; the proof bounds the endpoint contribution exponentially in d.The passages state that Φ is analytic in the interior and that the endpoint contribution can be exponentially small.Source: We furthermore have the identity · p. 41

J.2. Spherical Spectral Theory

  • For spherical CK/NTK kernels, Theorem H.13 states a fixed-degree spectral limit as d→∞, with Φ^(ℓ) denoting the ℓth derivative.The supplied theorem statement omits the displayed limit formula.Source: §J.2 · p. 43

J.3. Gaussian Spectral Theory

  • In the Gaussian spectral theory, K is positive definite if and only if every kernel Aℓ in its decomposition is positive definite.The supplied passages also state a positive-semidefinite Hilbert-Schmidt decomposition criterion, but omit its displayed decomposition formula.Source: §J.3 · pp. 44–45

Subclaim 3.1

  • The stated object converges in probability—and therefore in distribution—to the delta distribution centered at (0, 1, 1).Source: Subclaim 3.1 · p. 48

K. The {0, 1}d vs the {±1}d Boolean Cube

Related papers

Loading 1907.10599v4…