PaperlayerView exact revision on arXiv

Source-linked AI summary

Population loss in shallow ReLU networks: Bias & families of critical points

Michael Field

arXiv:2609.30661v1cs.LG

TL;DR

Bias-free ReLU networks have a restricted function class, and prior results about their spurious minima do not straightforwardly extend to networks with bias. This paper derives biased population-loss and gradient formulas and reports that spurious-minimum families persist, with lower loss after bias is added. The paper’s examples and symmetry analysis have stated scope limits.

  • Problem

    Bias-free ReLU networks span a restricted function class, and their results on spurious minima cannot be extended to biased networks by simple stability arguments.

    Source: §2.1 · p. 9

  • Method

    The paper derives formulas for biased-network loss and gradients and uses symmetry and fixed-point spaces to search for critical-point families.

    Source: §1 · pp. 6–7

  • Results

    The studied spurious-minimum families persist with bias, and adding bias reduces loss, with the difference rapidly tending to zero as d grows.

    Source: §1 · pp. 7–8

  • Takeaways & Limitations

    The analyzed bias-free spurious-minimum examples extend nontrivially to biased networks, rather than disappearing in the cases studied.

    Source: §1 · p. 7

    The paper does not present a detailed analysis of the Hessian spectrum.

    Source: §12 · p. 57

Abstract

from arXiv · show

The main result presented is a formula for the population loss in the student-teacher kernel model that is applicable to shallow ReLU networks with bias. This extends previous work of Choo and Saul (2009) and Brutzkus and Globerson (2017). The formula makes essential use of Owen's T-function. The necessary theory of the T-function is given and a high precision coding using MPFR for the T-function, based on an algorithm of Komelj (2023), is available on request. It is shown that various families of spurious minima described in past papers of Arjevani and the author extend to biased networks and that the loss is always strictly decreased when bias is added. The change in landscape geometry caused by adding bias appears to be relatively mild. Only the simplest examples are described in this paper where it is assumed that the number of inputs is equal to the number of neurons (this restriction is for reasons of length). A review of relevant previous results on unbiased networks is included. Aside from Gaussian statistics, the main mathematical tools and ideas come from analytic geometry (analytic and subanalytic sets, the Curve Selection Lemma).

1. Introduction

The paper develops tools for extending results on spurious minima from bias-free to biased ReLU networks, where the correspondence is nontrivial. It derives formulas for population loss and its gradients, then examines how critical-point families behave with bias.

1. Introduction

  • Examples are restricted to k = d, and the paper does not compute Hessian spectra.The stated reason for the k = d restriction is to keep the paper to a reasonable length.Source: §1 · p. 5
  • The paper derives explicit formulas for population loss and its gradients in shallow ReLU networks with bias, building on the bias-free formula of Choo and Saul.The derivation uses Cartesian coordinates and reports cancellations of error-function and trigonometric terms in the final computation.Source: §1 · p. 6
  • The integral needed for the biased loss formula is not expressible using standard functions, but is evaluated with Owen’s T-function, inverse tangents, and an error function.A high-precision T-function computation based on Komelj’s algorithm is described.Source: §1 · p. 6
  • The paper reports that previously studied families of spurious minima persist when bias is added, while the loss decreases and the biased–unbiased difference rapidly tends to zero as dimension grows.The introduction also reports nonzero bias gradients at the corresponding zero-bias critical points.Source: §1 · pp. 7–8

2.1. Shallow Neural Networks.

  • Bias-free ReLU networks span a restricted function class, and their local minima generally are not critical points of the biased network at zero bias.Thus, persistence of spurious-minimum families cannot be inferred by simple stability arguments.Source: §2.1 · pp. 9–10

2.2. Statistical learning and a reduction.

  • The paper generalizes the Choo and Saul loss formula and analyzes the regularity of the biased population loss to extend results on spurious minima and fractional power series.The biased setting has regularity properties quite different from the bias-free case.Source: §2.2 · p. 12

3. Bias

The section sets up the biased loss as a Gaussian integral over the region where both ReLU units are active. It lays out the integral components and their evaluation, including special treatment of the angle limits.

3. Bias

  • The section gives an explicit formula for the loss contribution f(w, v) when biases are included.It follows the approach of Choo and Saul for the zero-bias case.Source: §3 · p. 12
  • The expectation is integrated over the region where both biased ReLU arguments are positive, using coordinates on the plane spanned by w and v.The section uses normalized bias parameters to simplify the computations and formulas.Source: §3 · pp. 13–14

3.1. Computation of f if the network is not biased⋆.

  • For zero bias, the Cartesian-coordinate computation evaluates integrals of x1x2 and x2^2; adding bias requires five integrals, including x1^2 and 1.The latter two integrals use Owen’s T-function.Source: §3.1 · pp. 17–18

3.2. Evaluation of the defining integrals for f(w, v): I.

  • The biased loss integral is reduced by linearity to integrals of x1^2, x1x2, x1, x2, and 1 over the active region.The section evaluates several terms using standard functions, while the integral involving 1 is not expressible in elementary functions.Source: §3.2 · pp. 18–19, §3.3 · p. 24

3.3. Evaluation of the defining integrals for f(w, v): II.

  • The section establishes continuity of the integral quantities in the bias parameters, including their limiting values as the angle approaches 0 or π.The limiting cases are treated separately from the formulas for angles in (0, π).Source: §3.3 · pp. 23, 26

4. The Owen T-function

The section introduces Owen’s T-function and the generalized T-function used to express integrals arising in the biased loss calculation. It also notes that numerical and analytical work near zero bias requires care.

4. The Owen T-function

  • Owen’s T-function is presented in terms of the standard normal distribution, with properties and integral formulas used in the paper’s calculations.The section favors error-function expressions over CDF expressions for a more concise formula suited to the applications.Source: §4 · pp. 26–27

4.1. The Generalized Owen T-function [21].

  • The generalized T-function is analytic, although its displayed decomposition includes terms that are not continuous across all of R^3.The section cautions that discontinuous terms in the formula require care in numerical simulations and studies of bifurcation from zero bias.Source: §4 · p. 27, §4.1 · pp. 28–29

5. Evaluation of the defining integrals for f(w, v): III

This section develops properties of the generalized T-function representation used in evaluating the defining integrals. It distinguishes global behavior from behavior on restricted angle ranges and at endpoints.

5. Evaluation of the defining integrals for f(w, v): III

  • The decomposition’s symmetry is consistent with the identity f(w, v) = f(v, w).Source: §5 · pp. 29–30
  • H is continuous as defined by its integral, although terms 2–5 in its inverse-tangent and Owen’s T-function decomposition may be undefined when γ = 0.Source: §5 · p. 30
  • H is analytic but not subanalytic on R^2 × (0, π), while it is subanalytic when the angle is restricted to any closed interval [a, b] inside (0, π).Source: §5 · p. 30
  • H is not analytic at the endpoint angles θ = 0 and θ = π.Source: §5 · p. 32
  • S is real analytic and bounded on R^2 × (0, π), but it is not subanalytic.The closure of its graph is also not subanalytic.Source: §5 · p. 32

6. Evaluation of f(w, v) concluded

The section concludes the evaluation of f(w, v) with a formula, then sets out conventions and pairwise notation for the formulas used with network rows and target rows.

6. Evaluation of f(w, v) concluded

  • The section derives a formula for f(w, v) using A = −cot(θ) and B = csc(θ).Source: §6 · p. 35
  • The formula for f(w, v) can also recover the stated f(w, w) result and the cases θ ∈ {0, π}.Source: §6 · p. 35

6.1. Notational conventions & formulas for f(wi, vj), f(wi, wj).

  • The notation treats V as the target matrix and W as a k × d matrix with k ≥ d and no parallel rows, defining pairwise quantities for network and target rows.Source: §6.1 · pp. 35–36

7. Gradient of f(w, v) with respect to (w,

This section develops the gradient structure of the biased ReLU kernel loss, including its normalized-bias form, zero-bias reduction, behavior near parallel angles, and a coordinate transformation for studying critical points. It also examines zero-bias embeddings of bias-free critical points, finding that in the investigated non-degenerate cases they are not critical, despite small bias derivatives.

7. Gradient of f(w, v) with respect to (w,

  • The vectors j(w, v) and J(w, w′) are unit vectors perpendicular to w; {w, j} forms an orthonormal basis for the span of w and v.The section’s derivative strategy minimizes T-function evaluations by differentiating H’s defining integral rather than its Owen T-functions and inverse tangents.Source: §7 · p. 38, Some frequently used derivatives · p. 39

7.1. Derivatives of H(

  • The derivative with respect to w is singular when θ is 0 or π, but the singular term cancels in the full gradient for zero bias and not for nonzero bias.For nonzero w and v, f is C1 in β without restriction when λ lies in (0, π); at λ = 0 or π, the stated C1 condition depends on β.Source: §7.1 · p. 41, §8 · p. 44

9.1. The formula for ∂f ∂w(w, v) (normalized bias

  • The normalized-bias gradient is a relatively simple linear combination of w and v, with no cot(λ) or csc(λ) products except in an exponential term.When the bias is zero, the formula reduces to the gradient derived by Brutzkus and Globerson.Source: §9.1 · pp. 45–46

9.2. Angles close to 0 and π.

  • Near λ = 0 or π, the lemmas control a potential pathology from exponential factors and support the Curve Selection Lemma in cuspidal regions, including cusps tangent to β = 0.The discussion connects this boundary behavior to critical-point families whose rows converge to ± a target row, and suggests bias can make convergence to a non-target-parallel row harder.Source: §9.2 · p. 48

10.1. Table defining terms used in the formula for grad(L)(W).

  • The coordinate transformation maps non-degenerate critical points to corresponding non-degenerate critical points with similar linearizations and the same spectrum.The transformation assumes W has no zero rows; the induced vector fields on the fixed-point space need not be gradient fields under the standard Euclidean identification.Source: §11 · p. 55
  • In all investigated examples, a non-degenerate bias-free critical point does not remain critical when embedded in the biased network with bias set to zero.This failure means the bias-free network space is not invariant under the biased network's gradient dynamics in those cases.Source: §11 · p. 55
  • The bias derivatives at these points are nonzero but small, decaying asymptotically like a fractional power of d^-1.For small d, the derivatives are typically significantly smaller than the asymptotics suggest.Source: §11 · pp. 55–56
  • Small bias derivatives leave open the possibility of a nearby biased-network critical point without drastic changes in landscape geometry near the bias-free point.The passage presents this as a tractable transition when the derivatives are nonzero but small.Source: §11 · pp. 55–56
  • The discussion expects corresponding spurious minima in biased networks to have lower loss than in bias-free networks, and points to examples illustrating this expectation.The rationale is that adding bias is extra structure and would be counterintuitive if it increased loss.Source: §11 · p. 56

12. Symmetry and grad(L) in a network with bias

This section develops symmetry-based tools for analyzing critical points in biased ReLU networks, using group invariance and fixed-point spaces to reduce the analysis to lower-dimensional settings. It expresses gradients and population loss in rowtype coordinates, including formulas for loss and bias derivatives.

12. Symmetry and grad(L) in a network with bias.

  • The analysis uses high-symmetry targets to obtain sharp results that persist under symmetry-breaking perturbations, but this paper does not detail the Hessian spectrum.The section notes Hessian-spectrum computation and applications to nearby asymmetric targets as examples of the approach.Source: §12 · p. 57

12.1. Symmetries of the loss L in a biased network.

  • For V = Id with equal biases, the loss is permutation-invariant and its gradient is equivariant under row and column permutations.The gradient is defined and continuous when W has no zero rows.Source: §12.1 · p. 58

12.2. Isotropy groups of critical points when V = Id.

  • Adding a neuron typically turns a non-degenerate critical point into a 1-dimensional simplex of degenerate critical points, but a zero-loss global minimum instead becomes a connected simplicial complex of global minima.In the latter case, no new non-degenerate critical points defining the global minimum are created.Source: §12.2 · p. 59
  • Gradient descent initialized in a fixed-point space stays there, allowing critical-point searches to begin in typically lower-dimensional spaces.The fixed-point-space invariance also holds for biased networks.Source: §12.2 · pp. 59–60

12.3. Families of subgroups of ∆Sd ⊂Sr k × Sc

  • For the described subgroup families, adding bias increases the fixed-point-space dimension by the number of rowtypes, f + m.The fixed-point space is coordinatized by block values; the stated dimension increment is f + m.Source: Review of notation and conventions · pp. 61–62
  • Symmetry forces equal norms and consistent angles among rows of the same rowtype, while non-degenerate symmetric critical points can deform into asymmetric ones as the target loses symmetry.The deformation claim also applies when there are additional neurons or bias.Source: Review of notation and conventions · pp. 62–63, Properties of rowtypes · p. 63
  • The block coordinatization map is not an isometry, so taking a Euclidean-coordinate gradient in those coordinates does not recover grad(L).The section gives two remedies: change the range-space inner product or redefine the map to be an isometry.Source: Properties of rowtypes · pp. 64–65

12.5. Critical point equations for grad(L) on M(k, d + 1)G∆

  • Rowtype-based gradient components reduce the critical-point equations on fixed-point spaces; at critical points, grad(L)'s linearization is diagonally similar to the vector-field linearization, so Sylvester’s test applies without a Hessian.The rowtype reformulation accounts for multiple rows of the same type; the linearization need not be symmetric.Source: §12.5 · p. 65, Properties of rowtypes · p. 65

12.6. Critical point equations associated to rowtype i /∈rs.

  • For rowtypes i ∉ rs, the component formula uses f critical-point equations, while rowtypes i ∈ rs require one extra equation.The equations are formulated separately by rowtype, rather than by individual row.Source: §12.5 · p. 65, §12.6 · p. 67

12.7. Critical point equations associated to rowtype i ∈rs.

  • For symmetric rowtypes i ∈ rs, the gradient-component equations include additional terms that vanish when i ∉ rs.The rowtype-specific equations support analysis and simulation for large d; the source notes that their greater complexity is a cost of symmetry.Source: §12.7 · pp. 69–71

12.8. The terms Ai, Bi when i ∈rs and V = Id.

  • For V = Id, the Ai and Bi expressions extend the nonsymmetric-rowtype formulas to symmetric rowtypes and apply across all rowtypes.The symmetric rowtype blocks need not have equal entries throughout their diagonal block.Source: §12.8 · pp. 71–72

12.9. Population loss with symmetric target.

  • For a symmetric target V = Id with zero bias, the population loss and its derivatives with respect to bias are given in rowtype form.The loss expression includes contributions from extra neurons, and the bias-derivative formula is indexed by rowtype.Source: §12.9 · pp. 74–75

13. Review of results for symmetric target Id

This section reviews critical-point families for the symmetric identity target and gives corresponding biased-network equations, then develops fixed-point and fractional-power-series tools to describe their behavior. It distinguishes type I and type II families and illustrates their different loss and convergence properties.

13. Review of results for symmetric target Id

  • The review covers unbiased networks with target Id, their critical-point equations and infinite families, and corresponding biased-network equations when the target bias is zero.Source: §13 · p. 77

13.1. Fixed point spaces and fractional power series.

  • Fixed-point equations can be followed as dimension d increases, with the implicit function theorem providing a route when the Jacobian remains nonsingular.Source: Discussion · p. 78
  • Although subanalytic theory yields fractional power series for unbiased families, biased critical-point equations are not subanalytic, and unbiased critical points generally do not remain critical when bias is added.Source: Discussion · p. 79

13.2. A family of spurious minima with isotropy ∆Sd, k = d.

  • For k = d and isotropy ∆Sd, the fractional-power-series family is defined for d ≥ 4 and forms a curve of spurious minima for d ≥ 8.Source: §13.2 · p. 79

13.3. Fractional power series (FPS): types I and II.

  • Type II families are spurious local minima for d ≥ 6, while the described family is a non-degenerate critical point for d from 3 to 5 and bifurcates near d = 5.58.Source: §13.3 · p. 81
  • Type II losses typically approach zero as d grows, whereas type I losses approach a positive constant; the illustrated type I series includes d^-1/4 terms and converges slowly.Source: §13.3 · p. 81
  • For unbiased spurious minima, setting network and target biases to zero generally still does not make the corresponding parameter point critical in the biased network.Source: §13.3 · p. 82
  • The examples also include a completely asymmetric target and critical point, and a partially asymmetric target with critical points admitting fractional-power-series representations.Source: §13.3 · p. 82

14.1. Computation of ∂L

This section estimates bias derivatives at critical-point families using fractional power series (FPS) and high-precision numerical computation. For the type II example, the derivative decays at least as fast as d^-2, while finite-d values can differ from that leading-order estimate.

14.1. Computation of ∂L

  • The section uses FPS expansions of critical points and inner products to estimate the derivative, with rowtypes used to parameterize β derivatives.The authors note that high-precision computation is faster and easier than adding another term to each of five FPS expansions.Source: §14.1 · pp. 83–84
  • For the type II family, ∂L/∂β0 decays at least as fast as d^-2, and the leading FPS coefficient is checked numerically.At d = 4, the derivative is approximately 2.8025 × 10^-2, compared with the D0 4d^-2 estimate of 3.34 × 10^-2.Source: §14.1 · pp. 83–84
  • The leading coefficient D0 4 of d^-2 for the derivative with respect to β0 is 0.541586362042490362453470777718114374878787486392237162895683618.The reported value is accurate to 64 decimal places.Source: §14.1 · p. 84

14.2. Results for isotropy ∆Sd, type I (type A in [4]).

The section reports high-precision derivative coefficients for type I critical-point families under two isotropy patterns. In the ΔSd case the derivative decays like d^-1; the following ΔSd−1 cases report coefficients for m = 0, 1, and 2.

14.2. Results for isotropy ∆Sd, type I (type A in [4]).

  • For a type I critical point with k = d and isotropy ΔSd, the derivative decays like d^-1, and its initial coefficient D2 is 0.144967736664468798796892840195837787855643034035842961341133133.Source: §14.2 · p. 85

14.3. Results for isotropy ∆Sd−1, type I, m = 0, 1, 2.

  • For type I critical points with isotropy ΔSd−1 and m = 0, the reported initial derivative coefficient D2 is 0.197646145812959995056847958971127.The table identifies the derivative as βi(c(d), 0), i ∈ 2.Source: §14.3 · p. 86
  • For type I critical points with isotropy ΔSd−1 and m = 1, the reported initial derivative coefficient D3 is 0.0774456350768987130272615429004301.Source: §14.3 · p. 86
  • For type I critical points with isotropy ΔSd−1 and m = 2, the reported initial derivative coefficient D4 is 0.18171273140268941937100246131777.Source: §14.3 · pp. 86–87

14.4. Results for isotropy ∆Sd−1, type II, m = 0, 1, 2.

The section reports derivative coefficients for type II critical points with isotropy ΔSd−1, then gives type I results under isotropy GΔ. For the latter families, the derivatives decay like d^-1, and the critical-point matrix has a specified limit as d grows.

14.4. Results for isotropy ∆Sd−1, type II, m = 0, 1, 2.

  • For type II critical points with isotropy ΔSd−1 and m = 0, the reported initial derivative coefficient D2 is −0.237257064180446401333830561616386.The section also reports D3 as −0.0804839616208356631748165154342071; the supplied passage does not identify its m case clearly.Source: §14.4 · pp. 87–88
  • For type II critical points with isotropy ΔSd−1 and m = 2, the section reports computed βi(c(d), 0) values.Source: §14.4 · p. 88

14.5. Results for isotropy G∆

  • As d grows, the type I example with isotropy Δ(Sd−2 × S2) has ξ0^0 → +1, ξ1^1 → −1, and all off-diagonal entries converge to zero.Source: §14.5 · p. 88
  • For type I critical points with isotropy GΔ, initial D2 coefficients are 0.197646145812959995056847958971127 for m = 0 and 0.477893257041395097510780683943154 for m = 1; the m = 1 derivative decays like d^-1.Source: §14.5 · pp. 88–89

14.6. Results for isotropy ∆(Sd−2 × S2), type II, m = 0, 1.

The section reports type II critical-point families with specified isotropy and fractional power series (FPS) behavior. For p > 2, it also describes how diagonal entries and derivatives behave as d grows.

14.6. Results for isotropy ∆(Sd−2 × S2), type II, m = 0, 1.

  • For isotropy ∆(Sd−2 × S2), the reported results cover only m = 0, 1; no FPS solution originating from the m = 1 type II solution was known.For the m = 0 type II point, the derivative decays like d^-1 and the section reports initial FPS derivative coefficients.Source: §14.6 · pp. 89–90

14.7. Results for isotropy ∆(Sd−p × Sp), type II, p > 2.

  • For p > 2, examples form two-parameter critical-point families with FPS representations and diagonal entries converging to ±1.When d is even and d = 2p, one family has half its diagonal entries converging to +1 and half to −1; its isotropy is not a proper maximal subgroup of S2p.Source: §14.7 · p. 90
  • For p = 3, 4 with d − p = p, the upper-block diagonal entries converge to +1 and derivatives decay like d^-1.The section gives initial FPS derivative coefficients 0.2119019935040464… and 0.1135488183150048….Source: §14.7 · pp. 90–91

14.8. Asymmetric targets and fractional power series.

The section uses path following to continue critical points as the target is deformed, including for asymmetric targets. It reports a likely non-degenerate spurious minimum for a fully asymmetric target and FPS expansions for asymmetric examples, while noting continuation difficulties near bifurcations.

14.8. Asymmetric targets and fractional power series.

  • Path following deforms the target along a linear path from V to T, continuing a critical point by Newton iterations at successive steps.The text says non-degenerate critical points remain spurious minima if the target perturbation is sufficiently small.Source: §14.8 · p. 91, Path following deformations of the… · p. 91
  • Near bifurcations, path following may be difficult; piecewise linear paths can help avoid nearby points or Newton non-convergence.The text identifies the type II bifurcation near d = 5.6 and the type I bifurcation near d = 7.Source: Path following deformations of the… · pp. 91–92
  • Even when the target and critical point have few or no symmetries, FPS expansions for critical points can still be obtained.The section illustrates this in both asymmetric-target examples.Source: Path following deformations of the… · p. 92
  • For a partially asymmetric target with a 6 × 6 block, the loss decays like d^-1 with initial coefficient 0.36076059….The continuation reached d = 9, but the authors were unable to reach d = 8, even with a piecewise linear path.Source: Path following deformations of the… · p. 93
  • For a target with no permutation symmetries, path following produced a critical point with no indicated bifurcation or stability change, likely a non-degenerate spurious minimum.Newton’s method converged at every step, with errors of 10^-616 or better using 2024-bit precision.Source: Path following deformations of the… · pp. 93–94

15. Families of spurious minima in biased ReLU networks

This section examines whether spurious-minimum families from unbiased ReLU networks extend to biased networks, focusing on two isotropic families in the case k = d and target V = Id. It reports an FPS-based result and discusses numerical difficulties and safeguards for approximate Newton methods.

15. Families of spurious minima in biased ReLU networks

  • The section restricts attention to k = d and does not analyze over-specified networks or Hessian spectra; its numerical arguments assume FPS exist.The paper notes that more formal implicit-function arguments for the families may be relatively simple, as in [4] and [5, Appendix A, remark 4].Source: §15 · p. 95

15.1. Isotropy ∆Sd, d = k: Type I in a network with bias.

  • The Type I analysis sets k = d and V = Id and studies critical points with ∆S_d isotropy, using equations that include differentiation with respect to the network bias.This is the simplest family, described as Type I (or A) in [4].Source: §15.1 · pp. 96–98

15.2. Numerics.

  • For the Type I family with isotropy ∆S_d, the biased critical point has lower loss than its unbiased counterpart, and their loss FPS share the first two terms.The computations suggest the corresponding critical points are natural extensions; the loss decrease is reported as always positive.Source: §15.2 · pp. 100–101, Table 16 · p. 100

15.3. Path based analysis.

  • Path-based tests for several dimensions returned to the original unbiased critical point as bias was reduced, without observed bifurcations or jumps.The loss increased near the biased point but did not decrease near the unbiased point in this example; the authors suggest a strictly decreasing trajectory may exist, though it need not be a straight line.Source: §15.3 · p. 103

15.4. Isotropy ∆(Sd−1 × S1), q1 = 1, d = k, Types I and II.

  • For isotropy ∆(S^(d−1) × S1), biased Type I and II families are constructed; their loss FPS begin with matching terms, and the Type I bias is negative for d ≥ 10.The families have reported coordinate matching patterns of (2, 0, 2, 2, 2) for Type I and (4, 2, 2, 4, 2) for Type II.Source: §15.4 · pp. 103–105

15.5. Isotropy ∆(Sd−2 × S2), q1 = 2, d = k, Types I and II.

  • For isotropy ∆(S^(d−2) × S2), Type I and Type II families of spurious minima are described, and the first two terms in the FPS expansions of L(ξ(d)) and L(eξ(d)) are equal.The families are (2, 0, 2, 2, 2, 2)-matched for Type I and (4, 2, 2, 4, 2, 2)-matched for Type II.Source: Type I family of spurious minima with… · pp. 105–106, §15.6 · p. 107
  • Assuming the FPS exists, the first two terms for L(ξ(d)) and L(eξ(d)) agree; at d=109, initial β_i(d) terms have opposite signs, and the families are (4,2,2,4,2,2)-matched.Source: §15.6 · p. 108

15.7. Comments on numerical methods: comparison with bias free networks.

  • Approximate Newton differentiation is challenging with bias: cotangent and cosecant singularities require fast β_i decay, and formula (5.21) may be poorly behaved.Source: §15.7 · pp. 108–109
  • At d=10512 and 16384-bit precision, Newton failed with h=10^-480 but converged with h=10^-600; exact derivatives of the T-function integral offer a resolution.This example uses the type I maximum-isotropy family for the unbiased network and assumes the γ terms are zero; h=10^-1000 yielded faster convergence with errors from about 10^-4400 to 10^-4934.Source: §15.7 · p. 109
  • Noisy data may prevent convergence to spurious minima when β_i terms do not decay fast enough with d, potentially removing such minima when k>d.Source: §15.7 · p. 109
  • Approximate Newton increments must be chosen to remain within the region of subanalyticity; critical-point data and tested MPFR T-function code are available on request.Source: §15.7 · p. 109

Appendix A. Review of the error function erf(x)

Appendix A reviews error-function properties and integral identities used in Section 3. It notes that a nonzero-bias integral needed there is represented using Owen’s T-function.

Appendix A. Review of the error function erf(x)

  • The error-function results in this appendix are used in Section 3.Source: Appendix A · p. 110
  • erf is odd, whereas erfc is neither odd nor even; the sign symmetry of the relevant Gaussian integrals carries through to erf but not erfc.Source: Appendix A · p. 110
  • Integrals involving erf(ax + b) can often be handled by integration by parts and the identity in (A.60).Source: Appendix A · p. 111
  • A nonzero-bias integral used to compute f(w, v) is represented with Owen’s T-function because it cannot be evaluated using only erf and exponentials.Source: Appendix A · pp. 111–112

Appendix B. High precision computation of Owen’s

Appendix B describes high-precision MPFR computation of Owen’s T-function based on Komelj’s algorithms. It gives parameter-dependent algorithm choices and a stopping rule for the series computation.

Appendix B. High precision computation of Owen’s

  • The authors wrote MPFR code for high-precision T(h, a) computation based on Komelj’s method, which supports fast high-precision evaluation across real h and a.They note that Owen’s T-function was not available as a standard built-in function in MPFR or other high-precision packages, as far as they knew.Source: Appendix B · p. 113
  • The computation uses algorithms (40a) and a modified (40c), with (40a) fastest for |a| ≤ 1 and (40c) fastest for a > 1.Source: Appendix B · p. 113
  • For a < −1, the second algorithm computes T(h, −a), then obtains T(h, a) = −T(h, −a).Source: Appendix B · p. 113
  • The positive-term series computation stops when the next partial sum is smaller than the current one, and takes the current sum as its result.Source: Appendix B · p. 114
  • The implementation avoids MPFR’s incremental gamma function because its computation can be slow for large arguments.Source: Appendix B · p. 114

Appendix C. Generalized T-function

Appendix C discusses expressions for the generalized T-function and their relation to Owen’s formulas. It reports analytical and high-precision agreement between two expressions.

Appendix C. Generalized T-function

  • The appendix recalls the generalized T-function defined in.Source: Appendix C · p. 115
  • An alternative expression uses four instances of T, but identities from [18, Table II, 2.8,2.9] reduce it to inverse tangents and error functions, yielding the formula in.Source: Appendix C · p. 116
  • Two expressions for the generalized T-function agree analytically and to 615 decimal places at 2048-bit MPFR precision.Source: Appendix C · p. 116
Loading 2609.30661v1…