PaperlayerView exact revision on arXiv

Source-linked AI summary

Path Integral Approach to Random Neural Networks

A. Crisanti, H. Sompolinsky

arXiv:1809.06042v2cond-mat.dis-nn

TL;DR

Heuristic approaches to random neural-network dynamics do not systematically justify their assumptions or generally determine DMF stability. This paper uses path integrals to derive DMF equations and analyze fluctuations, obtaining stability conditions and equations for the maximal Lyapunov exponent.

  • Problem

    Heuristic Gaussian-based analyses do not systematically justify their assumptions and generally cannot derive stability conditions for DMF equations.

    Source: §I · p. 2

  • Method

    The paper formulates random-network dynamics as a path-integral dynamical field theory and derives the DMF equations from its saddle point.

    Source: §VII · p. 15

  • Results

    The DMF equations become exact at large N, and fluctuation analysis yields stability conditions and equations for the maximal Lyapunov exponent.

    Source: §VII · pp. 15–16

  • Takeaways & Limitations

    Fluctuations around the saddle point provide stability conditions for DMF solutions in the studied network.

    Source: §VII · p. 16

    The analysis focuses on simple network architecture and odd-symmetric sigmoidal nonlinearity.

    Source: §VII · p. 15

Abstract

from arXiv · show

In this work we study of the dynamics of large size random neural networks. Different methods have been developed to analyse their behavior, most of them rely on heuristic methods based on Gaussian assumptions regarding the fluctuations in the limit of infinite sizes. These approaches, however, do not justify the underlying assumptions systematically. Furthermore, they are incapable of deriving in general the stability of the derived mean field equations, and they are not amenable to analysis of finite size corrections. Here we present a systematic method based on Path Integrals which overcomes these limitations. We apply the method to a large non-linear rate based neural network with random asymmetric connectivity matrix. We derive the Dynamic Mean Field (DMF) equations for the system, and derive the Lyapunov exponent of the system. Although the main results are well known, here for the first time, we calculate the spectrum of fluctuations around the mean field equations from which we derive the general stability conditions for the DMF states. The methods presented here, can be applied to neural networks with more complex dynamics and architectures. In addition, the theory can be used to compute systematic finite size corrections to the mean field equations.

I. INTRODUCTION

The introduction motivates a systematic path-integral treatment of random neural-network dynamics, then develops a dynamical field theory for their correlations and responses. Its example extends a simple rate-based model to random asymmetric connectivity.

I. INTRODUCTION

  • The example extends Amari’s rate model to fully connected random asymmetric independent couplings, with second-moment scaling keeping inputs O(1) as N grows.The model represents each neuron’s activity with a continuous spin and uses a sigmoid response.Source: §I · pp. 1–2
  • Heuristic DMF derivations make stability analysis, response and correlation calculations, extensions, and finite-size corrections difficult, motivating a systematic path-integral approach.The paper presents this approach for random neural networks.Source: §I · p. 2

A. Path Integral and Dynamical Field Theory

  • The path-integral strategy constructs a generating functional for dynamical correlations and responses, then averages over couplings to obtain a dynamical field theory.The field-theory formulation uses collective correlation and response fields.Source: A. Path Integral and Dynamical Field… · pp. 2–4

B. Thermodynamic Limit and Dynamical Mean Field Theory.

In the thermodynamic limit, the generating functional is dominated by a stationary point, reducing the network dynamics to single-site processes whose Gaussian field is determined self-consistently.

B. Thermodynamic Limit and Dynamical Mean Field Theory.

  • As N grows, the integral is dominated by the largest exponent, and the stationary-point action determines the thermodynamic-limit description.Normalization implies that the response-field order parameter is zero at the self-consistent solution.Source: B. Thermodynamic Limit and Dynamical… · p. 4
  • The network’s dynamics is fully described by single-site dynamical processes in the limit N →∞.The effective process retains network-level memory through a Gaussian field whose covariance is computed self-consistently.Source: B. Thermodynamic Limit and Dynamical… · pp. 4–5
  • The central DMFT equations combine the effective stochastic process with the self-consistency condition for its Gaussian field.The Gaussian field carries memory of other sites through its self-consistent covariance.Source: B. Thermodynamic Limit and Dynamical… · p. 5

III. SOLUTION OF DMFT EQUATIONS.

The DMFT equations relate field and activity correlations self-consistently. In steady state, the correlation functions become time-translation invariant, and acceptable solutions form a one-parameter family.

A. DMFT equations

  • Equations (33) and (37) constitute the model’s self-consistent DMFT equations, with the synaptic field a zero-mean Gaussian process.The field correlation and activity correlation are related nonlinearly.Source: A. DMFT equations · p. 5

B. Steady State Solutions

  • In steady state, correlations depend on τ=ta−tb, and acceptable bounded solutions form an even one-parameter family ∆(τ; ∆0).The initial value ∆0 parameterizes the family, while the derivative at τ=0 is zero.Source: B. Steady State Solutions · pp. 5–6

1. Fixed Point Solution

The DMFT admits a zero fixed point and, for odd-symmetric transfer functions, a nonzero static solution when g > 1. Its time-dependent solutions can be understood through a self-consistent potential and energy, yielding distinct regimes mapped by the phase diagram.

1. Fixed Point Solution

  • For odd-symmetric transfer functions, a nonzero static fixed-point solution appears when g > 1, but it is unstable.The zero fixed point exists for all g.Source: §1 · p. 6

2. Time-dependent solution: Potential and Energy

  • All bounded energy-conserving orbits with ∆(0) = ∆0 and zero initial slope are possible DMFT solutions, with behavior set by a potential that depends on ∆0.The dependence on ∆0 reflects the self-consistent nature of the DMFT.Source: §2 · p. 6

3. Phase Diagram

  • On the phase diagram, f marks time-independent solutions; between f and d solutions are positive periodic; on d they decay to zero; below d they are sign-changing periodic.For g > 1, all curves collapse and only the static solution ∆(τ)=∆0=0 survives.Source: Figure 2 · p. 7

IV. FLUCTUATIONS AROUND THE DMFT AND SOLUTION SELECTION

The section distinguishes saddle-point stability from which state attracts network dynamics. It uses two identical replicas and fluctuations of their DMFT action to formulate stability tests.

IV. FLUCTUATIONS AROUND THE DMFT AND SOLUTION SELECTION

  • All the identified DMFT solutions are stable saddle points, so fixed points, limit cycles, and chaos remain possible steady states; selecting an attractor is a separate question.The attractor question asks which state the dynamics reaches with probability one as N → ∞.Source: §IV · p. 8

A. Two replica formalism

  • Two identical copies with the same initial value yield a 2-replica DMFT whose order parameters are independent of replica index.The replicas are used to study the response and stability of network dynamics.Source: §IV · p. 8, A. Two replica formalism · p. 8

1. Fluctuations of the replicated DMFT:

  • Stability of the replicated stationary point requires that the operator A have no zero eigenvalue; the test does not require evaluating M.The fluctuation analysis separates intra-replica from inter-replica fluctuations.Source: §1 · p. 9

B. Stability of the time-independent solution

The stability analysis distinguishes the zero solution from nontrivial time-independent solutions and separates fluctuations within and between replicas. For tanh, the static solution is stable to within-replica fluctuations but unstable to between-replica fluctuations when g > 1.

B. Stability of the time-independent solution

C. Stability of time-dependent solutions

The stability analysis reduces fluctuations around time-dependent DMF solutions to a one-dimensional eigenvalue problem. Periodic solutions are unstable, whereas for g > 1 the time-decaying solution is the only stable attractor; its Lyapunov exponent is used to establish chaos.

C. Stability of time-dependent solutions

  • Equation (90) always has a zero eigenvalue with eigenfunction Ψ0(τ) = ∂τ∆(τ), corresponding to marginally positive Λ = 2δ as δ → 0+.The stability calculation restricts attention to fluctuations depending only on τ = ta − tb.Source: C. Stability of time-dependent solutions · pp. 10–11
  • Periodic solutions are unstable because the eigenvalue Λ = ε + 2δ passes continuously through zero for arbitrarily small δ.The periodic quantum potential produces eigenvalue bands that allow this crossing.Source: C. Stability of time-dependent solutions · p. 11
  • The time-decaying solution is stable because its quantum-potential eigenvalues are isolated, so Λ = ε0 + 2δ cannot be zero as δ → 0+.Its ground-state eigenvalue ε0 is negative.Source: C. Stability of time-dependent solutions · p. 11
  • For g > 1, only the time-decaying solution is a stable attractor; the paper then uses its Lyapunov exponent to establish that it is chaotic.The analysis also notes that the Lagrangian can evaluate correlation and response functions of fluctuations around the mean field.Source: C. Stability of time-dependent solutions · p. 11

V. MAXIMUM LYAPUNOV EXPONENT

  • The maximum Lyapunov exponent measures sensitivity to infinitesimal initial perturbations, and its large-time value gives the dominant exponential growth rate of linear response.At N → ∞, the response quantity self-averages and the exponent becomes non-random.Source: §V · p. 12

A. Intuitive calculation of the Lyapunov exponent.

The paper first motivates an intuitive calculation of the Lyapunov exponent from averaged linear-response fluctuations, then derives the calculation systematically using dynamical field theory (DFT). The DFT result links the exponent to the lowest eigenvalue of a one-dimensional Hamiltonian and distinguishes the stable fixed point below g = 1 from chaotic dynamics above it.

A. Intuitive calculation of the Lyapunov exponent

  • The intuitive calculation squares the linear-response equation and averages it to obtain an equation whose solution is expressed through the Green function of Hτ.It assumes the averaged product φ′(t)φ′(t′) can be replaced by ∂C(t − t′)/∂∆(t − t′), and that the cross term vanishes or disappears at large N.Source: A. Intuitive calculation of the… · pp. 12–13
  • The intuitive equation yields the correct λ but is not systematic, leaves approximations difficult to control, and may be hard to construct for more complex cases.The paper therefore derives such fluctuation equations systematically within DFT.Source: A. Intuitive calculation of the… · p. 13

B. DFT calculation of the Lyapunov exponent

  • In the DFT calculation, the squared spin susceptibility is computed using two identical replicas and reduced to an equation involving Hτ.The resulting expression is identical to the earlier equation apart from a constant relating the two susceptibilities.Source: B. DFT calculation of the Lyapunov… · pp. 13–14
  • For g < 1, λ = −1 + g < 0 and ∆ = 0 is stable; for g > 1, the stable decaying solution has positive λ and is chaotic.The DFT calculation gives λn = −1 + √(1 − ϵn), with the maximal exponent determined by the lowest eigenvalue ϵ0 of Hτ.Source: B. DFT calculation of the Lyapunov… · p. 14

VI. EXPLICIT EXPRESSIONS FOR λ IN TIME DEPENDENT STATE

The paper obtains the Lyapunov exponent by solving the DMFT equation and finding the lowest eigenvalue of an associated quantum-mechanical problem. Near g = 1 from above, memory decay slows while the chaotic attractor approaches the zero fixed point.

VI. EXPLICIT EXPRESSIONS FOR λ IN TIME DEPENDENT STATE

  • To calculate λ for a time-dependent state, solve the DMFT equation and then find the lowest eigenvalue of the associated quantum-mechanical problem.The paper notes that this is difficult for arbitrary g > 1, but analyzes the limits g → 1+ and g → ∞.Source: §VI · p. 14

A. Limit g →1+

  • As σ → 0+, the amplitude of ∆(τ) vanishes linearly with σ while the characteristic decaying time diverges as σ⁻¹; as g → 1+, the dynamics slows down.The chaotic attractor goes continuously to the fixed point ∆ = 0 at the critical point g = 1.Source: A. Limit g →1+ · pp. 14–15
  • Near g = 1+, the rate λ⁻¹ of trajectory divergence scales as the square of the memory-decay rate.This is the paper’s stated relation for the onset of chaos.Source: A. Limit g →1+ · p. 15

B. Limit g →∞

At large g, the quantum potential forms a deep, narrow well and the Lyapunov exponent’s rate of trajectory divergence vanishes even though memory decay remains finite. The discussion presents path integrals as a systematic route to DMF equations and fluctuation stability analysis, while noting the model’s restricted architecture and nonlinearity.

B. Limit g →∞

  • For g ≫ 1, VQM(τ) forms a deep, narrow well near τ = 0, and its ground-state eigenfunction is localized within width O(1/g).Outside that region, the eigenfunction decays exponentially fast.Source: B. Limit g →∞ · p. 15
  • As g → ∞, the rate of exponential trajectory divergence vanishes while the memory-decay rate remains finite.The large-g DMFT correlation decays exponentially for τ ≫ 1 with a characteristic time.Source: B. Limit g →∞ · p. 15

VII. DISCUSSION

  • The discussion limits its concrete analysis to simple-architecture networks with odd-symmetric sigmoidal nonlinearity, while the path-integral derivation makes the DMF equations exact as N → ∞.It also derives response and correlation functions and stability conditions from fluctuations around the saddle point.Source: §VII · pp. 15–16
  • The authors identify systematic finite-size corrections and perturbation analysis as possible path-integral applications for deterministic random networks.They describe these directions as topics for future exploration.Source: §VII · p. 16

Appendix A: Chaotic behaviour in the Ising limit

The appendices derive results used in the paper’s DMF analysis, including chaotic behavior in the Ising limit, path-integral correlation calculations, and Gaussian-average relations for the potential.

Appendix A: Chaotic behaviour in the Ising limit

  • The maximal Lyapunov exponent is obtained from tangent-vector time evolution.Source: Appendix A · p. 16
  • In the Ising limit, the dynamics is chaotic, with spin or field autocorrelation decaying exponentially over a finite characteristic time.The derivation assumes large gain and large network size, approximating the tangent-vector evolution with products of independent random matrices.Source: Appendix A · p. 16

Appendix B: ⟨SS⟩correlation function

  • Averages over solutions of the dynamical equation are represented as path integrals weighted by the dynamical action, then coupling averages introduce auxiliary fields.The appendix uses these relations to obtain expressions for the correlation function.Source: Appendix B · pp. 16–17

Appendix C: Averages in the DMFT.

  • The DMF averages are evaluated using Fourier representations and Gaussian integrals, providing relations used to derive the potential and its derivatives.For matching-parity functions the average need not vanish; with ψ(x)=φ(x), the expression for C(∆, ∆0) in the main text is recovered.Source: Appendix C · pp. 17–18
  • For an odd gain function, Φ(x) is even, which determines the stated form of the potential V(∆; ∆0).Source: Appendix C · p. 18

Appendix D: Proof of Eq. (109)

The proof evaluates the four-point correlation by reusing the earlier procedure and applying integration by parts, with surface terms and cross terms vanishing.

Appendix D: Proof of Eq. (109)

  • The four-point correlation in Eq. (109) is evaluated by following the procedure used earlier and applying an identity.Source: Appendix D · p. 18
  • Integration by parts drops surface terms, and the symmetrised delta is defined as δab,cd = δacδbd + δadδbc.Source: Appendix D · pp. 18–19
  • The proof also uses a vanishing surface term and states that cross terms vanish.Source: Appendix D · p. 19

Related papers

Loading 1809.06042v2…