PaperlayerView exact revision on arXiv

Source-linked AI summary

On the equivalence of Hopfield Networks and Boltzmann Machines

Adriano Barra, Alberto Bernacchia, Enrica Santucci, Pierluigi Contucci

arXiv:1105.2790v3cond-mat.dis-nncs.AI

TL;DR

The paper asks how a hybrid Boltzmann Machine relates to Hopfield networks in modeling state probabilities. It analyzes the hybrid network and shows that, after marginalization, its visible-unit distribution matches the Hopfield distribution, with implications for simulation and hidden-layer sizing.

  • Problem

    The paper addresses how a hybrid Boltzmann Machine relates to Hopfield networks, within the broader goal of modeling probability distributions of observed systems.

    Source: §1 · pp. 1–2

  • Method

    The authors study a machine with continuous hidden and binary visible units, use stochastic stability to analyze its thermodynamics, and marginalize over hidden variables.

    Source: §2 · pp. 3–4, §3.1 · p. 7

  • Results

    The marginalized HBM and Hopfield network have the same probability distribution, while the HBM stores NP synapses rather than the Hopfield network’s N(N −1)/2 synapses.

    Source: §2 · p. 4, §4 · p. 11

  • Takeaways & Limitations

    The Hopfield phase transition corresponds to choosing the HBM hidden-to-visible size ratio: too few hidden units constrain learning, while too many overfit observed data.

    Source: §4 · p. 11

    The rigorous analysis of interacting hidden layers remains under investigation; for small ε, the authors obtain exact bounds using a first-order expansion.

    Source: §3.3 · p. 10

Abstract

from arXiv · show

A specific type of neural network, the Restricted Boltzmann Machine (RBM), is implemented for classification and feature detection in machine learning. RBM is characterized by separate layers of visible and hidden units, which are able to learn efficiently a generative model of the observed data. We study a "hybrid" version of RBM's, in which hidden units are analog and visible units are binary, and we show that thermodynamics of visible units are equivalent to those of a Hopfield network, in which the N visible units are the neurons and the P hidden units are the learned patterns. We apply the method of stochastic stability to derive the thermodynamics of the model, by considering a formal extension of this technique to the case of multiple sets of stored patterns, which may act as a benchmark for the study of correlated sets. Our results imply that simulating the dynamics of a Hopfield network, requiring the update of N neurons and the storage of N(N-1)/2 synapses, can be accomplished by a hybrid Boltzmann Machine, requiring the update of N+P neurons but the storage of only NP synapses. In addition, the well known glass transition of the Hopfield network has a counterpart in the Boltzmann Machine: It corresponds to an optimum criterion for selecting the relative sizes of the hidden and visible layers, resolving the trade-off between flexibility and generality of the model. The low storage phase of the Hopfield model corresponds to few hidden units and hence a overly constrained RBM, while the spin-glass phase (too many hidden units) corresponds to unconstrained RBM prone to overfitting of the observed data.

1 Introduction

Generative models aim to reproduce the probability distribution of observed systems, and Boltzmann Machines do so by marginalizing over hidden units. The paper studies a hybrid machine whose visible-unit distribution maps to a Hopfield network, linking the two models’ thermodynamics and retrieval behavior.

1 Introduction

  • Boltzmann Machines model state distributions by marginalizing over hidden units, while Restricted Boltzmann Machines use a bipartite topology that supports efficient learning.Ising-type networks, including Hopfield models and Boltzmann Machines, are used as generative models.Source: §1 · pp. 1–2
  • The hybrid Boltzmann Machine has analog hidden units and binary visible units, whose marginalized distribution is equivalent to a Hopfield network with visible units as neurons and hidden units as learned patterns.The mapping offers a new way to simulate Hopfield networks and interpret their spin-glass transition.Source: §1 · p. 2
  • Stochastic stability is used to study thermodynamics with two hidden-unit sets, extending the analysis to correlated pattern sets whose interaction acts as retrieval noise.The two non-interacting sets correspond to uncorrelated patterns; the analysis assumes replica symmetry.Source: §1 · p. 2

2 Statistical equivalence of HBM and Hopfield networks

The hybrid machine has binary visible units and continuous hidden units arranged in a bipartite network. Marginalizing over hidden units yields the Hopfield distribution, and simulations show retrieval behavior consistent with Hopfield predictions across noise levels and pattern counts.

2 Statistical equivalence of HBM and Hopfield networks

  • The HBM uses binary visible activities σ_i and continuous hidden activities z_μ and τ_ν, with reciprocal inter-layer connections but no intra-layer connections.The section focuses on one hidden layer, z, for parsimonious notation.Source: §2 · p. 3
  • Visible-unit input sums weighted hidden activities, while hidden-unit input sums weighted visible activities through the synaptic matrix ξ.The hidden activities evolve continuously, whereas visible activities change in discrete steps.Source: §2 · p. 3
  • The hidden-unit dynamics include leakage, visible-driven input, and Gaussian noise; the analysis assumes hidden diffusion is much faster than visible-unit updates.For fixed visible activity, hidden-unit equilibrium is Gaussian around the input signal, and hidden units are treated as independent.Source: §2 · p. 3
  • After marginalizing hidden variables, the HBM and Hopfield network have the same probability distribution, with HBM hidden-unit weights corresponding to stored patterns.The number of Hopfield patterns equals the number P of hidden units.Source: §2 · p. 4
  • At β = 0.5 retrieval is absent; at β = 2 it is lost near P ≃ 0.06N = 60, while at β = 10 it persists near P = 0.14N = 140, matching Hopfield predictions.The simulations use N = 1000 visible units and vary P across 50, 100, 150, and 200.Source: §2 · p. 4, Figure 2 · p. 5

3 Thermodynamic theory of HBM

Marginalizing the hybrid Boltzmann Machine’s analog hidden units gives visible-unit thermodynamics equivalent in distribution to a Hopfield network. The section uses stochastic stability and free-energy minimization to study independent and interacting hidden layers.

3 Thermodynamic theory of HBM

  • Marginalizing the hidden variables yields a Hopfield Hamiltonian with the same thermodynamics as the HBM, although corresponding observables are equivalent in distribution, not pointwise.The connection links Hopfield retrieval through free-energy minimization with HBM learning through log-likelihood estimation.Source: §3 · p. 6
  • The thermodynamic calculation averages observables over both state configurations and random synaptic weights to characterize a typical HBM.Free-energy minima with respect to the order parameters identify states with significant probability.Source: §3 · pp. 6–7

3.1 Multiple-layer stochastic stability

  • Multiple-layer stochastic stability introduces fictitious external fields that reproduce the averages of recurrent internal inputs, but not their second- and higher-order statistics.The method perturbs each of the three layers to simplify the free-energy calculation.Source: §3.1 · pp. 7–8

3.2 Free energy minimization and phase transition

  • The analysis locates the transition to zero visible-layer overlap, where neurons are uncorrelated and activity is random, by examining divergence of the rescaled overlap.The critical region is studied as a function of α, β, and γ.Source: §3.2 · p. 9
  • For two unconnected hidden layers, the equivalent Hopfield network stores P + K independent patterns, consistent with and generalizing earlier results.The hidden units are conditionally independent in this case.Source: §3.2 · p. 9

3.3 Analysis of interacting hidden layers

  • For interacting hidden layers, rigorous analysis remains under investigation; the reported treatment uses first-order expansion for small ε.It marginalizes each hidden layer in turn and averages the resulting expressions.Source: §3.3 · p. 10
  • For small interaction strength ε, the interacting HBM is approximated to first order by a Hopfield model with an additional noise source.At ε = 0, the standard Hopfield model is recovered; the noise’s effect on retrieval depends on the load of the other hidden layer.Source: §3.3 · p. 11

4 Conclusions

The paper establishes an exact mapping between Hopfield networks and hybrid Boltzmann Machines, with a more economical storage and update scheme. It also connects the Hopfield phase transition to layer-size trade-offs and describes how hidden-layer interactions affect retrieval.

4 Conclusions

  • The exact Hopfield–HBM mapping permits updates of N + P units and storage of NP synapses, versus N neurons and N(N − 1)/2 synapses for Hopfield networks.P is the number of stored patterns.Source: §4 · p. 11
  • The HBM phase transition corresponds to an optimal hidden-to-visible layer-size trade-off between an over-constrained model and one that overfits and cannot generalize.Too few hidden units correspond to Hopfield’s low-storage phase; too many correspond to its spin-glass phase.Source: §4 · p. 11
  • Disconnected hidden layers contribute linearly to Hopfield capacity, whereas interacting hidden layers act as an effective noise source for retrieval..Source: §4 · p. 11
  • The authors argue that stochastic stability is a tractable mathematical framework for studying Boltzmann Machine thermodynamics..Source: §4 · p. 11
  • The work connects mathematical physicists studying spin glasses with computer scientists studying machine learning and artificial intelligence..Source: §4 · p. 11

Appendix 1

Appendix 1 calculates the interpolating free energy at t = 0. It uses one-body factorization to separate contributions by layer and selects parameters to simplify the calculation.

Appendix 1

  • At t = 0, the interpolating free-energy calculation involves only one-body terms..Source: Appendix 1 · p. 12
  • Additivity of the logarithm and one-body factorization rewrite the expression as three terms, one for each layer..Source: Appendix 1 · p. 12
  • The parameter choice a = βq̄ and h = ϵ = β(1 − q̄) simplifies the calculation, after which the integrals and sums yield an expression..Source: Appendix 1 · p. 12

Appendix 2

Appendix 2 derives the t-derivative of the interpolating free energy by assembling seven terms. With a replica-symmetric treatment, it reduces the derivative to overlap fluctuations and an average term that integrates directly.

Appendix 2

  • The derivative of Ã(β, t) contains seven terms because t appears seven times in the exponential.The derivation reports the result for each term.Source: Appendix 2 · p. 12
  • The seven contributions are combined to obtain the derivative expression..Source: Appendix 2 · p. 13
  • Choosing convenient parameters expresses the whole derivative as the source of overlap fluctuations..Source: Appendix 2 · p. 13
  • Under replica symmetry, the order-parameter fluctuation term is neglected, while the t-independent average term integrates over 0 to 1 to itself..Source: Appendix 2 · p. 13

Appendix 3

Appendix 3 treats retrieval by adding an external-field multiplier to the interpolating free energy. It derives a replica-symmetric free energy and the stationarity equations used to solve for its order parameters.

Appendix 3

  • For Gaussian stored patterns in the thermodynamic limit, retrieval is not spontaneous, but a suitable Lagrange multiplier in the interpolating free energy can force it..Source: Appendix 3 · p. 13
  • The first condensed pattern’s Mattis magnetization is denoted m1, and M1 is its replica-symmetric approximation.The first pattern is chosen because patterns are fully permutation-invariant.Source: Appendix 3 · p. 13
  • The final replica-symmetric free energy includes fluctuations of m1 around M1.The appendix introduces the expression by analogy with the calculation in Section 4.Source: Appendix 3 · p. 13
  • The free energy is minimized with respect to the replica-symmetric order parameters q̄, p̄, r̄, and M by imposing their stationarity conditions..Source: Appendix 3 · p. 13
  • These conditions yield a system of integrodifferential equations to be satisfied simultaneously, which can be solved numerically..Source: Appendix 3 · p. 14
Loading 1105.2790v3…