PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Prototype Emergence in Modern Hopfield Models

Aditya Cowsik, Adithya Sriram

arXiv:2609.12079v1cond-mat.dis-nncs.AIcs.LG

TL;DR

How associative memories can learn hierarchical correlations and reconstruct patterns beyond stored examples remains an important question for understanding generative models. The paper analyzes a dense Hopfield network with polynomial activation, finding that ancestor prototypes can emerge as stable memories through prototype reconstruction. It also reports sensible emergent patterns on Fashion-MNIST, while noting blur on more complex data with spatial symmetries.

  • Problem

    How associative memory models learn hierarchical correlations and generalize beyond training examples to reconstruct new patterns remains an open question relevant to understanding generative models.

    Source: pp. 1–2

  • Method

    The paper analyzes a hierarchical dense Hopfield memory model with polynomial activation, using stability of ancestor prototypes as a minimal model of generalization.

    Source: pp. 2–3

  • Results

    Ancestor patterns in the hierarchy emerge as stable memory minima, and prototype reconstruction provides a form of generalization beyond stored examples.

    Source: pp. 2, 5

  • Takeaways & Limitations

    In this model, stored examples can serve as examples from which the network reconstructs more general concepts through prototype reconstruction.

    Source: p. 5

    Fashion-MNIST emergent memories are sensible, but on more complicated datasets such as those with spatial symmetries they become washed out and blurry.

    Source: p. 5

Abstract

from arXiv · show

Hierarchical correlations are a universal feature of any realistic model of data, and the question of how associative memory models may learn these correlations and generalize beyond them to construct new sensible images is an important step towards understanding more complex modern architectures such as diffusion models. We consider a hierarchical model for memories which are sampled and stored in a dense Hopfield network with polynomial activation. We analytically derive conditions for each level of this hierarchy to be locally stable - that is they are local energy minima. We use prototype reconstruction as a minimal model of generalization and we find that it takes only a quasi-polynomial amount of information to generalize beyond particular memories and even particular groups in the hierarchy. We observe a qualitatively analogous phase diagram in the number of memories, sharpness of the activation function (polynomial degree) for data from Fashion-MNIST.

S1. MOMENT FORMULAS AND DIAGRAMMATIC FORMALISM

This section defines the hierarchical memory model and develops approximations for moments of the energy gap. The resulting moment estimates provide the basis for assessing prototype stability.

S1. MOMENT FORMULAS AND DIAGRAMMATIC FORMALISM

  • The model starts from a random root, generates descendants by element-wise noise corruption, and stores only the leaf patterns in the network.Each new layer generates K −1 descendants per prototype; the tree has height h and even activation degree n.Source: S1. MOMENT FORMULAS AND DIAGRAMMATIC… · p. 7
  • Energy-gap statistics are the target of the analysis, and the derivation combines helper lemmas to establish the moment approximation.The text relates patterns on the tree through their path distances and overlaps.Source: S1. MOMENT FORMULAS AND DIAGRAMMATIC… · pp. 7–9
  • The section derives an approximation for the mth moment of the energy gap, using a restricted term in a parameter regime where its relative error is controlled.The approximation restricts the sum to the term with all k_a = 1; higher correlators are also approximated using powers of the mean argument.Source: S1. MOMENT FORMULAS AND DIAGRAMMATIC… · pp. 7–8
  • The approximation has a small relative error vanishing with N under the stated parameter setting.The cited formula sets k = (1,1,...,1) and ε = o(1).Source: S1. MOMENT FORMULAS AND DIAGRAMMATIC… · p. 8

A. Diagrams and Feynman Rules

The diagrammatic formalism represents terms in energy-gap moments as weighted paths through the memory tree. Its rules specify how to construct, simplify, weight, and sum those diagrams.

A. Diagrams and Feynman Rules

  • Each term in the moment expansion is represented by a tree diagram connecting the probe to selected leaf memories.The diagrams encode shared ancestry and path relationships among the probe and leaves.Source: A. Diagrams and Feynman Rules · p. 9
  • Solid lines encode extensive overlaps, while dashed lines encode single-site overlaps along paths through the tree.The solid-line weight includes N^(n−1) and a q^(n−1) factor depending on path distance; dashed lines can be reduced to nearest-neighbor factors.Source: A. Diagrams and Feynman Rules · p. 10
  • The extensive-overlap factor belongs to the whole solid path, whereas its q-dependent factor accumulates with path distance.Branches irrelevant to the diagram weight can be omitted.Source: A. Diagrams and Feynman Rules · pp. 9–10
  • Diagram construction marks the probe, selects m leaf replicas, draws their paths, cancels overlapping dotted lines, assigns weights, and sums over leaf placements.For a homogeneous tree, the placements are grouped by their hierarchy of most recent common ancestors.Source: A. Diagrams and Feynman Rules · p. 10

A. Diagrams and Combinatorics

The section counts the tree configurations contributing to first and second energy-gap moments, then uses those counts to form the variance and coefficient of variation. It simplifies the combinatorics under a large-degree assumption.

A. Diagrams and Combinatorics

  • For the first moment, leaf placements are counted by their distance from the probe, with the count depending on whether the probe is a leaf, internal node, or root.The count is denoted Cℓ(d).Source: A. Diagrams and Combinatorics · pp. 11–12
  • Second-moment diagrams have tripod structure, and their counts depend on distances from the probe and two targets to their junction.The squared mean has the same combinatorial structure as the second moment, enabling a general variance expression.Source: A. Diagrams and Combinatorics · p. 11
  • The coefficient of variation is obtained from the variance divided by the squared mean, but moment-error bounds do not directly transfer to a relative-error bound on the variance.The formulas use approximations for both the second moment and squared mean.Source: A. Diagrams and Combinatorics · pp. 11–12
  • For K ≫ 3, the combinatorial factors K −1, K −2, and K −3 are replaced by K to simplify the expressions.The text describes this as a reasonable approximation for large training sets and an upper bound on the variance.Source: A. Diagrams and Combinatorics · p. 13

B. Coefficient of Variation and Stability

Because the mean energy gap is positive for prototypes at every level, their relative stability is determined by the coefficient of variation. The section uses an approximate variance bound to study that quantity and bound instability probability.

B. Coefficient of Variation and Stability

  • The mean energy gap is positive for prototypes at every level, so relative stability is determined by the coefficient of variation.The analysis uses an approximate variance formula to bound the squared coefficient of variation, plotted for different levels in Fig. 2.Source: B. Coefficient of Variation and… · p. 14
  • The squared coefficient of variation is formed from the variance divided by the squared mean, with a square root yielding the coefficient of variation.The mean-squared expression also requires care when simplifying the combinatorics.Source: B. Coefficient of Variation and… · p. 14
  • A rigorous upper bound on the coefficient of variation supports a Chebyshev bound on the probability that some spin flip has a negative energy gap.A negative gap for any flip means the pattern is not a local minimum.Source: B. Coefficient of Variation and… · p. 14

1. Prototype Reconstruction 0 < ℓ< h

The paper treats stability of interior ancestor prototypes as prototype reconstruction, a minimal form of generalization. In a specified regime and with suitable scaling, an ancestor prototype’s probability of instability vanishes, yielding a finite density of emergent minima.

1. Prototype Reconstruction 0 < ℓ< h

  • At large K, the interior-prototype coefficient of variation approaches a constant; minimizing its controlling term gives K* ∝ q^(-4n/3).For interior prototypes, the dominant K^h terms cancel in the large-K limit.Source: §1 · pp. 14–15
  • As q → 1, perfect correlations merge all tree levels, leaving examples to reinforce only the single root memory.This is the limit in which the expression for interior-prototype stability vanishes.Source: §1 · p. 14
  • With appropriate scaling in q^-2 ≪ K ≪ q^(-2n+1), the probability of a spin flip lowering an interior prototype’s energy vanishes.The theorem specifies K ∝ N·κ(N), with κ(N) arbitrarily slowly growing, and n and h proportional to log N.Source: §1 · p. 14
  • Any given ancestor prototype has vanishing instability probability, implying a finite density of emergent stable minima in the energy landscape.Stability of prototypes with 0 < ℓ < h is treated as prototype reconstruction and a form of generalization.Source: §1 · p. 15

2. Memorization: ℓ= h

Leaf-memory stability concerns recall rather than generalization. Its coefficient of variation vanishes in the large-K limit under the stated condition on K and q.

2. Memorization: ℓ= h

  • Leaf-node stability is a recall phenomenon, rather than prototype reconstruction or generalization.The leaf corresponds to ℓ = h.Source: §2 · p. 15
  • The h → ∞ limit requires Kq^(2n−1) < 1; otherwise the infinite series diverges.The condition follows from terms involving the K^(2s) coefficient.Source: §2 · p. 15
  • When Kq^(2n−1) → 0, the leaf-memory coefficient of variation vanishes in the large-K limit.A finite expression as h → ∞ requires Kq^(2n−1) < 1.Source: §2 · p. 15

3. Root Prototype: ℓ= 0

Root stability depends on the number of examples relative to correlation strength. The root is the only absolutely stable memory at fixed n, h, and q in the large-K limit.

3. Root Prototype: ℓ= 0

  • For Kq² > 1, the sum defining root stability converges as h grows.All leaves derive from the same root and reinforce its stability.Source: §3 · p. 15
  • At large K, the root’s coefficient of variation vanishes, making it the only absolutely stable memory at fixed n, h, and q.Source: §3 · p. 15
  • For Kq² < 1, the root’s coefficient of variation diverges because there are too few examples to reinforce it through correlations.This regime combines sufficiently weak correlations with insufficient training examples.Source: §3 · p. 15

S3. ADDITIONAL PROOFS AND DERIVATIONS

The additional proofs organize expectation calculations by separating terms with distinct indices from those with collisions. A helper lemma bounds the nonnegative collision contribution.

S3. ADDITIONAL PROOFS AND DERIVATIONS

  • Splitting the expectation sum into all-distinct and colliding indices makes the all-distinct terms factorize under linearity of expectation.Source: S3. ADDITIONAL PROOFS AND DERIVATIONS · p. 16
  • The derivation uses d_a = d(ξ^μ_a, σ) as shorthand and counts subset configurations using associated Stirling numbers of the second kind.The sums are replaced by counts when they depend on subsets only through their size.Source: S3. ADDITIONAL PROOFS AND DERIVATIONS · pp. 16–17
  • Lemma 7 bounds the nonnegative collision contribution after the indices are divided into colliding and distinct groups.The collision indices are further partitioned according to which indices coincide.Source: S3. ADDITIONAL PROOFS AND DERIVATIONS · p. 16

A. Proof of Theorem 3

The proof compares general index configurations with the all-k_a=1 term and shows that this term approximates the moment; positivity of the relative error yields a lower bound.

A. Proof of Theorem 3

B. Proof of Theorem 4

The proof splits the expansion into distinct-index and collision terms, then bounds the relative error under a growth constraint.

B. Proof of Theorem 4

C. Proof of Theorem 5

The proof bounds the variance by controlling its relative error and exploiting positive correlations; terms with s=0 contribute zero.

C. Proof of Theorem 5

D. Proof of Theorem 6

The proof bounds the squared coefficient of variation and applies a union bound over spins; under the stated parameter scaling, its error term vanishes.

D. Proof of Theorem 6

Loading 2609.12079v1…