PaperlayerView exact revision on arXiv

Source-linked AI summary

Bias-Induced Crossover in Absolute Capacity of Dense Associative Memory

Yuto Sakurai, Takeaki Shimokawa, Kazunori Iwata, Kazushi Mimura

arXiv:2609.17477v1cond-mat.dis-nncs.LG

TL;DR

The paper asks how bias affects the absolute capacity of dense associative memory under a single-site error criterion. It analyzes centered biased patterns with a conditioned-Gaussian signal-to-noise method and finds bias-dependent capacity powers, a crossover near the unbiased limit, and recovery of the unbiased order under activity control.

  • Problem

    The paper examines how bias in centered binary patterns changes the absolute capacity of dense associative memory, which had mainly been analyzed for unbiased patterns.

  • Method

    The paper uses signal-to-noise analysis with finite-size conditioned-Gaussian crosstalk distributions, asymptotic matching, simulations, and an activity-dependent control potential.

  • Results

    For fixed q < 1/2, capacity is O(N^(n/2)) for even n ≥ 4 and O(N^((n+1)/2)) for odd n ≥ 5, while n = 3 remains O(N^2/ln N).

  • Takeaways & Limitations

    The crossover is caused by a bias-dependent conditional crosstalk mean, and activity control restores O(N^(n−1)/ln N) within the conditioned-Gaussian approximation.

Abstract

from arXiv · show

The absolute capacity of dense associative memory has mainly been analyzed for unbiased patterns. Here we examine the effect of bias in centered binary patterns under the Krotov-Hopfield single-site criterion $P_{\mathrm{error}}=1/N$, where $P_{\mathrm{error}}$ is the probability that a single-site flip lowers the energy of a stored pattern and $N$ is the number of neurons. Each pattern component takes $1-q$ with probability $q$ and $-q$ otherwise, where $0<q\le1/2$. For polynomial interactions of order $n$, a signal-to-noise analysis gives an absolute capacity of order $N^{n-1}/\ln N$ at $q=1/2$. For fixed $q<1/2$, however, the capacity is $O(N^{n/2})$ for even $n\ge4$ and $O(N^{(n+1)/2})$ for odd $n\ge5$. For $n=3$, both the unbiased and fixed-bias capacities remain $O(N^2/\ln N)$. For $n\ge4$, these different asymptotic forms imply a nonuniform large-$N$ limit near $q=1/2$. Asymptotic matching predicts a bias-induced crossover in the region $1-2q=O(\ln N/N^{\lfloor n/2\rfloor-1})$. The crossover originates from a bias-dependent crosstalk mean that reduces the stability of sites carrying the more frequent value $-q$. Computer simulations are compared with the finite-size conditioned-Gaussian predictions. An activity-dependent control potential that cancels the conditional crosstalk mean restores the $N^{n-1}/\ln N$ capacity for fixed $0<q<1/2$ within the conditioned-Gaussian approximation.

I. INTRODUCTION

The paper studies absolute, single-site stability capacity in dense associative memory with centered biased patterns, contrasting it with relative retrieval capacity and prior unbiased results. It develops a finite-size and asymptotic analysis of how bias changes crosstalk distributions and capacity.

  • I. INTRODUCTION: The absolute-capacity criterion sets P_error = 1/N, where P_error is the probability that a single-site flip lowers a stored pattern’s energy.This is an ensemble-averaged single-site criterion, not a simultaneous-stability condition for every stored memory.
  • I. INTRODUCTION: For polynomial interactions of order n, unbiased dense associative memory has absolute capacity O(N^(n−1)/ln N), whereas its storage capacity is O(N^(n−1)).The paper distinguishes stability-based absolute capacity from dynamical retrieval and relative capacity.
  • I. INTRODUCTION: Centered biased patterns use values 1−q and −q with probabilities q and 1−q, respectively, giving zero mean but unequal component probabilities and magnitudes when q < 1/2.The usual unbiased case is recovered up to scale at q = 1/2.
  • I. INTRODUCTION: For q < 1/2, conditioning crosstalk on the retrieved-site value produces unequal distributions, so the more frequent −q component can control the negative energy-gap tail.The conditional distributions coincide by symmetry at q = 1/2.
  • I. INTRODUCTION: The analysis combines a finite-size binomial formulation, conditioned-Gaussian crosstalk approximation, asymptotic capacity calculation, crossover analysis, simulations, and an activity-dependent control potential.The model defines an energy gap for a single-site flip and counts the site as unstable when that gap is negative.

III. ANALYSIS

The analysis decomposes the single-site energy gap into signal and crosstalk, retains exact finite-size binomial structure, and approximates only the sum of crosstalk contributions by a Gaussian.

  • III. ANALYSIS: The finite-size calculation conditions on M, the number of other retrieved-pattern sites equal to 1−q, before computing conditional error probabilities and averaging over M.M follows a binomial distribution with parameters N−1 and q.
  • III. ANALYSIS: The single-site energy gap at a retrieved pattern is separated into a signal term and crosstalk noise, with instability defined by a flip that lowers the energy.The capacity is estimated from the resulting single-site error probability.
  • III. ANALYSIS: Each non-condensed pattern contributes crosstalk through its overlap with the retrieved state, with conditional moments determined from binomial variables and the retrieved-site value.Conditioning preserves the finite-N distribution of an individual crosstalk contribution.
  • III. ANALYSIS: The method applies a Gaussian approximation only to the sum of the K−1 crosstalk contributions, retaining exact binomial counts, the signal, and one-term finite-size distributions.This yields a conditioned energy-gap distribution and finite-size error probability that are then averaged over M and the site value.

B. Large-N Distributions

The large-N analysis keeps separate conditional energy-gap distributions for the rare and frequent site values, because bias separates their crosstalk behavior and can make one lower tail dominant.

  • B. Large-N Distributions: At q = 1/2 the conditional energy-gap distributions coincide, whereas bias separates them according to the retrieved-site symbol.The more frequent −q component is the relevant conditional case for q < 1/2.
  • B. Large-N Distributions: The total variance includes both fluctuations of the conditional mean with activity M and the common shift shared by the K−1 crosstalk terms.Replacing M by its mean would omit these contributions.
  • B. Large-N Distributions: The single-site error probability is formed by calculating lower-tail probabilities for each conditional site value and then combining them with their probabilities q and 1−q.The dominant conditional tail determines the asymptotic stability estimate.

C. Dominant Conditional Tail

For fixed bias, the dominant conditional tail changes the capacity power for n ≥ 4, while n = 3 retains the unbiased order; matching these regimes produces a smooth crossover near q = 1/2.

  • C. Dominant Conditional Tail: For fixed 0 < q < 1/2, even n ≥ 4 gives K_max ∼ 2q (n−1)!!(1−2q)N^(n/2), changing the unbiased power O(N^(n−1)/ln N).The decreasing conditional mean for the frequent component vanishes at the leading-order capacity balance.
  • C. Dominant Conditional Tail: For fixed 0 < q < 1/2, odd n ≥ 5 follows the same mean-balance mechanism and has a different power of N from the unbiased capacity.The conditioned crosstalk overlap contributes an additional leading odd-moment term.
  • C. Dominant Conditional Tail: For n = 3, the crosstalk variance rather than its mean determines the leading balance, so biased and unbiased capacities retain the same N^2/ln N order.The n = 3 coefficient tends to 1/6 as q approaches 1/2, matching the unbiased result.
  • C. Dominant Conditional Tail: The resulting finite-size change is a smooth crossover between large-N regimes, not a thermodynamic phase transition.The paper studies n = 4 and n = 5 as the lowest orders where the asymptotic powers differ.

A. Interaction Order

The cubic model retains the same asymptotic capacity order for unbiased and fixed-biased patterns, whereas orders n≥4 exhibit distinct powers of N connected by a finite-size crossover near q=1/2.

  • A. Interaction Order: For n = 3, both unbiased and fixed-bias capacities are O(N^2/ ln N), so the cubic model has no crossover between different powers of N.The fixed-bias coefficient approaches the unbiased coefficient as q increases toward 1/2.
  • A. Interaction Order: For even n≥4, the unbiased and fixed-bias powers are n−1 and n/2, while for odd n≥5 they are n−1 and (n+1)/2.The fixed-bias power is smaller in both parity classes, producing two asymptotic forms for every n≥4.
  • A. Interaction Order: The crossover width is O(ln N/N) for n = 4, 5 and O(ln N/N^2) for n = 6, 7, becoming narrower as interaction order increases.This width is an asymptotic matching estimate within the Gaussian signal-to-noise approximation.
  • A. Interaction Order: For the quartic model, the crossover occurs when 1−2q = O(ln N/N), where the signal and bias-dependent crosstalk mean are comparable at the unbiased capacity.The two asymptotic estimates need only be comparable in order, not exactly equal.
  • A. Interaction Order: The normalized capacity R_N equals 1 at q = 1/2 but is O(ln N/N) for fixed q<1/2 when n = 4 or 5.Thus the normalized capacity tends to zero at fixed bias while remaining unity in the unbiased case.

V. NUMERICAL RESULTS

Finite-size conditioned-Gaussian predictions are compared with simulations for cubic, quartic, and quintic models. Simulations reproduce the main N-, q-, and crossover-dependence, with deviations where the Gaussian approximation becomes inaccurate or simulations are limited.

  • V. NUMERICAL RESULTS: Simulations reproduce the conditioned-Gaussian theory’s dependence on N and q for the cubic and quartic capacity curves.Figures 2 and 3 compare theoretical solutions of Eq. (30) with explicitly generated binary-pattern simulations.
  • V. NUMERICAL RESULTS: At small q, the conditioned-Gaussian quartic result can reach K_max = 1 while simulations give a larger value because the activity and relevant pattern load are small.A direct discrete evaluation would be required in this region.
  • V. NUMERICAL RESULTS: The quartic normalized capacity approaches Y_N = 1 for X_N≪1, decreases through X_N = O(1), and approaches Y_N = X_N^-1 for large X_N.The crossover occurs over a similar X_N range for different N, consistent with the predicted scaling variables.
  • V. NUMERICAL RESULTS: The quartic simulations reproduce the plateau and decrease through the crossover region, but deviate from finite-size theory at larger X_N.The theoretical curves are obtained by solving Eq. (30), rather than joining asymptotic forms.
  • V. NUMERICAL RESULTS: The quintic theory likewise decreases through X_N = O(1) toward Y_N = X_N^-1, while N = 100 simulations reproduce the plateau and crossover but depart at large X_N.The predicted O(ln N/N) width was not separately measured in simulations, and quintic simulations used only one system size.

VI. ACTIVITY CONTROL

An activity-dependent potential is introduced as a static energy control that cancels the conditional crosstalk mean. Within the conditioned-Gaussian approximation, this restores the unbiased capacity order for fixed bias.

  • VI. ACTIVITY CONTROL: The control is implemented through a static energy potential V that uses the same function for every state and does not refer to the stored-pattern index.Its local increment acts as an activity-dependent threshold.
  • VI. ACTIVITY CONTROL: A constant threshold cannot remove activity-dependent crosstalk-mean variation, whereas an activity-dependent threshold can cancel the coherent common fluctuation.The constant threshold can shift the average energy gap but cannot eliminate its variation with realized activity.
  • VI. ACTIVITY CONTROL: The activity-dependent control cancels the common activity fluctuation in the conditional crosstalk means.This cancellation equalizes the controlled conditional variances while shifting the two conditional gap means by opposite threshold terms.
  • VI. ACTIVITY CONTROL: For every fixed 0<q<1/2, the activity-dependent control restores the N^(n−1)/ln N capacity order within the conditioned-Gaussian approximation.At q = 1/2, the controlled result reduces to the unbiased capacity expression.
  • VI. ACTIVITY CONTROL: The controlled capacity recovery is qualitatively related to an activity-controlled result in prior work, but the state, control formulation, and capacity criterion differ.The comparison therefore concerns a qualitative relation rather than an identical model or criterion.

VII. DISCUSSION

The discussion identifies the bias-induced crossover as a consequence of an unbalanced conditional crosstalk mean and describes an activity-dependent control potential that restores the unbiased capacity scaling within the conditioned-Gaussian approximation.

  • For fixed q < 1/2, the more frequent −q component determines the lower tail, while activity control removes its conditional crosstalk mean and balances the conditional error fluxes.The control potential restores N^(n−1)/ln N capacity for fixed q within the conditioned-Gaussian approximation.
  • The crossover is caused by the bias-dependent crosstalk mean, which makes the more frequent component’s lower tail determine stability despite centered patterns.Centering removes the pattern mean but does not make the two component values equivalent.
  • The fixed-q sparse limit is not established: taking q → 0 after N → ∞ is not equivalent to the joint limit q = q_N → 0.A uniform Gaussian approximation would require conditions such as q_N N → ∞, and sparse models may introduce additional logarithmic factors.
  • The controlled model remains energy-based because its potential depends only on input activity, but moderate-deviation justification and controlled-model simulations remain open.The current restoration result relies on a conditioned-Gaussian tail approximation at probability 1/N.
  • For n = 3, unbiased and fixed-bias capacities share the N^2/ln N order, whereas n ≥4 exhibits different powers and a crossover near q = 1/2.For quartic and quintic models, the changes occur in the region 1−2q = O(ln N/N), and simulations reproduce the crossover over accessible sizes.

Appendix A: Conditional Moments and Covariances

The appendix derives conditional moments and the full finite-size variance by separating signal, crosstalk, and covariance contributions, including activity-mediated dependence between terms.

  • For fixed q < 1/2, a two-dimensional central limit theorem supports the conditional Gaussian treatment, while the unbiased case has no activity fluctuation.At q = 1/2, only the second component of the covariance structure is needed.
  • The energy gap is decomposed into a signal term and crosstalk noise, whose conditional means depend on the retrieved-site value and condensed-pattern activity.For ξ_i^μ = −q, the nonlinear conditional crosstalk factor changes sign and exchanges q with 1−q.
  • For even n ≥4, k = 2 supplies the leading crosstalk term; for odd n ≥5, skewness at k = 2 and the k = 3 term contribute at the same order.The odd-moment contribution is needed for the leading odd-order calculation.
  • The complete variance includes diagonal crosstalk variance, signal variance, signal–crosstalk covariance, and covariances between distinct crosstalk terms.These terms are combined through the law of total variance because crosstalk contributions share a random conditional mean.
  • For n = 3, the leading crosstalk mean is independent of activity, so the diagonal crosstalk variance determines the capacity scale.In the crossover window, common-activity covariance is asymptotically smaller than diagonal crosstalk variance and does not change the matching scale.

Appendix B: Comparison of Conditional Tails

The appendix compares conditional Gaussian tails and shows that the frequent −q component controls the fixed-bias absolute-capacity criterion, with a separate behavior for n = 3.

  • For even n ≥4 and odd n ≥5, capacity is asymptotically set by the point where the conditional mean for ξ_i^μ = −q vanishes.The other conditional mean remains O(N^(n−1)) with a smaller standard deviation, so its squared signal-to-noise ratio is at least O(N).
  • For n = 3, the two conditional tails share the leading crosstalk variance, but the rare component has a larger leading signal, so the frequent component still determines the criterion.The difference between the squared signal-to-noise ratios is a positive constant times ln N.
  • The fixed-q tail comparison is not uniform in the crossover window, where the conditional balances become comparable.The fixed-bias argument therefore cannot by itself justify the crossover behavior near q = 1/2.

Appendix C: Asymptotic Capacity

The appendix derives asymptotic capacity behavior separately for unbiased patterns, fixed bias, and the cubic case. It also identifies the resulting capacity relations summarized in Subsection III D.

  • Unbiased patterns: At q = 1/2, the activity produces no conditional-mean fluctuation, while the signal scales as N^(n−1) and the crosstalk variance as K N^(n−1).The capacity condition uses µ²/σ² = 2 ln N + O(ln ln N).
  • Fixed bias: For fixed 0 < q < 1/2 and even n ≥4, the leading frequent-component conditional mean scales as 2q (n−1)!!(1−2q)N^(n/2).The Gaussian-tail condition contributes only a relative o(1) correction at this scale.
  • Fixed bias: For odd n ≥5, the k = 2 skewness and k = 3 terms contribute at the same order before comparison with the complete variance.The appendix treats their sum as the relevant fixed-bias contribution.
  • Cubic model: For n = 3, the crosstalk mean is subleading at the capacity scale, so the leading signal and conditioned variance determine the result.The resulting logarithmic expression is recorded as Eq. (C8).
  • Appendix C: Asymptotic Capacity: The appendix derives the asymptotic capacities for the unbiased, fixed-bias, and cubic cases, summarized in Subsection III D.The derivation uses conditional means, variances, Gaussian-tail conditions, and capacity roots.

Appendix D: Activity Control

The activity-control analysis removes conditional crosstalk means and the associated covariance contribution. It then derives the controlled variance and capacity scaling while showing that signal fluctuations are asymptotically negligible.

  • Activity Control: The control removes the K2 crosstalk contribution, leaving an exactly specified remaining variance for fixed activity M = m.The cancellation follows from vanishing conditional means and independence of different non-condensed patterns at fixed M.
  • Large-N capacity: The leading variance of one centered crosstalk contribution scales as n²(2n−3)!!{q(1−q)}^(2n−1)N^(n−1).This expression enters the controlled large-N variance calculation.
  • Large-N capacity: At capacity scale K = O(N^(n−1)/ln N), signal fluctuations contribute only O(ln N/N) relative to the crosstalk variance.The total crosstalk variance is O(N^(2n−2)/ln N), whereas each signal variance is O(N^(2n−3)).
  • Gaussian-tail condition: The controlled conditional error probabilities are both of order 1/N and correspond to positive Gaussian quantiles.The leading capacity equations are obtained from these Gaussian-tail relations.
Loading 2609.17477v1…