Memorization to Generalization: Emergence of Diffusion Models from Associative Memory

1 Introduction

• The paper casts diffusion-model training as writing data into associative memory and generation as recall, which can retrieve training samples or produce unseen samples.This framing applies associative-memory theory to the memorization-to-generalization transition.Source: §1 · p. 2
• The paper addresses limited exploration of the intermediate transition regime, which generalization-centric studies have not fully examined.Those studies typically treat memorization as a small side effect addressed by substantially increasing training data.Source: §1 · p. 2
• Spurious states, historically detrimental to associative-memory recall, are presented as the first signatures of generalization in generative modeling.The paper notes they may also occur during full generalization, including in Stable Diffusion [21].Source: §1 · p. 2
• Diffusion models transition from memorizing training points to generalizing as training data increases, with spurious patterns appearing at the transition.The paper reports sizable attraction basins for these patterns even though they do not correspond to memorized data points.Source: §1 · p. 2
• Figure 1 illustrates the changing energy landscape and compares memorized, spurious, and generalized samples across four datasets.Panel B shows each target image alongside its nearest neighbors from the training and synthetic sets.Source: Figure 1 · p. 3
• The DenseAM perspective links two research areas and can illuminate diffusion models’ computational properties near the memorization-generalization boundary.The paper presents this as a close parallel between DenseAMs and diffusion models.Source: §1 · p. 3

2 Diffusion Models and Dense Associative Memories

• The section introduces a theoretical link between diffusion models and DenseAMs through an energy perspective.It motivates the comparison by asking whether spurious states can occur in diffusion models trained with score matching.Source: §2 · pp. 4–5
• A diffusion model’s reverse denoising process depends on estimating the score ∇_xt log p_t(x_t), which equals the negative energy gradient.The score is estimated with a neural network trained using denoising score matching across multiple time steps.Source: §2 · p. 4
• In the DenseAM comparison, diffusion-model data points play the role of memories, forward noise variance acts like effective temperature, and the reverse process corresponds to memory retrieval.The effective temperature controls separation between memories.Source: §2 · p. 5
• For two stored patterns, some finite-β configurations yield one local energy minimum rather than two, and that minimum can differ from both patterns.The paper identifies such a minimum as a spurious state; in associative-memory models these states emerge when memory load crosses critical capacity, which depends on pattern correlations.Source: §2 · p. 5

3 Discussion

• As training data increases, models progress from memorized training points to spurious attractors absent from the data, then to generalized states on a continuous manifold.The 2D toy model illustrates isolated basins at low K, spurious patterns around K = 9, and a flat manifold at large K.Source: §3 · p. 5, Figure 2 · p. 7
• Memorized samples recur in both training and generated sets, spurious samples recur only in generated samples, and generalized samples recur in neither.Generalized samples occupy flat manifolds, so successive generations differ rather than producing duplicates.Source: §3 · pp. 5–6
• CIFAR-10 and FASHION-MNIST examples show spurious generations recurring synthetically despite having no exact match in their training sets.Examples include a plane with an unusually large wing and a shirt with one sleeve.Source: §3 · p. 6
• As training-set size grows, memorized samples decline, generalized samples increase, and spurious samples peak at the transition boundary.This pattern quantifies the three categories as a function of training data size.Source: §3 · p. 6
• Spurious samples have smaller attraction-basin log-volumes than memorized samples but larger ones than generalized samples, and higher curvature than generalized samples.Generalized samples have near-zero basin volume; memorized and spurious samples tend to have fewer near-zero curvature singular values.Source: §3 · p. 6
• Careful duplicate-based analysis focused on small datasets because large-scale models require substantial training compute and still larger synthetic sets, while curvature analysis is more feasible.The paper also examines Stable Diffusion candidates using score-function Jacobian singular values.Source: §3 · p. 6
• In Stable Diffusion examples, intermediate-curvature candidates resemble training images but differ in details, while flat-curvature candidates vary more.The reported differences include rug patterns and curtain colors; these observations qualitatively align with the small-dataset findings.Source: §3 · p. 7

4 Results

• Diffusion models trained with standard methods generate spurious samples at the onset of the memorization-to-generalization transition, with well-defined attraction basins.The theoretical prediction assumes a score network that learns the true noisy empirical distribution; the empirical finding demonstrates the phenomenon in practical settings.Source: §4 · p. 7
• The paper establishes spurious states as attractors but does not claim they must be mixture states; how they are constructed depends on the score-modeling network.The authors suggest that characterizing this transition may inform future methods to mitigate memorization.Source: §4 · p. 8

4.1 Memorization-to-Generalization Transition

• In the toy model, increasing training-set size moves the energy landscape from memorized minima through emergent spurious minima toward a generalized manifold of minima.At K = 2 the model memorizes; at K = 9 the first spurious states appear; at K = 1000 behavior is fully generalized.Source: §4.1 · p. 8
• The sample-type metrics classify memorized samples by proximity to training data and spurious samples by repeated synthetic-set occurrence without training-set duplication.The basin radius is tied to the largest perturbation from which reverse diffusion can still recover the target pattern.Source: §4.1 · p. 10
• Across datasets, memorization declines and generalization rises as training size increases, while spurious patterns peak near the phase boundary.The figure also reports that memorized and spurious samples have statistically larger average basin log-volumes than generalized samples.Source: Figure 4 · p. 11

4.2 Energy Curvature

• Memorized and generalized samples often separate in curvature spectra, with curvature decreasing at training points as dataset size grows and the learned landscape becomes smoother.Separation among all three sample types is not consistent at every training size, likely because transitional phases are geometrically complex.Source: Figure 5 · p. 12, §4.2 · p. 13
• In a Stable Diffusion case study, memorized samples have the largest curvature, generalized samples the smallest, and candidate spurious samples lie between them.The authors suggest text conditioning constrains the generative manifold and makes the model more prone to high-curvature modes, increasing memorization risk.Source: Figure 5 · p. 12, §4.2 · p. 13

4.3 Relative Energy Across the Transition

• During the transition, spurious states have the lowest energy of the three sample types; as training size grows, memorized-sample energy rises while generalized-sample energy falls.The relative gaps are measured against training samples, whose energy is set to zero.Source: §4.3 · p. 14, Figure 6 · p. 14

5 Conclusion

• Spurious states mark the onset of generalization in diffusion models, connecting their generation process to DenseAM memory retrieval.The paper characterizes these states through their attraction basins, energy properties, and computational behavior.Source: §5 · p. 14
• The paper reports that stable attractors also persist in Stable Diffusion models trained on massive datasets.Source: §5 · p. 14

A.1 Overview

• As β approaches infinity, the toy model’s fully generalized phase has a continuous manifold of local minima centered at R = 1.The energy no longer depends on polar angle ϕ.Source: §A.1 · p. 22

A.2 Experimental Details

• The toy dataset comprises uniformly sampled points on the unit circle, with nested training subsets formed by adding distinct samples without replacement.A 60,000-point dataset was created with a fixed random seed.Source: §A.2 · pp. 22–23

A.3 Log-Likelihood of Diffusion Models

• For toy-model samples, the diffusion-model log-likelihood uses the instantaneous change-of-variable formula, numerically integrating the divergence to T = 1 and adding the prior logarithmic likelihood.Because the toy example is low-dimensional, the Jacobian trace is computed directly without noise estimators rather than using the Hutchinson trace estimator.Source: §A.2 · p. 24, §A.3 · p. 27

A.2 Experimental Details

• The toy score model is an MLP with Fourier timestep features, four residual blocks in each encoder and decoder, and a 2D output layer.Models used continuous-time training with σ = 1 and 800,000 iterations; reverse-SDE samples used 1,000 Euler-Maruyama steps.Source: §A.2 · p. 25

B.1 Selection of Points

• The transition search first doubles training-set sizes from 500, then samples 30 inclusive linear points between critical points A and B, with five points on each outer region.A marks the initial memorization drop and B its plateau; Table 1 reports these critical points by dataset.Source: §B.1 · p. 27, Table 1 · p. 27

B.2 Selection of Distance Metric

• For CIFAR10 and LSUN-CHURCH, memorization and spurious detection use LPIPS with an AlexNet backbone; MNIST and FASHION-MNIST use L2 distance.LPIPS is selected for high-dimensional images, while L2 is used for the simpler single-channel datasets.Source: §B.2 · pp. 27–28

B.3 Selection of Threshold Values

• The δm and δs thresholds are manually selected from bimodal distance histograms and checked against inspected samples to reduce false positives.The authors note that FASHION-MNIST’s gray-histogram bimodality may be weak, though sufficient to select δs.Source: §B.3 · p. 28

B.4 Model and Training Details

• DDPM-based PixelCNN++ U-Nets use fixed variances, 1,000 timesteps, and a shared learning rate, with channel multipliers adjusted for dataset complexity.The setup omits random flipping; LSUN-CHURCH images are center-cropped and down-sampled to 64 × 64.Source: §B.4 · p. 28
• On CIFAR10, increasing training-set size reduces memorized samples, while spurious patterns rise and fall around the transition and generalized patterns increase.Smaller models transition earlier; larger models can memorize more training data before the transition.Source: Figure 7 · p. 29

B.5 Additional Results

• Spurious-sample peaks occur at K = 2568 versus K = 5862 for smaller and larger CIFAR10 U-Nets, and K = 4862 versus K = 6086 for LSUN-CHURCH.The transition occurs earlier with U-Net (64) than with the larger U-Net models.Source: §B.5 · pp. 29–30
• At CIFAR10 K = 5862, the spurious fraction rises from 0.04 with 4× synthetic sets to roughly 0.06 with 8× sets.The memorization-generalization trend and spurious-sample emergence remain despite the larger synthetic-set multiplier.Source: §B.5 · p. 30, Figure 9 · p. 31

B.6 Additional Visualizations of Distance Histogram

• Distance histograms distinguish memorized samples below δm from generalized and spurious samples above it, while δs separates peaks in synthetic-neighbor distances.The figures show examples and nearest neighbors for LSUN-CHURCH, FASHION-MNIST, and MNIST.Source: Figure 10 · p. 34, Figure 11 · p. 35, Figure 12 · p. 36

B.7 Additional Examples of the Three Distinct Sample Types

• Memorized samples duplicate training data, and duplicates can also appear in the synthetic set during strong memorization.The figure shows nearest neighbors from both training and synthetic sets, although the detection metric does not use the synthetic set.Source: Figure 13 · p. 37
• Spurious patterns emerge as generalization begins and training samples start mixing, but they are not yet unique enough to count as generalized samples.Duplicates may appear several times in the synthetic set at this early stage.Source: Figure 14 · p. 38
• Generalized samples are novel and have little or no resemblance to their nearest neighbors in the training or synthetic sets.Source: Figure 15 · p. 39

C.1 Algorithm Details

• Critical time is estimated by perturbing an image at sampled timesteps, denoising each perturbation with DDIM, and tracking when recovery fails.LPIPS with a VGG-16 backbone compares recovered and original images; recovery probability across trials is checked against a threshold to stop the search.Source: §C.1 · p. 40

• For each training-data size, the analysis uses up to 512 samples per type and computes the logarithmic N-ball volume from the critical time and perturbation radius.Non-training samples are sorted by distance before selection; the radius is the Euclidean distance between an image and its recoverable perturbation.Source: §C.2 · p. 42
• The critical-time search uses stopping probability 0.8, 20 perturbations, 10 DDIM steps, and stride 10; statistics are omitted when a type has under 0.1% of its synthetic set.Source: §C.2 · p. 42
• CIFAR10 and LSUN-CHURCH use distance thresholds 0.03 and 0.12, while MNIST and FASHION-MNIST both use LPIPS threshold 0.1.The latter threshold was selected by comparing candidate values from 0.01 to 0.15 at three data sizes per dataset, with manual checks of recovered images.Source: §C.2 · p. 42

• For CIFAR10 and LSUN-CHURCH, the plots show average log-volume by sample type as training-data size grows, alongside training-sample averages and transition plots.The caption says critical-time variation, which affects radius, leads to different log-volume trends across sample types.Source: Figure 16 · p. 43
• For FASHION-MNIST and MNIST, the plots likewise show average log-volume by sample type across training-data sizes, with training averages and transition plots for context.The caption also notes that variation in critical time and radius leads to distinct trends across sample types.Source: Figure 17 · p. 44

C.4 Average Critical Time Results

• Average critical times are plotted across training-data sizes for CIFAR10, LSUN-CHURCH, FASHION-MNIST, and MNIST, with transition plots guiding where the transition occurs.The captions describe an overall decrease in training-sample critical time and distinct critical times for each sample type at each data size.Source: Figure 18 · p. 45, Figure 19 · p. 46

D.1 Overview

• The cited stable algorithm computes singular values of the score-function Jacobian, and negative semi-definiteness is given as a stability condition for the support-score Jacobian.The overview calls the relevant time-dependent points the stable latent set of the diffusion process [31].Source: §D.1 · p. 47
• The sorted eigenvalue spectrum reflects local manifold geometry: tangent-aligned perturbations have small eigenvalues, while orthogonal perturbations have high eigenvalues.A sharp drop in the sorted spectrum can estimate manifold dimensionality [57].Source: §D.1 · p. 47

D.2 Experimental Details

• At t0 = 2, the score Jacobian spectra use up to 512 samples per type, omit the top five singular values, and compare up to 2,048 training points.For Stable Diffusion, the authors instead use Algorithm 3 at t0 = 5, skip the top 50 singular values, and cap the plotted y-axis at 1500.Source: §D.2 · pp. 47–48

D.3 Curvature Results

• In the intermediate regime, memorized samples have the highest curvature, spurious samples the next highest, and generalized samples the lowest; CIFAR10 spectra remain overlapped through the transition.At large K, spectra for all three sample types become more distinguishable across the reported datasets.Source: Figure 20 · p. 50, Figure 21 · p. 51, Figure 22 · p. 52

D.4 Additional Examples of Stable Diffusion

• Stable Diffusion candidate memorized and spurious samples are stable attractors, with singular values larger than those of candidate generalized samples.Memorized candidates have much larger singular values than generalized candidates, while spurious candidates lie between them.Source: Figure 24 · p. 54

E.1 Overview

• The method projects a diffusion model’s potential onto a one-dimensional generative trajectory between target and reference images, using circular interpolation.The section derives a potential function from the reverse SDE and uses it to express the generative process.Source: §E.1 · pp. 55–56

E.2 Experimental Details

• The relative energy-gap estimates use up to 512 samples per sample type and up to min(K, 2048) training points, omitting sets below 0.1% of their corresponding synthetic set.A randomly selected training image serves as the reference for the relative-energy calculation.Source: §E.2 · p. 56

E.3 Discussion

• Spurious samples’ relative potential resembles that of training samples in the intermediate phase, then diverges as generalization advances; the generalized-to-training energy gap becomes the narrowest.This pattern emphasizes changes in the energy landscape as training-data size increases.Source: §E.3 · p. 57

E.4 Average Relative Energy Gap Results

• Memorized samples’ energy gap is initially close to zero and quickly diverges as K grows, while spurious samples resemble training data energetically in the intermediate phase.In the later generalization stages, the generalized-to-training energy gap decreases.Source: Figure 26 · p. 59

E.5 Average Relative Energy Gap Results with Different References

• Across four reference images per dataset, relative energy-gap values vary slightly but retain the same general trend.The reported comparisons cover CIFAR10, LSUN-CHURCH, FASHION-MNIST, and MNIST.Source: Figure 27 · p. 60, Figure 28 · p. 61, Figure 29 · p. 62

F Hardware Details

• Training took roughly 18–22 hours per CIFAR10 model, 48–103 hours per LSUN-CHURCH model, and about 17 hours per FASHION-MNIST or MNIST model.Each model was trained on one NVIDIA Tesla V100 GPU.Source: F Hardware Details · p. 64