PaperlayerView exact revision on arXiv

Source-linked AI summary

From Modes to Memories: Characterizing the Scale-Space Dynamics of Diffusion Models

Cristina López Amado, Marco Fumero, Francesco Locatello

arXiv:2609.39648v1cs.LGcs.AIcs.CV

TL;DR

Rather than reading diffusion models only through their full sampling process, the paper studies the denoiser at each fixed noise scale. It defines persistence as a critical scale and reports that this tracks several memorization mechanisms in controlled tests and identifies fully and partially memorized Stable Diffusion pairs.

  • Problem

    Diffusion models are commonly read through samples or loss across a full noise schedule, while this paper asks how model structure and memorization vary at fixed noise scales.

    Source: §1 · p. 1

  • Method

    The paper iterates each fixed-noise denoiser as a deterministic map and measures example persistence with σc, extending the measure to caption–image pairs.

    Source: §1 · p. 2

  • Results

    Controlled experiments show σc tracks duplication, overfitting, and outlier memorization, while Stable Diffusion tests detect fully and partially memorized image–caption pairs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The measure supports interpretations of conditional retention in terms of caption–image interaction and of where memorized content lies within an image.

    Source: §1 · p. 2

    The theory is proved for the exact denoiser; formalizing its transfer to trained networks and quantifying localization precision and generality remain future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

Diffusion models are typically viewed as stochastic processes that transform noise into data. We take a complementary perspective: a diffusion model defines a family of deterministic dynamical systems indexed by noise scale. At each fixed scale $\sigma$, we treat the denoiser as a self-map and study its dynamics. For an exact denoiser, fixed points correspond to critical points of the smoothed data density, while attractors correspond to its modes; as $\sigma$ increases, sample-level modes merge into progressively coarser ones. This suggests a geometric view of memorization: examples that receive excess probability mass due to duplication or overfitting, as well as outliers, should remain distinguishable under stronger smoothing than ordinary examples. We quantify this persistence by the critical scale $\sigma_c$, the largest noise scale at which an example is retained by the fixed-scale dynamics. In conditional models, the same construction extends naturally to image--caption pairs. Experiments in controlled settings and on large-scale models show that $\sigma_c$ tracks memorization arising from duplication, overfitting, and outliers, and identifies both memorized and partially memorized examples in Stable Diffusion. Moreover, $\sigma_c$ yields interpretable measures of the image spatial distribution and caption dependence of memorization.

1 Introduction

The paper treats a diffusion model as deterministic dynamics at each noise scale, where attractors of an exact denoiser track modes of the smoothed data density. It uses persistence across scales to characterize memorization and tests the measure in controlled experiments and Stable Diffusion.

1 Introduction

  • At fixed noise, iterating the denoiser defines deterministic dynamics whose exact attractors are smoothed-density modes that merge into coarser modes as noise rises.The scale-space view complements reading a model through samples or loss across its full noise schedule.Source: §1 · pp. 1–2
  • Critical scale σc measures how long an example remains retained, linking persistence to excess probability mass from duplication or overfitting and to isolation as an outlier.In Figure 1, the memorized image's σc is 6.7× that of the control image.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Controlled experiments show that σc detects memorization induced by duplication, overfitting, and planted outliers.These interventions test whether the mechanism transfers from theory to trained networks.Source: §1 · p. 2
  • On Stable Diffusion, σc detects fully and partially memorized image–caption pairs and supports interpretations of caption dependence and memorized image regions.The measures characterize whether retention comes from the caption, image, or their interaction.Source: §1 · p. 2

2 Diffusion models as a family of dynamical systems

At each noise level, the denoiser can be studied as a deterministic map; for an exact denoiser, its dynamics are tied to modes of the smoothed data density. The paper uses the critical scale to measure example persistence and relates it to probability mass, distance, and caption effects.

2 Diffusion models as a family of dynamical systems

  • The denoiser is a family of noise-indexed maps, and fixing σ and iterating the denoiser on its own output defines a deterministic dynamical system.The denoiser's minimizer under the stated training objective is the posterior mean.Source: §2 · pp. 2–3
  • For an exact denoiser, one map step is gradient ascent on the smoothed density, Mσ(x) = x + σ^2∇log pσ(x).The exact-denoiser assumption is what supports this explicit description.Source: §2 · p. 3
  • For an exact denoiser, fixed points are critical points of the smoothed density and attractors are its modes; as σ increases, individual attractors merge into population modes and eventually one attractor near the data mean.Figure 2 illustrates the small-, intermediate-, and large-scale regimes on an 8-mode Gaussian mixture.Source: §2 · p. 3, Figure 2 · p. 3

2.1 Memorization and the critical scale

  • The critical scale is the largest σ at which an example's orbit remains close to it under iteration; above it, the orbit may be drawn toward an attractor shared with other data.The retention test initializes the dynamics at the example and checks whether the orbit stays within the escape radius for the iteration budget.Source: §2.1 · p. 4
  • Doubling neighbor distance doubles σc, while doubling copies raises it only a few percent; at small mass, the theoretical scale grows linearly with distance and logarithmically with mass.In the 2-D example, the datum repeated 15 times retains its attractor until merging with neighboring sub-clusters.Source: §2.1 · p. 4, Figure 3 · p. 4
  • For conditional models, memorization is a caption–image-pair property, and the caption gap compares conditional and unconditional critical scales.For a single iteration, when the caption leaves other examples' relative weights unchanged, a positive gap is equivalent to the caption raising the model's mass on that example.Source: §2.1 · p. 5
  • The theory concerns the exact denoiser, so trained networks need not reproduce its numerical critical-scale values, although the authors expect the mass-or-isolation mechanism to carry over.The paper reports experiments on trained networks and notes that formalizing this transfer remains open.Source: §2.1 · p. 5

3 Controlled experiments

Controlled CIFAR-10 interventions test whether critical scale detects memorization from duplication, rarity, and overfitting. It tracks all three mechanisms, including rare additions that sampling does not detect.

3 Controlled experiments

  • σc detects memorization from duplication, overfitting, and planted outliers in controlled CIFAR-10 experiments.The experiments isolate these mechanisms through training interventions because they are confounded in pretrained models.Source: §3 · p. 5
  • With model capacity and training budget fixed, reducing training-set size moves the model from generalization to memorization, which σc tracks even where sampling no longer finds copies.The intervention shows that an isolated outlier can be memorized without being sampled because σc depends on distance as well as mass.Source: §3 · p. 5, §3.2 · p. 7

3.1 Memorization from data duplication

  • AUC 1.000 separates duplicated images at m ≥8 from m = 1; in the duplicate-trained model median σc is 1.04–1.43 for m ≥8 versus 0.142 at m = 1.In the control model, the same images have σc between 0.12 and 0.27 at every multiplicity; Figure 4 compares groups across models and iteration budgets.Source: §3.1 · p. 5, Figure 4 · p. 6

3.2 Memorization of rare samples

  • The σc detection threshold falls with atypicality—from m = 8 for CIFAR-10 and CIFAR-100 to m = 4 for SVHN and m = 1 for colourised MNIST.Sampling detects added images only from m ≈8 for every source; among never-sampled additions, σc flags 88.2% for MNIST, 48.6% for SVHN, 25.6% for CIFAR-10, and 22.0% for CIFAR-100.Source: §3.2 · pp. 6–7, Figure 5 · p. 6

4 Detecting Memorization in Stable Diffusion

The Stable Diffusion experiments evaluate critical-scale scores on memorized and control image-caption pairs, then examine spatial localization and image-driven retention. The scores distinguish memorized examples, and local and conditional-unconditional comparisons provide interpretable evidence about where retention lies and whether captions contribute.

4 Detecting Memorization in Stable Diffusion

  • The evaluation uses 70 matching-verbatim and 24 template-verbatim examples, with 94 training-member controls; global scores measure image-caption retention, while local scores test retention at each latent location.Each memorized image is tested with its own, swapped, and empty caption; controls use their own and the empty caption.Source: §4 · p. 7

4.1 Critical scale detects memorization

  • Local |∆log σc| achieves AUCs of 0.997 for MV and 0.993 for TV versus controls, and 0.935 for separating MV from TV; for TV, it raises AUC from 0.879 to 0.993.All scores separate memorized examples from controls, while the signed gap performs best for caption swaps; Table 1 reports detection tasks and score configurations.Source: §4.1 · p. 7

4.2 Local critical scale localizes memorized regions

  • Local critical-scale gaps align with regions consistently reproduced across generations, while variable regions generally have gaps near zero.In partially memorized images, non-memorized content disappears at smaller noise scales than memorized regions.Source: §4.2 · p. 8

4.3 Critical-Scale Gap Identifies Image-Driven Memorization

  • Strongly negative mismatched-caption gaps provide evidence of unconditional image retention: those cases also have lower inpainting RMSE and larger unconditional σc.Gaps near zero are ambiguous: both dynamics may retain the image, or neither may; unconditional inpainting helps evaluate image-driven retention.Source: §4.3 · pp. 8–9

5 Related work

Related work connects diffusion models to denoising and multiscale dynamics, studies memorization and generalization, and develops detection methods. This paper differs by iterating fixed-scale denoisers and measuring pair-specific retention across noise scales.

Dynamical and multiscale views of diffusion models

  • Prior multiscale studies follow generation trajectories, whereas this work fixes the noise scale and iterates the denoiser to track attractors and their persistence.Mean shift links Gaussian-kernel density estimates to mode-seeking dynamics [Fukunaga and Hostetler, 1975, Cheng, 1995, Comaniciu and Meer, 2002], and posterior-mean denoising connects these dynamics to smoothed-density modes.Source: §5 · p. 9

Memorization and generalization in diffusion models

  • Prior work studies diffusion memorization through training dynamics, reverse trajectories, and autoencoder fixed points; this work instead tracks attractor persistence along a scale axis.Related findings include memorization following good generation during training [Bonnaire et al., 2025] and reverse-trajectory steering into memorized-image basins [Jain et al., 2025].Source: §5 · p. 9

Detecting memorization in diffusion models

  • Detection methods include matching or extracting generations, prompt-based scores, and image- or region-level geometry; this score tests whether a candidate image-caption pair is retained, not training membership.Comparing conditional and empty-caption critical scales also detects unconditional-branch memorization.Source: §5 · p. 9

6 Conclusions

The paper frames diffusion denoisers as fixed-scale dynamical systems and measures memorization through the critical scale at which an example remains an attractor. On Stable Diffusion, the measure detects fully and partially memorized pairs, with conditional comparisons and local variants offering interpretations of caption interaction and image regions.

6 Conclusions

  • On Stable Diffusion, σc detects fully and partially memorized image-caption pairs; comparing conditional and unconditional scales isolates caption-image interaction, while local variants identify memorized image regions.The precision and generality of localization still need quantification.Source: §6 · pp. 9–10
  • The theory applies to exact denoisers; although the mechanism transfers to trained networks, formally closing that gap remains future work.The conclusion also identifies further work on localization precision and generality.Source: §6 · p. 10

AI use statement

The selected material documents AI-assisted implementation and editing and the work’s public-data scope, while presenting exact-denoiser dynamics and a mass-monotonicity result for the critical scale.

AI use statement

  • Generative AI assisted code implementation and modification and language editing; the authors designed the methodology and experiments and reviewed and validated all AI-assisted code and text.The tools did not produce research results on their own or replace the authors’ scientific judgment.Source: AI use statement · p. 11

Ethics statement

  • The work uses public datasets and models and previously identified memorized prompts, involves no human subjects, and collects no new data; the authors note possible misuse to recover training data.The experiments score already-public images, and the authors release no new private or copyrighted content.Source: Ethics statement · p. 11

Reproducibility statement

  • The experiments use PyTorch and HuggingFace diffusers, public checkpoints and recipes, with models, schedules, estimator settings, and filtering details listed in Appendix F.The authors state they will open-source the experimental codebase upon acceptance.Source: Reproducibility statement · p. 11

A Denoising dynamics in Stable Diffusion

B Proofs

  • The formal setup assumes p is supported on a compact convex set and defines the exact denoiser as a posterior mean; an attractor has Jacobian eigenvalues of modulus below one.The set assumption covers image data in [−1, 1]^d and any finite training set by taking its convex hull.Source: B Proofs · p. 15

B.1 The exact map (Section 2)

  • For exact denoisers, iteration is Gaussian mean shift: fixed points are critical points of pσ and attractors are its nondegenerate modes, with attractors merging as σ increases.At small σ, every training example is an attractor; above half the data diameter, one attractor remains.Source: §B.1 · p. 16

B.2 The critical scale and mass monotonicity (Sections 2.1–2.1)

  • In pθ = (1 − θ)q + θδx*, displacement decreases and σc is nondecreasing with θ; retention sets need not be intervals, so the authors verify them empirically.Duplicating a datum m times among N others corresponds to θ = m/(N + m).Source: §B.2 · p. 18

B.3 Two atoms (Theorem 5)

In the two-atom model, a lighter example retains its own attractor only below a critical noise scale that increases with its mass and separation. The analysis extends this balance intuition to training sets and conditional models, while identifying limits of the heuristic and of guidance above weight 1.

B.3 Two atoms (Theorem 5)

  • The probe retains its own attractor exactly below a critical scale that increases with its mass; for small mass, the scale grows linearly with separation and logarithmically with mass.The two-atom result describes an exact threshold; its leading-order formula gives scaling rather than the exact value.Source: §B.3 · p. 19
  • In the two-atom model, the lighter atom's attractor disappears by merging with the saddle as increasing noise flattens the map.At the critical scale, the escape test returns this saddle-node scale only if the attractor's displacement is within the escape radius.Source: §B.3 · pp. 19–20

B.4 From two atoms to a training set

  • For a training set, the leading-order balance heuristic compares the probe's copies with the other rows' kernel mass at the probe, but omits an escape threshold.Lumping all other rows at the nearest-neighbor distance overcounts their competition; an equivalent kernel distance summarizes their mass at the balance scale.Source: §B.4 · p. 21

B.5 Conditional models and guidance (Section 2.1)

  • The conditional construction extends the exact-denoiser results to p(x|c), with captions entering mean shift through image prior weights.When a caption changes only the probed image's weight, its critical-scale gap is positive exactly when that weight is higher under the caption.Source: §B.5 · p. 21
  • For guidance weights from 0 to 1, the dynamics correspond to a tilted density; above 1, the Jacobian may be indefinite and attractors can overshoot or oscillate.Only the unguided conditional map describes fixed points of the conditional density itself.Source: §B.5 · p. 22

C Remarks on the critical scale

The critical scale is computed by bisection when retention is monotone, although monotonicity is checked rather than assumed. The appendix compares this scale for trained networks with an exact training-set denoiser and finds the latter bounds measured trained-model values.

C Remarks on the critical scale

  • Bisection in log σ returns the critical scale exactly when the escape event is monotone; the authors checked trained models and observed no violations.For K > 1, the escape event depends on the whole orbit and has no analogous closed form.Source: C Remarks on the critical scale · p. 22
  • The trained denoiser matched the exact map's attractor counts at 80% of 90 scales, with mean adjusted Rand index 0.99 across induced training-set partitions.The lowest adjusted Rand index was 0.76 at the smallest scales, where single data split; the duplicated datum retained its attractor over the same scale range in both maps.Source: Figure 8 · p. 23
  • The defined iteration feeds the clean estimate back without adding noise, unlike renoising iterations with noise-dependent endpoints or generative round trips.These alternatives have different fixed points and are not equivalent to the paper's map.Source: C Remarks on the critical scale · p. 23

D.1 The theory on the exact denoiser

For the exact denoiser, the critical scale follows a distance-and-multiplicity law, while trained networks retain images to different degrees relative to that exact-map baseline. The appendix finds that this baseline bounds measured network retention and distinguishes fixed-point retention from transient retention.

D.1 The theory on the exact denoiser

  • The exact-denoiser critical scale is a saddle-node transition, unchanged across tested escape radii and iteration budgets.Training images remain exact fixed points below the transition.Source: §D.1 · p. 23
  • 1.95 per distance doubling versus 1.06 per multiplicity doubling is the fitted increase in σc, close to the predicted 2 and 1.05.The exact denoiser follows the predicted law without fitting; 32 copies increase σc by a factor of 1.26.Source: §D.1 · p. 24

D.2 Trained networks against the exact denoiser

  • No measured training image exceeds κ = 1; the largest values are 0.78 for duplication, 0.82 for overfitting, and 0.65 for rarity.The exact empirical denoiser therefore bounds every trained model measured.Source: §D.2 · p. 24
  • Across interventions, the exact-map retention ceiling varies much less than network σc; κ captures how much of that ceiling the network implements.The ceiling varies by a factor of 4.9–10.3 in group medians, while σc varies by a factor of 335.Source: §D.2 · pp. 24–25
  • Network retention can be transient: increasing the iteration budget releases ordinary and held-out images, unlike memorized images retained as fixed points.At m = 800, median σc is 1.431 at both K = 50 and K = 500; for m = 1 and held-out images, it falls by factors of 165 and 193.Source: §D.2 · p. 25

E.1 Memorization from data duplication (Section3.1)

With data duplication, σc increases with multiplicity but saturates, and separates duplicated from non-duplicated examples. In the overfitting sweep, σc tracks training-image retention and can still distinguish training from held-out images after copy sampling finds few or no copies.

E.1 Memorization from data duplication (Section3.1)

  • In the duplication experiment, 16,605 of 50,000 generated samples are copies; the model's FID is 4.70 for non-duplicated samples versus 3.90 for the control.Including the copies in the FID computation raises it to 26.46.Source: §E.1 · p. 25
  • A hundredfold increase in copies changes σc by less than a factor of 1.4, while κ separates duplicated from non-duplicated images rather than ordering them by multiplicity.κ is 0.028 at m = 1, 0.19 at m = 8, and 0.15 at m = 800; its correlation with multiplicity is ρ = −0.01.Source: §E.1 · pp. 25–26

E.2 From memorization to generalization: memorization from overfitting

  • At N = 2000, σc separates training from held-out images with AUC 0.897 even though only 4 of 50,000 samples are copies.For N ≤ 500, 52–100% of samples are copies and AUC is 1.00; for N ≥ 10^4, no samples are copies and σc is at the held-out level.Source: Figure 11 · p. 26, §E.2 · p. 27
  • The median training-image σc falls from 4.67 at N = 100 to 0.056 at N = 5 · 10^4, matching the held-out value of 0.060.The decrease is concentrated between N = 500 and N = 2000 rather than following a power law.Source: §E.2 · p. 27
  • Retention can outlast generation: σc detects memorization in a regime where the copy rate is already zero.The overfitting results report no copies at N ≥ 10^4 while σc remains informative at N = 2000.Source: §E.2 · p. 27

E.3 Memorization of rare samples (Section3.2)

In rare-sample experiments, σc detects added images even when sampling does not, with paired comparisons revealing effects that single-model comparisons may miss. The measured increase also relates to image isolation, distance, and multiplicity, subject to the chosen escape-test settings.

E.3 Memorization of rare samples (Section3.2)

  • σc flags 8 of 12 colour-MNIST images with one copy and all 12 with two, although no added image with m ≤ 4 is sampled more than 0.25 times per 10^5 samples.At m = 8, each image is generated 1.3–13.6 times; at m ≥ 16, every added image is generated.Source: §E.3 · p. 27
  • For two CIFAR sources with m ≤ 4, single-model AUC is only 0.36–0.66, yet paired comparisons detect the added images.The median ratio of σc in the model trained on an image to that in the model not trained on it rises from 1.02 for CIFAR-10 at m = 1 to 2.5 for SVHN at m = 4 and 3.0 for MNIST at m = 1.Source: §E.3 · p. 27, Figure 12 · p. 28
  • For m ≤ 4, the increase in σc after adding an image correlates with its distance to the nearest training image (Spearman ρ = 0.33, p = 5 · 10^-5, n = 144).The passage contrasts this with sampling, whose probability of reaching the basin grows with multiplicity rather than isolation.Source: §E.3 · p. 27
  • In a joint fit over 288 added images, each kernel-distance doubling corresponds to a factor of 4.25 [3.4, 5.3] in σc, versus 1.49 [1.44, 1.54] per multiplicity doubling.Distance is the stronger factor in trained models, as in the memorizing solution.Source: §E.3 · p. 27
  • The separation is stable for r ∈ [0.1, 0.5] and K ∈ [1, 20], but the primary escape radius r = 0.02 is unsuitable for the public model's noise schedule.At r = 0.02, escape occurs near σ ≈ 0.025, below the informative range.Source: §E.3 · p. 28

F Experimental details

The experimental details convert noise-prediction models into fixed-scale denoisers and specify the procedures used to estimate retention across CIFAR-10 and Stable Diffusion experiments. Additional Stable Diffusion analyses include qualitative orbit grids and sensitivity checks for the iteration count and escape radius.

F Experimental details

  • DDPM and Stable Diffusion are treated as denoisers by rewriting discrete noise-prediction corruption as y = x0 + σε and defining Dσ(y) = y − σϵθ(√ᾱt y, t).Source: F Experimental details · p. 28
  • The CIFAR-10 protocols vary multiplicity, training-set size, or added-image rarity, with σc estimated by bisection and sampling-based copy criteria specified for each experiment.The duplication control is a released checkpoint rather than an exactly matched model; rarity images are matched in RMS norm and total variation, with colour MNIST noted as an atypicality extreme.Source: F Experimental details · pp. 28–29
  • Stable Diffusion evaluation deduplicates memorized-image entries and removes mismatched image-caption pairs because the method evaluates memorization at the pair level.The filtering removes repeated templates and cases where a caption reproduces a different memorized image.Source: F Experimental details · p. 29, Figure 13 · p. 30
  • Local retention is tested separately on each four-dimensional latent patch, using K iterations over a logarithmic grid of 24 noise levels and estimating σc between the first escape and the previous grid value.Unless otherwise stated, local-gap experiments use K = 16 and r = 0.1.Source: F Experimental details · p. 29

G Stable Diffusion: additional experiments

  • Orbit grids expand Figure 1 for the same memorized and control pairs: each row shows an orbit at one noise scale, while the last column shows orbit endpoints across scales.Source: G Stable Diffusion: additional… · p. 29

G.1 Ablation

Supplementary analyses find that the local critical-scale gap is particularly robust to parameter choices and that retention varies with memorization type, caption conditioning, and image region. Near-zero caption gaps remain ambiguous because they can arise whether both branches retain an image or neither does.

G.1 Ablation

  • Detection performance is stable near K = 16; global scores are more sensitive to K and r, while the local gap remains robust across radii.The primary choices are r = 0.25 for global scores and r = 0.1 for the local gap; global-score sensitivity is especially pronounced on TV examples.Source: §G.1 · p. 30

G.2 Further experimental results

  • Most caption-swapped pairs have negative signed gaps, especially TV examples; taking absolute values obscures their separation from original pairs.The reported signed-gap setting uses K = 16 and r = 0.25.Source: §G.2 · p. 30

G.3 Detecting memorized regions: additional qualitative examples

  • Local gaps are large in regions reproduced consistently across generations, while variable regions generally have near-zero gaps, reflecting the spatial extent of memorized content.Additional TV examples show the same alignment between local gaps and consistent versus variable content.Source: §G.3 · pp. 30–31, §G.5 · p. 34

G.4 Further analysis of mismatched caption gap

  • Near-zero caption gaps can mean either both branches retain an image or neither does; more negative gaps align with more faithful reconstructions.Absolute critical scales and inpainting performance distinguish the two near-zero-gap regimes.Source: §G.4 · p. 31

G.5 Further qualitative examples

  • Across examples, retention is strongest for MV images, followed by TV images and controls; within TV images, memorized regions persist longer than variable regions.The examples also show that TV regions such as a floor, bedsheet pattern, or curtain can disappear before memorized content.Source: §G.5 · pp. 31–32
  • Training captions increase retention for MV pairs, whereas control pairs escape at similar scales under captioned and unconditional conditioning.The MV pair has a large positive caption gap; the control gap is close to zero.Source: §G.5 · pp. 31, 32, 40
Loading 2609.39648v1…