PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Hallucinations in Diffusion Models through Mode Interpolation

Sumukh K Aithal, Pratyush Maini, Zachary C. Lipton, J. Zico Kolter

arXiv:2406.09358v2cs.LG

TL;DR

Diffusion models can generate artifacts outside the support of their training data, but the source and behavior of these hallucinations require explanation. The paper studies mode interpolation using Gaussian, Shapes, Hands, and recursive-training experiments, and uses reverse-sampling trajectory variance to detect hallucinations. On the Hands dataset, the trajectory-variance metric eliminated 80% of hallucinated samples while retaining 81% of in-support samples.

  • Problem

    Diffusion models can generate artifacts outside the support of their training data, but the source and behavior of these hallucinations require explanation.

    Source: §1 · pp. 1–2

  • Method

    The paper studies mode interpolation using Gaussian, Shapes, Hands, and recursive-training experiments, and uses reverse-sampling trajectory variance to detect hallucinations.

    Source: §1 · p. 2, §5.2 · p. 9, §7 · p. 11

  • Results

    On the Hands dataset, the trajectory-variance metric eliminated 80% of hallucinated samples while retaining 81% of in-support samples.

    Source: §5.2 · p. 9

  • Takeaways & Limitations

    Diffusion models can indicate when they hallucinate, and filtering those generations can mitigate model collapse during recursive synthetic-data training.

    Source: §1 · p. 2

    The metric depends critically on selecting appropriate timesteps, and its behavior across schedules and sampling algorithms needs further analysis.

    Source: B Limitations and Broader Impact · p. 16

Abstract

from arXiv · show

Colloquially speaking, image generation models based upon diffusion processes are frequently said to exhibit "hallucinations," samples that could never occur in the training data. But where do such hallucinations come from? In this paper, we study a particular failure mode in diffusion models, which we term mode interpolation. Specifically, we find that diffusion models smoothly "interpolate" between nearby data modes in the training set, to generate samples that are completely outside the support of the original training distribution; this phenomenon leads diffusion models to generate artifacts that never existed in real data (i.e., hallucinations). We systematically study the reasons for, and the manifestation of this phenomenon. Through experiments on 1D and 2D Gaussians, we show how a discontinuous loss landscape in the diffusion model's decoder leads to a region where any smooth approximation will cause such hallucinations. Through experiments on artificial datasets with various shapes, we show how hallucination leads to the generation of combinations of shapes that never existed. Finally, we show that diffusion models in fact know when they go out of support and hallucinate. This is captured by the high variance in the trajectory of the generated sample towards the final few backward sampling process. Using a simple metric to capture this variance, we can remove over 95% of hallucinations at generation time while retaining 96% of in-support samples. We conclude our exploration by showing the implications of such hallucination (and its removal) on the collapse (and stabilization) of recursive training on synthetic data with experiments on MNIST and 2D Gaussians dataset. We release our code at https://github.com/locuslab/diffusion-model-hallucination.

1 Introduction

The paper identifies hallucinations as out-of-support samples caused by mode interpolation, where diffusion models smoothly connect distinct training modes. Controlled experiments explain this mechanism, demonstrate its manifestations, and motivate detection through sampling-trajectory variance.

1 Introduction

  • The paper terms this failure mode hallucination and explains it through mode interpolation between distinct data modes.The proposed explanation focuses on interactions among modes rather than only mode collapse or loss of diversity.Source: §1 · pp. 1–2
  • In disjoint Gaussian modes, a discontinuous score structure is approximated smoothly, causing interpolation through regions absent from the training data.The authors study this mechanism with 1D and 2D mixture-of-Gaussian setups.Source: §1 · p. 2
  • Hallucinated samples show high reverse-trajectory variance near the end of sampling, enabling detection with sensitivity and specificity above 0.92.The trajectory-variance metric is motivated by the observation that diffusion models often indicate when they leave the data support.Source: §1 · p. 2
  • The proposed detection mechanism mitigates model collapse during recursive retraining on 2D Grid of Gaussians, Shapes, and MNIST.The study also examines hallucinated hand generations and applies the metric to detect them.Source: §1 · p. 2

1.1 Hallucination in Diffusion Models

  • Diffusion models can generate samples outside the training distribution, including repeated shapes absent from the SIMPLE SHAPES data and hands with extra fingers.The toy dataset permits each shape at most once per image, yet generated samples can contain multiple occurrences; analogous hand artifacts appear in real-image experiments.Source: Figure 1 · p. 2, §1 · p. 2, §1.1 · p. 3

2 Related Work

Related work situates the paper within diffusion-model foundations, recursive-training collapse, and known image-generation failures. The paper distinguishes its focus by studying interactions among modes and the emergence of novel modes during recursive training.

Diffusion Models

  • Diffusion models use forward noising and learned reverse denoising, with connections to score-based modeling, SDEs, and hierarchical VAEs.The paper frames mode interpolation as a newly identified diffusion phenomenon.Source: §2 · p. 3

Recursive Generative Model Training

  • Prior recursive-training studies report collapse into low diversity or low-fidelity samples, with real-data mixing commonly used for mitigation.Theoretical work also identifies strong synthetic-data requirements when real-data support is absent.Source: §2 · p. 3
  • This paper extends recursive-training analysis beyond collapse toward existing modes by examining interactions between modes and the development of novel modes.The controlled experiments provide a different perspective on why recursive collapse occurs.Source: §2 · p. 3

Failure Modes of Diffusion Models

  • Known diffusion failures include distorted hands and legs, rare-concept failures, ignored spatial relationships, and confused attributes.These failures motivate broader investigation of diffusion-model behavior.Source: §2 · pp. 3–4

Hallucination in Language Models

  • Unlike language-model hallucination work, this paper studies out-of-support image generations and proposes identifying their causes and mitigating them.The related work notes that hallucinations in language models can be factually incorrect, instruction-inconsistent, or logically wrong.Source: §2 · p. 4

3 Definitions and Preliminaries

The preliminaries define diffusion as iterative noising followed by learned denoising and connect its score representation to gradients of log density. These formulations establish the notation used to analyze mode interpolation.

Diffusion Models

  • The forward process adds Gaussian noise for T timesteps, with β_t controlling variance and x_T approaching N(0, I).The process begins with x_0 sampled from the real data distribution q(x).Source: §3 · p. 4
  • Forward diffusion permits direct sampling of x_t from x_0 through a Gaussian transition determined by α_t and the cumulative schedule ᾱ_t.The closed form avoids simulating every preceding timestep.Source: §3 · p. 4
  • The reverse process learns p_θ(x_{t−1}|x_t), a Gaussian denoising transition parameterized by a neural network.Its mean and covariance define the learned step from a noisier sample to a cleaner one.Source: §3 · p. 4
  • DDPMs predict noise ε_θ(x_t,t), which can be used to form a predicted clean sample x̂_0.The original DDPM fixes the variance schedule, while later methods may learn it.Source: §3 · p. 4

Connections to Score Based Generative Models

  • The score function is the gradient of log density, and diffusion obtains its time-dependent estimate from the predicted noise.This links diffusion denoising to score-based sampling methods.Source: §3 · p. 4

4 Understanding Mode Interpolation and Hallucination

The paper defines hallucinations as samples outside the data support and argues that diffusion models create them by smoothly interpolating between nearby modes. Experiments trace this mechanism from Gaussian mixtures and shape representations to human-hand images.

4.1 1D GAUSSIAN Setup

  • Diffusion models generate nonzero-density samples between nearest Gaussian modes, despite those regions having nearly zero probability under the true distribution.The interpolated density decreases with more training data, greater mode separation, and more sampling timesteps.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 2D GAUSSIAN Grid

  • In a 2D Gaussian grid, interpolation appears between nearest modes, while rotating the grid eliminates interpolation along the x-axis where modes are not nearest neighbors.The interpolated samples have near-zero probability in the original distribution.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 What causes mode interpolation?

  • Smooth neural-network scores approximate sharp score-function jumps between disjoint modes, creating uncertainty regions that produce interpolated samples.Sampling with the true score function removes the observed mode interpolation.Source: §4.3 · pp. 6–7, Figure 4 · p. 6

4.4 SIMPLE SHAPES

  • In the Simple Shapes representation space, interpolation creates a region containing two squares even though that combination was absent from the training data.The intermediate bottleneck region lies between representations of squares at the top and bottom of the image.Source: §4.4 · p. 7

4.5 Mode Interpolation in Real World datasets: HANDS

  • Training on 5,000 high-quality hand images still produces hands with additional or missing fingers, demonstrating mode interpolation in a real-world dataset.The generated artifacts are not explained by blurred images in this experiment.Source: §4.5 · p. 7, Figure 5 · p. 7

5 Diffusion Models know when they Hallucinate

Diffusion models generate hallucinations when reverse-diffusion trajectories pass through uncertain regions between modes, producing unstable predictions near the final sampling steps. Variance in these trajectories provides a practical detector that separates hallucinated from in-support samples across synthetic and real-world datasets.

5.1 Variance in the trajectory of prediction

  • High trajectory variance distinguishes hallucinated samples from in-support samples because hallucinated predictions remain unstable during the final reverse-diffusion steps.Non-hallucinated trajectories converge with low variance, whereas hallucinated trajectories show instability in predicted x_0 values.Source: §5.1 · p. 8, §5 · p. 8

5.2 Metric for detecting hallucination

  • The detection metric computes variance across predicted x_0 values between a chosen starting timestep T1 and ending timestep T2.The variance can be computed from stored reverse-diffusion predictions or by repeatedly applying forward diffusion between T1 and T2.Source: §5.2 · p. 9
  • 95% of hallucinated SIMPLE SHAPES samples were filtered while retaining 95% of in-support samples using variance from timesteps T = 850 to T = 700.This interval was selected because images generally stabilize around T = 700 in the 1000-step reverse process.Source: §5.2 · p. 9
  • 95% of 1D-Gaussian hallucinations were filtered while retaining 98% of in-support samples using variance from the final 15 reverse-diffusion steps.Hallucinations were defined as samples with negligible probability under the true distribution, such as values beyond 6σ from the mean.Source: §5.2 · p. 9
  • 96% of 2D-Gaussian hallucinations were filtered while retaining 95% of in-support samples using mean per-dimension variance from the final 20 steps.Source: §5.2 · p. 9
  • On manually labeled hand images, the metric eliminated 80% of hallucinations while retaining 81% of in-support samples, despite the harder real-world detection setting.Hallucinated examples included missing or additional fingers, and their trajectories showed higher variance than in-support samples.Source: §5.2 · p. 9

6 Implications on Recursive Model Training

Recursive training amplifies diffusion-model hallucinations because each generation learns from outputs increasingly contaminated by interpolated or otherwise unsupported samples. Variance-based pre-emptive filtering reduces hallucinations and slows recursive model collapse across the studied datasets.

Implications on Recursive Model Training

  • Recursive synthetic-data training is motivated by the risk of mode collapse and bias amplification when future models train on machine-generated data.The paper emphasizes interactions between modes and mode interpolation rather than only collapse to a single mode.Source: §6 · p. 9

2D GAUSSIAN

  • By Generation 20 of recursive 2D-Gaussian training, interpolated samples increasingly dominate and the original grid modes have almost collapsed into a single mode.Each generation samples 100k examples and trains the next DDPM solely on the preceding generation’s outputs.Source: §6 · p. 10, Figure 8 · p. 10

SIMPLE SHAPES

  • SIMPLE SHAPES begins with about 5% hallucinated samples, but their proportion increases exponentially under iterative self-training.The compounding effect arises because later models learn from distributions increasingly dominated by hallucinated images.Source: §6 · p. 10

Mitigating the curse of recursion with pre-emptive detection of hallucinations

  • Variance-based filtering outperforms random filtering across 2D Gaussians, SIMPLE SHAPES, and MNIST, reducing hallucinations across generations and limiting model collapse.On MNIST, the method also selects high-quality samples without much loss in diversity.Source: §6 · pp. 10–11, Figure 9 · p. 10

7 Discussion

The paper frames diffusion hallucinations as mode interpolation caused by smooth approximations to disjoint score functions, and evaluates the phenomenon across synthetic and real-world settings. It proposes trajectory variance as a detector and connects hallucinations to recursive-training collapse.

7 Discussion

  • Across 1D Gaussian, 2D Grid of Gaussians, Shapes, and Hands datasets, diffusion models learn smoothed approximations of disjoint score functions, leading to mode interpolation.Source: §7 · p. 11
  • The proposed trajectory-variance metric identifies hallucinated samples and is evaluated for mitigating recursive generative-model training effects.Source: §7 · p. 11
  • The paper presents mode interpolation as a potential explanation for additional fingers generated by large-scale generative models, while positioning this finding as a hypothesis for future work.Source: §7 · p. 11

A.1 Gaussian experiments

The Gaussian experiments use a DDPM noise-prediction setup with a linear noise schedule, fully connected architecture, MSE training, and Adam optimization. The hallucination metric uses the final 15 reverse-diffusion steps for 1D Gaussian and final 8 steps for the 2D Gaussian Grid.

A.1 Gaussian experiments

  • The synthetic Gaussian models use 10,000 training epochs, batch size 10,000, MSE noise prediction, and Adam with learning rate 0.001.The default setup uses a linear noise schedule from β0 = 0.001 to β1 = 0.2 and T = 1000 diffusion steps.Source: §A.1 · p. 15
  • The Gaussian DDPM architecture consists of an initial fully connected layer, three normalization-LeakyReLU blocks, and a final fully connected output layer.Source: §A.1 · p. 15
  • The trajectory metric uses t = 0 to t = 15 for 1D Gaussian and t = 0 to t = 8 for the 2D Gaussian Grid.Source: §A.1 · p. 15

A.2 Shapes

The appendix specifies separate diffusion-model training setups for Shapes, MNIST, and hands experiments, including architectures, schedules, sampling procedures, and computational budgets.

A.2 Shapes

  • Shapes uses an unconditional 64 × 64 DDPM with a U-Net, cosine noise schedule, T = 1000 training timesteps, and 250 sampling steps.The training set contains 5000 grayscale images.Source: §A.2 · p. 15

A.3 MNIST

  • MNIST uses classifier-free guidance with a conditional DDPM, a 256-feature U-Net, T = 500, and 50 training epochs across four GPUs.Variance filtering uses 10 timesteps between t = 100 and t = 150.Source: §A.3 · p. 15

A.4 HANDS

  • The hands model trains for 200k iterations with a 256-channel U-Net, attention and residual blocks, cosine scheduling, and 500 respaced sampling timesteps.The implementation follows the official DDPM-IP repository and uses a learning rate of 1e-4.Source: §A.4 · p. 15

B Limitations and Broader Impact

The broader-impact discussion frames hallucination detection as useful for understanding and controlling diffusion-model failures, while noting limits from data scale and metric design.

Broader impact

  • Removing hallucinated hands could make AI-generated images harder to identify, although the authors argue that understanding these failures supports better practical control.The discussion connects hallucination detection with both synthetic-image forensics and model reliability.Source: B Limitations and Broader Impact · p. 16
  • More training data may make future generative models more robust to hallucinations independently of this work.The authors identify poorly modeled hands as a current giveaway of AI-generated images.Source: B Limitations and Broader Impact · p. 16

Limitations

  • The proposed metric depends critically on selecting appropriate timesteps, and its behavior across schedules, sampling algorithms, and real-world datasets remains underexplored.The authors identify broader analyses of trajectory regions and real-world mode interpolation as future work.Source: B Limitations and Broader Impact · p. 16

C Additional Experiments and Figures

Additional experiments test the generality, data dependence, representation-space mechanism, and trajectory signatures of mode interpolation across Gaussian, shape, hand, and recursive-training settings.

Variational diffusion models

Gaussian experiments

Shapes representation space

  • Representation-space analysis shows the two-square region interpolating between one-square regions located in the bottom and top halves of the Shapes dataset.The analysis uses the U-Net bottleneck to expose the interpolation structure.Source: Figure 10 · p. 17

Gaussian experiments

  • The four-Gaussian experiment shows more interpolated samples between the closer modes µ1 = 1 and µ2 = 2 than between µ2 = 2 and µ3 = 4.The two-Gaussian experiments also compare different training-set sizes and random seeds.Source: Figure 15 · p. 18, Figure 14 · p. 18

Recursive generative training

  • The recursive-training MNIST experiment evaluates FID, precision, and recall across generations while applying variance and random filtering.The figure reports that filtering can discard low-quality samples while maintaining sufficient diversity.Source: C Additional Experiments and Figures · p. 16, Figure 16 · p. 18

Hallucination examples

  • Additional figures provide examples of hallucinated images and compare hallucinated with in-support samples through their predicted x0 trajectories.The trajectory visualizations are intended to contrast unstable and consistent predictions.Source: Figure 17 · p. 19, Figure 18 · p. 20, Figure 19 · p. 21

Hands

  • Hands experiments separately show hallucinated images with extra fingers and in-support generated hands.These figures illustrate the image-level manifestation of diffusion-model hallucinations.Source: Figure 20 · p. 22, Figure 21 · p. 23

Related papers

Loading 2406.09358v2…