Source-linked AI summary
Score-Control for Hallucination Reduction in Diffusion Models
Mahesh Bhosale, Naresh Kumar Devulapally, Abdul Wasi, Chau Pham, Vishnu Suresh Lokhande, David Doermann
TL;DR
Diffusion hallucinations place probability mass in low-density or off-support regions, undermining reliability and trust. The paper formalizes this density-based problem, links it to score smoothness, and introduces VSM to reduce it; experiments report reduced hallucinations while preserving fidelity and diversity.
Problem
Diffusion hallucinations remain underexplored despite undermining trust, particularly in common text-conditioned image generation settings.
Method
The paper defines hallucinations through low-density regions, connects off-support mass to score smoothness, and introduces Variance-Guided Score Modulation to counteract excessively smooth scores.
Results
VSM consistently reduces hallucinations across synthetic and real-world datasets while preserving high fidelity and diversity.
Takeaways & Limitations
VSM provides a practical, architecture-agnostic strategy for mitigating diffusion hallucinations and supports controlled evaluation through ChessImages and Cards.
Takeaways & Limitations
The theoretical characterization assumes a minimum model-density value on the boundary of the data high-density region and bounds on score magnitude and Lipschitz smoothness.
Abstract
from arXiv · showhide
Diffusion models have emerged as the backbone of modern generative AI, powering advances in vision, language, audio and other modalities. Despite their success, they suffer from hallucinations, implausible samples that lie outside the support of true data distribution, which degrade reliability and trust. In this work, we first empirically confirm previously proposed hypothesis that score smoothness causes hallucinations in Image Generation diffusion models and provide a density-based perspective. We further formalize this notion by linking the hallucinations probability mass to lipschitz constant of the learned score function. Motivated by this, we introduce a Variance-Guided Score Modulation (VSM) strategy that controls the score Jacobian, in turn reducing score smoothness and better approximating the ground truth score that decreases hallucinations. Empirical results on synthetic and real-world datasets demonstrate that our approach reduces hallucinations (up to ~25%) while maintaining high fidelity and diversity, providing a principled step toward more reliable diffusion-based image generation. We also propose two benchmark datasets with extreme semantic variation for systematic hallucination evaluation. Code and Datasets are publicly available at https://github.com/bhosalems/VSM.
1 Introduction
Diffusion hallucinations undermine trust, yet remain underexplored in common text-conditioned image generation. This work formalizes hallucinations through density and introduces a training-time mitigation method.
- Motivation: Diffusion models support image, audio, text, and biomedical generation, but their growing adoption makes generation reliability increasingly important.Recent systems have improved fidelity, controllability, and latency, while Adobe Firefly reports 22B+ generated assets as of April 2025.
- Problem: Hallucinations degrade sample quality and undermine trust in model generations, but diffusion-model hallucinations remain largely underexplored.Prior studies address structural hallucination, mode interpolation, or temperature scaling without covering the common text-conditioned image-generation setting comprehensively.
- Approach: The paper formalizes hallucinations from a density-based perspective and introduces a simple training-time method to reduce them during image generation.This frames hallucinations as probability mass in regions outside the effective data support.
- Contributions: The paper links score-field smoothness to non-zero off-support probability mass and proposes Variance-Guided Score Modulation to counteract excessively smooth scores.VSM increases local score curvature through a Jacobian-based penalty and applies time-dependent weighting toward late denoising steps.
- Results: VSM reduces hallucinations by up to ∼26% across multiple existing datasets while preserving high fidelity and diversity.The paper also introduces ChessImages and Cards, each designed with ∼1044 semantic classes for controlled hallucination evaluation.
2 Related Work
Prior work studies diffusion hallucinations through local denoising, mode interpolation, attention scaling, text-specific bias, or specialized reasoning settings. These approaches often require annotations, lack mitigation, or do not target general text-to-image generation.
- Hallucination mitigation: Local denoising can mitigate structural hallucinations but requires expert mask annotations for medical data, unlike VSM.The cited approach operates over estimated out-of-distribution regions.
- Hallucination analysis: Mode interpolation explains hallucinations through smooth score approximations but does not propose a mitigation technique.This provides a conceptual account without a corresponding intervention.
- Hallucination mitigation: Temperature-scaled self-attention suppresses early-stage noise, but prior work does not address the common text-conditioned image-generation setting.The introduction also characterizes this line of work as lacking mitigation for that setting.
- Text hallucination: The Local Dependency Ratio frames text hallucination as local generation bias, but its analysis is limited to images containing text.The cited work argues that stronger global dependencies help.
- Specialized settings: Sequential generation with Spatial Reasoning Models targets structured reasoning and is less applicable to general text-to-image generation.Its MNIST Sudoku benchmark is contrasted with ChessImages, which has a much larger semantic space.
3 Hallucinations in Diffusion Models
The paper defines hallucinations as generated samples in low-density regions of the data distribution and further separates non-hallucinated samples into memorized and generalized regions. These mutually exclusive regions support measuring both hallucination reduction and its effects on memorization and generalization.
- Hallucinated samples: The data distribution defines an ε-hallucination set H_ε consisting of points whose effective data density is at most ε.When the data distribution has global support, a vanishingly small ε defines an effective support.
- Hallucinated samples: A generated sample is hallucinated if it belongs to H_ε and non-hallucinated otherwise.Setting ε = 0 identifies regions where the data density is zero.
- Memorization and generalization: Non-hallucinated samples are divided into memorization and generalization regions using a distance-based proximity threshold to the training set.Memorized samples lie in M, while generalized samples lie in G := X \ (H_ε ∪ M).
- Partitioning: The regions H_ε, M, and G are mutually exclusive and partition the sample space.The paper treats ε and δ as fixed hyperparameters when using these regions.
- Hallucination probability: Hallucination probability measures the model probability assigned to the hallucination region H_ε.The paper uses this probability together with memorization and generalization metrics to assess mitigation side effects.
4 Methods
The method links hallucinations to score-field smoothness, formalizes off-support probability through the score’s Lipschitz constant, and introduces VSM to counteract this smoothness during training. Experiments manipulate smoothing and evaluate practical curvature and variance-learning components for score control.
- 4.2 Motivation: Diffusion models learn smoothed approximations of sharp score fields, so experiments vary weight regularization and training-set size to measure effects on hallucinations.The study uses a 1D Gaussian mixture with component means {1.0, 1.5, 2.0} and shared standard deviation 0.35.
- 4 Methods: The evaluation includes datasets designed for semantic hallucination studies, including Cards and ChessImages with vast semantic spaces and rapid training-free detection.The dataset table identifies semantic classes as valid, interpretable configurations.
- 4.2 Motivation: A theoretical lower bound shows that off-manifold model density depends on the learned score’s magnitude bound and Lipschitz constant.The bound applies to an off-manifold point at distance δ_x from a high-density data region.
- 4.2 Motivation: Lower score smoothness, corresponding to a larger effective Lipschitz constant, reduces exponential probability-mass leakage into off-manifold regions.The paper notes that precisely locating support boundaries in high dimensions requires unknown data-support information, motivating global modulation.
- 4.3 Variance Guided Score Modulation: VSM uses variance learning to construct a tractable diagonal curvature proxy instead of computing the full high-dimensional score Jacobian.The proxy retains a sample-dependent diagonal term derived from the learned reverse conditional covariance.
- 4.3 Variance Guided Score Modulation: The training objective augments denoising loss with a variational variance-learning term and a smoothness penalty, while time-dependent scaling strengthens VSM near low-noise sampling stages.The schedule uses tunable strength ρ and avoids the overly aggressive weighting of a fully inverse schedule.
5 Experiments
Experiments evaluate VSM with automated hallucination detection across synthetic, real-world, and highly diverse datasets, using pixel- and latent-space diffusion models. VSM consistently reduces hallucinations while preserving fidelity, diversity, and generalization, with performance depending on regularization strength and schedule.
- Results: VSM reduces score error and hallucination rates across 1D/2D Gaussian mixtures and Hands-11K.These experiments directly measure score error and define hallucinations outside effective data support.
- Results: VSM consistently lowers hallucination rates across DDPM and LDM evaluations spanning MNIST, Shapes, Hands-11K, Cards, ChessImages, and ImageNet-1K.ImageNet-1K uses improved precision and recall in CLIP and Inception feature spaces alongside FID because explicit detectors are unavailable at that scale.
- Results: VSM improves precision and recall relative to LDM-UC, while retaining stronger recall than a baseline whose higher precision comes with a substantial recall drop.Higher precision reflects support inclusion, while higher recall reflects coverage of distributional modes.
- Generalization vs Memorization: VSM reduces invalid chessboards and increases valid novel boards relative to LDM, enabling evaluation of generalization beyond hallucination rates.ChessImages separates invalid, memorized, and generalized valid samples using rule-checkable legality and a very large semantic space.
- Fine-tuning: Variance-head-only fine-tuning with VSM reduces hallucinations versus fine-tuning without VSM while preserving fidelity and diversity.This supports using VSM as a corrective mechanism for pretrained checkpoints rather than requiring only from-scratch training.
- Ablation Studies: The regularizer has a sweet spot at 𝜌= 0.1, while the inverse square-root schedule achieves the best MNIST C-FID, FLD, and hallucination rate.Excessive regularization can overpower diffusion loss, and moderate late-stage penalty growth outperforms linear and fully inverse schedules.
6 Conclusion
The paper presents a density-based account of hallucinations, attributing off-support leakage to excessive score smoothness and proposing VSM to increase the score Jacobian. Experiments show reduced hallucinations while preserving fidelity and diversity, though elimination and reliable natural-image measurement remain unresolved.
- Excessive score smoothness causes probability mass to leak into off-support regions at an exponential rate controlled by the score’s Lipschitz constant.
- VSM increases the score Jacobian through an architecture-agnostic method to suppress off-support leakage and mitigate hallucinations.
- VSM consistently reduces hallucinations across synthetic, real-world, and challenge benchmarks while preserving high fidelity and diversity.
- The method mitigates rather than eliminates hallucinations, while natural-image understanding and reliable hallucination metrics remain open problems.
A Towards Zero Hallucinations during generation
The paper’s iterative training strategy filters generated cards and retrains on valid outputs, progressively reducing hallucinations. Its analysis connects this behavior to score regularity and evaluates score differences, Jacobian modulation, and variance-based implementation choices.
- A Towards Zero Hallucinations during generation: 1.07% hallucination rate is reached by iteration-6, down from 7.98% in iteration-1, while non-hallucinated outputs rise to 98.93%.Each iteration generates 15,000 cards, filters hallucinated samples, and appends valid cards before retraining from scratch.
- B More Details on Proposition 4.1: The theoretical analysis assumes compact data support and locally Lipschitz, bounded learned scores, with other regularity properties following from the DDPM’s Gaussian structure.
- C Score difference correlates with Hallucinations: The method estimates ground-truth scores in one- and two-dimensional datasets from closed-form probability densities, while image settings lack the ground-truth noised posterior at inference time.
- D Details on implementation of LVSM: VSM implementation penalizes small score Jacobians, using centered finite differences in low dimensions and learned denoising variances for high-dimensional images.
E More details on the ChessImages dataset
ChessImages uses automated visual parsing and rule-based validation to identify generated boards that violate image or chess semantics. The evaluation compares methods by valid novel boards, while some FEN-dependent legality rules are excluded.
- E More details on the ChessImages dataset: The validator uses visual and rule-based checks, and Figure 8 illustrates images flagged for violations of the chess-library status rules.
- E More details on the ChessImages dataset: A generated chessboard is invalid when FEN parsing scores below 50% or the parsed position violates legality checks such as king, pawn, piece-count, or overlap constraints.
- E More details on the ChessImages dataset: Template matching recovers only PiecePlacement, so castling-rights, en-passant, and opposite-check rules are ignored.
- E More details on the ChessImages dataset: The methods are compared by the total number of valid novel boards generated, supporting evaluation of robust augmentation on ChessImages.
F Effect of Dataset Size
The study finds that reducing training-set size increases hallucinations, whereas changing denoising-step count shows no discernible relationship with hallucination rate on Chess. The remaining passages specify prompt and digit-generation inputs rather than additional dataset-size findings.
- F Effect of Dataset Size: Shrinking the training set from the full dataset to nested 75%, 50%, and 25% subsets increases hallucination incidence by reducing support from diverse examples.
- F Effect of Dataset Size: Denoising-step count has no discernible relationship with hallucination rate on Chess, although fewer steps can reduce overall image fidelity.
- F Effect of Dataset Size: The conditional generation setup uses a default Hands prompt or class-embedded MNIST prompts, while prompt tuning randomly selects among 20 prompts per dataset.
I More details on the Cards dataset
This section presents additional samples of hallucinated images from the Cards dataset. The samples are identified using the hallucination rules described in the main paper.
- Figure 9 presents additional samples of images identified as hallucinated.The figure organizes examples according to the hallucination criteria described in the paper.
- The hallucination examples are evaluated using rules defined in the main paper.
- The section provides qualitative evidence for hallucination cases in the Cards dataset.
J Implementation Details
The implementation uses established codebases for the 1D, 2D, and image experiments, with experiments run on eight Nvidia A6000 GPUs. Quantitative evaluations use six seeds with dataset-specific sample counts.
- The 1D and 2D experiments build upon code from [3], while image experiments use [25] for variance learning and L_VSM.
- All experiments are conducted using 8 Nvidia A6000 GPUs.
- Image results average six seeds with 100 generated images per seed, while 1D and 2D results average six seeds with 1 million points per seed.
- The LDM baseline uses the codebase provided by [28].
K Additonal qualitative samples on ImageNet-1K
Additional ImageNet-1K comparisons contrast an LDM baseline without L_VSM against the proposed method with L_VSM. The baseline frequently generates deformed or incompletely denoised samples, whereas the proposed method mitigates these cases and produces more coherent images.
- ImageNet-1K qualitative comparisons use the LDM model without L_VSM as the baseline and the proposed method with L_VSM as the comparison.The baseline is shown in red and the proposed method in green.
- Figure 9 contains generated images marked hallucinated according to reasons listed at the bottom of each row.
- Figure 10 pairs generated chessboard configurations with corresponding FEN strings that symbolically represent board states.
- The baseline frequently produces deformed objects and incompletely denoised samples.These failures result in images that deviate from the training data distribution.
- The proposed method mitigates these failure cases and generates more coherent, well-formed samples.