Source-linked AI summary
DynHD: Hallucination Detection for Diffusion Large Language Models via Denoising Dynamics Deviation Learning
Yanyu Qian, Yue Tan, Yixin Liu, Wang Yu, Shirui Pan
TL;DR
Hallucination detection in D-LLMs is difficult because informative tokens are sparse within fixed-length outputs and uncertainty evolves across denoising steps. DynHD filters structural tokens, learns reference uncertainty dynamics, and detects deviations from them; experiments report state-of-the-art performance, including a 12.2% average AUROC margin over TraceDet. Its evaluation is mainly limited to factual QA settings and two representative D-LLM backbones.
Problem
D-LLM hallucination signals are distributed across informative tokens and denoising trajectories, while existing methods may ignore token importance and process-level dynamics.
Method
DynHD constructs semantic-aware entropy evidence by filtering structural tokens, then compares observed denoising dynamics with learned reference dynamics.
Results
12.2% average AUROC margin over TraceDet is reported across three QA benchmarks and two D-LLM backbones, without external retrieval or expensive repeated sampling.
Takeaways & Limitations
Denoising dynamics can serve as hallucination-detection signals when combined with semantic-aware evidence construction and trajectory-deviation modeling.
Takeaways & Limitations
Evaluation mainly covers factual QA scenarios and two representative D-LLM backbones, leaving broader generation settings and generality for further evaluation.
Abstract
from arXiv · showhide
Diffusion large language models (D-LLMs) have emerged as a promising alternative to auto-regressive models due to their iterative refinement capabilities. However, hallucinations remain a critical issue that hinders their reliability. To detect hallucination responses from model outputs, token-level uncertainty, such as entropy, has been widely used to indicate potential factual errors. Nevertheless, unlike auto-regressive models that generate tokens sequentially, D-LLMs generate fixed-length sequences simultaneously, where only a small subset of tokens is informative for hallucination detection. Thus, aggregating uncertainty over all tokens can be suboptimal. Moreover, the evolution trend of uncertainty throughout the diffusion process can also provide valuable signals, highlighting the necessity of modeling its denoising dynamics for hallucination detection. In this paper, we propose DynHD, which bridges these gaps from both spatial (token sequence) and temporal (denoising dynamics) perspectives. To handle the information density imbalance across tokens, we propose a semantic-aware evidence construction module that extracts hallucination-indicative signals by removing task-invariant structural tokens and emphasizing the uncertainty of the remaining informative tokens. To model denoising dynamics for hallucination detection, we introduce a reference evidence generator that learns the expected evolution trajectory of uncertainty evidence, along with a deviation-based hallucination detector that makes predictions by measuring the discrepancy between the observed and reference trajectories. Extensive experiments demonstrate that DynHD consistently outperforms state-of-the-art baselines while achieving higher efficiency across multiple benchmarks and backbone models. The code is available at: https://github.com/qyy11-com/DynHD.
1 Introduction
DynHD addresses hallucination detection in D-LLMs by filtering low-information tokens and modeling uncertainty evolution across denoising steps. It achieves state-of-the-art performance across multiple benchmarks and backbones.
- Motivation: D-LLMs generate fixed-length sequences whose token information density is uneven, with structural padding tokens carrying little hallucination-related information.This makes all-token uncertainty aggregation potentially obscure informative signals.
- Motivation: TraceDet aggregates uncertainty across all tokens, causing factual and hallucinated entropy profiles to become indistinguishable.The limitation is attributed to ignoring semantic importance among generated positions.
- Denoising dynamics: Factual and hallucinated samples exhibit different denoising evidence trends, while entropy-curve range, trend, and shape vary across tasks and questions.These observations motivate modeling continuous denoising dynamics rather than selecting only a few diffusion steps.
- DynHD: DynHD filters structural padding tokens, constructs entropy evidence with multivariate statistical modeling, and compares observed dynamics with learned reference dynamics.Its deviation-based detector identifies hallucinations from discrepancies between observed and expected evidence trajectories.
- Results: 12.2% average AUROC margin over TraceDet is reported across TriviaQA, HotpotQA, and CommonsenseQA using Dream-7B and LLaDA-8B.The experiments also report no need for external retrieval or computationally expensive repeated sampling.
2 Preliminaries
The preliminaries describe D-LLMs as iterative denoising systems that produce token-level uncertainty trajectories, then formulate hallucination detection as supervised binary classification. The detector uses the question, final response, and collected uncertainty trajectory.
- Diffusion denoising mechanism: D-LLMs iteratively denoise fixed-length sequences from a highly masked initialization to a final output across descending diffusion steps.The sequence state spans vocabulary positions over the total denoising trajectory.
- Uncertainty trajectory: At each denoising step, the model predicts categorical token distributions from logits and computes position-wise uncertainty using token entropy.The entropy values form a sequence of token uncertainties for that step.
- Uncertainty trajectory: The raw uncertainty trajectory collects token-entropy sequences across all descending denoising steps.Fixed-length generation can add non-semantic end-of-text padding to sequences that terminate early, introducing noise.
- Detection problem: Hallucination detection is formulated as binary classification of question-response pairs, with hallucinated responses labeled 1 and factually correct responses labeled 0.The goal is to learn a detector minimizing empirical risk on labeled training data.
- Detection problem: The detector bases its decision on the question, final response, and collected uncertainty trajectory.The preliminaries identify this decision rule through a classification loss formulation.
3 Methodology
DynHD constructs hallucination evidence by filtering structural tokens, summarizing uncertainty across multiple granularities, and modeling how this evidence evolves during denoising. It then compares the observed evidence trajectory with a question-conditioned reference to detect hallucinations.
- Semantic-aware Evidence Construction: DynHD first removes structural padding tokens from fixed-length sequences before constructing hallucination evidence.The filtering retains structurally valid positions and preserves potentially meaningful lexical and formatting tokens.
- Statistical Evidence Construction: At each denoising step, DynHD condenses the filtered uncertainty field into a three-dimensional statistical evidence vector.The vector captures mean, peak, and top-k characteristics of the entropy distribution.
- Statistical Evidence Construction: Global, peak, and regional uncertainty statistics capture macro-level convergence, localized factual errors, and phrase-level uncertainty patterns.These statistics operate at different granularities and help prevent isolated factual mistakes from being hidden by fluent contexts.
- Denoising Dynamics Modeling: The step-wise statistics form an evidence trajectory from initial noise to the final generation, representing semantic uncertainty resolution throughout denoising.This trajectory preserves temporal information rather than reducing detection to a single denoising step.
- Denoising Dynamics Modeling: A question-conditioned reference path models expected evidence evolution, while a deviation-based detector predicts hallucinations from discrepancies between observed and reference trajectories.The reference accounts for evolution patterns associated with different question-and-answer scenarios.
4 Experiments
DynHD is evaluated across multiple D-LLMs, factuality benchmarks, baselines, ablations, generalization settings, and diagnostic visualizations. It achieves strong detection performance while maintaining efficiency and robustness across remasking strategies.
- Experimental Setup: Experiments use TriviaQA, HotpotQA, and CommonsenseQA with LLaDA-8B-Instruct and Dream-7B-Instruct, following TraceDet’s evaluation protocol.AUROC is the primary metric, with GPT-4o-mini correctness labels agreeing with human judgments at 94%.
- Main Results: DynHD achieves the highest performance in almost all settings, exceeding the strongest baseline by an average of 12.2% absolute AUROC on LLaDA-8B-Instruct and reaching 84.3% average AUROC on Dream-7B-Instruct.The comparison spans two D-LLMs and three factuality QA datasets with varying generation lengths.
- Inference Efficiency: DynHD occupies the upper-left performance–efficiency region, matching fast baselines such as CCS while outperforming computationally expensive multi-sample methods in efficiency.The reported balance supports real-time monitoring without extra delays.
- Cross-dataset Generalization: DynHD yields the highest average zero-shot cross-dataset AUROC of 72.9%, including 85.5% AUROC for TriviaQA-to-HotpotQA transfer versus 73.1% for TraceDet.Performance remains robust on challenging transfers, although fixed options reduce observable uncertainty signals in some settings.
- Ablation and Diagnostics: Ablations show that token filtering, combined statistical evidence, temporal aggregation, attention weighting, and trajectory regularization each contribute to detection performance.Removing token filtering or evidence components degrades results, while late-stage weighting captures stagnation and rebound patterns.
- Trajectory and Remasking Analysis: Entropy trajectories show factual samples smoothly approaching zero, whereas hallucinated samples can rebound near the end; performance remains competitive across remasking strategies.These findings support modeling trajectory shape and late denoising dynamics rather than relying only on absolute uncertainty.
5 Conclusion
DynHD detects D-LLM hallucinations by comparing observed evidence trajectories with learned references. It combines semantic-aware evidence construction with dynamical deviation learning and reports superior performance with high computational efficiency.
- Conclusion: DynHD uses trajectory-based deviation modeling to identify factual errors by comparing actual hallucination evidence against a learned reference.The method captures late-stage stagnation and uncertainty rebound patterns.
- Conclusion: Combining semantic-aware evidence construction with dynamical deviation learning exploits both spatial uncertainty patterns and temporal denoising behaviors.The framework filters structural tokens and models deviations in denoising trajectories.
- Conclusion: Experiments on mainstream models show that DynHD achieves superior performance with high computational efficiency.The conclusion characterizes the approach as a practical framework for hallucination detection.
Limitations
The evaluation focuses mainly on factual question answering and uses two representative D-LLM backbones with commonly used generation lengths. Broader generation settings, model families, decoding configurations, multilingual settings, and adaptability beyond in-domain factual samples remain to be examined.
- Evaluation scope: The evaluation mainly covers open-domain, multi-hop, and commonsense question answering.These benchmarks provide clear correctness criteria for controlled hallucination-detection evaluation.
- Evaluation scope: Broader evaluation is needed for longform summarization, multi-turn dialogue, mathematical reasoning, code generation, and structured prediction.These settings may involve more diverse factual boundaries and formatting patterns.
- Model and decoding scope: The experiments use two representative D-LLM backbones and commonly used generation lengths.Future work could examine broader model families, decoding configurations, and multilingual settings.
- Method scope: The evidence dynamics generator relies on in-domain factual samples to establish reference trajectories, which may affect adaptability.
Ethics Statement
The paper reports no anticipated ethical risks or conflicts of interest because it concerns scientific questions without human subjects, animals, or environmentally sensitive materials.
- Scope: The research focuses exclusively on scientific questions.
- Participants and materials: The study involves no human subjects, animals, or environmentally sensitive materials.
- Ethical assessment: The authors foresee no ethical risks or conflicts of interest.They state a commitment to scientific integrity and ethics.
Artifact Use and Licensing
The work uses publicly available datasets, models, and baseline methods for research and evaluation, while citing their creators and following applicable artifact licenses and terms.
- Artifacts: The study uses publicly available datasets, models, and baseline methods.
- Use: The artifacts are used only for research and evaluation purposes.
- Licensing: The authors cite the original creators and do not redistribute the original datasets or model weights.
- Licensing: Artifact use follows the corresponding licenses and terms of use released by the providers.
B.2 Complexity Analysis
DynHD adds little computational overhead because it operates on pre-computed uncertainty fields without re-invoking the backbone. Its detection process has an O(T · l) footprint, smaller than the underlying D-LLM’s O(T · l^2) self-attention complexity.
- Overall overhead: DynHD’s computational overhead is negligible relative to D-LLM inference because it uses pre-computed uncertainty fields.The method does not re-invoke the backbone.
- Spatial filtering: Spatial filtering costs O(T · l log k) across a full trajectory using position-wise masking and Top-k statistics.The sequence length l is typically smaller than the vocabulary size |V|.
- Temporal modeling: Temporal modeling costs O(T ·(d·h+h^2)) on condensed 3D evidence vectors.The dimensions d and h are small constants independent of LLM scale.
- Overall complexity: O(T · l) is DynHD’s overall footprint, versus O(T · l^2) for the underlying D-LLM’s self-attention.
C.1 Datasets and Benchmarks
The evaluation uses three factuality-oriented QA benchmarks, seven baselines spanning output- and latent-centric paradigms, and GPT-4o-mini correctness labels. Additional analyses examine annotation reliability, hyperparameter search, denoising examples, and dynamical evidence separation.
- Datasets and Benchmarks: Experiments cover TriviaQA, HotpotQA, and CommonsenseQA, targeting entity retrieval, multi-hop reasoning, and commonsense logic.
- Baselines: Seven baselines are organized into output-centric methods that analyze generated text or probabilities and latent-centric methods that probe internal representations.
- Baselines: TraceDet provides the primary trajectory-based comparison by uniformly aggregating the denoising path to analyze uncertainty evolution.
- Evaluation Protocol: GPT-4o-mini supplies reference-grounded correctness labels, with manual examination finding 94% agreement between automatic and human judgments.
- Denoising Examples: Examples show factual cases with steady late-stage entropy decrease and hallucinations with instability, including rebounds caused by unresolved constraints.
- Evidence Analysis: Filtering clarifies uncertainty data, separating factual downward paths from hallucination paths that bounce at the end.
D.3 Sensitivity of λ1 and λ2
The sensitivity analysis shows that DynHD performs best when both path and rebound regularizers are used, while remaining effective across a broad parameter range. Reference-curve comparisons likewise distinguish factual convergence from hallucination rebounds.
- Regularizer Sensitivity: DynHD works best when λ1 (Path) and λ2 (Rebound) are both used at approximately 0.1 to 0.3 on TriviaQA.
- Regularizer Sensitivity: Performance remains strong across a wide range of λ1 and λ2 values, indicating limited sensitivity to specific settings.
- Denoising Dynamics: At 64 denoising steps, factual answers follow the reference curve closely, whereas hallucinations show a clear bounce.