Source-linked AI summary
OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models
Jianjiang Yang, Peihang Li, Shanqing Xu, Mengchen Qian, Lu Zhang, Meng Luo
TL;DR
MLLM hallucination detection remains fragmented across modalities and task directions, despite hallucinations threatening safety-critical deployment. OmniHallu unifies claim-level benchmark evaluation and modality-aware multi-agent verification across six tasks, with experiments showing consistent gains and modality-dependent weaknesses.
Problem
Existing hallucination detection mainly targets single modalities or comprehension, leaving cross-modal generation and shared hallucination patterns insufficiently covered.
Method
OmniHallu combines a 10,000-sample claim-annotated benchmark with atomic claim decomposition, modality-specific expert verification, structured aggregation, and preference-optimized verification.
Results
Across all six tasks, the multi-agent framework improves Mac.F1 by +3.4 to +8.1 points over the strongest baseline, with audio receiving the largest gains.
Takeaways & Limitations
Detection benefits from claim decomposition, modality-aware evidence, and structured reasoning, while temporal and audio-grounded verification remain persistent weaknesses.
Takeaways & Limitations
Performance depends partly on expert capability and reasoning-model choice, with current audio experts weaker than vision experts and shared expert errors unresolved by voting.
Abstract
from arXiv · showhide
While Multimodal Large Language Models (MLLMs) have achieved remarkable progress across diverse tasks, they suffer from hallucinations where generated outputs contradict or misrepresent input semantics. Existing research typically addresses hallucination detection within a single modality or task type, limiting generalizability. We introduce OmniHallu, a unified hallucination detection framework spanning both comprehension and generation tasks across image, video, and audio modalities. We contribute OmniHallu-Bench, a 10,000-sample benchmark with claim-level human annotations covering six cross-modal tasks: image-to-text (I2T), video-to-text (V2T), audio-to-text (A2T), text-to-image (T2I), text-to-video (T2V), and text-to-audio (T2A). Our multi-agent architecture decomposes model outputs into atomic claims, verifies them through modality-specific experts, and aggregates evidence via structured reasoning. We further propose a preference-optimized trainable verifier that approximates the multi-agent decision boundary, reducing expert calls by 66% with minimal performance loss. Extensive experiments reveal a consistent modality-dependent performance gradient and provide fine-grained insights into cross-modal hallucination patterns.
1 Introduction
OmniHallu addresses hallucination detection as a cross-modal, bidirectional problem spanning comprehension and generation. It introduces a unified benchmark and modality-aware framework with claim-level verification and a trainable efficiency mechanism.
- MLLM hallucinations affect comprehension and generation across modalities, creating a barrier to safety-critical deployment.
- Existing detectors usually isolate one modality or task, leaving cross-modal and generation hallucinations insufficiently evaluated.
- OmniHallu-Bench provides 10,000 human-annotated samples across six bidirectional image, video, and audio tasks.
- OmniHallu combines claim decomposition, modality-specific verification, and evidence aggregation for unified cross-modal comparison.
- The preference-optimized verifier reduces expensive expert calls by 66% with minimal performance loss.
2 Related Work
Prior hallucination research spans multiple modalities, tasks, detection paradigms, and tool-augmented systems, but largely evaluates each modality and task type independently. OmniHallu extends structured, tool-assisted reasoning toward unified cross-modal detection.
- Prior work studies hallucinations in vision-language, video-language, audio-visual, text-to-image, and text-to-video systems.
- Most existing research evaluates modalities and task types independently, limiting insight into cross-modal regularities.
- Detection methods have progressed from self-consistency checks to structured pipelines using claim decomposition, verification, and cross-system consistency.
- Tool-augmented systems provide external verification capabilities that OmniHallu extends through unified reasoning-based aggregation across modalities.
3 Task Formulation and Hallucination Taxonomy
The paper formulates hallucination detection across text, image, video, and audio inputs and outputs for comprehension and generation. It defines hallucinations at the semantic-claim level and organizes them into four cross-modal error types.
- Comprehension maps image, video, or audio to text, whereas generation maps text to image, video, or audio.
- A hallucinated output contains a semantic claim unsupported by or contradicting the input.
- The benchmark comparison distinguishes fact-checking or detection function, response-, segment-, or claim-level granularity, and explanatory rationale availability.
- The taxonomy covers object, attribute, relation, and event hallucinations across all six tasks.
4 OmniHallu-Bench
OmniHallu-Bench is a stratified, human-verified benchmark spanning six cross-modal tasks, with atomic claim annotations and measured hallucination rates.
- Dataset design: The dataset is stratified across modalities, tasks, and hallucination types to support balanced cross-modal analysis.Comprehension comprises 60% of samples and generation 40%, while image, video, and audio follow a 5:3:2 ratio.
- Annotation and quality control: 24.3% of initial samples were removed after three annotators required full consensus on claim fidelity and hallucination labels.Pre-filter Fleiss’ κ was 0.89 for image, 0.86 for video, and 0.83 for audio.
- Dataset statistics: 42.8% of retained samples contain hallucinations, with higher rates for generation tasks than comprehension tasks.Generation and comprehension hallucination rates are 48.1% and 39.2%, respectively.
- Dataset statistics: Atomic claims average between 3.1 and 5.6 per sample across the six task directions.V2T has the highest average at 5.6 claims, while T2A has the lowest at 3.1.
5 Multi-Agent Hallucination Detection
OmniHallu detects hallucinations by decomposing outputs into atomic claims, verifying them with modality-aware experts, and aggregating evidence into final judgments. A GRPO-trained verifier reduces reliance on expensive expert ensembles while preserving uncertainty-triggered escalation.
- Framework overview: The framework uses three stages: atomic claim decomposition, modality-aware expert verification, and reasoning-based aggregation.The aggregator combines expert judgments, evidence, and confidence signals into a final label and rationale.
- Atomic Claim Decomposition: Atomic Claim Decomposition converts captions or prompts into semantically discrete, self-contained, verifiable claims.Against human references on 300 samples, GPT-4.1 decomposition achieved mean coverage of 0.92 and mean redundancy of 1.15.
- Modality-Aware Expert Verification: Verification specializes by modality: image experts inspect visual evidence, video experts retrieve temporal evidence, and audio experts evaluate acoustic content.Video claims become targeted QA queries, while image object claims use Grounding DINO and other claim types use MLLM ensembles.
- Reasoning-Based Aggregation: Expert judgments are combined by equal-weight majority voting before reasoning-based aggregation produces the final label.The design prioritizes transparency because reliable cross-model confidence calibration is unavailable.
- Trainable Verifier: The verifier maps UNSUP to hallucinated, SUP to nonhallucinated, and ABS to full expert verification.This routing makes abstention an explicit fallback for uncertain cases.
- Trainable Verifier: The preference-optimized verifier uses GRPO rewards for label accuracy, expert-consensus consistency, and calibration.Confident predictions above 0.85 skip expert calls, while abstentions receive full ensemble verification.
6 Experiments
Across six cross-modal tasks, OmniHallu consistently outperforms baselines, with performance varying by modality and task direction. Ablations, cost analysis, and failure analysis identify claim decomposition, multi-expert verification, and modality-specific bottlenecks as central factors.
- 6.1 Main Results: +3.4 to +8.1 Mac.F1 points over the strongest baseline are achieved across all six tasks, with statistically significant improvements.The comparison is reported at claim level across the six-task benchmark, with p < 0.01 using 10,000 bootstrap resamples.
- 6.1 Main Results: Audio is hardest at 73.4–74.1 Mac.F1, yet gains reach +7.8 points for A2T and +8.1 points for T2A, the largest absolute improvements.The passage attributes the challenge to limited auditory expert-tool availability and describes the multi-agent architecture as especially valuable when individual models are weaker.
- 6.1 Main Results: Performance follows an image > video > audio gradient, while comprehension exceeds generation within each modality, including I2T 82.95 versus T2I 81.81 and V2T 77.14 versus T2V 75.63.The reported explanation combines modality complexity, expert-model maturity, and the greater difficulty of verifying generated media against textual prompts.
- 6.2 Ablation Studies: Removing atomic claim decomposition reduces Mac.F1 by 6.6–7.9 points, exceeding the 4.5–5.3-point loss from removing multi-expert voting.The ablation attributes the larger degradation to losing fine-grained hallucination localization, identifying decomposition quality as the primary pipeline bottleneck.
- 6.3 Reasoning Model and Cost Analysis: The GRPO verifier reduces expert calls by 66% and costs about $0.05 per sample, versus 5–8 calls and about $0.12 for the full pipeline.The filter reduces latency from approximately 15 seconds to 6 seconds per sample; the aggregator mode reaches 83.15 Mac.F1, while the filter reaches 80.52 Mac.F1.
- 6.5 Fine-Grained Analysis: Relation hallucinations receive the largest average improvement, +9.2 points, while errors remain modality-specific: perception-limited for images, reasoning-limited for video, and tool-limited for audio.The analysis links relation gains to complementary evidence across experts and identifies small or occluded objects, temporal misalignment, and weak source separation as key failure sources.
7 Conclusion
OmniHallu and OmniHallu-Bench provide a unified, claim-level approach to hallucination detection across cross-modal comprehension and generation. The results identify modality- and error-type-specific weaknesses and motivate stronger temporal, auditory, and compositional verification.
- 7 Conclusion: OmniHallu and OmniHallu-Bench unify claim-level hallucination detection across cross-modal comprehension and generation.The framework places diverse tasks and modalities under a common evaluation protocol for systematic comparison.
- 7 Conclusion: Claim decomposition, modality-aware evidence, and structured reasoning improve hallucination detection across heterogeneous scenarios.
- 7 Conclusion: Temporal and audio-grounded verification remain persistent weaknesses in cross-modal hallucination detection.
- 7 Conclusion: Detection difficulty shifts from object and attribute errors toward event and relation errors.
- 7 Conclusion: Future progress requires stronger temporal, auditory, and compositional verification capabilities rather than generic detectors alone.
Limitations
The framework’s taxonomy and text-centric scope simplify some modality-specific phenomena and exclude nontext pairings. Its performance also depends on the reasoning model and available experts, with shared expert errors remaining uncorrected.
- Taxonomy granularity and scope: The four-type taxonomy compresses modality-specific phenomena and compound errors.Video events and audio errors could be subdivided into more specific categories.
- Taxonomy granularity and scope: The text-centric formulation excludes nontext pairings such as image-to-audio and video-to-image.These pairings may require different decomposition strategies.
- Model and expert dependence: Performance depends on the reasoning model and available experts, with modality differences partly reflecting tool capability.Vision experts are substantially larger than current audio experts.
- Model and expert dependence: Majority voting cannot correct errors shared by all experts.
Ethical Considerations
The paper reports that benchmark sources are publicly available and generated content is disclosed and human-audited. It recommends transparent documentation and human review before high-stakes use.
- Ethical Considerations: Benchmark sources are publicly available, and model-generated content is disclosed and human-audited.
- Ethical Considerations: The authors recommend transparent documentation and human review before high-stakes use.
A Voting Mechanism Design
The voting mechanism uses equal-weight voting because selected experts have comparable performance in their respective domains. This choice favors transparency and avoids task-specific calibration.
- A Voting Mechanism Design: Equal-weight voting is used because the selected experts have comparable performance in their respective domains.
- A Voting Mechanism Design: Unequal weighting would require reliable cross-model calibration, whereas equal voting is transparent and needs no task-specific tuning.
B Additional Ablation Studies
Additional ablations show that video hallucination-detection performance depends on model capacity and that T2V hallucination prevalence rises with prompt complexity.
- The section includes an ablation varying the number of expert agents using Mac.F1.
- Mac.F1 falls from 76.58/74.78 to 62.54/59.25 when video-task experts shrink from 72B/78B to 7B/8B.
- T2V hallucination prevalence increases from 23.2% for low-complexity prompts to 40.3% for high-complexity prompts.Medium-complexity prompts yield 28.7% prevalence.