PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding

Zongxia Li, Xiyang Wu, Guangyao Shi, Yubin Qin, Hongyang Du, Fuxiao Liu, Tianyi Zhou, Dinesh Manocha, Jordan Lee Boyd-Graber

arXiv:2505.01481v4cs.CVcs.LG

TL;DR

Familiar real-world video evaluations leave open whether VLMs reason from visual cues or rely on learned correlations, while real-world out-of-distribution tests are costly to construct. VideoHallu uses expert-curated synthetic videos with controlled violations to evaluate VLMs and examines post-training strategies. Models often miss these violations, while GRPO fine-tuning improves VideoHallu accuracy without degrading real-world performance.

  • Problem

    Real-world video evaluations resemble training data, leaving uncertain whether VLMs reason from visual cues or exploit familiar correlations; constructing real-world out-of-distribution tests is costly.

    Source: §1 · pp. 1–2

  • Method

    VideoHallu evaluates VLMs with expert-curated question-answer pairs about controlled violations in synthetic videos, and the paper tests GRPO post-training with synthetic data.

    Source: §1 · pp. 1–2

  • Results

    VLMs frequently miss synthetic-video violations, while GRPO fine-tuning improves VideoHallu accuracy without degrading real-world performance.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Takeaways & Limitations

    VideoHallu exposes gaps in VLM recognition of synthetic-video abnormalities, and structured physics and commonsense reasoning data can improve performance on these tasks.

    Source: §5 · p. 9

    Chain-of-thought prompting can amplify reliance on language priors and increase hallucination risk on synthetic videos that conflict with real-world expectations.

    Source: §3.1 · pp. 5–6

Abstract

from arXiv · show

Vision-Language Models (VLMs) have achieved strong results in video understanding, yet a key question remains: do they truly comprehend visual content or only learn shallow correlations between vision and language? Real visual understanding, especially of physics and common sense, is essential for AI systems that interact with the physical world. Current evaluations mostly use real-world videos similar to training data, so high benchmark scores may not reflect real reasoning ability. To address this, we propose negative-control tests using videos that depict physically impossible or logically inconsistent events. We introduce VideoHallu, a synthetic dataset of physics- and commonsense-violating scenes generated with Veo2, Sora, and Kling. It includes expert-annotated question-answer pairs across four categories of violations. Tests of leading VLMs (Qwen-2.5-VL, Video-R1, VideoChat-R1) show that, despite strong results on benchmarks such as MVBench and MMVU, they often miss these violations, exposing gaps in visual reasoning. Reinforcement learning fine-tuning on VideoHallu improves recognition of such violations without reducing standard benchmark performance. Our data is available at https://github.com/zli12321/VideoHallu.git.

1 Introduction

The introduction argues that familiar real-world video tests cannot establish whether VLMs reason from visual cues or rely on learned correlations. It introduces VideoHallu’s synthetic negative controls and reports frequent model failures alongside gains from GRPO post-training.

1 Introduction

  • Real-world evaluations closely aligned with training data leave unresolved whether VLMs reason from visual cues or exploit familiar vision-language correlations.Constructing out-of-distribution tests using real-world videos is costly and impractical.Source: §1 · pp. 1–2
  • Synthetic video generation enables controlled tests with physically impossible or logically inconsistent events beyond familiar training distributions.The paper describes modern video generators as able to produce photorealistic but physically impossible scenes.Source: §1 · pp. 1–2
  • VideoHallu provides 3K expert-annotated QA pairs across alignment, consistency, commonsense, and physical-reasoning categories.The paper uses the benchmark to evaluate leading VLMs and analyze their failure modes.Source: §1 · p. 2, Figure 2 · p. 3
  • Top-performing models achieve only ∼50% accuracy on VideoHallu, frequently hallucinating in counterintuitive scenarios.The reported examples include failures to detect physically or logically inconsistent events.Source: §1 · p. 2
  • GRPO post-training with synthetic data improves VideoHallu visual reasoning while maintaining real-world performance.The paper presents this as a post-training strategy alongside supervised fine-tuning.Source: §1 · pp. 2–3

2 VideoHallu: Evaluating VLMs’ Synthetic Video Understanding

VideoHallu’s data pipeline creates synthetic videos with deliberate abnormalities, then uses expert review to build adversarial questions grounded in what the videos show. The resulting dataset has non-overlapping training, validation, and test splits.

2 VideoHallu: Evaluating VLMs’ Synthetic Video Understanding

  • Questions contrast visually grounded answers with language-prior expectations, targeting counterintuitive events and detection of visual or logical inconsistencies.The paper distinguishes counterintuitive questions about implausible events from critical-thinking questions about inconsistencies.Source: §2 · p. 3
  • Five experts created 141 adversarial prompts that generated 987 videos across seven video models.The prompt design followed reviewed abnormality categories.Source: §2 · p. 3
  • Experts reviewed videos and wrote natural-language questions and ground-truth answers for visually clear but semantically abnormal content.The QA pairs probe violations of common sense, physics, or prompt-video matches.Source: §2 · pp. 3–4
  • The dataset contains 3,233 QA pairs split into 800 training, 908 validation, and 1,525 test pairs, with no video overlap across splits.Videos average 96.0 frames, about 5.3 seconds at 23 FPS.Source: §2 · p. 4

3 Experiment and Results

Leading VLMs struggle to detect abnormalities in synthetic videos. The experiments compare SFT and GRPO and examine how synthetic training data affects performance.

3 Experiment and Results

  • Evaluation covered 17 leading VLMs, and GPT-4o-mini judge ratings achieved 97% agreement with human annotations on 200 sampled answer pairs.CoT models were prompted to reason step by step before producing a final answer.Source: §3 · p. 4

3.1 Limitations of VLMs in OOD Data

  • Below 55% accuracy, leading VLMs scored only slightly above the 50% random baseline on VideoHallu.Table 1 evaluates alignment, physics understanding, spatial-temporal consistency, and commonsense reasoning.Source: Table 1 · p. 4, §3.1 · p. 5
  • Video-R1 and VideoChat-R1 showed limited synthetic-video improvement over Qwen2.5-VL-7B, with minimal or worse alignment and commonsense performance.The authors report that training on real-world videos alone inculcates real-world reasoning patterns.Source: §3.1 · p. 5

3.2 Visual Learning or Pattern Matching? Questioning RL’s True Impact on Vision Models

  • GRPO outperformed SFT on out-of-distribution and critical visual understanding tasks across Qwen2.5-VL-7B and LLaVA-One-Vision.The authors interpret the result as evidence that GRPO transfers better to novel scenarios than SFT's surface-pattern learning.Source: Table 2 · p. 7, §3.2 · p. 7
  • Real-world-only training improved synthetic-video understanding by just 0.07%, while mixing real and synthetic videos produced the strongest results.The authors say synthetic data supports domain-specific reasoning, while real-world data adds diversity that stabilizes training and robustness.Source: §3.2 · p. 7, Table 2 · p. 7
  • Post-training performance was also evaluated on MVBench and MMVU to examine whether synthetic-video gains coexist with real-world video understanding.The supplied passages name the benchmarks but do not report their results.Source: §3.2 · p. 8, Table 3 · p. 8
  • The authors attribute synthetic-video hallucinations to models relying on learned physics and commonsense priors instead of visual evidence.They report that R1-trained models can lean on linguistic priors, making their chain-of-thought responses susceptible to hallucinations in counterintuitive scenes.Source: §3.2 · p. 8
  • The authors report that small, well-annotated datasets with positive and negative examples and reasoning-rich training can improve abnormality detection, including for smaller models.They identify data quality and coverage, not just fine-tuning method, as drivers of gains.Source: §3.2 · p. 8

4 Related Work

Prior video-hallucination benchmarks mainly use real-world factual data, while hallucination in generative video understanding remains underexplored. Work on VLM post-training likewise focuses predominantly on real-world imagery and videos.

Hallucinations in VLMs

  • Video hallucination benchmarks evaluate entities and events, but most rely on real-world factual data rather than synthetic videos.The paper describes hallucinations as outputs that contradict or misrepresent target text, images, or videos; video understanding is more challenging than static-image understanding because of spatial-temporal information and contextual cues.Source: §4 · pp. 8–9

Reinforcement Learning for Post-training of Vision-Language Models

  • Prior reinforcement-learning post-training work predominantly targets real-world imagery and videos, leaving generated videos comparatively little explored.Video-R1 uses 260K video and image samples with data-type-specific rewards; VideoChat-R1 combines video-centric instruction tuning with RLHF for interactive dialogue.Source: §4 · p. 9

Video Generation Models and Synthetic Content Monitoring

  • Synthetic video generation has advanced, but systematic VLM datasets and evaluation protocols for understanding generated videos remain lacking.The passage also identifies scalability challenges for monitoring generated content, as manual annotation and evaluation become infeasible at scale.Source: §4 · p. 9

5 Conclusion

VideoHallu evaluates visual commonsense and physics reasoning with counterfactual synthetic videos and expert-annotated questions. The paper reports accuracy improvements after GRPO fine-tuning with real and synthetic videos, while noting that annotation and training costs constrain scalability.

5 Conclusion

  • GRPO fine-tuning with both real and synthetic videos improves accuracy on VideoHallu.The conclusion frames this as evidence for the value of structured physics and commonsense reasoning data on synthetic video tasks.Source: §5 · p. 9
  • VideoHallu uses expert-annotated, reasoning-intensive QA pairs across alignment, spatial-temporal consistency, commonsense, and physics categories to assess abnormality detection.The dataset is designed to evaluate VLM visual commonsense and physics reasoning through counterfactual scenarios.Source: §5 · p. 9
  • High-quality annotation and fine-tuning at scale are costly, and limited data and compute constrain further progress.Source: §5 · p. 9
  • The authors propose expanding abnormality QA datasets and exploring adversarial QA and prompt decomposition to reduce reliance on human annotations.Source: §5 · p. 9

A More Synthetic Video Examples

The appendix presents selected model-evaluation cases across VideoHallu subcategories. Each case pairs prompts and video frames with questions, ground truth, and model answers, highlighting hallucinations and critical context.

A More Synthetic Video Examples

B Theoretical Problem Formulation

The formulation treats synthetic-video evaluation as maximizing divergence between VLM and human answers while constraining language-only judgments to remain human-aligned. It also requires the video to contain human-detectable inconsistencies relative to an embedded context.

B Theoretical Problem Formulation

  • The objective maximizes contextual distance between VLM and human outputs for a synthetic video V and query Q.The formulation defines fVLM, fLLM, and fHuman as the VLM, its language-model backbone, and a human expert, respectively.Source: B Theoretical Problem Formulation · p. 22
  • The constraints keep the language-only model aligned with human judgment within tolerance ϵ while requiring the video to differ from the context by more than threshold δ.Source: B Theoretical Problem Formulation · p. 22
  • Context C is embedded within video V to preserve coherence, and the formulation maps context and query elements into natural language.Source: B Theoretical Problem Formulation · p. 22

C Video Understanding and Evaluation Categorization/Motivation

VideoHallu categorizes evaluation questions across alignment, spatial-temporal consistency, common-sense reasoning, and physics to identify visual abnormalities beyond superficial video-quality metrics.

Categorization purpose

  • The categories target visual abnormalities rather than relying on superficial measures such as frame consistency or resolution.Targeted adversarial questions are designed to assess whether models detect and interpret these issues.Source: C Video Understanding and Evaluation… · p. 23

Alignment

  • Alignment covers entity counts, appearance and classification, and the spatial relationships described in the prompt.It checks whether the video matches the prompt without omissions or discrepancies.Source: C Video Understanding and Evaluation… · p. 23

Spatial-Temporal Consistency

  • Spatial-temporal consistency examines camera and entity movement over time, including abrupt changes, transformations, and abnormal appearances or disappearances.The category includes camera, spatial, and temporal dynamics.Source: C Video Understanding and Evaluation… · p. 23

Common Sense Reasoning

  • Common-sense reasoning tests the use of everyday knowledge and reasoning cues to detect conflicts between the visual context and expected events.Its subcategories cover everyday phenomena and interpreting emotional or environmental clues.Source: C Video Understanding and Evaluation… · p. 23

Physics

  • Physics questions probe inconsistencies in conservation, physical constraints, motion, and physics-driven state transitions.Examples include gravity, rigid bodies blocking motion, reflection, heat effects, and phase transitions.Source: C Video Understanding and Evaluation… · p. 23

D Prompt Templates

The appendix provides separate prompt templates for reasoning before answering, direct answers, and LLM-based judging.

CoT prompt

Direct-answer prompt

LLM-as-a-judge prompt

  • Table 6 provides the prompt template for an LLM acting as a judge.The appendix identifies it as the LLM-as-a-judge prompt template.Source: Table 6 · p. 24

E Categorization Breakdown Results

The section uses radar-chart breakdowns to supplement aggregate results and compare model performance across VideoHallu categories and fine-tuning configurations.

Supplementary analysis

Figure 17

  • Across subcategories, evaluated VLMs are weaker on physics and commonsense reasoning than on alignment tasks.Figure 17 separates smaller models, 7B–38B models, R1 fine-tuned models, and large black-box VLMs.Source: Figure 17 · p. 25

Figure 18

  • Fine-tuning on commonsense and physics data with RFT improves reasoning and overall synthetic-video-understanding performance over Qwen2.5-VL-7B.Figure 18 compares previous SOTA models, sub-dataset fine-tuning, and full-dataset curriculum learning.Source: Figure 18 · p. 25

F Common Sense and Video-dependent Question-Answering

VideoHallu separates language-only commonsense questions from video-dependent questions, and the reported results show that models struggle most with open-ended abnormality detection.

Question types

Results

  • All six evaluated MLLMs achieve 100.0% accuracy on commonsense-only questions, but all score below 55% on counterintuitive questions and none exceeds 25% on critical thinking.Table 7 distinguishes questions answerable from language priors alone from those testing counterintuitive phenomena and abnormality detection.Source: F Common Sense and Video-dependent… · p. 26, Table 7 · p. 26
  • Gemini-2.5-Pro has the highest critical-thinking accuracy at 23.3%, followed by Video-R1 at 16.7%.The authors describe this as suggesting some benefit from CoT prompting, while noting that CoT does not consistently improve abnormality detection.Source: F Common Sense and Video-dependent… · p. 26

Question types

  • Counterintuitive questions provide contextual hints, whereas open-ended critical-thinking questions require models to identify and reason about abnormalities unaided.The latter are more vulnerable to hallucinations when video understanding is incomplete.Source: F Common Sense and Video-dependent… · p. 26
Loading 2505.01481v4…