Source-linked AI summary
Let the Bullets Fly: Multimodal Fake News Detection with Temporal-Aligned Generative Danmaku
Xiansheng Luo, Chaowei Zhang, Zewei Zhang, Yi Zhu, Jipeng Qiang
TL;DR
The paper addresses the latency of real Danmaku for real-time fake news detection. It proposes Genda to generate temporally aligned pseudo-Danmaku and DM-FEND to integrate it with multimodal content; DM-FEND consistently outperforms baselines on FakeSV and FakeTT.
Problem
Existing fake-news approaches do not capture how collective user responses evolve temporally alongside multimedia content, while real Danmaku accumulates too slowly for real-time detection.
Method
Genda predicts reaction timing and intensity and generates aligned Danmaku, while DM-FEND uses it to guide interactions among video, audio, text, and Danmaku.
Results
DM-FEND consistently outperforms baseline methods on the Chinese FakeSV and English FakeTT benchmarks.
Takeaways & Limitations
Temporally aligned Danmaku modeling improves multimodal fake-news detection and provides dynamic supervision before real user-interaction streams accumulate.
Takeaways & Limitations
Existing approaches remain limited in capturing temporal user-interaction dynamics and the evolution of collective responses alongside content.
Abstract
from arXiv · showhide
The social interactions among crowds via \textit{Danmaku} (a.k.a., bullet comments) on modern multimedia platforms can facilitate both viewpoint conflicts and consensus, providing fine-grained discriminative social signals that can benefit fake news detection. However, the inherent accumulation latency of \textit{Danmaku} in real-world scenarios violates the real-time necessity of fake news detection, making the studies of \textit{Danmaku}-related fake news detection underexplored. To break this violation, we simulate this temporal-aware user interactive process by proposing a novel temporal \textbf{Gen}erative \textbf{da}nmaku framework, called \textbf{Genda}, which consists of: (1) a \textit{Danmaku} Trigger for predicting the timing and intensity of user reactions; and (2) a \textit{Danmaku} Generator for synthesizing corresponding semantic and emotional expressions, thereby mutually constructing a temporally aligned and human-like pseudo \textit{Danmaku} streams. To make the generated \textit{Danmaku} useful for identifying fake news videos, we further design a \textit{Danmaku}-guided Temporal Multimodal fake news detection model - \textbf{DM-FEND}, which enables fine-grained multimodal interactions among video, audio, text, and \textit{Danmaku}, enhancing dynamic modalities alignment and semantic noise inhibition. The experimental results demonstrate that \emph{DM-FEND} consistently outperforms state-of-the-art baselines across both Chinese (FakeSV) and English (FakeTT) benchmarks. Further ablations validate the crucial role of temporal \textit{Danmaku} modeling in enhancing robustness and discriminative capability. Finally, this study offers a bright and robust solution for multimodal fake news detection in modern social interactive fashions by bridging the temporal inconsistency between news and user behaviors.
1 Introduction
The paper argues that temporally synchronized Danmaku provides social and content-aligned signals for real-time fake news detection, then introduces Genda and DM-FEND to generate and exploit those signals.
- Fast-paced multimedia news platforms intensify fake-news spread and create a need for urgent, real-time detection.
- Content-centric detectors can miss societal context when manipulations leave few visual artifacts or genuine videos are miscontextualized.
- Genda predicts when users react and how strongly, then generates semantic and emotional Danmaku aligned with video frames.
- DM-FEND uses generated Danmaku to guide segment-level interactions among video, audio, and text while suppressing decision-irrelevant information.
- Across two short-video datasets, DM-FEND consistently outperforms fine-tuned unimodal, prompt-based, and state-of-the-art baseline methods.
2 Fake News Detection in the Era of LMs
Prior fake-news detectors emphasize content authenticity or auxiliary social signals, but existing approaches do not model the temporal evolution of user interactions. The paper addresses this gap with temporally grounded Danmaku generation and multimodal integration.
- Mainstream detectors analyze misleading patterns, semantic inconsistencies, and manipulation cues across multimedia content.
- Generation-based approaches add supplementary supervision such as reasoning chains, counterfactual samples, or extra multimodal content.
- Existing approaches neither capture temporal user-interaction dynamics nor learn how collective responses evolve alongside content.
- Genda produces pseudo-Danmaku streams, while DM-FEND integrates them with video, audio, and text at the clip level.
3 Why does Danmaku work?
An empirical study finds that Danmaku follows structured temporal evolution and aligns more strongly with local video segments than with entire videos, supporting its use as a fine-grained social signal.
- Both real and fake videos show Danmaku density rising rapidly near the beginning, peaking at t≈0.07, and declining afterward.
- Local alignment reaches 0.749 average semantic similarity, compared with 0.686 for global alignment.
- These structured temporal patterns and strong local alignment support using Danmaku to capture temporal multimodal interactions.
4 Genda: Temporal Generative Danmaku
Genda represents videos as temporal segments, predicts ordinal reaction intensity with a Danmaku Trigger, and conditions a Danmaku Generator on content, timing, intensity, and user style to synthesize aligned responses.
- Genda overview: Genda comprises video understanding, a Danmaku Trigger, and a Danmaku Generator for modeling temporal user reactions.
- Problem definition: The task models each short video as multimodal content divided into temporally ordered segments with associated Danmaku signals.
- Video understanding: Videos are partitioned into N non-overlapping segments, each with fixed duration Δt=4 seconds, and represented using sampled frames and segment-level multimodal reasoning.
- Danmaku Trigger: The Danmaku Trigger estimates reaction intensity as a continuous latent score and maps it to five ordinal levels using monotonic thresholds.
- Danmaku Trigger: The five intensity levels are None, Minimal, Normal, Noticeable, and Heated, with threshold counting producing the final ordinal prediction.
- Danmaku Generator: The Danmaku Generator conditions responses on segment semantics, predicted intensity, temporal position, and latent user style to produce diverse, human-like streams.
5 Danmaku-guided Fake News Detection
DM-FEND uses temporally aligned Danmaku as supervision for unimodal enhancement and segment-level multimodal alignment, then jointly learns video-level representations and fake-news classification.
- DM-FEND extracts segment-level representations from text, video, audio, and Danmaku in a unified multimodal space.
- Danmaku-guided Unimodal Learning: Danmaku-conditioned saliency scores identify high- and low-saliency tokens for complementary masking and reconstruction.The method replaces selected tokens with a learnable mask embedding and re-encodes them.
- Danmaku-guided Unimodal Learning: The reconstruction objective recovers masked high-saliency content, aligns original and noise-masked representations, and regularizes saliency distributions.These terms support robustness and discriminative unimodal representations by distinguishing informative content from noise.
- Danmaku-guided Multimodal Learning: Danmaku-guided contrastive learning aligns same-segment modality pairs with associated Danmaku while separating mismatched pairs.The modality pairs are text-video, text-audio, and video-audio.
- Danmaku-guided Multimodal Learning: Segment-level multimodal representations are temporally aggregated into video-level features and jointly optimized with classification and similarity objectives.The unified formulation jointly optimizes representation learning, cross-modal alignment, and final decision-making under Danmaku supervision.
Implementation Details: We use Qwen2.5-VL-7B-Instruct1
DM-FEND is evaluated on FakeSV and FakeTT against unimodal, language-model, and multimodal baselines, with ablations examining temporal Danmaku generation and guided learning components. Results show consistent gains across datasets, while case studies and PCA visualizations examine temporal localization and feature separation.
- Results and Analysis: DM-FEND achieves the best performance across all evaluation metrics on both FakeSV and FakeTT.On FakeSV, it reaches 87.01% accuracy and 86.68% Macro-F1; on FakeTT, it surpasses ExMRD by 3.63% accuracy and 3.73% Macro-F1.
- Results and Analysis: Unimodal models and large language models perform worse than multimodal approaches on the evaluated benchmarks.Text-BERT reaches 81.36% accuracy on FakeSV and 76.54% on FakeTT, while GPT-5-mini reaches 64.49% accuracy on FakeTT.
- Results and Analysis: DM-FEND models temporally aligned Danmaku as dynamic supervision for fine-grained segment-level alignment and semantic noise suppression.The framework facilitates interactions among video, audio, text, and Danmaku, while guided unimodal learning strengthens salient cues and suppresses low-saliency tokens.
- Ablation Study: Removing all proposed components lowers accuracy to 81.49% on FakeSV and 77.58% on FakeTT, whereas individual modules consistently improve performance.The CM-only variant raises FakeSV accuracy by 1.63%, and the DA-only variant improves FakeTT Mis-F1 by 6.47%.
- Ablation Study: Combining all three components improves accuracy over the base model by 5.52% on FakeSV and 5.85% on FakeTT.The full model also shows the most distinct real–fake feature separation, with minimal overlap and well-structured clusters in Fig. 3.
- Case Study on Genda: The case study shows pseudo Danmaku becoming more specific across clips and highlighting deceptive signals when critical text appears.The generated stream provides fine-grained cues that help localize moments indicative of misleading or manipulated content.
7 Conclusion
The paper introduces Genda to reconstruct temporally aligned Danmaku signals and DM-FEND to use them for multimodal fake news detection. Together, these components target real-time detection while modeling fine-grained interactions among video, audio, text, and Danmaku.
- Genda simulates user reactions during video playback with a Danmaku Trigger and Danmaku Generator, reconstructing realistic interaction signals without real-time feedback.
- DM-FEND supports clip-level interactions among video, audio, text, and Danmaku for temporal multimodal fake news detection.
- DM-FEND uses Danmaku guidance to enhance unimodal representations and strengthen cross-modal alignment through Danmaku-aware multimodal learning.