Source-linked AI summary
FakeSV: A Multimodal Benchmark with Rich Social Context for Fake News Detection on Short Video Platforms
Peng Qi, Yuyan Bu, Juan Cao, Wei Ji, Ruihao Shui, Junbin Xiao, Danding Wang, Tat-Seng Chua
TL;DR
Fake-news detection on short-video platforms lacks comprehensive datasets and multimodal methods that use social context. The paper constructs FakeSV and proposes SV-FEND, which models cross-modal correlations and social-context signals; SV-FEND outperforms other methods, although the task remains challenging and benefits from external fact-checking.
Problem
Fake-news video detection lacks comprehensive large-scale datasets and methods that use all available modalities, including social context.
Method
The paper constructs FakeSV and proposes SV-FEND, which uses co-attention for multimodal content and self-attention to fuse social-context features.
Results
SV-FEND performs much better than other methods, while the best average accuracy on FakeSV remains below 0.8.
Takeaways & Limitations
FakeSV supports multimodal benchmark comparisons and analysis of fake-news characteristics, while the results highlight the difficulty of short-video fake-news detection.
Takeaways & Limitations
Qualitative analysis shows limitations when fake news requires external information, motivating combinations of detection and fact-checking.
Abstract
from arXiv · showhide
Short video platforms have become an important channel for news sharing, but also a new breeding ground for fake news. To mitigate this problem, research of fake news video detection has recently received a lot of attention. Existing works face two roadblocks: the scarcity of comprehensive and largescale datasets and insufficient utilization of multimodal information. Therefore, in this paper, we construct the largest Chinese short video dataset about fake news named FakeSV, which includes news content, user comments, and publisher profiles simultaneously. To understand the characteristics of fake news videos, we conduct exploratory analysis of FakeSV from different perspectives. Moreover, we provide a new multimodal detection model named SV-FEND, which exploits the cross-modal correlations to select the most informative features and utilizes the social context information for detection. Extensive experiments evaluate the superiority of the proposed method and provide detailed comparisons of different methods and modalities for future works.
1 Introduction
Short video platforms are important news channels but create distinctive fake-news detection challenges because videos combine heterogeneous modalities and can be easily edited. FakeSV and SV-FEND address these gaps with rich social-context data and multimodal modeling.
- Motivation: Fake news video detection must select informative clues across multiple modalities because video content and titles jointly describe verifiably false news.Short-video editing can alter frames or titles, weakening visual-truthfulness cues and making content-only analysis inadequate.
- Research gaps: Existing datasets are generally small and lack simultaneous information about news content, user comments, and publisher profiles.This limits fair comparisons among methods using different modalities.
- Contributions: FakeSV provides a large-scale Chinese short-video dataset with complete news content and rich social context for detection and exploratory analysis.The dataset is intended to support research on fake-news characteristics, diffusion, and intervention.
- Contributions: SV-FEND uses co-attention to enhance multimodal content representations and self-attention to fuse them with social-context features.The paper evaluates SV-FEND against existing baselines and compares methods and modalities on FakeSV.
2 Related Work
Prior fake-news video research relies on small, heterogeneous datasets and preliminary methods, often using only partial modalities. These limitations prevent effective fair comparison and motivate FakeSV with an all-modality baseline evaluation.
- Datasets: Existing fake-news video datasets are typically small, platform-specific, and limited in available features, with many containing fewer than 1,000 instances.Their construction is also more burdensome than that of text-based or text-image datasets because videos require viewing and understanding.
- Techniques: Most video fake-news detection methods remain preliminary and rely on hand-crafted features such as metadata, title linguistics, and comment credibility.This contrasts with the more developed neural modeling of text-image correlations.
- Research gap: Because existing methods use different subsets of modalities and are evaluated on different datasets, important clues are missed and comparisons are ineffective.The paper therefore reimplements representative baselines on FakeSV while proposing a model that considers all involved modalities.
3 Dataset Construction
FakeSV is constructed by mining fact-checking sources, retrieving candidate short videos, manually annotating them, and organizing balanced labels across events. The resulting dataset supports multimodal and social-context research.
- Data Collection: The construction process begins with fact-checking websites and uses extracted, deduplicated, and paraphrased event descriptions to guide video retrieval.The paper crawled 99,970 debunking articles from January 2011 to January 2022 and used BERT-based clustering to remove duplicate events.
- Data Collection: FakeSV crawls fields covering news content and social context, including information about videos, comments, and publishers.The dataset is designed to support comparisons across modalities and analyses of fake-news diffusion.
- Dataset Statistics: The dataset records event and time distributions for its three principal video classes.Figure 3 presents how FakeSV is distributed across events and years.
- Annotation: Manual annotation classified 11,603 videos shorter than five minutes into fake, real, debunked, and other categories.After balanced sampling, the dataset contained 1,827 fake, 1,827 real, and 1,884 debunked videos across 738 events.
4 Data Analysis
FakeSV analysis identifies distinct differences between fake and real short news videos across content, social context, and propagation. These patterns provide multimodal clues for detection, including title characteristics, frame quality, publisher behavior, audience responses, persistence after debunking, and duplication.
- News Content: Fake news videos show lower frame quality, more emotionally distinctive speech, and shorter, emptier titles than real news videos.Fake titles emphasize emotional and spoken words, while fake-video speech shows stronger emotional preferences.
- Social Context: Fake news publishers are usually unverified and show more subscribing but less publishing, liking, and fan accumulation than real-news publishers.
- Social Context: After controlling for publisher fan counts, fake news videos receive more likes, and 18% contain doubtful comments versus 4% of real-news videos.
- Propagation: 39% of 434 events with debunking videos later produced fake news videos, especially around current or long-standing hot events.
- Propagation: Fake news videos have higher repetition than real news videos, whereas real events produce more diverse imagery from different witnesses.
5 Method
SV-FEND represents short news videos with textual, audio, visual, comment, and publisher features, then models cross-modal and social-context correlations for fake-news classification.
- 5.2 Multimodal Feature Extraction: SV-FEND extracts six modalities—text, audio, keyframes, video clips, comments, and user profiles—for multimodal news-video representation.The framework combines content and social-context features before classification.
- 5.3 Cross-Modal Feature Fusion: Two cross-modal transformers model interactions among text, audio, and keyframes to capture mutually informative signals.The audio-enhanced textual representation is subsequently interacted with frame features.
- 5.3 Cross-Modal Feature Fusion: A self-attention transformer dynamically fuses the six content and social-context features into a multimodal representation.The concatenated feature sequence is fed into a standard transformer layer before classification.
- 5.4 Classification: A fully connected layer with softmax maps the multimodal representation to real and fake-news probabilities and trains with binary cross-entropy.The predicted vector contains probabilities for labels 0 and 1, corresponding to real and fake news videos.
6 Experiments
Experiments benchmark single- and multimodal methods on FakeSV using five-fold event-level cross-validation. SV-FEND performs best overall, while modality analyses show that news content and complementary visual data are especially informative, though temporal evaluation and case studies expose remaining challenges.
- 6.1 Baselines: The benchmark combines metadata, comments, titles, transcripts, keyframes, video clips, and audio using handcrafted features, pretrained encoders, attention, and standard classifiers.These baselines cover single-modality and multimodality settings for comprehensive comparison.
- 6.2 Experimental Settings: Five-fold cross-validation uses event-level 4:1 training-testing splits with no event overlap and reports accuracy, macro precision, macro recall, and macro F1.The protocol addresses performance variation under different data splits.
- 6.3 Experimental Results: SV-FEND performs much better than other methods on FakeSV, validating its ability to capture important multimodal clues for fake news video detection.The best average accuracy on FakeSV remains below 0.8, underscoring the dataset’s difficulty.
- 6.3 Experimental Results: All modalities contribute to SV-FEND’s best performance, with news content more effective than social context and comments playing the least role.Title and transcript, as well as keyframes and video clips, exhibit both homogeneity and complementarity.
- 6.3 Experimental Results: SV-FEND remains superior under a chronological 70%:15%:15% temporal split, although temporal performance is higher because long-standing fake-news events are present.This split evaluates detection of future fake news using only previously emerging data for training.
- 6.4 Case Studies: A missed case illustrates that one-sided user-uploaded videos can require external information, motivating combined fake-news detection and fact-checking.The example depicts a police drill as a real-life incident.
7 Conclusion and Potential Applications
The paper concludes by presenting FakeSV as a benchmark with rich news-content and social-context features, alongside SV-FEND and extensive comparisons. It also identifies applications in debunked-content intervention, fake-news evolution, and user trustworthiness analysis.
- Conclusion: FakeSV provides a benchmark with abundant news-content and social-context features, while extensive experiments compare methods and modalities.The dataset also supports research beyond fake news video detection.
- Potential Applications: FakeSV can support detection and intervention for previously fact-checked fake news because it provides corresponding debunking articles and videos.The paper discusses personalized recommendations of debunked videos as an application.
- Potential Applications: Video-similarity-based restoration of secondary-editing chains can support analysis of fake-news event lifecycles and publisher or spreader intentions.This application uses propagation structure to study how fake news evolves.
- Potential Applications: Combining published videos with user profiles can support trustworthiness and susceptibility analysis, including cross-platform comparisons.The proposed analyses concern users’ preferences and susceptibility to fake news across platforms.
Ethical Considerations
The dataset uses public publisher information to study behavioral differences between fake- and real-news publishers. The authors anonymized the data, documented collection and use, and restrict it to academic research.
- Ethical Considerations: Publisher profiles are collected from public information to analyze behavioral differences between fake- and real-news publishers.The data collection follows prior work on user profiles in fake news detection.
- Ethical Considerations: The authors anonymized the collected data and clearly stated what information was collected and how it was used.These practices are described as part of the dataset’s responsible handling.
- Ethical Considerations: The data is intended for academic research and should not be used outside academic research purposes.This defines the stated use boundary for the dataset.
A Dataset Construction
FakeSV combines official fact-checking resources, structured extraction procedures, and trained annotation to construct a challenging dataset. Its construction uses debunked events so fake and real videos can be compared within the same events.
- A Dataset Construction: FakeSV construction uses official fact-checking sites, regular expressions for key-sentence extraction, and predefined queries spanning multiple domains.These resources are documented through collection tables and query examples.
- A Dataset Construction: Nine postgraduate annotators received Chinese guidelines and sample videos, completed pilot testing, and achieved Cohen’s Kappa of 0.89.Videos were assigned according to different events for annotation.
- A Dataset Construction: The dataset uses a fact-checking-based strategy that can obtain larger datasets with more credible annotations than retrieving videos around broad events and judging them directly.The alternative strategy depends more heavily on retrieval ability and annotator judgment because direct evidence is limited.
- A Dataset Construction: Real videos are selected from events that were debunked rather than from general real-news events, making fake-versus-real discrimination possible within the same events.The authors describe this setting as more challenging and valuable for studying fake news videos.
B Data Analysis
FakeSV analysis identifies distinctive title, emotion, and publisher timing patterns in fake news videos, alongside regional differences among rumormongers.
- Textual and emotional patterns: Fake news videos have shorter, emptier titles, with more like emotion, while real-news titles show more disgust.The analysis derives title-emotion intensities from a Chinese affective lexicon covering seven emotion types.
- Textual and emotional patterns: Fake and real videos differ in title and transcript length, with transcripts averaging 211 words versus 33 for titles.The transcript analysis is motivated by its greater descriptive detail about news events.
- Spatiotemporal: Fake news videos emerge more often during off-hours, whereas real and debunking videos are more common during working hours.All three classes share a trough from 0:00–8:00 and peaks around 11:00 and 15:00.
- Spatiotemporal: Rumormongers are geographically concentrated in Guangdong and several central-coast provinces.The distribution is based on publishers’ current IP locations.
C Experimental Settings
The experiments combine pretrained multimodal feature extractors, transformer-based fusion, and several single- and multimodal baselines under specified training and preprocessing settings.
- SV-FEND implementation: SV-FEND uses pretrained BERT, VGGish, VGG19, and C3D features with co-attention and self-attention transformer blocks.The co-attention block uses four heads and hidden size 128, while the self-attention transformer uses two heads.
- SV-FEND implementation: The model caps video frames, audio frames, and comments at 83, 50, and 23, respectively, using uniform sampling and zero-padding when needed.Training runs for 30 epochs with early stopping to reduce overfitting.
- Baseline implementation: Single-modality baselines include handcrafted-feature SVM, Text-CNN, BERT, visual attention models, C3D+Att, and audio-emotion features.The baselines use specified pretrained visual models and OpenSmile audio-emotion features.
- Baseline implementation: Multimodal baselines are reimplemented from their published descriptions because original code is unavailable.The authors state that these implementations will be released to support fair comparisons.