Source-linked AI summary
Weak Supervision for Fake News Detection via Reinforcement Learning
Yaqing Wang, Weifeng Yang, Fenglong Ma, Jin Xu, Bin Zhong, Qiang Deng, Jing Gao
TL;DR
Fake news detection needs fresh labeled data, but manual annotation is costly and existing labels can become outdated. WeFEND uses user reports as weak supervision, then applies reinforcement learning to select higher-quality samples for a content-based detector. Experiments on WeChat news show that WeFEND achieves the best performance against state-of-the-art methods.
Problem
Deep fake news detectors need substantial labeled data, but manual annotation is expensive and existing annotations may not represent newly emerging events.
Method
WeFEND automatically labels unlabeled news from user reports, uses reinforcement learning to select high-quality weakly labeled samples, and trains a content-based fake news detector.
Results
WeFEND achieves the best performance compared with state-of-the-art fake news detection methods on a large WeChat official-account dataset.
Takeaways & Limitations
User reports can serve as weak supervision for enlarging fake news detection training data when paired with quality-oriented sample selection.
Takeaways & Limitations
The framework assumes a small set of expert-labeled samples and focuses on news articles with user feedback comments.
Abstract
from arXiv · showhide
Today social media has become the primary source for news. Via social media platforms, fake news travel at unprecedented speeds, reach global audiences and put users and communities at great risk. Therefore, it is extremely important to detect fake news as early as possible. Recently, deep learning based approaches have shown improved performance in fake news detection. However, the training of such models requires a large amount of labeled data, but manual annotation is time-consuming and expensive. Moreover, due to the dynamic nature of news, annotated samples may become outdated quickly and cannot represent the news articles on newly emerged events. Therefore, how to obtain fresh and high-quality labeled samples is the major challenge in employing deep learning models for fake news detection. In order to tackle this challenge, we propose a reinforced weakly-supervised fake news detection framework, i.e., WeFEND, which can leverage users' reports as weak supervision to enlarge the amount of training data for fake news detection. The proposed framework consists of three main components: the annotator, the reinforced selector and the fake news detector. The annotator can automatically assign weak labels for unlabeled news based on users' reports. The reinforced selector using reinforcement learning techniques chooses high-quality samples from the weakly labeled data and filters out those low-quality ones that may degrade the detector's prediction performance. The fake news detector aims to identify fake news based on the news content. We tested the proposed framework on a large collection of news articles published via WeChat official accounts and associated user reports. Extensive experiments on this dataset show that the proposed WeFEND model achieves the best performance compared with the state-of-the-art methods.
Introduction
WeFEND addresses the shortage and noise of labeled data for fake news detection by using user reports as weak supervision and reinforcement learning to select higher-quality samples. Experiments on WeChat news and user feedback show strong detection performance.
- Motivation: Social media enables fake news to spread rapidly, creating an urgent need for early automatic detection.Rapid dissemination can produce social panic and financial loss even after correct information becomes available.
- Motivation: Deep learning improves fake news detection but requires large amounts of expensive, manually labeled data.The dynamic nature of news also makes existing annotations less representative of newly emerged events.
- Motivation: User reports provide scalable weak annotations, but their labels and explanations can be noisy or unreliable.The central challenge is converting weak annotations into training labels while selecting high-quality samples.
- Approach: WeFEND combines an annotator, reinforced selector, and fake news detector to exploit user feedback for content-based detection.The annotator assigns weak labels to unlabeled articles, while the selector chooses samples for detector training.
- Contributions: Reinforcement learning enables WeFEND to select high-quality samples, improving fake news detection performance.The framework is designed to enlarge the training set through automatic annotation while filtering lower-quality weakly labeled data.
- Results: On a large-scale WeChat official-account dataset, WeFEND significantly outperformed state-of-the-art fake news detection models.The evaluation used news articles and associated user reports collected from WeChat official accounts.
Related Work
Prior work uses social context or news content for fake news detection, but social context may arrive too late and user feedback may be unreliable. WeFEND incorporates report explanations as weak supervision and uses reinforcement learning for performance-driven sample selection.
- Fake News Detection: Related fake news detection work includes rumor detection, spam detection, and methods based on social context or news content.The paper defines fake news as intentionally fabricated news that can be verified as false.
- Fake News Detection: Social context features require accumulated engagement data, limiting timely detection of newly emerged fake news.Examples include follower counts, hashtag retweets, and network structure.
- Weak Supervision: Manual labeling is expensive, while user feedback is valuable but potentially unreliable because user reliability is unknown.Uneven user participation makes reliability estimation difficult.
- Weak Supervision: WeFEND uses users’ explanatory report messages as weak supervision for content-based fake news detection.These explanations can contain information about why users consider an article suspicious.
- Reinforcement Learning: The framework applies reinforcement learning to select high-quality samples using a performance-driven data-selection method.The paper argues that prediction probabilities are unsuitable selection criteria when the distribution of news changes over time.
Methodology
WeFEND combines report-based weak annotation, reinforcement-learning sample selection, and content-based fake-news detection. The selector evaluates samples using quality, diversity, labels, and detection-performance feedback before adding selected data to detector training.
- WeFEND comprises an annotator, data selector, and fake news detector that jointly expand and improve training from weakly labeled news.The annotator assigns weak labels from user reports, the selector filters samples, and the detector predicts fake-news probabilities from news content.
- A CNN-based textual feature extractor supplies representations to both the annotator and the fake news detector.The extractor processes news content or report messages represented as sequences of word embeddings.
- The annotator uses user reports and labeled examples to assign weak labels to unlabeled news, enlarging the training set at low cost.Report messages are aggregated into sample representations before the annotator outputs fake-news probabilities.
- The reinforced selector retains or removes each weakly labeled sample within a sequential decision process modeled as a Markov Decision Process.Its reward is the change in fake-news detection accuracy after selecting a bag of samples.
- Each selector state combines annotator probability, detector probability, maximum cosine similarity, and the current sample’s weak label.These features represent sample quality, diversity, and class balance relative to already chosen samples.
- The selector maximizes expected cumulative reward, using summed accuracy differences to make reinforcement-learning training more efficient.The framework also divides inputs into small bags, providing additional feedback during selector training.
Reinforced Weakly-supervised Fake News Detection Framework
The complete framework first trains an annotator on labeled reports, then assigns weak labels to unlabeled news and selects a subset with the reinforced selector. The fake news detector is trained on both selected weakly labeled data and original labeled data.
- The framework pretrains the annotator on labeled report data and assigns weak labels to the unlabeled news set.The annotated unlabeled set is then passed to the reinforced selector for sample selection.
- The reinforced selector chooses high-quality samples from the weakly labeled dataset, producing the selected set used by the detector.Selection follows the procedure summarized in Algorithm 1.
- The detector is trained on both selected automatically annotated data and the original manually labeled data.Its final loss contains separate sublosses for these two data sources, with λ_l and λ_u controlling their balance.
- Algorithm 2 alternates annotator training and weak-label assignment with selector-based data selection and fake news detector updates.The detector update uses the original and selected datasets together.
Experiments
Experiments evaluate WeFEND on a WeChat dataset containing user reports, comparing supervised, semi-supervised, weakly supervised, and automatically annotated settings. WeFEND achieves the strongest reported performance while addressing temporal distribution change and noisy weak labels.
- Dataset: The WeChat dataset contains news and user reports from WeChat Official Accounts, with manually verified samples split by non-overlapping post timestamps.Training data span March–September 2018, while testing data span September–October 2018.
- Experimental Settings: The experiments compare supervised, semi-supervised, weakly supervised, automatically annotated, and feature-based learning settings.The weakly supervised setting trains an annotator on reported samples, assigns weak labels to unlabeled news, and selects validation subsets.
- Performance Comparison: WeFEND achieves the best results across Accuracy, AUC-ROC, precision, recall, and F1 on the WeChat dataset.The reported comparison covers the evaluated approaches in Table 2.
- Performance Comparison: CNN improves Accuracy and AUC-ROC by around 27% and 35%, respectively, over the best traditional machine-learning baseline LIWC-RF.Deep learning models, including LSTM, CNN, and EANN, are reported to outperform traditional models.
- Semi-supervised Setting: LSTMsemi increases Accuracy by 3% and AUC-ROC by 5% compared with supervised LSTM when unlabeled news are incorporated.The experiment attributes this improvement to enlarging the training set with unlabeled data.
- Weak Supervision: Weakly supervised CNNweak and LSTMweak outperform their supervised versions, while WeFEND improves precision and achieves the best overall performance among baselines.The reinforced selector is intended to reduce the effect of noisy weak labels; without it, increasing coverage raises recall but lowers fake-news precision.
- Temporal Distribution: Detector accuracy is around 90% on the same-time set but around 70% on the different-time set, confirming changing news distributions.Report-based annotation achieves similar performance across the two time sets, indicating that its annotation quality does not change with time.
- Importance of Reinforced Selector: The reinforced selector yields stably higher average accuracy than the version without reinforcement learning after 12 epochs.The ablation compares five runs over the first 30 epochs and attributes the improvement to the reinforced selector.
Conclusions
The paper addresses the difficulty of obtaining continuously labeled, high-quality samples for dynamic fake-news detection by combining weak supervision with reinforcement learning. WeFEND integrates an annotator, reinforced selector, and detector, and experiments on WeChat data show significantly improved detection performance.
- Dynamic news makes continuously labeled, high-quality training samples infeasible to obtain, especially for deep learning models.
- WeFEND leverages user reports as weak supervision for fake news detection.
- The framework integrates an annotator, reinforced selector, and fake news detector.The annotator labels unlabeled articles, the selector chooses high-quality samples, and the detector predicts news labels.
- The enhanced training set produced by the annotator and selector yields significantly improved fake-news detection performance on a WeChat dataset.