Source-linked AI summary
Defending Against Neural Fake News
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, Yejin Choi
TL;DR
Online disinformation is an urgent problem, and advances in language generation may enable realistic, targeted neural fake news. The paper introduces Grover as a controllable generator and detector, finding that Grover detects neural fake news with 92% accuracy and that generation artifacts remain detectable.
Problem
Online fake news intentionally deceives audiences, while advances in natural language generation may enable realistic propaganda at scale, creating a need to understand and respond to neural fake news.
Method
The paper frames fake-news generation and detection as an adversarial security problem and develops Grover for efficient, controllable generation and discriminative detection of multi-field news documents.
Results
92% accuracy: Grover detects neural fake news, outperforming existing discriminators at 73% accuracy; exposure bias and mitigating sampling strategies leave detectable artifacts.
Takeaways & Limitations
The best generators can also be effective detectors, supporting the paper’s conclusion that releasing strong generators is important for defense against neural fake news.
Takeaways & Limitations
A human-seeded generation workaround leaves a human-written beginning that may create distributional artifacts, while models conditioned on their own predictions may avoid exposure bias but often perform poorly on language tasks.
Abstract
from arXiv · showhide
Recent progress in natural language generation has raised dual-use concerns. While applications like summarization and translation are positive, the underlying technology also might enable adversaries to generate neural fake news: targeted propaganda that closely mimics the style of real news. Modern computer security relies on careful threat modeling: identifying potential threats and vulnerabilities from an adversary's point of view, and exploring potential mitigations to these threats. Likewise, developing robust defenses against neural fake news requires us first to carefully investigate and characterize the risks of these models. We thus present a model for controllable text generation called Grover. Given a headline like `Link Found Between Vaccines and Autism,' Grover can generate the rest of the article; humans find these generations to be more trustworthy than human-written disinformation. Developing robust verification techniques against generators like Grover is critical. We find that best current discriminators can classify neural fake news from real, human-written, news with 73% accuracy, assuming access to a moderate level of training data. Counterintuitively, the best defense against Grover turns out to be Grover itself, with 92% accuracy, demonstrating the importance of public release of strong generators. We investigate these results further, showing that exposure bias -- and sampling strategies that alleviate its effects -- both leave artifacts that similar discriminators can pick up on. We conclude by discussing ethical issues regarding the technology, and plan to release Grover publicly, helping pave the way for better detection of neural fake news.
1 Introduction
The paper frames AI-generated neural fake news as an emerging disinformation threat and introduces Grover to study controllable generation and detection. Grover-generated disinformation can appear highly trustworthy, while its own discriminative use achieves strong detection accuracy.
- Online fake news intentionally deceives audiences to gain advertising revenue, influence opinions, and potentially affect elections.
- Progress in natural language generation may let malicious actors produce realistic-looking propaganda at scale.
- Grover generates complete, controllable news articles, including titles, sources, dates, authors, and article bodies.
- Humans rate Grover-generated disinformation as more trustworthy than human-written disinformation.
- 92% accuracy: Grover detects neural fake news better than existing deep pretrained-language-model discriminators, which achieve 73% accuracy.
- Exposure bias and sampling strategies that mitigate it both leave artifacts detectable by strong discriminators.
- The paper also examines ethical responsibilities and the potential negative implications of releasing neural-disinformation models.
2 Fake News in a Neural and Adversarial Setting
The paper treats fake-news generation and detection as an adversarial security problem involving attackers and verifiers. This framing highlights escalating attacks and defenses, alongside the scope and limits of existing verification practices.
- The paper presents a framework for anticipating deep-model disinformation attempts and designing corresponding verification responses.
- Scope of fake news: Fake news includes forms ranging from satire to propaganda; this paper focuses on text-only news articles containing purposefully false information.
- Fact checking and verification: Existing verification services rely on manual fact-checking, but cognitive biases can make people believe fake news that fits their worldview.
- Framework: The framework models fake-news generation and detection as an adversarial game between an adversary and a verifier.
- Framework: Adversaries seek realistic stories matching specified attributes, while verifiers classify stories as real or fake with few examples from a specific adversary.
- The dual objectives of attackers and defenders create an escalating arms race requiring preparation for increasingly strong adversarial attacks.
3 Grover: Modeling Conditional Generation of Neural Fake News
Grover is designed for controllable generation of complete news articles by modeling structured metadata and body fields together. Its conditional design improves language modeling when metadata is provided, while decoding choices affect generation quality.
- Grover: Modeling Conditional Generation of Neural Fake News: Grover generates realistic, controlled articles by producing metadata and body fields, rather than treating an article as only unstructured text.The model can generate or condition on domains, dates, authors, headlines, and bodies.
- Grover: Modeling Conditional Generation of Neural Fake News: Grover uses field-specific start and end tokens to generate a target field from a sorted set of contextual fields.This supports flexible decomposition of the multi-field article distribution during inference.
- Grover: Modeling Conditional Generation of Neural Fake News: Grover can generate an article body, authors, and a revised headline from partial metadata, enabling controllable article construction.Figure 2 illustrates body generation with missing authors, author generation, and headline regeneration.
- Grover: Modeling Conditional Generation of Neural Fake News: 10% field dropout and 35% body-only dropout expose Grover to both conditional and unconditional generation during training.Fields are randomly partitioned into two sets, and the model predicts one set after the other using cross-entropy.
- 3.1 Language Modeling results: measuring the importance of data, context, and size: 0.6–0.9 perplexity points is the improvement Grover obtains when conditioned on metadata, with Grover-Mega reaching 8.7 perplexity conditionally.Perplexity also decreases with model size, and Grover outperforms similarly sized GPT2 models on these news evaluations.
- Grover: Modeling Conditional Generation of Neural Fake News: Nucleus Sampling (top-p) is used for open-ended generation because likelihood-maximization methods can produce degenerate text.The selected decoding strategy samples from the smallest set of probable words whose cumulative probability reaches threshold p.
4 Humans are Easily Fooled by Grover-written Propaganda
Human evaluation finds that Grover-written propaganda can appear more plausible than human-written propaganda. The gain in overall trustworthiness is accompanied by lower consistency between generated content and its supplied metadata.
- 4 Humans are Easily Fooled by Grover-written Propaganda: Grover-written propaganda is rated more plausible than original human-written propaganda, although Grover-written news does not match human-written news overall.The comparison covers human and machine versions of reputable news and propaganda articles.
- 4 Humans are Easily Fooled by Grover-written Propaganda: 2.64 versus 2.85 is the average consistency score for machine-written versus human-written propaganda relative to headline, date, and author.The difference is reported on the same one-to-three scale and is statistically significant at p ≤ 0.01.
5 Neural Fake News Detection
The paper evaluates neural fake-news detection as a semi-supervised adversarial problem, comparing pretrained discriminators with Grover-based discrimination. Grover is strongest at detecting its own generations, while weaker-generator data improves detection when target examples are scarce.
- 5 Neural Fake News Detection: 73% accuracy is achieved by the best existing deep pretrained discriminators when given 5000 Grover generations and unlimited real news.The detection setting reflects limited fake examples from a specific adversary and abundant real-news data.
- 5.2 Discrimination results: Grover performs best at detecting Grover’s fake news: Grover-based discrimination reaches roughly 90% accuracy across generator and discriminator sizes, while larger generators reduce accuracy below 81% and larger discriminators raise it above 98%.The results show a strong effect of relative model size in the generator–discriminator arms race.
- 5.2 Discrimination results: Grover performs best at detecting Grover’s fake news: Paired detection is easier than unpaired detection, indicating that independent classification is harder to calibrate.In the paired setting, the discriminator compares real and machine-written articles sharing metadata.
- 5.2 Discrimination results: Grover performs best at detecting Grover’s fake news: Grover outperforms other discriminators, including BERT, at detecting Grover’s generations, suggesting that matching the generator’s inductive bias is useful.This pattern holds even when architecture size and, for BERT, news-domain adaptation are controlled.
- 5.2 Discrimination results: Grover performs best at detecting Grover’s fake news: 78% accuracy is obtained with 16–256 Grover-Large examples plus weak supervision, versus around 50% without weak supervision.As Grover-Mega examples increase, accuracy converges to around 92%.
6 How does a model distinguish between human and machine text?
Grover distinguishes human from machine text using artifacts created by exposure bias and variance-reduction sampling. Random sampling increasingly leaves the human-language distribution, while carefully tuned truncation makes discrimination hardest.
- Exposure Bias: Exposure bias creates generation artifacts because models train on human-written text rather than their own generations.Random sampling can therefore move increasingly out-of-distribution as sequences lengthen.
- Exposure Bias: Randomly sampled text has higher perplexity than human-written text, with the gap increasing by body position.At p=0.96, limiting variance lowers perplexity and limits its growth.
- Variance Reduction: Variance clipping creates detectable artifacts because Nucleus Sampling removes the distribution’s final 1-p tail from every token choice.For human text, the probability that all n tokens fall within the top-p mass is p^n, which approaches zero as n increases.
- Variance Reduction: Discriminator visibility depends on whether its language model matches the generator’s worldview and can identify the truncated tail.This difference may help explain BERT’s lower discrimination performance.
- Variance Reduction: p=0.94–0.98 is the discrimination sweet spot, where unpaired validation accuracy is hardest across discriminators.Both insufficient and excessive variance reduction make generated text easier to distinguish.
- Overall Interpretation: Grover is strongest at detecting Grover because it is best positioned to recognize whether the generator truncated the distribution tail.The analysis frames exposure bias and variance reduction as a double-bind for generation and detection.
7 Conclusion: a Release Strategy for Grover
The paper argues that neural disinformation poses a real threat and that releasing strong generators can improve defenses. It identifies practical deployment safeguards, ethical concerns, and several boundaries for future work.
- Conclusion: Grover provides a concrete threat model for controllable neural fake news, while its own capabilities suggest that the threat is real and dangerous.The model can rewrite propaganda, with humans rating the rewritten versions as more trustworthy.
- Threat Scale: Training Grover-Mega cost $25k over two weeks, while acquiring the required Common Crawl data cost $10k in AWS credits.The training process can be massively parallelized, and greater investment could produce more powerful generators.
- Release Strategy: Releasing generators gives researchers access to effective detectors, whereas keeping them private would leave little recourse against adversarial attacks.The authors therefore released their models to researchers.
- Future Work: BERT is weaker than left-to-right models such as Grover at detecting Grover generations, and alternative generation objectives may evade Grover-based discrimination.The paper notes that self-conditioned training might reduce exposure bias but can lower language-task performance.
- Scope: The study covers adversaries generating complete news articles from minimal context, not comments, dialogue agents, sentence edits, images, or video.The authors call for these additional threat models to be studied for better defenses.
- Scope: Machine-generated truthful news is outside the main study, although templated-news domains in the dataset are easy for Grover to spoof convincingly.The paper also notes that Grover can help spot human-written fake news.
- Future Work: Current discriminators rely mainly on distributional features rather than evidence-based assessments of truth.Scaling knowledge-based verification to entire articles without paired evidence remains open.
- Deployment: Platforms could scan news with ensembles of generative and shallow models, but humans should remain involved because false flags and social biases are possible.The proposal adapts existing neural scanning practices used for uploaded videos.
A Optimization Hyperparameters
The appendix specifies optimization choices for training Grover, emphasizing memory-efficient optimization for the largest model.
- Optimization: Adafactor is used because Adam’s optimizer memory scales linearly with parameter count, making Grover-Mega training nearly impossible.Adafactor factors second-order momentum parameters into a tensor product of two vectors.
B Real News and Propaganda Websites
The experiments compare mainstream news sites with selected propaganda and misinformation sites. BERT discrimination uses metadata and article body as separate input spans and adapts to the news domain.
- Data Sources: Real-news sources include The Guardian, Reuters, The New York Times, The Atlantic, USA Today, Huffington Post, and NBC News.These sites define the mainstream news set used in generation experiments.
- Data Sources: Propaganda sources include Breitbart, Infowars, WND, Big League Politics, and Natural News.The selected sites were associated with misinformation or propaganda.
- BERT Setup: BERT receives metadata and article body as two spans separated by its special input structure, using the cased model for cased generations.Metadata fields are prefixed by names such as [title].
- BERT Setup: BERT is adapted to the news domain on RealNews and extended beyond its original 512-token sequence length for Grover’s longer generations.Grover generations contain up to 1024 BPE tokens, whereas BERT was pretrained with at most 512 WordPiece tokens.
D Hyperparameters for the Discriminators
The discrimination experiments used length-limited articles and model-specific training settings, while human evaluation measured style, content, trustworthiness, and metadata consistency.
- Discrimination setup: Discrimination inputs were limited to 1024 BPE tokens because the discriminators handled documents up to 1024 words.Longer documents were empirically easier for the models to discriminate.
- Discriminator hyperparameters: BERT discrimination models used Adam, a 2e-5 learning rate, batch size 64, five epochs, and 20% linear warm-up.
- Human evaluation: Human raters scored article style, content coherence, and overall trustworthiness on a 1–3 Likert scale.The highest score indicated consistency, coherence, or trustworthiness, respectively.
- Metadata evaluation: Metadata consistency was evaluated by asking how well the article body matched its headline, authors, and publication date.Each question used the same 1–3 response scale.
F Examples
The examples illustrate Grover-generated news, propaganda, identity-spoofed opinion, political reporting, movie reviews, and business news, alongside human-rating comparisons.
- Human comparisons: Figures 8 and 9 compare human-written and Grover-written articles about shared headlines or propaganda topics using average human ratings.The examples cover a Guardian news article and Natural News propaganda article.
- Controlled propaganda: Grover generates controlled anti-vaccine propaganda from a headline about a link between autism and vaccines.The example continues the teaser figure’s hypothetical claim.
- Identity spoofing: Grover-Mega produces a realistic-looking editorial seemingly written by New York Times columnist Paul Krugman.The example supplies the New York Times domain, author, date, and a Trump-impeachment headline.
- Political news: Grover-Mega generates fake political news in the Washington Post style about the House voting to impeach Donald Trump.The generated article includes a headline stating that Congress voted to impeach Trump and that he vowed to veto the resolution.
- Other generated content: Additional examples include a positive LA Times-style review of Sharknado 6 and a TechCrunch-style article about an “Uber for Dogs” startup.These examples demonstrate opinion spam and fake business news generation.