Fact-Enhanced Synthetic News Generation

• Synthetic text can be realistic yet abused to generate and amplify disinformation, creating challenges for the news ecosystem and accurate detection.Source: Introduction · p. 1
• Fake and real news can both contain supplementary facts, while existing synthetic-news methods may contradict claims or omit essential details.Source: Introduction · p. 1
• The stated generation challenges are to keep news related to its claim or context and include supplemental factual information.Source: Introduction · p. 2
• FACTGEN combines a pseudo-self-attentive language model, external fact retrieval, and claim reconstruction to generate fact-consistent, fact-enriched news.The retriever supplies candidate facts, while the reconstructor predicts masked claim tokens from generated content and unmasked claim tokens.Source: Introduction · p. 2
• The authors report experiments on real-world datasets using quantitative and qualitative metrics to evaluate generation effectiveness and a defense.Source: Introduction · p. 2

Methodology

• FACTGEN generates news from a claim and retrieved facts, while a masked-language objective reconstructs the claim from generated content and unmasked fragments.The combined objective weights claim reconstruction with λ and is minimized end-to-end.Source: Methodology · p. 2

Preliminary

• Self-attentive language models encode inputs into key-value pairs and queries, then produce outputs using query-key dot products to weight values.The input hidden states are projected into key, value, and query spaces.Source: Preliminary · p. 2

Proposed Method

• FACTGEN’s pseudo-self-attentive model extends decoder key-value pairs with claim and fact representations, so decoding uses the input as well as previous output.The claim and facts are marked separately; fact representations use randomly initialized projections.Source: Proposed Method · p. 3, Figure 2 · p. 3
• The fact retriever selects top-k1 documents by tf-idf cosine similarity, then top-k2 sentences by cosine similarity between BERT sentence encodings.This adds related external facts to the source input.Source: Proposed Method · p. 3
• The claim reconstructor predicts masked claim tokens from unmasked fragments and mean-pooled generated-output states, without requiring prior input knowledge.It uses pseudo-self-attention to project the output representation into key-value pairs.Source: Proposed Method · p. 3, Figure 3 · p. 3

Training Schedule

• FACTGEN uses two training stages because the generation and reconstruction improvements cannot be trained together directly by minimizing the combined objective.The schedule jointly trains the modules, first on claim/news pairs and then on the corpus with retrieved facts; the authors cite warm starts and gradient-explosion avoidance as benefits.Source: Training Schedule · pp. 3–4

Experiments

• Experiments assess generated news for fluency, consistency, richness, and trustworthiness on two real-world datasets.The evaluation uses both quantitative and qualitative measures.Source: Experiments · p. 4

Dataset

• GossipCop pairs claims averaging 30 words with 250-word content; CNN/DailyMail uses highlights averaging 56 tokens as claims and 790-token content as targets.Claims and content longer than 100 and 300 words, respectively, are truncated; GossipCop is split 75%/15%/10% for training, validation, and testing.Source: Dataset · p. 4
• The external fact corpus comprises factual sentences from the training data, avoiding whole-news copying and test-time data leakage.The paper focuses on text facts, while noting the approach could extend to tabular data or knowledge graphs.Source: Dataset · p. 4

Experiment Settings

• FACTGEN uses a four-block, 12-head encoder, a median pretrained GPT-2 decoder, and a three-block claim-reconstruction module.It retrieves 10 documents and 5 sentences, and uses top-p sampling with p = 0.9.Source: Experiment Settings · p. 4

Automatic Evaluation

• Automatic evaluation measures fluency with BLEU, consistency with the proportion of stance predictions that agree, and richness with unique named entities.The stance detector is fine-tuned on Fake News Challenge data and achieves 0.93 test accuracy.Source: Automatic Evaluation · p. 4

Human Evaluation

• Two linguistically trained annotators with no advanced knowledge about the generated content’s source evaluate 100 CNN/DailyMail samples for fluency, richness, consistency, and trustworthiness.The human evaluation has 7,200 questions, each scored from 1 to 3, with 3 best and 1 worst.Source: Human Evaluation · pp. 4–5

Baseline Methods

• CopyTransformer is a transformer sequence-to-sequence model with a pointer network that copies source words into its output.Source: Baseline Methods · p. 5
• Conv Seq2Seq is a convolutional sequence-to-sequence model designed to generate claim-consistent stories.Source: Baseline Methods · p. 5
• PPLM controls topic and content in a pretrained language model without fine-tuning.Source: Baseline Methods · p. 5
• GPT-2 is a large pretrained language model and transformer decoder; the median model size is used for fair comparison.Source: Baseline Methods · p. 5
• Grover generates news conditioned on its title, authors, and website domain, using the same architecture as GPT-2.Source: Baseline Methods · p. 5

Experimental Results

• FACTGEN achieves the best fluency performance by human evaluation on CNN/DailyMail and by BLEU on both datasets.The reported human fluency comparison is on CNN/DailyMail.Source: Experimental Results · p. 5
• 42% improvement over the best baseline on GossipCop automatic consistency and 6% over the best baseline on CNN/DailyMail human consistency.The authors attribute the increase to claim reconstruction improving output coverage of input information.Source: Experimental Results · p. 5
• FACTGEN ranks first for richness on CNN/DailyMail and second on GossipCop; the smaller GossipCop candidate-document pool (7,331 < 278,408) limits fact retrieval.The authors say the fact retriever cannot retrieve enough related facts from the smaller corpus and claim reconstruction rejects inconsistent facts during generation.Source: Experimental Results · p. 5
• Human trustworthiness evaluations indicate that FACTGEN can generate high-quality text overall.The authors say this result helps them understand differences between machine-generated and true news.Source: Experimental Results · p. 5

Case Study

• FACTGEN’s sample adds Iraq and UK nuclear-related context to the Iran agreement, while Grover’s added facts are unrelated and GPT-2 and PPLM provide little supplemental information.The case study describes FACTGEN’s added context as making the generated news more convincing.Source: Case Study · p. 5

Ablation Study

• λ = 0.001 performs best across the automatic evaluations, while increasing λ greatly decreases fact richness.The paper attributes this trade-off to claim reconstruction constraining coverage around the input.Source: Ablation Study · p. 5, Figure 4 · p. 5
• Removing claim reconstruction or the Fact Retriever decreases performance, though claim reconstruction improves consistency at the cost of fact richness.Without claim reconstruction, fact information is richest but consistency is lowest; without the Fact Retriever, BLEU and consistency are comparatively strong but richness is worst.Source: Ablation Study · pp. 5–6
• Two-stage training outperforms single-stage training across all reported automatic and human evaluation categories.The single-stage comparator directly uses claims and external facts in the first stage.Source: Ablation Study · p. 6

Difficulty of Defending Synthetic Fake News

• On 300 fake generated news items and 300 human-written real-news items from GossipCop, fake-news detection methods perform worse than neural text classification (RoBERTaa > EANN, MWSS-CNN), indicating difficulty detecting fake synthetic news.Training included an extra 100 fake synthetic news pieces.Source: Difficulty of Defending Synthetic Fake… · p. 6

Defending Against Synthetic Fake News

• FACTGENdef achieves the best accuracy in the reported comparison, using 300 synthetic fake-news pieces and 300 human-written real-news pieces for fine-tuning.It uses a FACTGEN checkpoint at iteration 20k and represents the input with hY.Source: Defending Against Synthetic Fake News · p. 6

Synthetic News Generation

• Synthetic-news systems have used rules and templates; Grover models news metadata autoregressively, while structured-data-to-text work generates sports reports from score data [Lepp¨anen et al. 2017a] [Zellers et al. 2019a] [Wiseman, Shieber, and Rush 2017].The structured-data approach uses copy mechanisms and source reconstruction, while another method separates planning and realization into two stages [Puduppully, Dong, and Lapata 2018].Source: Related Work · pp. 6–7

Synthetic/Fake News Detection

• Fake-news detection approaches use linguistic content representations, reconstruct user comments, or stylometric features, while visualization tools help non-experts inspect generated text [P´erez-Rosas et al. 2017] [Schuster et al. 2020] [Gehrmann, Strobelt, and Rush 2019].Source: Related Work · p. 7
• Neural generation detectors fine-tune classifiers on earlier generator checkpoints, while another approach identifies the NLP method that produced text [Zellers et al. 2019a] [Solaiman et al. 2019].Source: Related Work · p. 7

Conclusion and Future Work

• FACTGEN targets fact-consistency and fact-richness, and the paper reports that FACTGENdef performs strongly at detecting synthetic fake-news content.Source: Conclusion and Future Work · p. 7
• The authors propose adding tabular or knowledge-graph facts to retrieve up-to-date information during generation.Source: Conclusion and Future Work · p. 7
• They also propose exploring style control to make generated content more prone to spread.Source: Conclusion and Future Work · p. 7

Human Evaluation Question

• Human evaluation uses four questions scored from 1 to 3, covering fluency, richness, consistency, and trustworthiness.The fluency item asks whether the article appears human-written; the others ask about extra information, consistency with the input, and trust in the output.Source: Human Evaluation Question · p. 9

Synthetic News Generation

• FACTGEN is compared with five baselines, and retrieval uses the top 10 documents and five sentences per document on both datasets.The datasets are GossipCop and CNN/DailyMail; semantic-similarity parameters match RoBERTa.Source: Synthetic News Generation · p. 9

Synthetic News Detection

• The synthetic-news detection setup uses EANN and MWSS models, alongside a RoBERTa detector for distinguishing human-written from machine-generated content.The appendix identifies online sources for EANN and MWSS-CNN and introduces the RoBERTa hyperparameter settings.Source: Synthetic News Detection · p. 9
• The RoBERTa model is fine-tuned for 10 epochs, with early stopping after 3 epochs without validation-set progress, batch size 10, learning rate 5e-5, and inputs padded to 300 tokens.Source: Synthetic News Detection · p. 9

Appendices on Ethics Statement

• Journalists could use FACTGEN to generate fact-enriched, claim-based news, but outputs still require manual checking.The statement frames journalism assistance as a possible use.Source: Appendices on Ethics Statement · p. 9
• FACTGENdef is presented as a possible defense, but it relies on semantic information rather than veracity; future work should verify factual correctness through a claim-checking pipeline.The proposed pipeline includes check-worthy sentence extraction, verified-claim matching, and prediction.Source: Appendices on Ethics Statement · p. 9
• The authors propose publicly releasing code but restricting generator and discriminator checkpoints to academic use.Source: Appendices on Ethics Statement · p. 9