Source-linked AI summary
Check-COVID: Fact-Checking COVID-19 News Claims with Scientific Evidence
Gengyu Wang, Kate Harwood, Lawrence Chillrud, Amith Ananthram, Melanie Subbiah, Kathleen McKeown
TL;DR
Fact-checking COVID-19 news claims requires matching everyday language to formal scientific evidence, a challenge that surface forms alone cannot address. Check-COVID pairs claims with scientific evidence and tests fact-checking systems; the experiments show limited transfer from existing corpora and difficulty producing human-aligned evidence.
Problem
Automated fact-checking must verify everyday-language news claims against formal, jargon-heavy scientific evidence rather than relying only on surface forms or linguistic features.
Source: §1 · p. 1Method
Check-COVID pairs 1,504 COVID-19 news claims, both extracted and composed, with sentence-level evidence selected by annotators from scientific articles.
Source: §1 · pp. 1–2Results
Existing fact-checking corpora yield only small improvements on Check-COVID, and GPT-3.5 has limitations in providing human-aligned evidence for its labels.
Source: §1 · p. 2Takeaways & Limitations
The benchmark supports evaluating how systems handle both extracted and composed news claims against scientific evidence.
Source: §1 · p. 2The corpus is relatively small and COVID-19-specific, while CORD-19’s size makes retrieval evaluation and full-pipeline NOTENOUGHINFO evaluation difficult.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
We present a new fact-checking benchmark, Check-COVID, that requires systems to verify claims about COVID-19 from news using evidence from scientific articles. This approach to fact-checking is particularly challenging as it requires checking internet text written in everyday language against evidence from journal articles written in formal academic language. Check-COVID contains 1, 504 expert-annotated news claims about the coronavirus paired with sentence-level evidence from scientific journal articles and veracity labels. It includes both extracted (journalist-written) and composed (annotator-written) claims. Experiments using both a fact-checking specific system and GPT-3.5, which respectively achieve F1 scores of 76.99 and 69.90 on this task, reveal the difficulty of automatically fact-checking both claim types and the importance of in-domain data for good performance. Our data and models are released publicly at https://github.com/posuer/Check-COVID.
1 Introduction
Check-COVID frames COVID-19 news-claim verification against scientific articles as a challenge of matching everyday language to formal scientific evidence. It introduces a dataset spanning extracted and composed claims and evaluates baseline systems, finding that existing corpora transfer only modestly.
1 Introduction
- Check-COVID targets the difficult task of checking everyday-language news claims against jargon-laden evidence from an evolving body of scientific articles.Successful verification can require aligning vernacular claims with scientific terminology and commonsense knowledge.Source: §1 · p. 1
- The dataset contains 1,504 COVID-19 claims paired with sentence-level scientific evidence, including 322 extracted claims and 1,182 composed claims.Human annotators select evidence from abstracts in the CORD-19 biomedical literature corpus.Source: §1 · pp. 1–2
- The authors adapt SciFact’s annotation method to news by collecting claims and cited abstracts, then manually identifying claim-specific evidence sentences.The dataset includes both verbatim news claims and annotator-reworded claims.Source: §1 · p. 2
- Existing fact-checking corpora yield only small performance improvements on Check-COVID, while GPT-3.5 has limitations in providing human-aligned evidence for its labels.The experiments use adapted fact-checking systems and GPT-3.5.Source: §1 · p. 2
- The paper evaluates adaptation to both claim types and presents a baseline for future work on this benchmark.Its baseline adapts a system with an abstract retriever, rationale selector, and label predictor.Source: §1 · p. 2
2 Related Work
Prior COVID-19 fact-checking datasets vary in whether they include evidence and in the sources of their claims. Check-COVID differs from similar news-claim work in how it obtains evidence and in including a third veracity label.
2 Related Work
- Many COVID-19 datasets contain claims without evidence, or source claims mainly from social media or scientific texts; some news-based datasets have only a few hundred examples.The paper notes that small news-based datasets can challenge deep-learning training.Source: §2 · p. 2
- Unlike Sarrouti et al. (2021), Check-COVID draws evidence from citations in the source news articles rather than retrieving it with extracted claims as queries.The authors say this makes its claim-evidence pairings closer to those an average reader encounters online.Source: §2 · p. 2
- Check-COVID includes NOTENOUGHINFO, a label for claims that cannot always be fact-checked and that the authors consider important to real-world performance.Source: §2 · p. 2
3 Check-COVID Dataset
Check-COVID is a dataset of 1,504 COVID-19 claims from news, paired with sentence-level scientific evidence and veracity labels. It includes both extracted and composed claims, annotated through a process designed to produce evidence-backed examples.
3 Check-COVID Dataset
- 1,504 claims are paired with sentence-level scientific evidence and veracity labels, with balanced label counts and train/dev/test splits that do not share abstracts.The dataset also provides the abstracts from which evidence is drawn, a subset of CORD-19.Source: §3 · p. 2
3.1 Data Source
- Claims come from citances in news that cite CORD-19 journal articles; the corpus is restricted to well-regarded news sites, with manual annotation and negation checks for evidence relevance.The authors describe the restriction as increasing the likelihood that claims have relevant scientific evidence.Source: §3.1 · p. 3
3.2 Claims in Check-COVID
- Claims are atomic factual statements about COVID-related scientific entities or processes, and the dataset includes composed and extracted forms; models trained on one category do not simply adapt to the other.Composed claims are annotator-written from news paragraphs, while extracted claims are detected from news; the composed-to-extracted ratio is 3.67:1. Refuted examples are created by manually negating supported claims while asking annotators to avoid simple negators such as “not” and “no”.Source: §3.2 · p. 3
3.3 Annotation Procedure
- Annotators select or write claims, create their negations, label claims as supported or refuted, and identify minimal evidence rationales; NOTENOUGHINFO examples pair composed claims with non-evidence sentences from their abstracts.Those non-evidence sentences have lexical and topic overlap with the claim, making the examples difficult.Source: §3.3 · pp. 3–4
3.4 Quality Control
- Annotation quality is controlled through training and practice, review and revision by NLP graduate students, peer review, expert monitoring, and author proofreading.The paper notes that agreement scores are difficult to calculate because annotators write their own claim versions.Source: §3.4 · p. 4
3.5 Dataset Statistics
- Extracted claims are twice as long as composed claims, while the higher frequency of “not/no” in REFUTE claims does not correspond to a clear pattern of easier REFUTE prediction.The authors suggest the length difference indicates simpler wording in composed claims; “not/no” may come from source SUPPORT claims or the negation process.Source: §3.5 · p. 4
4 Check-COVID Task and Challenges
The task is to retrieve COVID-19 scientific abstracts, select relevant evidence sentences, and classify claims as supported, refuted, or lacking enough information. The examples require medical, temporal, and numeric reasoning, and performance declines on temporal and numeric subsets.
4 Check-COVID Task and Challenges
- Fact-checking requires retrieving related scientific abstracts, selecting evidence sentences, and deciding whether the evidence supports, refutes, or provides insufficient information about a claim.The paper says this formulation matches the SciFact task format.Source: §4 · p. 4
- Examples require connecting medical terminology to biological implications, comparing temporal durations or frequencies, and translating and comparing numeric values.For example, the paper contrasts testing twice weekly with once every two weeks, and compares incidence values of 29.2% and 8.1%.Source: §4 · pp. 4–5
- Macro F1 on the composed test set processed through the Vespa pipeline fell from 63.34 to 29.80 for the temporal subset and to 33.87 for the numeric subset.The analysis covers 147 temporal-reasoning claims and 93 numeric-reasoning claims across 450 dev and test examples.Source: §4 · p. 5
5 Baseline System
The baseline adapts a supervised fact-checking pipeline with abstract retrieval, rationale selection, and label prediction. It retrieves COVID-19 literature from CORD-19, then identifies relevant sentences and classifies their relation to the claim.
5 Baseline System
- The baseline adapts a supervised pipeline with abstract retrieval, rationale selection, and label prediction, trained on combinations of fact-checking corpora and Check-COVID.The pipeline draws on systems presented in DeYoung et al. (2019), Wadden et al. (2020), and Wang et al. (2021).Source: §5 · p. 5
- Vespa retrieves and ranks CORD-19 abstracts for a claim with BM25, passing the three highest-scoring abstracts to rationale selection.The authors use a searchable COVID-19 literature corpus that can be continually updated.Source: §5 · p. 5
- A fine-tuned RoBERTa model selects relevant evidence sentences, and another predicts whether the selected rationale supports, refutes, or provides insufficient information about the claim.The label predictor concatenates multi-sentence rationales when needed.Source: §5 · p. 5
6 Experiments
Experiments reveal substantial losses when evidence is retrieved rather than supplied, and GPT-3.5 trails the trained baseline. Performance also varies with claim type and training data.
6 Experiments
- The experiments compare Check-COVID training alone or with FEVER and SciFact, and evaluate full pipelines using Oracle or Vespa abstracts.The best dev-set rationale-selection and label-prediction models were used in the test-set pipeline evaluation.Source: §6 · p. 5
6.1 Rationale Selection Evaluation
- For composed claims, harder rationale metrics peaked at 38.31 global recall-focused F1 and 31.13 strict accuracy, while easier metrics peaked at 52.18 standard F1 and 75.16 intersection accuracy.The harder-metric leaders were Check-COVID-only and FEVER plus Check-COVID; the easier-metric leader used FEVER, SciFact, and Check-COVID.Source: Table 2 · p. 6, §6.1 · p. 7
- Selected single-sentence rationales outperformed random gold sentences for label prediction, while intersection accuracy exceeded strict accuracy by 40 points.The authors note that high intersection scores may mask rationales that include extra sentences.Source: Table 4 · p. 7, §6.1 · p. 7
- FEVER most often boosted rationale-selection performance, SciFact did not always help, and FEVER plus Check-COVID scored best across extracted-claim metrics.The authors suggest FEVER’s size helps, whereas the genre difference between journal-based SciFact claims and news claims limits transfer.Source: §6.1 · p. 7
6.2 Label Prediction Evaluation
- Label prediction peaked at 83.76 macro F1 for composed claims and 85.19 macro F1 for extracted claims; REFUTE and SUPPORT claims were similarly difficult.Training on composed and extracted examples together usually did not improve performance over training on either type alone.Source: §6.2 · pp. 7–8, Table 3 · p. 7
6.3 Pipeline Evaluation
- Oracle composed-only pipelines scored 87.04 macro F1 (allow NEI) and 91.55 (ignore NEI); extracted-only pipelines scored 92.27 and 89.95, respectively.These are the best results for the respective claim types in the full-pipeline evaluation.Source: Table 5 · p. 8, §6.3 · p. 8
- Vespa retrieval reduced F1 by 20–30 points in most settings, and Vespa pipelines benefited more from ignoring NOTENOUGHINFO.The authors suggest Vespa may retrieve less relevant abstracts, leading to more NOTENOUGHINFO predictions.Source: Table 5 · p. 8, §6.3 · p. 8
6.4 In-Context Learning with GPT-3.5
- GPT-3.5 scored below the trained baseline by 10–20 points on rationale selection, 15–30 on label prediction, and 6–20 in the full pipeline.The authors suggest it may struggle to ground natural-language claims in scientific jargon without task-specific fine-tuning.Source: §6.4 · p. 9
7 Conclusion
The paper presents Check-COVID as a benchmark for fact-checking everyday COVID-19 claims against scientific journal evidence. Its experiments show transfer from existing corpora remains difficult, motivating a unified model as future work.
7 Conclusion
- Check-COVID is a corpus and benchmark for checking everyday COVID-19 claims against scientific-journal evidence.The authors also present a strong baseline and evaluate adaptation to extracted and composed claims.Source: §7 · p. 9
- Experiments demonstrate difficulty transferring learning from existing fact-checking corpora to Check-COVID.Source: §7 · p. 9
- Future work will train a unified rationale-selection and label-prediction model to mitigate error propagation.Source: §7 · p. 9
Limitations
The authors identify limits in the corpus’s size, topic scope, evidence-retrieval evaluation, and source representation.
Limitations
- The relatively small corpus focuses specifically on COVID-19, limiting its coverage beyond that topic.The authors say these constraints call for systems with richer inductive biases and effective transfer from related corpora.Source: Limitations · p. 9
- CORD-19 may contain relevant abstracts beyond those selected, preventing abstract-precision measurement and full-pipeline evaluation of NOTENOUGHINFO.Source: Limitations · p. 9
- Claims and annotators come from western, English-language news sources, which may not represent the full range of online COVID-related misinformation.Source: Limitations · p. 9
Ethics Statement
The ethics statement warns that system outcomes depend on the source of truth and that fact-checking can be misused for propaganda or censorship.
Ethics Statement
- The system’s fact-checking results depend on its source of truth; replacing peer-reviewed science with an agenda-driven source could advance propaganda and misinformation.Source: Ethics Statement · p. 9
- Using fact-checking to flag or remove online content could enable unjust censorship when paired with the wrong source of truth.Source: Ethics Statement · p. 9
- Claims came from U.S. news sources leaning left or center, so system behavior on right-leaning claims may differ in ways that favor one set of views.Source: Ethics Statement · p. 9
A News Citance Sources
The section describes the news websites used to collect citances and points readers to the annotation interface and its accompanying instructions.
A News Citance Sources
- Citances came from medical-journal-linked news stories on The New York Times, The Washington Post, The Atlantic, CNN, NPR, and BBC.Source: A News Citance Sources · p. 11
- The authors note that most selected websites lean politically left, which could constrain the breadth of collected claims.Source: A News Citance Sources · p. 11
B Annotation Interface and Instruction Guide
- The paper presents the annotation interface and points readers to its instruction guide and video.Source: B Annotation Interface and Instruction… · p. 11
C Experiment with GPT-3.5
The GPT-3.5 experiments use prompts for rationale selection and label prediction in module and pipeline evaluations. The prompts differ in what they require when evidence is insufficient.
C Experiment with GPT-3.5
- GPT-3.5 prompts used two or three in-context examples with text-davinci-003, a 0.7 temperature, and a 256-token cap.Top-p was 1; frequency and presence penalties were 0.Source: C Experiment with GPT-3.5 · p. 11
- Module rationale selection returns supporting or refuting abstract sentences when available, and permits an empty selection otherwise.Source: C Experiment with GPT-3.5 · p. 11
- Pipeline rationale selection requires at least one abstract sentence, choosing the one most related to the claim when none supports or refutes it.Source: C Experiment with GPT-3.5 · p. 11
Prompt for Label Prediction in Module Evaluation
- Module label prediction distinguishes SUPPORT, REFUTE, and NOT ENOUGH INFO based on the claim and rationale.Source: Prompt for Label Prediction in Module… · p. 12
Prompt for Label Prediction in Pipeline Evaluation
- Pipeline label prediction asks whether the rationale supports or refutes the claim, without a NOT ENOUGH INFO option in the prompt.Source: Prompt for Label Prediction in… · pp. 12–13