PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Automated Fact-Checking for Public Health Claims

Neema Kotonya, Francesca Toni

arXiv:2010.09926v1cs.CLcs.AI

TL;DR

Automated fact-checking has focused largely on political claims, while explainable systems for expertise-dependent domains remain limited. The paper studies public-health fact-checking, constructs PUBHEALTH, and develops veracity-prediction and explanation-generation methods with coherence evaluation. Its results indicate that in-domain training improves veracity prediction accuracy and generated-explanation quality, while the study identifies variation in the intended purpose of labelled explanations and plans broader evaluation.

  • Problem

    Automated fact-checking has focused largely on political claims, with few explainable systems for domains requiring specialised expertise.

  • Method

    The paper constructs PUBHEALTH and develops public-health-specific veracity prediction and explanation-generation models, evaluating explanation coherence with humans and computational methods.

  • Results

    In-domain training improves veracity-prediction accuracy and the quality of generated explanations compared with generic training.

  • Takeaways & Limitations

    Explainable automated fact-checking can make gains for claims requiring specific expertise when models are trained on in-domain data.

  • Takeaways & Limitations

    The labelled explanations vary widely in their intended purpose across the crawled websites, and the study plans to examine explanation properties beyond coherence.

Abstract

from arXiv · show

Fact-checking is the task of verifying the veracity of claims by assessing their assertions against credible evidence. The vast majority of fact-checking studies focus exclusively on political claims. Very little research explores fact-checking for other topics, specifically subject matters for which expertise is required. We present the first study of explainable fact-checking for claims which require specific expertise. For our case study we choose the setting of public health. To support this case study we construct a new dataset PUBHEALTH of 11.8K claims accompanied by journalist crafted, gold standard explanations (i.e., judgments) to support the fact-check labels for claims. We explore two tasks: veracity prediction and explanation generation. We also define and evaluate, with humans and computationally, three coherence properties of explanation quality. Our results indicate that, by training on in-domain data, gains can be made in explainable, automated fact-checking for claims which require specific expertise.

1 Introduction

Automated fact-checking has advanced, but remains narrowly focused on political claims and lacks explainable systems. This study addresses both gaps in the public-health domain, where specialised expertise is essential.

  • Existing automated fact-checking research has improved veracity prediction and produced datasets, but mainly targets political or general claims.
  • Public-health fact-checking is challenging because claims require specific expertise and may be inaccessible to non-experts through jargon and biomedical terminology.
  • The study presents explainable automated fact-checking for claims requiring specialised expertise, using public health as its case study.
  • The proposed system must provide human-understandable judgments for predictions that are also understandable to people without subject-domain expertise.
  • PUBHEALTH is introduced as a public-health fact-checking dataset with journalist-produced gold-standard explanations, described as the first dataset of its kind.
  • The paper introduces public-health-specific veracity prediction and explanation generation, reports gains from in-domain data, and defines three coherence properties evaluable by humans and computationally.
  • The in-domain explanation model outperforms the general-purpose model on summarization evaluation and explanation quality.

2 Related Work

Related work has explored domain-specific biomedical language models and explainable fact-checking, but explainable systems have rarely addressed expertise-dependent public-health claims through accessible abstractive explanations. This paper combines extractive evidence selection with abstractive summarization and evaluates explanation coherence computationally and with humans.

  • Biomedical NLP: Domain-specific biomedical language models such as BIOBERT and SCIBERT improve on BERT for several in-domain NLP tasks.
  • Biomedical NLP: Because biomedical NLP models show domain-specific improvements, the authors hypothesize that public-health fact-checking may benefit from specialised language representations.
  • Explainable Fact-Checking: Prior explainable fact-checking work mostly produces extractive explanations, whereas this paper focuses on abstractive explanations formulated as summarization.
  • Explainable Fact-Checking: The proposed framework combines extractive and abstractive explanation generation rather than using a purely extractive model.
  • Explainable Fact-Checking: Abstractive explanations are hypothesized to improve accessibility for readers with limited subject knowledge of public-health claims.
  • Evaluation: Explanation-quality evaluation has included human judgments and established metrics, but only a few explainable fact-checking systems conduct thorough evaluation.
  • Evaluation: The paper operationalizes coherence through three concrete properties evaluated with human and computational methods.

3 The PUBHEALTH dataset

PUBHEALTH is a public-health fact-checking dataset of 11,832 claims with journalist-provided explanations supporting veracity labels. Its construction combines multiple sources with health-focused filtering, label standardization, and quality controls, while analyses characterize label balance and readability.

  • Data collection: 11,832 claims spanning biomedical subjects, healthcare policy, and other public-health stories form the PUBHEALTH dataset.Claims were collected from fact-checking, news, and news-review websites and paired with journalist explanations.
  • Data collection: Journalist-provided gold-standard texts explicitly support the fact-checking judgments, making PUBHEALTH distinctive among fact-checking datasets.The dataset is described as the first to include such gold-standard explanatory texts specifically for the judgment.
  • Data collection: 39,301 initially scraped claims came from five fact-checking websites, Associated Press and Reuters health content, and Health News Review.The collection included full article texts, explanation or summary texts, cited URLs, and accessible source texts.
  • Data processing and analysis: Data processing standardized labels, removed nonbiomedical claims, and excluded entries with incomplete or brief explanations.The final label set was true, false, mixture, and unproven; claims with unverifiable alternative labels were discounted.
  • Data processing and analysis: PUBHEALTH contains mostly false claims, with unproven claims least common, and its claims average a Flesch-Kincaid reading-ease score of 59.1.The 59.1 score corresponds to approximately a 10th–12th grade reading level and is described as the most challenging among compared large datasets.
  • Data processing and analysis: Claims were retained when their article text contained more than three unique terms from a 7,000-term public-health lexicon augmented with 65 COVID-19-related terms.The inclusion rule is formalized using article text Ac, claim text Tc, lexicon L, and the sets CA and CT.
  • Data processing and analysis: The dataset does not control the intended purpose of explanation texts, which varies widely across the crawled websites.This variation is identified as an aspect of explanation quality that the authors chose not to control.

4 Methods

The paper develops separate models for four-way veracity prediction and explanation generation from public-health claims and article evidence. Prediction ranks evidence before label classification, while explanation generation uses joint extractive-abstractive summarization fine-tuned with general or in-domain data.

  • Methods: The methods train a classifier for four-way veracity prediction and a summarization model for generating fact-checking explanations.The classifier predicts true, false, unproven, or mixture; the summarizer generates explanations from article text about claims.
  • 4.1 Veracity Prediction: Veracity prediction selects evidence sentences and then predicts the claim label from the claim and selected evidence.This two-part architecture is shown in Figure 1.
  • 4.1 Veracity Prediction: SBERT ranks article sentences by cosine similarity to the claim representation, and the model selects the top k = 5 sentences.The evidence pool includes fact-checking and news articles.
  • 4.1 Veracity Prediction: Label prediction fine-tunes BERT, SciBERT, and two BioBERT versions on PUBHEALTH for the downstream fact-checking task.The BioBERT versions differ in training steps and source texts.
  • 4.2 Explanation Generation as Abstractive Summarization: Explanation generation uses joint extractive-abstractive summarization to express judgments in simpler terms for claims with complex public-health terminology.The approach is motivated by the needs of users with limited domain expertise.
  • 4.2 Explanation Generation as Abstractive Summarization: EXPLAINERFC uses non-public-health training data, whereas EXPLAINERFC-EXPERT is fine-tuned on PUBHEALTH, with both evaluated on PUBHEALTH test data.Both models use all article sentences and the claim sentence, with evidence ranked by SBERT.

5 Results

The experiments evaluate veracity prediction and explanation generation, using automatic, human, and computational measures. Domain-specific models perform best for prediction, while EXPLAINERFC-EXPERT improves ROUGE scores and NLI reliably approximates some coherence properties.

  • 5.1 Prediction: The prediction evaluation compares two baselines and four BERT-based models using precision, recall, macro F1, and accuracy.One baseline randomly selects sentences for BERT, whereas the other fine-tunes BERT on the entire article text.
  • 5.1 Prediction: SCIBERT achieves the highest macro F1, precision, and accuracy among the four BERT-derived models on the test set.BIOBERT v1.1 ranks second on these scores, while standard BERT has the lowest recall and F1.
  • 5.2.1 Automated Evaluation: EXPLAINERFC-EXPERT achieves higher ROUGE-1, ROUGE-2, and ROUGE-L F1 scores than EXPLAINERFC.The comparison also includes LEAD-3 and an extractive ORACLE upper bound.
  • 5.2.1 Automated Evaluation: ROUGE does not fully capture explanation usefulness because the generated explanations come from heterogeneous sources and are not directly comparable.The study therefore supplements automatic evaluation with formal coherence properties and human assessment.
  • 5.3 Evaluation of Explanation Quality: The study defines strong global, weak global, and local coherence, measuring entailment or contradiction relative to the claim and among explanation sentences.Strong global coherence requires every explanatory sentence to entail the claim; weak global coherence prohibits sentences entailing its negation; local coherence prohibits pairwise sentence contradiction.
  • 5.3.1 Human & Computational Evaluations: NLI reliably approximates weak global and local coherence, but entailment is a poor approximation for strong global coherence.The authors report that a larger human evaluation is needed to verify these findings.

6 Conclusion and Future work

The paper concludes that PUBHEALTH supports explainable fact-checking for public-health claims requiring specialist expertise. It reports benefits from in-domain training and proposes extending the work across domains, explanation properties, and task congruity.

  • 6 Conclusion and Future work: PUBHEALTH is presented as the first fact-checking dataset to include explanations as annotations and as a sizeable resource for public-health fact-checking.The dataset supports claims requiring specific expertise to produce veracity predictions and explanations.
  • 6 Conclusion and Future work: Training veracity prediction and explanation generation models on in-domain data improves prediction accuracy and generated-explanation quality relative to generic language models without explanation.This is the paper’s central reported conclusion about domain-specific training.
  • 6 Conclusion and Future work: Future work includes specialist fact-checking in subjects beyond public health and explanation evaluation using properties such as actionability and impartiality.The authors also plan to study congruity between veracity prediction and explanation generation.

A.1 Dataset

The dataset analysis describes PUBHEALTH’s health-focused vocabulary, source formats, and distributions of claims and explanations. It also documents the lexicon construction and variation in scraped explanation formats.

  • Dataset analysis: Explanations were scraped from fact-checking, news, and news-review websites, whose formats differed slightly.Table 7 records the origins and formats of the explanations included in the dataset.
  • Dataset analysis: PUBHEALTH entries contain public-health vocabulary spanning biomedical subjects, healthcare policy, and related public-health stories.The lexicon was built from health-related terms scraped across six websites and supplemented with additional words.
  • Dataset analysis: Figure 4 shows the most common health terms in PUBHEALTH entry texts, including terms such as vaccine, virus, pandemic, and mental health.The figure focuses on health-lexicon vocabulary occurring more than 300 times in article texts.
  • Dataset analysis: Figure 5 shows token-count distributions for claims and explanations in the PUBHEALTH dataset.The analysis treats claim and explanation length as separate distributions.

A.2 Reproducibility

The appendix documents the validation search and experimental computing setup used for the paper’s models.

  • Hyperparameter validation searched batch sizes of 8, 16, and 32; learning rates of 1e-5, 5e-6, and 1e-6; and 2, 3, or 4 epochs.The veracity prediction model was optimized with cross-entropy loss.
  • The selected configuration used batch size 16, learning rate 1e-6, and 4 training epochs.
  • Experiments ran on a dual Intel Core i9-9900X CPU with an Nvidia GeForce RTX 2080 Ti GPU.Software-package information was provided through the paper’s GitHub repository.

A.3 Human Evaluation Questionnaire

The human evaluation questionnaire asks participants to assess explanation consistency and infer claim veracity from the complete explanation, using a public-health example.

  • Participants were shown claim-and-explanation pairs typical of the human evaluation questionnaire.The examples were related to the claim and explanation presented in the questionnaire.
  • One question asked whether any sentences or phrases in an explanation disagreed with each other, with Yes and No response options.
  • Another question asked participants to assign a veracity label based on the entire explanation, choosing Mixture, false, true, or unproven.
  • The example claim concerned new mosquito-borne illness findings, while the explanation described a confirmed Rhode Island eastern equine encephalitis case.

A.4 Coherence properties

The appendix illustrates the paper’s three explanation-coherence properties schematically: strong global coherence, weak global coherence, and local coherence.

  • Global coherence: Figure 6 schematically represents strong and weak global coherence, including examples where strong global coherence is not satisfied or is satisfied.
  • Global coherence: Figure 6 also includes an example in which weak global coherence is satisfied.
  • Local coherence: Figure 7 schematically represents local coherence, with an example where local coherence is satisfied.
Loading 2010.09926v1…