Source-linked AI summary
CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims
Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, Markus Leippold
TL;DR
Climate misinformation creates a need for reliable evidence verification, while existing FEVER claims do not represent real-world climate claims. The paper builds CLIMATE-FEVER by collecting and annotating Internet claims with retrieved Wikipedia evidence, then evaluates a FEVER-trained entailment predictor. The resulting dataset contains diverse, disputed claims and yields substantially lower baseline performance than FEVER, highlighting the complexity of real-world climate claim verification.
Problem
Existing FEVER claims are artificially constructed, leaving limited representation of complex real-world climate claims despite the public impact of climate misinformation.
Method
The paper adapts the FEVER methodology by collecting Internet climate claims, retrieving five Wikipedia evidence candidates per claim, and obtaining human labels for claim-evidence pairs.
Results
The dataset contains 1,535 claims and 7,675 annotated pairs; a FEVER-trained predictor achieves 38.78% label-accuracy on CLIMATE-FEVER versus 77.69% on the FEVER dev-set.
Takeaways & Limitations
CLIMATE-FEVER exposes diverse and disputed real-world climate claims as a challenge for evidence retrieval, language understanding, and future climate-AI research.
Takeaways & Limitations
Majority-vote aggregation cannot always produce a final verdict when retrieved evidence covers only some facets of a claim.
Abstract
from arXiv · showhide
We introduce CLIMATE-FEVER, a new publicly available dataset for verification of climate change-related claims. By providing a dataset for the research community, we aim to facilitate and encourage work on improving algorithms for retrieving evidential support for climate-specific claims, addressing the underlying language understanding challenges, and ultimately help alleviate the impact of misinformation on climate change. We adapt the methodology of FEVER [1], the largest dataset of artificially designed claims, to real-life claims collected from the Internet. While during this process, we could rely on the expertise of renowned climate scientists, it turned out to be no easy task. We discuss the surprising, subtle complexity of modeling real-world climate-related claims within the \textsc{fever} framework, which we believe provides a valuable challenge for general natural language understanding. We hope that our work will mark the beginning of a new exciting long-term joint effort by the climate science and AI community.
1 Introduction
CLIMATE-FEVER adapts FEVER-style claim verification to real-world climate claims, motivated by the public and policy impact of climate misinformation. It provides a dataset intended to support automated evidence retrieval and human assessment of claim veracity.
- Climate misinformation has confused the public, contributed to political inaction, and stalled support for climate-change mitigation measures.
- FEVER combines human-generated claims, Wikipedia evidence, and a shared task for evaluating claim validators.
- Because FEVER claims are artificially constructed, researchers began collecting real-world claims with evidence curated by human fact-checkers.
- The paper collects 1,535 Internet claims and produces 7,675 annotated claim-evidence pairs labeled SUPPORTS, REFUTES, or NOT_ENOUGH_INFO.The dataset uses the five most relevant Wikipedia evidence candidates for each claim.
2 Methodology
The methodology constructs CLIMATE-FEVER through claim collection, verifiability screening, evidence retrieval, and human labeling, then defines a two-stage validation baseline. Its retrieval system combines document search, sentence embeddings, and re-ranking to handle the scale and complexity of full Wikipedia articles.
- System architecture: The pipeline has two components: the Evidence Candidate Retrieval System (ECRS) retrieves evidence, while Entailment Prediction assigns SUPPORTS, REFUTES, or NOT_ENOUGH_INFO.
- Claim collection: Claims were gathered from scientifically informed and skeptical or denier sources using seed-keyword searches, yielding a balanced pool of more than 3,000 climate claims.
- Claim definition: A potentially verifiable claim must be a single well-formed sentence and subjectively investigable through evidence from a knowledge document collection.The claim must be consistent, unambiguous, and complete, while evidence must reduce uncertainty about its truthfulness or falsehood.
- Claim labeling: Climate scientists labeled claims for verifiability, and more than 1,535 claims with sufficient annotator consensus were retained.Each claim received up to five votes in Task 1.
- Evidence retrieval: ECRS retrieves top-10 articles with entity-linked BM25 search, top-100 sentences with FEVER-trained embeddings, then re-ranks claim-sentence pairs.The sentence-level preselection uses ALBERT embeddings and FAISS before pairwise re-ranking, addressing the intractable sentence volume of full Wikipedia articles.
- Evidence labeling: For each claim, annotators label five retrieved evidence sentences, producing five votes per claim-evidence pair and both micro-verdicts and aggregate claim labels.The resulting annotation set contains 7,675 claim-evidence pairs.
3 Discussion and future work
The dataset covers diverse and contested real-world climate claims, exposing difficulties for both aggregation and entailment prediction. The authors present these complexities as a basis for extending CLIMATE-FEVER and encouraging collaboration between climate science and AI.
- Dataset scope: More than 20 topics occur among the 1,535 claims, including Arctic climate change, sea-level rise, and climate change or global warming generally.
- Label distribution: The aggregate labels are 655 SUPPORTS (42.67%), 253 REFUTES (16.5%), 153 DISPUTED (9.97%), and 474 NOT_ENOUGH_INFO (30.88%).Each claim has five evidence candidates, and each evidence sentence received at least two votes on average.
- Disputed claims: Including claims with both supporting and refuting evidence captures disputed cases that the original FEVER dataset excludes.The authors regard these cases as especially useful because they appear common in real-world claims.
- Aggregation limitation: Majority-vote aggregation is limited because retrieved evidence may cover only some claim facets, making a final verdict epistemologically impossible in some cases.The dataset therefore provides both micro-verdict labels and claim labels for each claim-evidence pair.
- Annotation difficulty: Average Task-2 inter-annotator agreement is Krippendorff’s alpha = 0.334, indicating that humans also find support and refutation judgments non-trivial.Training and subject expertise were associated with higher agreement in some annotation slices.
- Baseline evaluation: The FEVER-trained predictor reaches label-accuracy = 38.78%, precision = 56.49%, and F1 = 32.85% on CLIMATE-FEVER, versus 77.69% label-accuracy on the original FEVER dev-set.The model particularly struggles with SUPPORTED claims and performs slightly better on REFUTED claims.
- Future work: Real-world climate claims introduce complexity and subtlety that motivate new strategies for addressing climate-related disinformation.
A Claim Labelling (Task 1)
Task 1 distinguishes verifiable claims from claims that are inconsistent, incomplete, ambiguous, or improperly formed. Verifiable claims are well-formed and likely to have retrievable supporting or refuting evidence.
- Verifiable claims are well-formed and have a high probability of retrievable supporting or refuting evidence.
- Claims are not verifiable when they contain multiple sentences rather than a single sentence.
- A claim containing inconsistent information, such as 0.0 cm of sea-level rise over 140 years, is not verifiable.
- A statement is incomplete when understanding it requires an unstated reference, such as interpreting “commercial operators” as air travel.
- A claim is ambiguous when it omits a subject, as in the statement about nature-based solutions receiving 2% of climate funding.
B Topic distribution
The collected climate claims cover a broad range of topics identified through clustering after textual preprocessing. These topics include specific climate domains and broader climate-change themes.
- More than 20 topics were identified among the 1,535 collected climate claims.
C Baseline evaluation
The baseline predicts SUPPORTS, REFUTES, or NOT_ENOUGH_INFO from claims and retrieved evidence. Dataset labels also represent disagreement when supporting and refuting micro-verdicts coexist.
- 77.58% label-accuracy was achieved by a FEVER-trained claim validation baseline on the FEVER dev-set.
- The entailment task uses a three-way classifier predicting SUPPORTS, REFUTES, or NOT_ENOUGH_INFO from a claim and concatenated evidence sentences.
- Average entropy is used as a surrogate for inter-annotator agreement, with high entropy indicating disagreement and low entropy indicating agreement.
- A claim is labelled DISPUTED when its micro-verdicts include both supporting and refuting evidence rather than only one verdict type or NOT_ENOUGH_INFO.
D.1 Supported claims
Supported examples show that evidence can validate claims about human contributions to warming and other climate statements, while some claims remain unsupported by retrieved evidence. The examples also illustrate disagreement arising from incoherent wording.
- Supported claims: The claim that more than 100 per cent of warming is due to human actions received a SUPPORTS verdict despite high inter-annotator disagreement.
- Supported claims: Evidence states that human activities are the dominant cause of warming since the 1950s and primarily explain warming over the past 50 years.
- Supported claims: The claim that half of the 1980–2002 warming trend was caused by Urban Heat Island received a SUPPORTS verdict from its evidence.
D.2 Refuted claims
The refuted-claims examples show that retrieved evidence can directly contradict climate claims, while annotation disagreement can arise from differing interpretations of the claim and evidence.
- The annotators marked the presented claims as refuted by retrieved evidence sentences.
- High inter-annotator disagreement arose when evidence concerned US public opinion rather than consensus among climate scientists.
- 97% consensus on human-caused global warming has been disproven was assigned the verdict REFUTES.
- Extreme weather isn’t caused by global warming was assigned the verdict REFUTES.
- The dataset authors argue that disputed examples should be added because FEVER lacks such cases.
Example D.5
Example D.5 illustrates a disputed climate claim whose evidence includes support, refutation, and insufficient information, producing a disputed verdict.
- The claim that carbon dioxide increases lagged temperature rises by 200 to 1000 years received the verdict DISPUTED.
- The evidence set also contained items labeled NOT_ENOUGH_INFO and REFUTES, creating contradictory support for the claim.
- Evidence supported the claim by reporting that CO2 increases were in phase with or lagged Antarctic temperature by less than 1000 years.
- Other evidence stated that carbon dioxide concentrations increased rapidly about 150 years ago after long-term stability.
Example D.6
Example D.6 shows that a claim about unchanged droughts and floods can receive both supporting and refuting evidence, resulting in a disputed verdict.
- The claim that droughts and floods have not changed since fossil-fuel use received the verdict DISPUTED.
- Supporting evidence reported little change in drought over the past 60 years.
- Refuting evidence described more frequent droughts and possible changes in floods, droughts, and tropical cyclones.
- Additional evidence linked environmental degradation and warming with intensification of tropical cyclones, floods, droughts, and forest fires.
D.4 Subtle cases of claims and evidences
The subtle-case examples demonstrate that claim verification depends on interpreting evidence scope and wording, not merely matching numeric details or related terms.
- Example D.7: The Greenland ice-sheet example was supported even though the claim said six metres while every evidence candidate mentioned about seven metres.
- Example D.7: The evidence states that complete Greenland ice-sheet melting would raise global sea level by more than 7 metres.
- Example D.8: The polar-ice example was refuted because evidence described continuing sea-ice decline and glacier retreat despite differing evidence scopes.
- Claim analysis: Table 2 lists 150 pruned terms from the 1,535 claims, sorted by document frequency.
- Claim analysis: Table 3 organizes the 1,535 claims into clusters, giving cluster sizes and their five most frequent terms.
- Baseline evaluation: Table 4 compares baseline classification reports on Climate-FEVER and the original FEVER development set using percentage metrics.
- Baseline evaluation: Table 5 compares normalized confusion matrices, with true labels as rows and predicted labels as columns.