Source-linked AI summary
ClimateBert: A Pretrained Language Model for Climate-Related Text
Nicolas Webersinke, Mathias Kraus, Julia Anna Bingler, Markus Leippold
TL;DR
General-language models struggle with specialized climate-related wording, while publicly available climate NLP models had not used climate domain-adaptive pretraining. The paper introduces CLIMATEBERT, continued-pretrained on over 2 million climate-related paragraphs, and reports lower masked-language-modeling loss and improved predictive performance across three climate NLP tasks.
Problem
Climate-related wording is specialized, and publicly available models had relied on general-language pretraining rather than climate domain-adaptive pretraining.
Method
CLIMATEBERT continues pretraining DistilROBERTA on climate-related text and evaluates multiple sample-selection strategies.
Results
Domain-adaptive pretraining decreases cross-entropy loss by 46–48% versus basis DistilROBERTA and improves predictive performance across three climate-related NLP tasks.
Takeaways & Limitations
CLIMATEBERT provides a publicly available climate-domain language model for downstream climate-related NLP research.
Takeaways & Limitations
Training CLIMATEBERT caused 115.15 kg CO2 emissions, and renewable-energy-powered servers were unavailable.
Abstract
from arXiv · showhide
Over the recent years, large pretrained language models (LM) have revolutionized the field of natural language processing (NLP). However, while pretraining on general language has been shown to work very well for common language, it has been observed that niche language poses problems. In particular, climate-related texts include specific language that common LMs can not represent accurately. We argue that this shortcoming of today's LMs limits the applicability of modern NLP to the broad field of text processing of climate-related texts. As a remedy, we propose CLIMATEBERT, a transformer-based language model that is further pretrained on over 2 million paragraphs of climate-related texts, crawled from various sources such as common news, research articles, and climate reporting of companies. We find that CLIMATEBERT leads to a 48% improvement on a masked language model objective which, in turn, leads to lowering error rates by 3.57% to 35.71% for various climate-related downstream tasks like text classification, sentiment analysis, and fact-checking.
1 Introduction
Climate-related NLP needs models that represent specialized wording more accurately than general-language models. The paper addresses this gap through climate-domain continued pretraining and evaluates the resulting models on downstream tasks.
- Motivation: Climate-related wording can vary substantially, limiting traditional dictionary and bag-of-words analyses.These methods do not adequately capture the domain’s specialized language.
- Approach: Domain-adaptive pretraining extends standard pretraining by adding pretraining on the downstream task’s language domain.The sequence is general-domain pretraining, domain-adaptive pretraining, then downstream-task training.
- Research gap: Before this work, publicly available models for climate NLP were generally pretrained on general language and fine-tuned on downstream tasks.The paper identifies the absence of a publicly available climate domain-adaptively pretrained model.
- Contribution: CLIMATEBERT is pretrained on climate-related news, corporate disclosures, and scientific articles.The model is intended for multiple climate-change NLP downstream tasks.
- Contribution: The paper evaluates CLIMATEBERT on three standard climate-related downstream tasks and releases its training code and model weights publicly.The stated tasks include text classification, sentiment analysis, and fact-checking.
2 Background
Climate NLP has progressed from keyword methods toward contextual transformer models, but climate-related information extraction remains challenging. The paper therefore examines whether domain-adaptive pretraining improves climate downstream-task performance.
- Domain-specific pretraining: Domain-adaptive pretraining continues pretraining a general language model on a downstream task’s language domain.This approach builds on general-domain pretraining before downstream-task training.
- Domain-specific pretraining: General-domain language models can perform worse on niche topics, motivating further pretraining in finance, legal, medical, and other domains.Prior work reports more accurate downstream models after domain-adaptive pretraining.
- Climate-related NLP: Traditional climate-text analysis often searches documents with predefined dictionaries or keywords without accounting for context.Words such as “environment,” “sustainable,” and “climate” can be ambiguous.
- Climate-related NLP: BERT-based models account for word context and have outperformed traditional approaches across various climate-related datasets.The cited research also characterizes climate information extraction as difficult because the topic is complex, fast-moving, ambiguous, and resource-scarce.
- Research question: The paper investigates whether domain-adaptive pretraining improves climate-related text classification, sentiment analysis, and fact-checking.This question responds to the lack of prior work examining domain-adaptive pretraining for these specific tasks.
3 CLIMATEBERT
CLIMATEBERT is built by continuing DistilROBERTA pretraining on a heterogeneous climate corpus and testing alternative sample-selection strategies. The design also adds frequent climate-specific tokens to the model vocabulary.
- Text Corpus: The CORP corpus contains 2,046,523 climate-related paragraphs from news articles, research abstracts, and corporate climate reports.The corpus excludes full research articles and Twitter data because of specificity and noise concerns.
- Text Corpus: CORP paragraphs are heterogeneous in length, with NEWS and REPORTS generally shorter and ABSTRACTS generally longer.The paper reports these descriptive differences across the three source datasets.
- Corpus comparison: The climate corpus has 57.05% vocabulary overlap with ROBERTA’s pretraining corpus, indicating dissimilarity between the domains.The authors use this overlap to motivate adding domain-specific vocabulary.
- Sample selection: CLIMATEBERT compares FULL-SELECT, SIM-SELECT, DIV-SELECT, and combined diversity-plus-similarity sample-selection strategies.Similarity and diversity strategies retain 70% of CORP, while FULL-SELECT uses all paragraphs.
- Vocabulary Augmentation: The vocabulary is expanded with the 235 most common climate-specific tokens, increasing DistilROBERTA’s vocabulary from 50,265 to 50,500.This lets the model explicitly learn representations for terminology frequent in climate text but uncommon in general text.
- Model Selection: All models start from DistilROBERTA and use masked language modeling with cross-entropy loss on randomly masked tokens.Downstream prediction uses the final-layer [CLS] representation with a task-specific feedforward layer.
4 Performance Analysis of Language Model
The language-model evaluation measures cross-entropy loss for predicting randomly masked tokens on held-out CORP paragraphs. Domain-adaptive pretraining substantially lowers this loss relative to the DistilROBERTA baseline.
- Evaluation: Evaluation uses a random 80% training and 20% validation split of CORP.The reported validation loss is cross-entropy for predicting randomly masked tokens.
- Results: 46–48%: domain-adaptive pretraining decreases cross-entropy loss compared with basis DistilROBERTA.The authors describe this as cutting the loss almost in half.
- Results: CLIMATEBERTF achieves the lowest validation loss among the evaluated models.The other CLIMATEBERT variants show similar results.
5 Performance Analysis for Climate-Related Downstream Tasks
CLIMATEBERT is evaluated across text classification, sentiment analysis, and climate-related fact-checking, with repeated validation runs. Domain-adaptive pretraining improves performance over the DistilROBERTA baseline across these tasks and metrics.
- Evaluation setup: The evaluation covers text classification, sentiment analysis, and fact-checking, using 60 repeated training and validation runs for each experiment.Each run uses a random 90% training set and 10% validation set.
- Sentiment Analysis: 7.42% lower average-F1 error rate and 7.33% lower cross-entropy loss are achieved by CLIMATEBERTF in sentiment prediction versus the DistilROBERTA baseline.All CLIMATEBERT models outperform DistilROBERTA on average F1 score and average cross-entropy loss.
- Fact-Checking: CLIMATE-FEVER evaluates whether evidence sentences support or refute climate-related claims after claims and evidence are concatenated with a [SEP] token.Evidence labeled NOT ENOUGH INFO is filtered out for comparison with prior results.
- Fact-Checking: CLIMATEBERTD+S raises average F1 from 0.748 to 0.757 on CLIMATE-FEVER while achieving similar cross-entropy loss to the DistilROBERTA baseline.The reported 0.757 average F1 exceeds the previous state-of-the-art score of 0.7182.
6 Carbon Footprint
The authors acknowledge CLIMATEBERT’s carbon footprint and describe steps to reduce emissions from model choice, data collection, and training procedures.
- 115.15 kg CO2 emissions resulted from training CLIMATEBERT.The estimate uses 0.7 kW, 350 combined training hours, and a 470 gCO2e/kWh emission factor.
- DistilROBERTA was selected because its smaller parameter count requires less training energy.The authors also avoided crawling huge amounts of data without considering quality.
- Exhaustive hyperparameter tuning was avoided because tens or hundreds of training runs would considerably increase CO2 emissions.The authors instead built on previous findings and used a limited set of combinations.
- Renewable-energy-powered servers were unavailable, so the authors donated Euro 100 to atmosfair while avoiding calling it CO2 compensation.They also rejected afforestation as the basis for the solution.
7 Conclusion
The conclusion presents CLIMATEBERT as a climate-domain language model trained on over 2 million paragraphs. Its domain-adaptive pretraining improves language-modeling performance and predictive performance across three downstream tasks.
- CLIMATEBERT is presented as the first language model pretrained on over 2 million climate-related paragraphs.
- The study evaluates corpus selection strategies to identify samples most helpful for later tasks.
- Domain-adaptive pretraining considerably lowers masked language modeling loss on the climate corpus.
- Predictive improvements appear across text classification, corporate risk and opportunity analysis, and climate-claim fact-checking.
A Climate Performance Model Card
The section presents a climate performance model card for CLIMATEBERT, following an established model-card framework.
- Table 9 presents CLIMATEBERT’s climate performance model card.
- The model card follows Hershcovich et al. (2022).
- The model card is identified as Table 9 in Appendix A.
B Annotation Guidelines
The annotation guidelines define climate relevance and classify climate-related paragraphs by sentiment as risk, opportunity, or neutral, using binary decisions and adjudication for ties.
- Annotation procedure: Annotators were instructed to use common sense when context was incomplete and record each annotation as a 0-1 decision.
- Adjudication: Close verdicts or ties were discussed by the authors before reaching agreement.
- Climate relevance: Annotators labeled paragraphs as climate-relevant or not climate-relevant using a Yes/No decision.Climate relevance includes topics such as climate policy, emissions, energy sources, transition risks, and physical risks.
- Sentiment labels: Risk denotes negative effects on an entity, opportunity denotes beneficial effects arising from climate change, and neutral denotes factual statements without positive or negative perspective.The sentiment categories are applied to climate change-related paragraphs concerning entities, society, or the environment.
C Added Tokens
The section presents climate-related vocabulary, including terms associated with emissions, environmental conditions, impacts, and sustainability. It also identifies sentiment annotation categories as Opportunity, Neutral, and Risk.
- Climate-related vocabulary includes terms such as “CO2,” “emissions,” “temperature,” “environmental,” and “greenhouse.”
- The listed vocabulary also covers climate impacts, processes, regions, variability, and renewable resources.
- The sentiment annotation task uses the categories Opportunity, Neutral, and Risk.