PaperlayerView exact revision on arXiv

Source-linked AI summary

Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining

Yongan Yu, Shantam Raj, Jingwei Ni, Ario Saeid Vaghefi, Dominik Stammbach, Markus Leippold

arXiv:2609.07798v1cs.CLcs.AI

TL;DR

How heterogeneous climate corpora should be combined during continued pretraining remains underexplored. The paper adapts ModernBERT-base on academic, climate-filtered web, and synthetic text, comparing joint training with parameter-space merging. Simple weight averaging achieves the best overall result, while academic climate data provides the strongest adaptation signal among evaluated sources.

  • Problem

    How to combine heterogeneous climate data sources during continued pretraining remains underexplored, despite climate NLP’s need to handle scientific, policy, and web text.

    Source: §1 · pp. 1–2

  • Method

    The study continues pretraining ModernBERT-base on academic climate, climate-filtered web, and synthetic text, comparing joint corpus training with parameter-space merging of specialized checkpoints.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    76.3 average F1: simple weight averaging of independently specialized checkpoints achieves the best overall performance across the evaluated climate NLP benchmarks.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Academic climate data provides the strongest adaptation signal, while broader mixtures often fail to improve and can degrade performance in this climate NLP evaluation.

    Source: §6 · p. 8

    Current climate NLP benchmarks offer limited evaluation of long-document understanding, cross-section reasoning, and information integration beyond ClimRetrieve.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Natural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, policy disclosures, and synthetic reports. However, how to effectively combine diverse domain corpora during continued pretraining (CPT) remains underexplored. We introduce Climate-ModernBERT, a family of climate-adapted encoder models obtained through continued pretraining of ModernBERT-Base on three climate corpora: academic climate text, climate-filtered web data, and synthetic climate documents. We systematically compare joint continued pretraining on corpus mixtures with parameter-space merging of independently specialized checkpoints. Across nine climate NLP benchmarks, our best model achieves 76.3 average F_1, improving significantly over a vanilla ModernBERT baseline by 2.8 points. Within the climate NLP setting, the results show that academic climate corpora provide the strongest adaptation signal among the evaluated sources, while parameter-space merging improves over joint multi-source training and better preserves complementary information from heterogeneous climate corpora. We release all Climate-ModernBERT variants and training checkpoints to support future research in climate NLP and domain-adaptive pretraining.

1 Introduction

Climate NLP must handle heterogeneous scientific, policy, and web text, yet how to combine such sources during continued pretraining remains underexplored. CLIMATE-MODERNBERT studies three climate corpora and two adaptation strategies, finding academic corpora strongest and parameter-space merging better than joint multi-source training.

1 Introduction

  • Climate NLP models must handle heterogeneous, noisy sources and diverse downstream tasks, while the best way to combine corpora during continued pretraining remains poorly understood.The sources span informal web discourse, scientific literature, and policy documents, and climate benchmarks include short and long documents with specialized terminology.Source: §1 · pp. 1–2
  • ModernBERT’s encoder design is presented as suitable for climate tasks requiring long-document processing and nuanced understanding across heterogeneous sources.Encoder-only models support non-generative tasks such as classification, retrieval, and information extraction.Source: §1 · p. 1
  • The study introduces CLIMATE-MODERNBERT, climate-adapted ModernBERT encoders trained on academic climate text, climate-filtered web data, and synthetic climate documents.It compares joint continued pretraining with parameter-space merging of independently specialized checkpoints.Source: §1 · p. 2, Figure 1 · p. 2
  • Academic climate corpora provide the strongest adaptation signal, and parameter-space merging outperforms joint multi-source training.The study evaluates 21 model variants across nine climate NLP benchmarks and reports over 810 fine-tuning runs.Source: §1 · pp. 2–3

2 Related Work

Prior work establishes the relevance of encoder models and continued pretraining for specialized NLP, but leaves open how heterogeneous sources should be integrated. Parameter-space merging offers an alternative to joint training that this study examines in climate NLP.

2.1 Encoder-only Models in Climate NLP

  • Climate encoder models support classification, retrieval, and disclosure analysis, while fine-tuned encoders can match or exceed larger decoder models on supervised tasks when labeled data are sufficient.ClimateBERT is cited as a widely adopted climate-domain encoder with strong classification and retrieval performance.Source: §2.1 · p. 3

2.2 Continued Pretraining for Domain Adaptation

  • Continued pretraining adapts pretrained language models to specialized domains, but how to combine multiple data sources during this process remains an open question.Climate NLP makes the question salient because its data span scientific, policy, web, and synthetic text with different distributions.Source: §2.2 · p. 3

2.3 Parameter-Space Model Merging

  • Parameter-space merging combines separately adapted checkpoints after training, but its use for heterogeneous corpora during continued pretraining remains underexplored.This study compares merging with joint continued pretraining under a matched data budget.Source: §2.3 · p. 3

3 CLIMATE-MODERNBERTS

The study constructs a 6.42B-token climate corpus from academic, filtered web, and synthetic sources, then adapts ModernBERT-base through context extension followed by learning-rate decay specialization. Context-extension-only checkpoints serve as an ablation.

3.1 Data Curation

  • The 6.42B-token climate corpus combines academic climate text, climate-filtered web data, and synthetic climate text.The academic corpus comprises 1.28B tokens, and the web filtering pipeline yields 2.59M candidate documents.Source: §3.1 · pp. 3–4
  • Synthetic climate text is generated from climate-domain seed excerpts in three communication perspectives: public awareness, industry, and environmental journalism.Seeds come from five academic journal subsets and the ClimateNews archive sampled across 22 years.Source: §3.1 · p. 4

3.2 Training Schedule

  • ModernBERT-base adaptation follows context extension (CX) with learning-rate decay (LRD) specialization; CX-only checkpoints are retained to assess LRD’s contribution.The CX+LRD checkpoints are the final climate-adapted models used in the main experiments.Source: §3.2 · p. 4
  • Training starts from the public ModernBERT-base context-extension checkpoint, which preserves a stable optimization landscape for continued training.The authors note that resuming from a post-LRD checkpoint would require reintroducing warm-up after decay reaches a near-zero learning rate.Source: §3.2 · p. 4
  • CX training uses 8,192-token sequences, followed by three additional LRD-specialization epochs with the other hyperparameters unchanged.CX runs for three epochs with a constant 3e−4 learning rate; LRD uses an initial 3e−4 learning rate and final factor 1e−3.Source: §3.2 · p. 4

4 Evaluation Setting

The evaluation compares climate-adapted checkpoints with ModernBERT-base and ClimateBERT under a shared fine-tuning protocol. It covers nine climate NLP benchmarks and contrasts joint corpus subsets with parameter-merging variants.

4 Evaluation Setting

  • Climate-adapted models are compared with the ModernBERT-base initialization and ClimateBERT, with all models fine-tuned under the same downstream protocol.The ModernBERT-base comparison isolates climate continued-pretraining effects; ClimateBERT supplies context against an established climate encoder.Source: §4 · p. 5

4.1 Evaluated Models

  • The evaluated model design varies corpus composition across all seven non-empty subsets of academic, synthetic, and filtered-web data, with CX-only checkpoints retained as an ablation.The main joint-training experiments use LRD-specialized checkpoints.Source: §4.1 · p. 5
  • The study also evaluates seven merged checkpoints formed by combining parameter deltas from the three independently trained single-source models.The four strategies are weight averaging, Task Arithmetic, TIES-Merging, and DARE-TIES; the merged models use the full corpus union, enabling comparison with joint training at matched data coverage.Source: §4.1 · p. 5

4.2 Downstream Tasks

  • Evaluation spans nine climate NLP benchmarks: six single-label tasks, two multi-label tasks, and one retrieval benchmark formulated as binary relevance classification.The datasets cover disclosure, climate commitments, sentiment, retrieval, environmental impact, and other settings across reports, policy pledges, and historical news.Source: §4.2 · p. 5
  • Test performance is reported with F1-based metrics after jointly fine-tuning encoder parameters and a task-specific classification head.Binary tasks and ClimRetrieve use positive-class F1, while multi-class and multi-label tasks use macro-F1; the shared protocol uses early stopping based on validation F1.Source: §4.2 · p. 6, Table 2 · p. 6

5 Experimental Results

Across climate NLP benchmarks, academic climate data provides the strongest adaptation signal, synthetic-data effects vary by task, and parameter-space merging outperforms joint multi-source training.

5.1 Joint Continued Pretraining

  • 75.3 average F1 is the highest among jointly trained variants for CX-only θ{A}; adding S and F lowers it to 74.8 and 74.0.The broader mixture lowers Commitments & Actions from 71.8 to 67.1.Source: §5.1 · p. 6
  • Adding synthetic text yields TCFD +2.4 (p_t < .001) and Specificity +2.2 (p_t = .001), but reduces Commitments & Actions −3.9 (p_t < .001).Retrieval performance remains statistically unchanged; the comparison uses ten shared fine-tuning seeds.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Parameter-space Merging

  • Removing A reduces average F1 by 4.0 points versus the full Soup, while removing S or F reduces it by 1.5 points.The drop-one results provide independent evidence that academic data contributes the strongest broadly transferable signal.Source: Figure 2 · p. 7, §5.2 · p. 7
  • Source task vectors have pairwise cosine similarities of 0.06–0.18, indicating largely distinct updates.Their complementarity provides a plausible explanation for why merging can recover information from multiple sources more effectively than joint optimization.Source: Figure 3 · p. 8, §5.2 · p. 8
  • 76.3 average F1 from uniform weight averaging exceeds the best individual source by 1.8 points and same-data joint training by 1.7 points.Five of seven merged models outperform the strongest single-source LRD checkpoint.Source: §5.2 · p. 8, Table 4 · p. 8

6 Conclusion

The conclusion presents Climate-ModernBERT, reports findings on corpus composition and parameter-space merging, and releases models and training resources for future research.

6 Conclusion

  • Climate-ModernBERT comprises climate-adapted ModernBERT-Base encoders continued-pretrained with LRD specialization on academic, climate-filtered FineWeb-Edu, and synthetic climate text.The conclusion summarizes results across 21 adapted variants and nine climate NLP benchmarks.Source: §6 · p. 8
  • Academic climate data provides the strongest adaptation signal, synthetic text has task-dependent effects, and simple weight averaging achieves the best overall performance at 76.3 average F1.The weight-averaged model outperforms both single-source models and joint training on the full corpus union.Source: §6 · p. 8
  • The authors release all Climate-ModernBERT variants, training checkpoints, and data-processing pipelines to support future climate NLP and domain-adaptive encoder research.Source: §6 · p. 8

Limitations

The climate NLP benchmarks provide limited coverage of long-document understanding and cross-section reasoning, and the study’s scope is limited to English climate text, one encoder family, and continued pretraining.

Limitations

  • Current climate benchmarks center on sentence- or passage-level classification, leaving long-document understanding and cross-section reasoning largely unevaluated beyond ClimRetrieve.The study identifies standardized long-context and document-centric benchmarks as a future research direction.Source: Limitations · p. 9
  • The released collection is trained only on English climate text and adapted from ModernBERT-Base, so its results do not establish effects across languages or encoder families.The authors frame observed corpus-composition and merging effects as findings within climate NLP, not universal principles of domain adaptation.Source: Limitations · p. 9
  • The analysis covers continued pretraining but not instruction-tuned encoder variants, whose representations and adaptation dynamics may differ from base models.The authors identify interactions between instruction tuning, corpus composition, and parameter-space merging as future work.Source: Limitations · p. 9

Ethical Considerations

The paper describes constraints on releasing climate text and outlines its data preparation, corpus sources, synthetic generation, and training setup.

Ethical Considerations

  • Only trained checkpoints and permitted data-processing pipelines are released, rather than raw academic climate text, to respect licensing and intellectual-property restrictions.The academic corpus includes copyrighted sources.Source: Ethical Considerations · p. 9

A Data Preprocessing Pipeline

  • The shared pipeline normalizes text and removes URLs; climate-filtered web data undergoes MinHash–LSH deduplication configured to retain pairs sharing approximately 80% of tokens.The pipeline uses NeMo Curator; the web-data configuration uses 24-character n-grams, 20 hash bands, and 13 signatures per band.Source: A Data Preprocessing Pipeline · p. 12

B Academic Climate Data: Sources and Statistics

  • The academic climate corpus combines journal articles, climate news, climate-focused arXiv preprints, and handbooks; it is not redistributed, and only permitted artifacts and aggregate statistics are released.The sources are accessed under institutional licenses, permissive arXiv terms, or non-commercial research-use terms.Source: B Academic Climate Data: Sources and… · p. 12

C Web Data: Keyword List for FineWeb-Edu Filtering

  • The FineWeb-Edu web pipeline retains documents containing any listed case-insensitive climate term, with terms drawn from GEMET and Sautner et al. (2023).The pipeline also applies deduplication using NeMo Curator.Source: C Web Data: Keyword List for… · p. 12

D Synthetic Climate Data: Seed Pool, Prompts, and Generation

E Training Setup: Full Configuration

  • Training variants differ only in data subset and learning-rate schedule, and final Composer checkpoints are converted to the Hugging Face Transformers FlexBERT implementation for release.The accompanying configuration table covers architecture and compute details.Source: E Training Setup: Full Configuration · p. 14, Table 8 · p. 14

F Context Extension-only Model Merging Analysis

The context-extension-only analysis tests whether checkpoint merging remains effective without LRD specialization and finds competitive benchmark performance.

F Context Extension-only Model Merging Analysis

  • The comparison evaluates three strategies on context-extension-only checkpoints: uniform weight averaging, Task Arithmetic with λ = 0.5, and norm-balanced merging.The main experiments instead use final CX+LRD checkpoints.Source: F Context Extension-only Model Merging… · p. 14
  • CX-only merging achieves competitive performance across climate NLP benchmarks, indicating that merging effectiveness is not exclusively dependent on LRD specialization.The analysis isolates context-extension-only checkpoints and evaluates weight averaging, Task Arithmetic with λ = 0.5, and norm-balanced merging.Source: F Context Extension-only Model Merging… · p. 14
  • The paper retains LRD-specialized checkpoints as its primary setting because they represent the final adapted models in the established ModernBERT training pipeline.Source: F Context Extension-only Model Merging… · p. 14

G Comparison Against the Post-LRD Public ModernBERT Checkpoint

The authors compare their pre-LRD initialization with the publicly released post-LRD ModernBERT checkpoint under the same nine-benchmark fine-tuning protocol.

G Comparison Against the Post-LRD Public ModernBERT Checkpoint

  • On the nine climate benchmarks, the final publicly released post-LRD ModernBERT-base checkpoint averages 72.7 F1, 0.8 points below θBASE (73.5), under the identical fine-tuning protocol.Source: G Comparison Against the Post-LRD… · pp. 14–15
  • The post-LRD checkpoint has its largest regressions on Commitments & Actions (−5.3), Retrieval (−3.6), and TCFD (−2.2), but gains on Sentiment (+3.1) and WFB (+0.8).Detection (−0.1) and WXImpactBench (−0.2) are essentially unchanged; NetZero (+0.2) and Specificity (+0.1) show modest gains.Source: G Comparison Against the Post-LRD… · p. 15

H Downstream Tasks: Detailed Descriptions

This appendix specifies the nine downstream benchmarks, describing their label spaces, target phenomena, and selected dataset construction details.

Single-label classification

  • Single-label tasks cover climate relevance and specificity, commitments versus actions, sentiment, net-zero or reduction targets, and TCFD categories (Bingler et al., 2024; Schimanski et al., 2023a).Climate Detection contrasts climate-related text with superficially similar unrelated text; specificity distinguishes concrete commitments from vague statements.Source: H Downstream Tasks: Detailed… · p. 15

Multi-label classification

  • Multi-label benchmarks test co-occurring nature dimensions in WFB Nature and six societal weather-impact categories in WXImpactBench (Schimanski et al., 2023b; Yu et al., 2025).WXImpactBench uses keyword extraction, event categorization, topic-aware article selection, and expert annotation.Source: H Downstream Tasks: Detailed… · p. 16

Retrieval

  • ClimRetrieve turns 16 questions over 30 reports (>8.5K question-passage pairs) into binary relevance classification by collapsing ordinal labels at threshold ≥1 (Schimanski et al., 2024).The report-level setup uses an 80/10/10 stratified split.Source: H Downstream Tasks: Detailed… · p. 16

Related papers

Loading 2609.07798v1…