BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP

1 Introduction

• Clinical information databases have become more comprehensive as NLP is applied to healthcare tasks including decision support, care-delivery analysis, and clinical-trial cohort selection.The cited examples are Cho et al. (2024) and Abdel-Jaber et al. (2022).Source: §1 · p. 1
• Encoder models use bidirectional attention for classification and span extraction, but have received less development than decoders, which are increasingly used for non-generative tasks.The passage attributes the shift at least partly to decoder investment in longer inputs, generalization, and zero-shot capabilities.Source: §1 · p. 1
• ModernBERT supports up to 8,192 tokens and a 50,368-token vocabulary, enabling whole clinical notes to be processed in one pass rather than fragmented.Its vocabulary is larger than BERT’s 30,000 tokens.Source: §1 · p. 2
• BioClinical ModernBERT uses two-step continued pretraining and offers base and large models, with released checkpoints and state-of-the-art performance across multiple benchmarks.The training data includes diverse clinical sources and realistic surrogate identifiers that support learning more natural entity-context representations.Source: §1 · p. 2

General domain encoders

• Encoder-only models remain useful in retrieval and RAG because of their inference efficiency, while ModernBERT adds longer context and updated architecture to older encoders.MosaicBERT, CrammingBERT, and GTE-en-MLM focus largely on retrieval or efficient training rather than broad downstream performance.Source: §2 · p. 2

Biomedical and clinical encoders

• BioBERT, Clinical BERT, and BioMed-RoBERTa adapt encoders using biomedical literature, MIMIC-III notes, or Semantic Scholar text, respectively.The passage reports that these models outperform general-domain encoders on tasks including concept extraction, temporal relation identification, and outcome prediction.Source: §2 · p. 2

Long context

• Clinical notes often exceed the 512-token limit of many encoders, motivating long-context models for information distributed across notes or documents.Clinical-Longformer and Clinical-BigBird are described as publicly available clinical models with extended sequence lengths [Li et al., 2022].Source: §2 · pp. 2–3

3.1.1 Biomedical Datasets

• The biomedical corpus uses PubMed abstracts and PMC full-text articles, totaling 50.7B tokens for pretraining.Source: §3.1.1 · p. 3

3.1.2 Clinical Datasets

• The clinical corpus combines MIMIC-IV with 20 publicly available, non-synthetic English-language datasets totaling 2.8B tokens, broadening institutional and clinical-context coverage.Some included datasets use synthetic PHI surrogates rather than masking tokens, addressing a cited limitation of placeholder-based training data.Source: §3.1.2 · p. 3

3.1.3 Training Schedule

• Training resumes from ModernBERT stable-stage checkpoints, first mixing biomedical and clinical text, then refining on clinical data alone.Phase 1 uses 160.5B tokens to mitigate biomedical knowledge forgetting; Phase 2 runs for three clinical-data epochs, with model-specific learning-rate schedules. A biomedical-only extension underperformed the clinical-data variant.Source: §3.1.3 · p. 4

3.1.4 Computing Resources

• Continued pretraining used eight NVIDIA H100 SXM5 GPUs and took approximately four days for base and eight days for large, excluding preprocessing and ablation studies.Source: §3.1.4 · p. 4

3.2.1 Models

• Comparators include Clinical-Longformer and Clinical-BigBird, trained on MIMIC-III with 4,096-token sequences, alongside BioBERT, BioMed-RoBERTa, and Bio_ClinicalBERT.The comparison also includes ModernBERT base and large to assess the added value of continued pretraining.Source: §3.2.1 · pp. 4–5
• Clinical ModernBERT was trained on 13B tokens with 128-token sequences; this work trains base and large models with 8,192-token inputs on a larger, more diverse corpus from pre-decay checkpoints.The passage notes that the earlier model’s shorter sequences may limit its ability to capture long-range dependencies.Source: §3.2.1 · p. 5

3.2.2 Tasks

• Evaluation covers five datasets: one multilabel classification task, one single-label classification task, and three NER tasks, with median test scores across five fine-tuning seeds.Classification uses weighted F1, while NER uses the seqeval F1 score.Source: §3.2.2 · p. 5

Classification tasks

• ChemProt is six-class single-label classification of chemical–protein interactions, while Phenotype predicts 14 phenotypes as multilabel labels, excluding the all-absent “None” category.Source: Classification tasks · p. 5

Named Entity Recognition Tasks

• NER evaluation covers COS, Social History, and DEID; tasks use token-level multi-label classification, with each dataset converted to BIO format.COS labels four clinical-event entities; Social History covers substance-use attributes and other social-history factors; DEID labels six PHI types.Source: Named Entity Recognition Tasks · pp. 5–6

3.2.3 Inference Speed

• Speed tests use six synthetic datasets of 8,192 documents: three fixed lengths and three variable-length sets centered at half each maximum sequence length.Fixed lengths are 512, 4,096, and 8,192 tokens; variable-length speeds are averaged over 10 runs.Source: §3.2.3 · p. 6

3.2.4 Computing Resources

• Downstream fine-tuning runs on one H100 PCIe GPU, while inference-speed tests use one A100 40GB GPU.Source: §3.2.4 · p. 6

4.1 Performance

• On 4 of 5 tasks, BioClinical ModernBERT base and large outperform other models; large scores F1 90.8% on ChemProt and 60.8% on Phenotype.The base model also scores 89.9% on ChemProt and 58.1% on Phenotype; the two models remain competitive on COS at 95.1%.Source: Table 2 · p. 6, §4.1 · p. 6
• On ChemProt, Phase 2 scores 89.9% (base) and 90.8% (large); large improves from 90.5% in Phase 1, above BioBERT (89.5%) and Clinical BERT (88.3%).ChemProt is used as a proxy for biomedical domain knowledge.Source: §4.1 · pp. 6–7

4.2 Inference Speed

• ModernBERT base throughput is 76.2, 73.8, and 71.1 kTok/s at fixed 512-, 4,096-, and 8,192-token lengths, versus BigBird’s 71.5 to 50.1 kTok/s.Source: §4.2 · p. 7
• ModernBERT records 75.1 kTok/s on the variable-length 512-token dataset and 73.7 kTok/s on the variable-length 8,192-token dataset.Source: §4.2 · p. 7
• The Lee et al. model’s throughput degrades on longer fixed-length inputs and is lower on variable-length sets; its configuration appears to follow BERT architecture.The authors suggest its variable-length performance indicates unpadding is not being used.Source: §4.2 · p. 7

5 Discussion

• BioClinical ModernBERT base and large outperform existing clinical and biomedical encoders on classification and named entity recognition tasks.The discussion attributes Phenotype gains to both clinical-language adaptation and the longer input sequences in that dataset.Source: §5 · p. 7, Table 3 · p. 7
• ChemProt performance changed by only -0.3 points for base and +0.3 points for large between Phases 1 and 2, while Clinical BERT scored 1.2 points below BioBERT.The authors suggest that including clinical data in the first pretraining stage helps prevent catastrophic forgetting during clinical adaptation.Source: §5 · p. 8
• Training across datasets with varied PHI handling strategies may improve de-identification performance by preserving semantic structure in some PHI representations.Some corpus datasets replace PHI with synthetic, realistic surrogates rather than masking tokens.Source: §5 · p. 8
• BioClinical ModernBERT maintains high efficiency across sequence lengths, aided by local sliding-window attention and unpadding that excludes padding tokens during inference.Two-thirds of attention layers use local windows; unpadding reduces memory use and speeds inference on variable-length inputs.Source: §5 · p. 8
• MIMIC-III and MIMIC-IV make up over 95% of the clinical training corpus, and English-only training limits direct applicability to other languages.The planned i2b2 benchmarks were unavailable during the project.Source: §5 · p. 8

6 Conclusion

• The two-phase continued-pretraining corpus combines biomedical data with 20 clinical datasets from multiple countries, institutions, and settings, and is described as the largest such encoder corpus to date.The authors release both base and large versions.Source: §6 · p. 8
• BioClinical ModernBERT achieves state-of-the-art performance across a broad set of clinical benchmarks and improved generalization across clinical NLP tasks.The authors link the design to integrating biomedical and clinical data early and using long-context modeling.Source: §6 · pp. 8–9
• The model supports inputs up to 8,192 tokens while remaining highly efficient at long input lengths.Source: §6 · p. 9
• The authors release code, models, and training checkpoints to support further research.Source: §6 · p. 9

A Encoders training corpora

• The table's breakdown highlights the diversity and scale of BioClinical ModernBERT's training data compared with prior models.Source: A Encoders training corpora · p. 13
• The composition is summarized separately across clinical, biomedical, and other sources.Source: A Encoders training corpora · p. 13
• Table 4 reports training-data composition by data type, with token counts expressed in billions.Source: Table 4 · p. 13

B Fine-tuning learning rates

• Table 5 reports the selected learning rates used to fine-tune encoders on each downstream task.Source: B Fine-tuning learning rates · p. 13, Table 5 · p. 13
• Separate learning-rate values are shown for the base and large model variants.Source: B Fine-tuning learning rates · p. 13
• The table organizes learning-rate choices by task and model size.Source: Table 5 · p. 13

• Table 6 reports the number of classes for classification datasets or BIO-tagged entity types for NER datasets.Source: C Downstream evaluation datasets… · p. 13, Table 6 · p. 13
• Table 6 includes the number of examples in each dataset split.Source: C Downstream evaluation datasets… · p. 13, Table 6 · p. 13
• Table 6 reports average tokens per example, computed with the ModernBERT tokenizer.Source: C Downstream evaluation datasets… · p. 13, Table 6 · p. 13