PaperlayerView exact revision on arXiv

Source-linked AI summary

Clinical ModernBERT: An efficient and long context encoder for biomedical text

Simon A. Lee, Anthony Wu, Jeffrey N. Chiang

arXiv:2504.03964v1cs.CLcs.AIcs.LG

TL;DR

Clinical NLP needs encoders that represent specialized biomedical language and handle long clinical sequences. Clinical ModernBERT adapts ModernBERT through pretraining on biomedical literature, clinical notes, and medical ontologies, and the authors report competitive or better benchmark results, including state-of-the-art performance on long-context tasks.

  • Problem

    Clinical notes can exceed the 512-token limit of BERT and its biomedical derivatives, while biomedical and clinical language requires domain-adapted representations.

    Source: §2.2 · p. 3

  • Method

    Clinical ModernBERT adapts ModernBERT with pretraining on biomedical literature, clinical notes, and structured medical ontologies.

    Source: §1 · p. 2

  • Results

    Clinical ModernBERT is competitive with or outperforms domain baselines across clinical NLP benchmarks and achieves state-of-the-art performance on long-context tasks such as i2b2 concept extraction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that adapting language-modeling principles through targeted masking, structured knowledge integration, and dense supervision yields competitive and often state-of-the-art biomedical NLP models.

    Source: §6.2 · p. 12, §6.4 · p. 13

Abstract

from arXiv · show

We introduce Clinical ModernBERT, a transformer based encoder pretrained on large scale biomedical literature, clinical notes, and medical ontologies, incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with their textual descriptions. Building on ModernBERT the current state of the art natural language text encoder featuring architectural upgrades such as rotary positional embeddings (RoPE), Flash Attention, and extended context length up to 8,192 tokens our model adapts these innovations specifically for biomedical and clinical domains. Clinical ModernBERT excels at producing semantically rich representations tailored for long context tasks. We validate this both by analyzing its pretrained weights and through empirical evaluation on a comprehensive suite of clinical NLP benchmarks.

1 Introduction

Encoder-only models remain useful for fine-grained semantic discrimination, and Clinical ModernBERT adapts ModernBERT’s architecture to biomedical and clinical language. The paper reports competitive or better benchmark performance and state-of-the-art results on long-context tasks.

Motivation

  • Bidirectional attention lets encoder-only transformers use preceding and succeeding context, producing semantically enriched embeddings suited to fine-grained discrimination.The introduction cites Flash Attention as an example of advances in encoder efficiency.Source: §1 · p. 1

Background

  • ModernBERT supports contexts up to 8,192 tokens and incorporates RoPE, GeGLU, bias-free linear transformations, and Flash Attention.Table 1 compares encoder architectures and design innovations across domains; Table 2 summarizes pretraining supervision, ontology integration, and task coverage.Source: §1 · p. 2, Table 1 · p. 2, Table 2 · p. 2

Contributions

  • Clinical ModernBERT is a domain-specialized encoder pretrained on biomedical literature, clinical narratives, and structured medical ontologies.It integrates ModernBERT architectural advances with domain-adaptive pretraining over clinical notes and medical ontologies.Source: §1 · p. 2

Results

  • Clinical ModernBERT is competitive with or outperforms domain baselines across retrieval, classification, and entity-recognition benchmarks, with state-of-the-art performance on long-context i2b2 concept extraction.The reported baselines include BioBERT, BioClinicalBERT, and Clinical Longformer.Source: §1 · p. 2

2 Related Works

Biomedical and clinical BERT variants adapt pretraining data, vocabulary, and structured knowledge to specialized language. ModernBERT adds an efficient long-context encoder design, while Clinical ModernBERT combines these directions for clinical and biomedical documents.

2.1 Biomedical and Clinical Adaptations of BERT

  • BioBERT continued BERT pretraining on biomedical literature, while SciBERT used scientific publications and a domain-specific vocabulary.Both lines of work were reported to improve performance on biomedical or scientific NLP tasks.Source: §2.1 · p. 3
  • ClinicalBERT trained on MIMIC-III notes, and BioClinicalBERT and BlueBERT extended clinical adaptation through pretraining on clinical notes or combined biomedical and clinical corpora.BioClinicalBERT continued BioBERT pretraining on MIMIC-III; BlueBERT combined PubMed and MIMIC-III data.Source: §2.1 · p. 3

2.2 Ontology-Enriched and Scaling Context Length BERT Variants

  • SapBERT used UMLS synonym mappings for entity embeddings, while DK-BEHRT included medical codes and descriptions during pretraining.The cited work reports a latent space of ICD-9 disease codes for DK-BEHRT.Source: §2.2 · p. 3
  • Clinical Longformer extended context to 4,096 tokens with sparse attention, addressing the 512-token limit of BERT and its biomedical derivatives.The passage identifies long patient narratives, longitudinal EHR entries, and radiology reports as settings for long-context processing.Source: §2.2 · p. 3
  • ModernBERT introduced RoPE, GeGLU, Flash Attention, and context lengths up to 8,192 tokens; Clinical ModernBERT builds on it for clinical and biomedical documents.The paper describes Clinical ModernBERT as a more efficient, long-context replacement for BioClinicalBERT, whose context is limited to 512 tokens.Source: §2.2 · p. 3

3 Methods

Clinical ModernBERT inherits ModernBERT’s architectural changes and adapts its pretraining to biomedical literature, clinical notes, and structured medical terminologies. Training uses a large, curated corpus and token-aware masking, with optimization choices and checkpoint tracking selected for pretraining.

3.1 Summarizing ModernBERT

  • RoPE rotates pairs of embedding dimensions so relative positional information enters the query-key inner product without explicit position embeddings.The passage describes this representation as preserving order-sensitive structure and supporting extrapolation beyond the training context length.Source: §3.1 · p. 4
  • ModernBERT uses GeGLU instead of ReLU-based feed-forward transformations, introducing learned multiplicative interactions between nonlinear and linear projections.The passage links GeGLU’s gating and smooth curvature to richer feature encodings and improved convergence dynamics.Source: §3.1 · p. 4
  • Removing bias terms reduces trainable parameters and modestly lowers memory and compute costs, with the passage also attributing faster convergence to this simplification.The stated rationale is that bias terms can have negligible impact when layer normalization is applied.Source: §3.1 · p. 4
  • Flash Attention computes self-attention blockwise with a tiling scheme that avoids materializing the full attention matrix, reducing memory use while preserving exact results.Fusing softmax and matrix multiplication enables streaming computation; memory usage scales near-linearly with sequence length.Source: §3.1 · p. 5

3.2 Pretraining Data Sources

  • Pretraining data spans about 40 million PubMed abstracts, MIMIC-IV discharge summaries and radiology reports, and structured codes and descriptions including ICD-9 through ICD-12 and CPT.The code-description pairs are converted into standardized natural-language representations across ICD versions.Source: §3.2 · pp. 5–6, Figure 1 · p. 6

3.3 Pre-training Optimization

  • The normalized and preprocessed corpus contains 13 billion tokens, with coherent clinical narratives upsampled and byte-pair tokenization initialized from ModernBERT-base.The corpus combines MIMIC-IV notes, PubMed abstracts, and structured medical ontologies.Source: §3.3 · p. 6
  • A custom collator applies token-aware masking to biomedical entities, decaying the masking rate from 30% to 15% over training.Early training emphasizes informative terms such as medication names, procedure codes, and morphological descriptors.Source: §3.3 · p. 6
  • Training ran for 150,000 steps with sequence length initially set to 128 tokens, StableAdamW, a cosine learning-rate schedule, and mixed precision; MLM accuracy and loss were tracked for checkpoint selection.The authors report that MLM performance plateaued beyond 150,000 steps and checkpoints were saved every 10,000 steps.Source: §3.3 · pp. 6–7

4 Experimental Setup

The experiments assess masked-token prediction, compare Clinical ModernBERT with general, biomedical, and long-context baselines across short- and long-context tasks, and probe code representations and inference efficiency.

4.1 Assessing Pre-training

  • Masked language modeling masks selected input tokens and trains the model to reconstruct them; top-k accuracy measures recovery at different ranking cutoffs.Top-1 reflects precision, while top-25 captures broader lexical recall.Source: §4.1 · pp. 7–8

4.2 Benchmarking Models

  • The baseline set spans general-purpose BERT, biomedical encoders, and models designed for longer contexts to assess domain adaptation, modernization, and sequence-length scaling.The comparison includes BERT-base, BioBERT, BioClinicalBERT, Clinical Longformer, and ModernBERT.Source: §4.2 · p. 8

4.3 Benchmarking Datasets

  • Evaluation combines short-context classification, biomedical NER, trial labeling, and sentence-level retrieval with full-document NER and long-range retrieval tasks.The long-context benchmarks use i2b2 tasks and other clinical corpora, supporting comparison of local and global context modeling.Source: §4.3 · p. 8

4.4 Medical Codes Visualization Protocol

  • The ICD-9 visualization compares t-SNE projections of code embeddings from Clinical ModernBERT and ModernBERT to examine alignment with clinical taxonomy.Code descriptions are encoded using each model’s [CLS] embeddings, then projected into two dimensions.Source: §4.4 · p. 8

4.5 Model Efficiency Benchmarking Protocol

  • Inference latency is measured for three models on 512-token synthetic clinical inputs, using mean runtime across three runs on one NVIDIA A100 GPU.Timing includes tokenization through final hidden-state output, excludes disk and network I/O, and is designed to isolate transformer-level inference cost.Source: §4.5 · pp. 8–9

5 Results

Clinical ModernBERT achieves strong outcomes across standard clinical and biomedical benchmarks, including leading EHR and retrieval results, competitive long-context NER, clearer ICD-code clusters, and the lowest measured runtime. Pretraining results also show sensitivity to masking strategy.

5.1 Effectiveness of Pre-training

  • Top-1 accuracy reaches 63.31% and top-25 accuracy approaches 88.10%, indicating strong recovery of clinically appropriate tokens among the highest-ranked predictions.The reported MLM loss declines during training as accuracy plateaus.Source: §5.1 · p. 9, Table 4 · p. 9
  • Removing token-aware masking lowers top-1 accuracy from 63.31% to 48.84% and top-25 accuracy below 59%; reducing masking from 30% to 15% yields top-1 accuracy of 58.22%.Both ablations underperform the baseline, with the larger decline following removal of token-aware selection.Source: §5.1 · p. 9, Table 5 · p. 9

5.2 Benchmark Across Multiple Standard Biomedical NLP Tasks

  • Clinical ModernBERT leads EHR AUROC (0.9769) and PMC-Patients NDCG@10 (0.2167) and MAP (0.1982), while BioBERT leads PubMed-NCT and MedNER F1.The model also exceeds BioBERT in PubMed-NCT accuracy and outperforms all models except BioBERT on MedNER F1.Source: §5.2 · pp. 9–10, Table 7 · p. 10, Table 6 · p. 10

5.3 Long Context Biomedical NLP Tasks

  • Clinical Longformer leads i2b2 2006, 2010, and 2014 F1, while Clinical ModernBERT leads on i2b2 2012 and ranks second on the other datasets.ModernBERT outperforms BioBERT and BioClinicalBERT on all four datasets.Source: §5.3 · p. 10, Table 8 · p. 11

5.4 Latent Space Visualizations for medical codes

  • Clinical ModernBERT forms cleaner, tighter ICD-chapter clusters than ModernBERT, with distinct separation for neoplasms, nervous-system disorders, and congenital anomalies.The comparison uses t-SNE projections of diagnosis-code embeddings.Source: §5.4 · pp. 10–11, Figure 2 · p. 11

5.5 Efficiency and Scalability of Encoding Embeddings

  • Clinical ModernBERT maintains the lowest runtime across tested data volumes, with BioClinicalBERT’s runtime nearly 60% higher at 100,000 points.The figure compares processing time for Distil-BERT, BioClinicalBERT, and Clinical ModernBERT as volume increases.Source: §5.5 · p. 11, Figure 3 · p. 12

6 Discussion

The discussion reports strong clinical benchmark performance, ablation evidence for domain-aware pretraining choices, and clinically meaningful structure in the model’s representations. It closes with future research directions and a summary of the approach’s results.

6.1 Domain Adaptation and Representational Quality

  • 63.31% top-1 MLM accuracy accompanies marked increases in top-5 to top-25 accuracies, which the authors interpret as narrowing toward semantically proximate candidates.The authors suggest that predictive precision reflects clinical syntax and terminology, likely related to corpus composition and token-aware masking.Source: §6.1 · p. 12
  • Clinical ModernBERT attains state-of-the-art AUROC in EHR classification, outperforms baselines on all retrieval metrics, and alternates with Clinical Longformer for best or second-best results across four i2b2 datasets.The i2b2 datasets are from 2006, 2010, 2012, and 2014.Source: §6.1 · p. 12

6.2 Ablation Findings and Pretraining Design

  • Removing token-aware masking caused a catastrophic drop across top-k thresholds, while reverting to 15% masking also degraded performance.The authors argue that targeted dense supervision and domain-specific curriculum construction matter for representational quality.Source: §6.2 · p. 12

6.3 Latent Structure and Qualitative Insights

  • Improved clustering by ICD category is presented as qualitative evidence that Clinical ModernBERT captures clinically meaningful structure and latent ontological relationships.The discussion also reports scalable dense-embedding efficiency outperforming BioClinicalBERT and Distil-BERT.Source: §6.3 · p. 13

6.4 Future Directions

  • Future work includes clinical scaling-law studies, comprehensive error analysis, MEDS-based benchmarking, and multimodal links between text and imaging or waveforms.The authors frame multimodal extension as a possible route to novel clinical decision-support capabilities.Source: §6.4 · p. 13
  • The authors conclude that targeted masking, structured knowledge integration, and dense supervision yield models that are competitive or state-of-the-art across many biomedical NLP tasks.The conclusion describes this as an adaptation of general language-modeling principles to the clinical domain.Source: §6.4 · p. 13

Biomedical NLP tasks (short-context)

The short-context task passages describe clinical and biomedical classification, entity-recognition, and retrieval benchmarks used to assess the model.

Biomedical NLP tasks (short-context)

  • The ED disposition task represents first-hour visit features as textual pseudo-notes and predicts whether a patient is admitted or discharged.The records include demographics, triage vitals, chief complaints, procedures, labs, and medication events.Source: Biomedical NLP tasks (short-context) · p. 17
  • PubMed-200k-RCT classifies roughly 200,000 abstract sentences by rhetorical function, using labels for Background, Objective, Methods, Results, or Conclusions.Its records include abstract ID, sentence ID, label, and sentence text.Source: Biomedical NLP tasks (short-context) · p. 17
  • MedNER frames annotated clinical and biomedical text as binary token-level entity recognition, labeling each token as part of a medical entity or not.The passage describes it as a testbed for entity-boundary detection and semantic salience under constrained supervision.Source: Biomedical NLP tasks (short-context) · p. 17
  • PMC-Patients supports patient-to-article and patient-to-patient retrieval using 167,000 patient summaries, over 3.1 million relevance annotations, and 293,000 similarity links.It provides predefined training, validation, and test splits.Source: Biomedical NLP tasks (short-context) · p. 17

Long Context Tasks

The i2b2 setup evaluates long-context named entity recognition on clinical notes while preserving document structure so the model can use its extended context capacity.

Long Context Tasks

  • The i2b2 setup preserves original document structure without chunking, enabling evaluation of long-context NER and contextual disambiguation across clinical notes.The datasets cover extraction of problems, treatments, tests, and temporal expressions from real-world clinical narratives.Source: Long Context Tasks · p. 17

B.1 MLM Accuracies and MLM Loss

Figure 4 compares MLM top-K accuracies and loss across three pretraining runs; higher learning rates yielded more stable convergence but noisier gradients.

B.1 MLM Accuracies and MLM Loss

  • Figure 4 reports top-K accuracy for k = 1, 5, 10, and 25 alongside MLM loss across three runs with different learning rates.The plotted rates, top to bottom, are 3 × 10−3, 5 × 10−4, and 1 × 10−5.Source: Figure 4 · p. 18
  • Higher learning rates yielded more stable convergence and avoided shallow local minima, suggesting improved exploration of the loss landscape.Source: Figure 4 · p. 18
  • Higher learning rates also introduced noisier gradient updates, consistent with the passage’s stated optimization intuition.Source: Figure 4 · p. 18
Loading 2504.03964v1…