Source-linked AI summary
Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences
Yikuan Li, Ramsey M. Wehbe, Faraz S. Ahmad, Hanyin Wang, Yuan Luo
TL;DR
Full self-attention limits clinical transformers because memory consumption grows quadratically with sequence length, restricting standard inputs to 512 tokens. The paper pretrains Clinical-Longformer and Clinical-BigBird on large-scale clinical corpora and evaluates them across clinical NLP tasks. Both models consistently outperform ClinicalBERT and other short-sequence transformers, including on long- and short-sequence benchmarks.
Problem
Quadratic memory growth in full self-attention limits standard clinical transformers to 512-token inputs, while long clinical documents often exceed that length.
Method
The paper pretrains Clinical-Longformer and Clinical-BigBird on approximately 2 million MIMIC-III clinical notes and adapts them to clinical NLP datasets.
Results
Clinical-Longformer and Clinical-BigBird consistently outperform ClinicalBERT and other short-sequence transformers across question answering, named entity recognition, and classification tasks.
Takeaways & Limitations
The models support clinical NLP on both long- and short-sequence benchmarks, with larger gains reported for datasets containing longer sequences.
Takeaways & Limitations
The study does not evaluate text generation or summarization because the models were not integrated into an encoder-decoder framework under the available GPU memory limits.
Abstract
from arXiv · showhide
Transformers-based models, such as BERT, have dramatically improved the performance for various natural language processing tasks. The clinical knowledge enriched model, namely ClinicalBERT, also achieved state-of-the-art results when performed on clinical named entity recognition and natural language inference tasks. One of the core limitations of these transformers is the substantial memory consumption due to their full self-attention mechanism. To overcome this, long sequence transformer models, e.g. Longformer and BigBird, were proposed with the idea of sparse attention mechanism to reduce the memory usage from quadratic to the sequence length to a linear scale. These models extended the maximum input sequence length from 512 to 4096, which enhanced the ability of modeling long-term dependency and consequently achieved optimal results in a variety of tasks. Inspired by the success of these long sequence transformer models, we introduce two domain enriched language models, namely Clinical-Longformer and Clinical-BigBird, which are pre-trained from large-scale clinical corpora. We evaluate both pre-trained models using 10 baseline tasks including named entity recognition, question answering, and document classification tasks. The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT as well as other short-sequence transformers in all downstream tasks. We have made our source code available at [https://github.com/luoyuanlab/Clinical-Longformer] the pre-trained models available for public download at: [https://huggingface.co/yikuan8/Clinical-Longformer].
1 Introduction
Full self-attention enables broad transformer success but makes memory usage grow quadratically with sequence length, limiting standard models to 512 tokens. The paper addresses this clinical long-sequence gap by introducing two domain-enriched sparse-attention models.
- 1 Introduction: 512-token limits make full-attention transformers impractical for long clinical documents because self-attention memory consumption grows quadratically with sequence length.Clinical discharge summaries average 1,435 words, exceeding this limit.
- 1 Introduction: Sparse-attention long-sequence transformers reduce memory growth from quadratic to linear in sequence length and support inputs up to 4,096 tokens.Their attention combines sliding windows with reduced global attention.
- 1 Introduction: The study examines long-sequence transformers in clinical and biomedical NLP, where such models had been rarely discussed.The motivation is to test their adaptability to clinical tasks.
- 1 Introduction: Clinical-Longformer and Clinical-BigBird are pretrained on large-scale clinical notes and publicly released for long clinical sequences.The models target clinical NLP tasks including question answering, named entity recognition, and document classification.
2 Related Work
Prior work established domain-enriched clinical and biomedical transformers and proposed several mechanisms for reducing attention memory. These approaches motivate long-sequence clinical models while exposing trade-offs between efficiency and bidirectional representations.
- 2 Related Work: ClinicalBERT and BioBERT demonstrated that pretraining on clinical narratives or biomedical publications can improve downstream clinical and biomedical NLP.These models enrich transformer representations with domain knowledge.
- 2 Related Work: Transformer-XL models long-term dependency through left-to-right segment recurrence but loses the bidirectional representation advantage of BERT-like models.The reported dependency modeling is 5.5× that of vanilla transformer models.
- 2 Related Work: The related work surveys attention mechanisms proposed to reduce the memory consumption of vanilla Transformer architectures.The supplied passage introduces Transformer-XL and Reformer as examples.
3 Material and Methods
The study pretrains clinical long-sequence models and evaluates them across downstream clinical NLP tasks using documented fine-tuning procedures. Source code and pretrained models are made publicly available for reproducibility and reuse.
- 3 Material and Methods: The methods introduce the clinical pretraining corpus, pretraining process, downstream fine-tuning tasks, and technical details needed for reproduction.The paper states that source code and pretrained models are publicly available.
3.1 Datasets
The models are pretrained on approximately 2 million clinical notes from MIMIC-III, a large EHR dataset containing narratives from more than 40,000 intensive-care patients. Minimal preprocessing removes protected-health-information placeholders and normalizes text.
- 3.1 Datasets: Approximately 2 million MIMIC-III clinical notes from over 40,000 intensive-care patients form the pretraining corpus.MIMIC-III is described as the largest publicly available EHR dataset containing clinical narratives.
- 3.1 Datasets: Preprocessing removes de-identification placeholders, non-alphanumeric characters except punctuation, uppercase letters, and extra whitespace.These are the four minimal preprocessing operations reported by the authors.
3.2 Pre-training
Clinical-Longformer and Clinical-BigBird adapt sparse-attention transformers for long clinical sequences. They are initialized from existing Longformer and BigBird weights and trained on clinical notes.
- Longformer and BigBird extend transformer input capacity to 4,096 tokens, eight times the conventional 512-token limit, using localized sliding-window and global attention.Their sparse attention mechanisms reduce the computational expense of full self-attention.
- Clinical-Longformer and Clinical-BigBird were initialized from pre-trained Longformer and BigBird weights, respectively.Clinical-Longformer used base Longformer weights, while Clinical-BigBird used the ITC version of BigBird.
3.3 Tasks
The study evaluates the proposed models across nine clinical NLP datasets covering question answering, named entity recognition, natural language inference, and document classification. The tasks span clinical notes, radiology reports, and academic medical abstracts.
- Nine datasets cover extractive question answering, named entity recognition, natural language inference, and document classification.The study uses these tasks to validate performance improvement, with dataset statistics and descriptions reported in Table 1.
- Question Answering: Clinical question answering is evaluated on three large emrQA subsets—Medication, Relation, and Heart Disease—using exact match and F1-score.emrQA contains more than 400,000 semi-automatically generated question-answer pairs, and extractive QA identifies answer spans from reference contexts.
- Named Entity Recognition: Named-entity recognition is evaluated on four i2b2 challenges covering PHI de-identification, medical concept extraction, and clinical concepts and events.The raw datasets were converted to IOB tagging format and split into train, development, and test sets.
- Document Classification: Document classification includes MIMIC-AKI, OpenI, MedNLI, and OHSUMed, spanning clinical prediction, chest-x-ray reporting, sentence-pair inference, and cardiovascular disease classification.MIMIC-AKI uses clinical notes from the first 24 hours after ICU admission, while OpenI is used only as a testing set because of its small sample size.
3.4 Baseline Models
The proposed models are compared against BERT, BioBERT, and ClinicalBERT, representing general, biomedical, and clinical pre-trained transformer baselines.
- Clinical-Longformer and Clinical-BigBird are compared with BERT, ClinicalBERT, and BioBERT.The comparison includes the pioneering general transformer, a biomedical variant, and a clinical variant.
- BioBERT was pre-trained on PubMed abstracts and PMC full-text articles, while ClinicalBERT was further pre-trained on MIMIC-III clinical notes.BioBERT initialized its weights from BERT, and ClinicalBERT initialized from BioBERT.
- BERT, BioBERT, and ClinicalBERT used masked language modeling without next sentence prediction in the latter two models.BERT used masked language modeling and next sentence prediction, whereas BioBERT and ClinicalBERT discarded next sentence prediction.
3.5 Experimental setup
The experimental setup uses token-level and sequence-level classification heads, with longer inputs for the proposed models than for the baseline transformers.
- Clinical-Longformer and Clinical-BigBird use a maximum sequence length of 3,072, compared with 384 for the three baseline models.This setup applies to token-level classification tasks including question answering and named entity recognition.
- For token-level classification, a classification head is added to each token representation produced by the transformer.The passage specifically includes question answering and named entity recognition.
- For sequence-level classification, predictions are generated from the [CLS] token placed at the beginning of each sentence or document.
4 Results and Discussion
Clinical-Longformer and Clinical-BigBird outperform short-sequence transformers across clinical question answering, named entity recognition, and document classification tasks. Improvements extend to short sequences and are stronger for datasets with longer sequences, while Clinical-Longformer generally performs better than Clinical-BigBird.
- Overall results: Across QA, NER, and classification, both long-sequence models generally outperform prior short-sequence transformers, with OHSUMed favoring BioBERT.QA gains include 2 percentage points in F1 across all emrQA subsets and 5 percentage points in EM on relations; NER gains exceed 2 percentage points across all four i2b2 datasets.
- Overall results: Clinical-Longformer and Clinical-BigBird also improve performance on short-sequence tasks, where inputs could fit entirely within BERT-like 512-token limits.The authors attribute this pattern to pre-training over longer contexts and learning more long-term dependencies.
- Sequence length effects: Performance improvements are larger on datasets with longer sequences, with i2b2 2014 showing nearly twice the gain of the other three NER subsets.The heart disease emrQA subset also shows a stronger Clinical-Longformer F1 improvement.
- Model comparison: Clinical-Longformer consistently outperforms Clinical-BigBird and requires less fine-tuning time, making it the recommended checkpoint for resource-limited users.The recommendation is explicitly tied to Clinical-BigBird's higher fine-tuning time.
- Limitations: The study omits text generation and summarization because encoder-decoder integration was not feasible under the available GPU memory limits.Future work also includes comparisons with architecture-compression approaches such as TinyBERT.
5 Conclusion
The study introduced two pre-trained language models for long-sequence clinical NLP and compared them with short-sequence transformer baselines. They achieved better results on both long- and short-sequence benchmark datasets, while broader model comparisons and clinical generation tasks remain for future work.
- Conclusion: Clinical-Longformer and Clinical-BigBird achieved better results than short-sequence transformer baselines on both long- and short-sequence benchmark datasets.The models were evaluated on named entity recognition, question answering, and document classification tasks.
- Future work: Future work will compare the models with memory-efficient architectures such as TinyBERT and assess their generalizability to clinical text generation and summarization.The study did not include generation or summarization experiments because of GPU memory limits.