PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparative Study of Pretrained Language Models for Long Clinical Text

Yikuan Li, Ramsey M. Wehbe, Faraz S. Ahmad, Hanyin Wang, Yuan Luo

arXiv:2301.11847v1cs.CLcs.AI

TL;DR

Long clinical texts challenge full self-attention because memory consumption grows quadratically with sequence length. The study introduces clinically pretrained long-sequence transformers and reports stronger downstream performance, while noting scope limitations related to model architectures and memory constraints.

  • Problem

    Full self-attention interacts across tokens, but its memory consumption grows quadratically with sequence length, motivating long-sequence transformers for clinical text.

  • Method

    The study introduces Clinical-Longformer and Clinical-BigBird, using sparse attention and large-scale clinical pretraining.

  • Results

    Clinical-Longformer and Clinical-BigBird learn more useful contextualized relationships than baseline models, with domain-specific pretraining supporting downstream accuracy improvements.

  • Takeaways & Limitations

    Clinical-Longformer and Clinical-BigBird provide a foundation for improving downstream clinical NLP performance through long-sequence, domain-specific pretraining.

  • Takeaways & Limitations

    The study applies Longformer and BigBird within scope constrained by memory limits, including limitations for encoder-decoder frameworks.

Abstract

from arXiv · show

Objective: Clinical knowledge enriched transformer models (e.g., ClinicalBERT) have state-of-the-art results on clinical NLP (natural language processing) tasks. One of the core limitations of these transformer models is the substantial memory consumption due to their full self-attention mechanism, which leads to the performance degradation in long clinical texts. To overcome this, we propose to leverage long-sequence transformer models (e.g., Longformer and BigBird), which extend the maximum input sequence length from 512 to 4096, to enhance the ability to model long-term dependencies in long clinical texts. Materials and Methods: Inspired by the success of long sequence transformer models and the fact that clinical notes are mostly long, we introduce two domain enriched language models, Clinical-Longformer and Clinical-BigBird, which are pre-trained on a large-scale clinical corpus. We evaluate both language models using 10 baseline tasks including named entity recognition, question answering, natural language inference, and document classification tasks. Results: The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT and other short-sequence transformers in all 10 downstream tasks and achieve new state-of-the-art results. Discussion: Our pre-trained language models provide the bedrock for clinical NLP using long texts. We have made our source code available at https://github.com/luoyuanlab/Clinical-Longformer, and the pre-trained models available for public download at: https://huggingface.co/yikuan8/Clinical-Longformer. Conclusion: This study demonstrates that clinical knowledge enriched long-sequence transformers are able to learn long-term dependencies in long clinical text. Our methods can also inspire the development of other domain-enriched long-sequence transformers.

Introduction

Full self-attention makes transformer memory use grow quadratically, limiting common models to 512 tokens and weakening long-document modeling. This study introduces clinical long-sequence models to address that gap and evaluates them across clinical NLP tasks.

  • Motivation: Full self-attention has quadratic memory growth, making training impractical and difficult within modern GPU limits.
  • Motivation: Clinical notes can exceed the 512-token limit, causing models to miss long-term dependencies and lose information.MIMIC-III discharge summaries average 2,984 tokens.
  • Prior approach: Long-sequence transformers replace full attention with sparse attention and can process up to 4,096 tokens.Their attention typically combines sliding windows with reduced global attention.
  • Research gap: Clinical long-sequence transformers had not been systematically explored despite their potential relevance to long clinical and biomedical documents.
  • Contribution: The study pre-trains Clinical-Longformer and Clinical-BigBird on large-scale clinical notes and evaluates them on question answering, named entity recognition, and document classification.

Background and Significance

Domain-enriched BERT variants improve clinical NLP but retain a 512-token input limit. Longformer and BigBird extend context through sparse attention, motivating their clinical adaptation.

  • Domain-enriched models: Existing domain-enriched models such as BioBERT and ClinicalBERT improve clinical and biomedical NLP but use vanilla BERT's 512-token limit.
  • Alternative approaches: Alternative approaches illustrate trade-offs: Transformer-XL learns longer dependencies but loses BERT-like bidirectional representations, while Reformer improves speed and memory efficiency.Transformer-XL is reported to learn 5.5 times longer dependencies.
  • Long-sequence transformers: Longformer and BigBird replace pairwise full attention with sliding-window and global attention mechanisms to reduce memory consumption.
  • Long-sequence transformers: Both models support sequences up to 4,096 tokens and improve performance on long-text question answering and summarization tasks.
  • Research gap: The adaptability of long-sequence transformers to clinical and biomedical documents had not been investigated, although such documents often exceed BERT-like length limits.

Materials and Methods

The study pre-trains Clinical-Longformer and Clinical-BigBird, then fine-tunes and compares them with short-sequence models across downstream clinical NLP tasks. The overall workflow is summarized in Figure 1.

  • Pipeline: The experimental pipeline pre-trains the clinical models and fine-tunes them for comparison with short-sequence models on downstream tasks.
  • Pipeline: Figure 1 presents the pipeline for pre-training and fine-tuning transformer-based language models.

Datasets

The pre-training corpus contains approximately 2 million MIMIC-III clinical notes from intensive-care patients. Preprocessing is intentionally minimal and removes identifiers and formatting noise.

  • Pre-training corpus: Approximately 2 million clinical notes from MIMIC-III form the pre-training corpus.
  • Pre-training corpus: MIMIC-III contains clinical narratives from more than 40,000 patients admitted to intensive care units.
  • Preprocessing: Preprocessing removes de-identification placeholders, non-alphanumeric characters, uppercase letters, and extra whitespace.
  • Preprocessing: The authors use minimal preprocessing because complicated preprocessing may not improve downstream performance and may reduce generalizability.

Pre-training

Clinical-Longformer and Clinical-BigBird are long-sequence, clinically pretrained transformer models designed to capture dependencies in long clinical text. They extend input capacity to 4,096 tokens and are pretrained on clinical notes for downstream evaluation.

  • 4,096 tokens is the extended input limit of the long-sequence models, compared with 512 tokens for conventional transformers.
  • Localized sliding windows, global attention, and BigBird’s additional random attention reduce the computational expense of full self-attention.
  • Clinical-Longformer and Clinical-BigBird are initialized from Longformer and BigBird checkpoints, respectively, then pretrained on clinical text.
  • The models use byte-level BPE tokenization and are pretrained for 200,000 and 300,000 steps, respectively, with a shared learning rate of 3e-5.
  • Pretraining evaluation masks 10% of tokens in held-out MIMIC-III documents and compares models using perplexity and top 5 accuracy.

Downstream Tasks

The study fine-tunes the clinical long-sequence models on 10 datasets spanning question answering, named entity recognition, natural language inference, and document classification. These benchmarks include clinical notes, reports, and sentence pairs evaluated with task-specific metrics.

  • 10 clinical NLP datasets cover extractive question answering, named entity recognition, natural language inference, and document classification.
  • Question Answering: emrQA contains more than 400,000 semi-automatically generated question-answer pairs and is used as a proof-of-concept benchmark because its answers and annotations have known limitations.
  • Question Answering: The question-answering evaluation uses Medication, Relation, and Heart Disease emrQA subsets with exact match and F1-score metrics.
  • Named Entity Recognition: The four i2b2 challenges evaluate clinical information extraction, including PHI de-identification, medical concept extraction, and timeline-relevant concepts and events.
  • Document Classification: Document classification includes MIMIC-AKI, OpenI, and other clinical datasets evaluated with AUC, F1, weighted AUC, or accuracy depending on the task.
  • Natural Language Inference: MedNLI uses sentence pairs labeled entailment, contradiction, or neutral and is evaluated with accuracy.

Baseline Models and Comparisons

The comparisons include general, biomedical, clinical, and long-sequence transformer baselines. ClinicalBERT provides a clinically pretrained short-sequence reference, while Longformer and BigBird supply long-sequence architectures for clinical pretraining.

  • Clinical-Longformer and Clinical-BigBird are compared with short-sequence models across the downstream experiments.
  • The baseline set includes BERT, ClinicalBERT, RoBERTa, and BioBERT, while static word-embedding models are excluded from comparisons.
  • ClinicalBERT is initialized from BioBERT and further pretrained on MIMIC-III clinical notes, improving MedNLI and four included i2b2 NER tasks.
  • Longformer and BigBird initialize from RoBERTa, and their reported downstream performance difference is minimal.
  • A hierarchical transformer is additionally tested on MIMIC-AKI using chunk-level BERT representations followed by a recurrent neural network.

Experimental setup

The experiments adapt sequence processing to each model’s input limit and task type. Long-sequence models process up to 4,096 tokens, whereas short-sequence baselines use 512-token chunks and pooling for long documents.

  • Long-sequence models process chunks of 4,096 tokens with 1,024-token strides, while short-sequence models use 512-token chunks with 128-token strides.
  • OpenI and MedNLI require no truncation or sliding windows because their maximum sequence lengths are below 512 tokens.
  • Documents exceeding the limit are truncated to the first 4,096 tokens for Clinical-Longformer and Clinical-BigBird.
  • For document classification, long-sequence predictions use the [CLS] output, whereas short-sequence documents are segmented and their snippet probabilities are pooled.
  • The short-snippet pooling strategy slightly outperforms maximum pooling in preliminary experiments.
  • Training uses four 32GB GPUs, model-specific batch sizes, half precision for the long-sequence models, and learning-rate selection from three values.

Results and Discussion

Clinical-Longformer and Clinical-BigBird outperform short-sequence and prior clinical transformer models across question answering, NER, and document classification tasks. Gains are generally larger for longer sequences, while Clinical-Longformer is slightly better than Clinical-BigBird in most experiments and more efficient to fine-tune.

  • Clinical-Longformer and Clinical-BigBird outperform short-sequence transformers across question answering, NER, and document classification benchmarks.They improve emrQA, i2b2 NER, and OpenI, MIMIC-AKI, and medNLI classification results.
  • Around 2 percent higher F1 scores occur across all three emrQA subsets, while stricter EM gains reach about 5 percent on the relations subset.The models match ClinicalBERT on EM for the other two emrQA subsets.
  • Clinical-Longformer leads short-sequence transformers by more than 2 percent on all four i2b2 NER datasets, and Clinical-BigBird beats ClinicalBERT and BioBERT throughout.
  • The long-sequence models also improve tasks whose inputs fit within 512 tokens, suggesting benefits beyond sequence-length capacity.The paper attributes these gains to clinical pretraining, longer-range dependency learning, and richer contextualization.
  • Performance gains are largest on longer datasets, including i2b2 2014, whose average sequence length is nearly twice that of the other NER subsets.
  • Clinical-Longformer is slightly better than Clinical-BigBird in most experiments, although differences are usually not statistically significant and BigBird costs more fine-tuning time and memory.The authors therefore recommend Clinical-Longformer when selecting between the two models.
  • The study excludes generative tasks such as text generation and summarization because the models were not integrated into encoder-decoder frameworks under available memory limits.

Conclusion

Clinical-Longformer and Clinical-BigBird are presented as pretrained models for long clinical text and outperform short-sequence transformer baselines on long- and short-sequence benchmarks. The authors identify future work in clinical text generation, summarization, and comparisons with other approaches addressing long-sequence memory consumption.

  • The study introduces Clinical-Longformer and Clinical-BigBird as pretrained models designed specifically for long clinical text NLP tasks.
  • The models are compared with ClinicalBERT and related BERT-variant transformers across named entity recognition, question answering, and document classification tasks.
  • Clinical-Longformer and Clinical-BigBird achieve better results than short-sequence transformer models on both long-sequence and short-sequence benchmark datasets.
  • Future studies will examine generalization to clinical text generation and summarization and compare the models with other approaches for reducing long-text memory consumption.

Data Availability

The benchmark datasets come from multiple publicly available clinical resources, including MIMIC-III, MIMIC-CXR, OpenI, and MedNLI. Access to some data requires applications and data-use agreements with the owners.

  • The benchmark datasets are derived from publicly available MIMIC-III, MIMIC-CXR, OpenI, and MedNLI resources.
  • Access requires applicants to apply and sign data user agreements with the relevant data owners.
Loading 2301.11847v1…