PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Clinical Concept Extraction with Contextual Embeddings

Yuqi Si, Jingqi Wang, Hua Xu, Kirk Roberts

arXiv:1902.08691v4cs.CL

TL;DR

Clinical concept extraction lacks established best practices for integrating newer contextual embeddings. This study compares traditional and contextual embeddings, open-domain and MIMIC-III pre-training, and pre-training duration across four corpora. Clinical contextual embeddings achieve new state-of-the-art performance across all tasks, while also encoding semantic information beyond traditional word representations.

  • Problem

    The study addresses the lack of common best practices for integrating advanced contextual embeddings into clinical concept extraction.

  • Method

    The study compares word2vec, GloVe, fastText, ELMo, and BERT using open-domain and MIMIC-III embeddings across four clinical concept extraction corpora.

  • Results

    Clinical contextual embeddings achieve new state-of-the-art performance across all four clinical concept extraction tasks.

  • Takeaways & Limitations

    Contextual embeddings encode semantic information not accounted for by traditional word representations.

  • Takeaways & Limitations

    BERT wordpiece tokenization does not exactly conform to traditional medical term morphology, raising a clinical-text tokenization concern.

Abstract

from arXiv · show

Neural network-based representations ("embeddings") have dramatically advanced natural language processing (NLP) tasks, including clinical NLP tasks such as concept extraction. Recently, however, more advanced embedding methods and representations (e.g., ELMo, BERT) have further pushed the state-of-the-art in NLP, yet there are no common best practices for how to integrate these representations into clinical tasks. The purpose of this study, then, is to explore the space of possible options in utilizing these new models for clinical concept extraction, including comparing these to traditional word embedding methods (word2vec, GloVe, fastText). Both off-the-shelf open-domain embeddings and pre-trained clinical embeddings from MIMIC-III are evaluated. We explore a battery of embedding methods consisting of traditional word embeddings and contextual embeddings, and compare these on four concept extraction corpora: i2b2 2010, i2b2 2012, SemEval 2014, and SemEval 2015. We also analyze the impact of the pre-training time of a large language model like ELMo or BERT on the extraction performance. Last, we present an intuitive way to understand the semantic information encoded by contextual embeddings. Contextual embeddings pre-trained on a large clinical corpus achieves new state-of-the-art performances across all concept extraction tasks. The best-performing model outperforms all state-of-the-art methods with respective F1-measures of 90.25, 93.18 (partial), 80.74, and 81.65. We demonstrate the potential of contextual embeddings through the state-of-the-art performance these methods achieve on clinical concept extraction. Additionally, we demonstrate contextual embeddings encode valuable semantic information not accounted for in traditional word representations.

1 Introduction

The study evaluates traditional and contextual embedding methods for clinical concept extraction across four corpora, including domain pre-training and pre-training-time effects. It reports state-of-the-art performance across all four corpora and examines the trade-off between clinical and open-domain pre-training.

  • The study evaluates word2vec, GloVe, fastText, ELMo, and BERT for clinical concept extraction.
  • The analysis spans four clinical concept corpora to assess the generalizability of embedding methods.
  • State-of-the-art results are achieved on all four clinical concept corpora.
  • The study compares pre-training on clinical corpora with pre-training on larger open-domain corpora, highlighting an important trade-off in clinical NLP.
  • The study analyzes how pre-training time affects performance when starting from pre-built open-domain ELMo and BERT models.

2 Background

Traditional word embeddings assign one stable vector to each word, whereas contextual embeddings adjust representations to surrounding meaning. The background also describes clinical concept extraction, contextual models, and related clinical and biomedical applications.

  • 2.1 Word Embedding Models: Word-level methods represent each word with a real-valued vector, while fastText additionally uses character n-grams to handle infrequent words.
  • 2.1 Word Embedding Models: Traditional representations conflate a word’s possible meanings, whereas contextual representations vary the vector according to context.
  • 2.1 Word Embedding Models: ELMo learns context-sensitive multilayer embeddings with a language-model objective during pre-training.
  • 2.1 Word Embedding Models: BERT uses a deep bidirectional transformer with multi-headed self-attention to encode long-distance contextual information.
  • 2.2 Clinical Concept Extraction: Clinical concept extraction identifies medical concepts such as problems, tests, and treatments from clinical notes, typically as a sequence-tagging task.
  • 2.2 Clinical Concept Extraction: Prior work applied contextual embeddings to clinical and biomedical concept extraction, including ELMo, BioBERT, and MIMIC-III-pretrained models.

3 Methods

The methods compare open-domain and MIMIC-III clinical embeddings across traditional and contextual architectures. Static embeddings use Bi-LSTM CRF, while ELMo and BERT provide contextual inputs for sequence labeling.

  • The study compares off-the-shelf open-domain embeddings with clinical-domain embeddings pretrained on MIMIC-III notes.
  • Traditional static embeddings are fed into a Bi-LSTM CRF, with infrequent words mapped to UNK and character embeddings added.
  • ELMo forms context-dependent embeddings from a character CNN and two-layer bidirectional language model before downstream sequence labeling.
  • BERTBASE and BERTLARGE general models receive additional Bi-LSTM layers, while clinical models are pretrained on MIMIC-III from the corresponding checkpoints.

4 Datasets and Experiments

The experiments compare embedding resources and clinical pre-training strategies across four concept-extraction datasets, using standardized downstream evaluation and reporting state-of-the-art results.

  • Datasets: Four shared tasks are evaluated: i2b2 2010, i2b2 2012, SemEval 2014 Task 7, and SemEval 2015 Task 14.The datasets differ in report counts and concept types, including PROBLEM, TEST, TREATMENT, and DISEASE DISORDER.
  • Clinical pre-training: Clinical embeddings are trained on 1,908,359 MIMIC-III notes containing 786,414,528 tokens and a 712,286-word vocabulary.Words are lowercased for traditional embeddings, while casing is preserved for ELMo and BERT pre-training.
  • Downstream model: Concept extraction uses a Bi-LSTM CRF architecture, with shared hyperparameters for traditional embeddings and ELMo and early stopping to limit overfitting.BERT fine-tuning additionally uses Xavier initialization, 800-step early stopping, and post-processing to align outputs with concept annotations.
  • Embedding resources: The study compares off-the-shelf General embeddings with MIMIC embeddings pre-trained on clinical notes, including traditional methods, ELMo, BERT, and BioBERT.BioBERT is a biomedical-literature model initialized from BERTBASE.
  • Evaluation: Evaluation uses precision, recall, and exact-matching F1 on the official test sets, with 10% of each official training set reserved for development.Pre-training costs are reported for ELMo, BERTBASE, and BERTLARGE at every 20,000-step checkpoint.
  • Results: 93.18 partial F1 is achieved by BERTLARGE(MIMIC) on i2b2 2012, exceeding the reported 92.29 partial-F1 state of the art.The reported state-of-the-art result is partial-matching F1 because i2b2 2012 state-of-the-art performance is reported only with that metric.

5 Results

Clinical-domain contextual embeddings achieved the strongest exact-matching F1 results across the evaluated concept-extraction tasks, with BERTLARGE(MIMIC) leading the reported benchmarks. Results also show category-specific gains and that downstream performance can vary with pre-training duration.

  • Clinical-corpus embeddings generally outperform the corresponding open-domain embeddings on the four clinical concept-extraction tasks.
  • 90.25 F1 is achieved by BERTLARGE(MIMIC) on i2b2 2010, improving 5.18 over GloVe(MIMIC)’s 85.07.
  • 80.91 F1 is achieved by BERTLARGE(MIMIC) on i2b2 2012, improving 5.64 over GloVe(MIMIC)’s 75.27.
  • 80.74 exact-matching F1 is achieved by BERTLARGE(MIMIC) on SemEval 2014, while GloVe(MIMIC) exceeds ELMo(General).
  • 81.65 F1 is achieved by BERTLARGE(MIMIC) on SemEval 2015.
  • On i2b2 2010 categories, BERT improves all three labels and makes TREATMENT outperform TEST, unlike ELMo and traditional embeddings.
  • For i2b2 2012 PROBLEM labels, BERTLARGE scores 86.1 versus GloVe’s 77.83, while ELMo leads both BERT models for CLINICAL DEPARTMENT and OCCURRENCE.
  • On i2b2 2010, ELMo reaches maximum F1 87.80 at 280K steps, while BERTBASE reaches 89.55 at 340K and then tends to decrease.

6 Discussion

Clinical-domain contextual embeddings consistently improve concept extraction, with BERT LARGE(MIMIC) reaching state-of-the-art results across all four tasks. Contextual representations also encode context-sensitive semantic information, while BERT wordpiece segmentation remains an open clinical-NLP question.

  • Embedding Performance: Clinical-domain pre-training consistently improves embedding performance, and contextual embeddings outperform traditional word representations.The study reports especially large improvements when a deep language model is pre-trained on a large corpus and then fine-tuned for the task.
  • State-of-the-art Comparison: BERT LARGE(MIMIC) achieved new state-of-the-art performances across all four clinical concept extraction tasks.Its F1-measures were 90.25, 93.18 (partial F1), 80.74, and 81.65.
  • Semantic Information from Contextual Embeddings: ELMo(General) mixes the two meanings of “cold,” whereas BERT representations form clearer temperature and symptom clusters.This clustering pattern is consistent with ELMo(General) performing more poorly on concept extraction than the other evaluated models.
  • Semantic Information from Contextual Embeddings: Contextual embeddings are not well-suited to context-free lexical similarity tasks, but their aggregate clusters preserve spatial relationships among related concepts.In the visualization, pain is nearby while herpes is distant.
  • Lexical Segmentation in BERT: BERT wordpiece tokenization does not always follow traditional medical morphology, motivating further experimentation with clinical tokenization.The paper specifically identifies deriving wordpieces from MIMIC-III as an area requiring additional study.

7 Conclusion

The paper evaluates traditional and contextual word representations for clinical concept extraction across four tasks, comparing clinical pre-training with open-domain models. It finds that clinical unsupervised pre-training yields higher performance and state-of-the-art results across all tasks, while contextual embeddings capture semantic information beyond traditional word representations.

  • 7 Conclusion: The study analyzes different word embedding methods and their effectiveness on four clinical concept extraction tasks.It compares traditional word representations with advanced contextual representations.
  • 7 Conclusion: Clinical pre-training produces higher performance than off-the-shelf open-domain embedding models and achieves state-of-the-art results across all tasks.The comparison focuses on pre-trained contextual embeddings from a large clinical corpus versus open-domain models.
  • 7 Conclusion: Contextual embeddings provide semantic information that traditional word representations do not account for.Figure 3 illustrates this using context-dependent representations of “cold.”
Loading 1902.08691v4…