PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review

Irene Li, Jessica Pan, Jeremy Goldwasser, Neha Verma, Wai Pan Wong, Muhammed Yavuz Nuzumlalı, Benjamin Rosand, Yixin Li, Matthew Zhang, David Chang, R. Andrew Taylor, Harlan M. Krumholz, Dragomir Radev

arXiv:2107.02975v1cs.CLcs.AI

TL;DR

EHRs are ubiquitous and clinically important, yet much of their unstructured text remains difficult to process and underused for secondary purposes. This survey synthesizes neural NLP methods across diverse EHR tasks, finding broad task-level progress while noting persistent data, privacy, interpretability, and domain-transfer limitations.

  • Problem

    About 80% of EHR data is unstructured text that remains difficult to process for secondary use, while privacy and annotation constraints complicate neural NLP.

  • Method

    The survey reviews neural NLP approaches for unstructured EHR text across classification, prediction, representation learning, extraction, generation, and related applications.

  • Results

    The reviewed studies cover broad EHR-NLP tasks, with many reported papers achieving performance at or near state of the art for their tasks.

  • Takeaways & Limitations

    The survey organizes recent methods, datasets, and tools to support research on neural NLP for unstructured EHR data.

  • Takeaways & Limitations

    Biomedical applications remain constrained by limited domain-specific textual data and limited interpretability of deep-learning methods.

Abstract

from arXiv · show

Electronic health records (EHRs), digital collections of patient healthcare events and observations, are ubiquitous in medicine and critical to healthcare delivery, operations, and research. Despite this central role, EHRs are notoriously difficult to process automatically. Well over half of the information stored within EHRs is in the form of unstructured text (e.g. provider notes, operation reports) and remains largely untapped for secondary use. Recently, however, newer neural network and deep learning approaches to Natural Language Processing (NLP) have made considerable advances, outperforming traditional statistical and rule-based systems on a variety of tasks. In this survey paper, we summarize current neural NLP methods for EHR applications. We focus on a broad scope of tasks, namely, classification and prediction, word embeddings, extraction, generation, and other topics such as question answering, phenotyping, knowledge graphs, medical dialogue, multilinguality, interpretability, etc.

1 Introduction

EHRs are central to healthcare but contain large amounts of difficult-to-process unstructured text. This survey reviews neural NLP methods across broad EHR applications while highlighting privacy, annotation, and interpretability challenges.

  • Unstructured text, including clinical notes and discharge summaries, represents about 80% of EHR data but remains difficult to process for secondary use.
  • Neural and deep-learning methods have outperformed traditional rule-based and statistical methods on many NLP tasks.
  • EHR NLP faces privacy constraints, limited labeled data, annotation difficulty, and interpretability challenges.
  • The survey provides broader coverage of EHR clinical-text applications and includes recent BERT-based models, alongside relevant datasets and tools.
  • The survey covers classification, prediction, representation learning, extraction, generation, question answering, phenotyping, knowledge graphs, and related applications.

2 Preliminaries

The preliminaries introduce core deep-learning and NLP concepts needed to understand neural approaches to clinical text. They cover architectures, word embeddings, Transformers, and transfer learning.

  • The survey introduces deep-learning architectures, word embeddings, Transformer models, and transfer learning as NLP preliminaries.

2.1 Basic Architectures

The section surveys basic neural architectures used in NLP, from representation-learning autoencoders and feature-learning CNNs to sequential RNNs and encoder-decoder models.

  • Autoencoders encode inputs into lower-dimensional representations and decode them to reconstruct the original data while preserving salient features.
  • Denoising autoencoders reconstruct uncorrupted inputs after randomly corrupting values, encouraging robust feature-dependency learning and reducing overfitting.
  • Variational autoencoders represent samples with probability distributions, improving latent-space robustness and enabling generation by sampling encodings.
  • Stacked autoencoders chain successive autoencoder units, with each layer reconstructing the preceding layer’s encoding.
  • CNNs learn progressively higher-level features by convolving input matrices with learned smaller filters.
  • RNNs process sequential inputs at discrete time steps using shared weights, making them suitable for variable-length NLP sequences.
  • LSTM, BiLSTM, and GRU cells use gates to control information flow and retain memory across multiple time steps.
  • Sequence-to-sequence models encode an entire input sequence before decoding an output sequence one token at a time.

2.2 Word Embeddings

Neural word-embedding methods replace sparse, semantics-blind token representations with dense vectors that capture contextual or global co-occurrence information, including document-level representations.

  • One-hot word vectors ignore semantic relationships and do not scale efficiently as vocabulary size grows.
  • Word2vec introduced skip-gram and CBOW models that learn word representations from surrounding context and support semantic analogies.
  • Other embedding methods learn from global word co-occurrence statistics, while doc2vec produces fixed-length embeddings for sentences, paragraphs, and documents.

2.3 Language Models

Language models address the limits of fixed word embeddings by learning representations from context. Transformer-based models, including GPT, BERT, RoBERTa, T5, and BART, extend this approach for efficient encoding, classification, and generation.

  • Language models create contextualized embeddings by predicting each word from the sequence preceding it.This lets embeddings reflect prior context rather than assigning one fixed representation to every occurrence.
  • ELMo introduced bidirectional LSTMs that incorporate context from both directions into each word embedding.Each embedding factors in every other word in the input, not only preceding words.
  • Transformers use self-attention to represent all input tokens in parallel rather than encoding them sequentially with an RNN.The architecture trains faster and yields better results than traditional attention-based seq2seq models.
  • GPT models are stacks of Transformer decoders that generate successive tokens using previously decoded tokens.GPT, GPT-2, and GPT-3 use this decoder-based language-model framework to produce word embeddings.
  • BERT combines bidirectional training with Transformer encoders and can be pretrained on raw text before task-specific fine-tuning.At publication, BERT had achieved state-of-the-art performance on 11 NLP tasks.
  • RoBERTa retains BERT’s architecture while changing training data, duration, masking, and sequence length; T5 and BART add decoders for generation.These variants adapt Transformer-based language modeling to improve performance or support text generation.

2.4 Transfer Learning

Transfer learning uses language-model pretraining to learn from abundant raw text before adapting models to downstream NLP tasks. Its value comes from learned semantic and grammatical representations and access to large unsupervised corpora.

  • Language models support transfer learning by pretraining on raw text and then adapting architectures such as ELMo and BERT to downstream tasks.Pretraining can use very large corpora, including the entirety of English Wikipedia.
  • Transfer learning is useful because language models learn semantics and grammar from unsupervised training data that is widely available.

3 Classification and Prediction

Neural methods support a broad range of EHR classification and prediction tasks, including text classification, word sense disambiguation, medical coding, outcome prediction, and de-identification. Across these applications, performance gains are substantial but task- and setting-dependent, with conventional baselines sometimes remaining competitive.

  • Medical Text Classification: Deep models, including word embeddings and CNNs, can outperform TF-IDF and topic-modeling features for clinical-text disease classification.
  • Word Sense Disambiguation: 96.82% macro accuracy was achieved by deepBioWSD, exceeding selected LSTM, SVM, and BiLSTM baselines that scored below or around 95%.Deeper or attention-based neural models improved average testing accuracy by about 2–4% over basic neural models.
  • Medical Coding: CAML uses convolution and attention to select document segments that trigger multi-label ICD-code predictions while providing explanations.
  • Medical Coding: On top-10 ICD-9 prediction, RNN-based methods outperformed logistic regression, random forest, and feed-forward baselines, whereas logistic regression won on top-50 codes.The contrast leaves room for improving deep-learning methods for multi-label ICD-9 coding.
  • Medical Outcome Prediction: MetaPred improved low-resource clinical risk prediction by about 0.3464 AUCROC and 0.4521 F1 on average over fully supervised models.It transfers knowledge from related risk-prediction tasks before fine-tuning or applying the model to target diseases.
  • Medical Outcome Prediction: Unstructured notes provide useful information for some predictions, such as readmission, but negligible additional value for others, such as mortality.A small group of selected sentences outperformed the entire note set for downstream prediction tasks.

4 Embeddings

Neural embeddings represent medical concepts, visits, and patients for downstream EHR tasks while preserving semantic, temporal, hierarchical, or textual information. Across reviewed approaches, these representations support prediction and retrieval, with several models outperforming earlier baselines.

  • 4.1 Medical Concept Embeddings: Clinical embeddings model biomedical text and medical concepts for downstream EHR applications.Cui2vec maps concepts to CUIs and learns embeddings from co-occurrence statistics using GloVe and word2vec.
  • 4.1 Medical Concept Embeddings: Temporal-event embeddings capture correlations and irregular timing among diagnoses, laboratory tests, and drug administrations.Liu et al. modify Phased LSTM with a time gate to represent heterogeneous temporal events for clinical endpoint prediction.
  • 4.2 Visit Embeddings: Med2vec represents visits and medical concepts as vectors while preserving visit order and concept co-occurrence.Its embeddings outperform previous models for predicting concepts in future visits and estimating current patient severity.
  • 4.2 Visit Embeddings: Hierarchical and multimodal patient representations address limitations of flattened visit concepts and combine information from text, codes, or tabular data.MiME models multilevel EHR relationships, while other systems derive visit representations from raw text or combine Transformer-based code embeddings with pretrained BERT text embeddings.
  • 4.3 Patient Embeddings: Deep Patient learns general-purpose patient embeddings with stacked denoising autoencoders and improves disease-prediction accuracy and F-score by 15% and 54%, respectively, over baselines.It extracts ICD-9 codes, medications, laboratory tests, and concepts from preprocessed EHRs and was evaluated on 76,214 test patients across 78 diseases.
  • 4.3 Patient Embeddings: Text-based and sequence-aware patient embeddings improve prediction outcomes, including a 0.03 F1-score gain and 8.0–13.2% average-precision gains over prior approaches.The reviewed systems learn representations directly from clinical text, model patient history with RNNs and attention, or preserve timing and disease sequences through BERT-style pretraining.

5 Information Extraction

Information extraction in EHRs covers identifying entities, linking them to medical knowledge bases, and extracting relations among clinical concepts. Neural and transfer-learning methods improve performance across these subtasks while addressing domain-specific vocabulary and limited annotations.

  • Information extraction identifies important content in unstructured text through named entity, event, and relation extraction subtasks.
  • Named Entity Recognition: 0.95 F1 was achieved on the i2b2 Medical Extraction Challenge by pretraining medical NER word embeddings on unannotated EHRs.The result improved several points over the previous state of the art.
  • Named Entity Recognition: Cross-specialty NER transfers feature representations and parameters so resource-poor specialties can use knowledge from specialties with more annotated EHRs.The approach targets vocabulary differences and scarce specialty-specific data.
  • Entity Linking: 0.92 MAP was achieved by LATTE on two biomedical entity-linking datasets through attention-based ranking of candidate entities.MEDTYPE additionally combines semantic-type disambiguation, Transformer encoders, and large-scale datasets.
  • Relation and Event Extraction: 0.62 to 0.65 F-measure improvement resulted from jointly modeling entity labeling and relation assignment with BiLSTM-based networks.Other work used CNN-RNN hybrids and BioBERT-based systems for clinical and biomedical relation extraction.
  • Medication Information Extraction: BERT-based medication extraction can compute daily dosage from unstructured medication instructions after identifying names, frequencies, routes, and dosage.Amazon Comprehend Medical also improved negation detection by over 1.5% F1 over a bidirectional LSTM-CRF baseline.

6 Generation

Neural generation methods create synthetic EHR data, clinical text, summaries, and medical reports to reduce workload and address EHR accessibility and confidentiality. Across generation tasks, Transformers, GANs, and reinforcement learning improve selected evaluation measures over stated baselines.

  • EHR Generation: NLG creates novel clinical text from existing documents, while synthetic data can help researchers avoid privacy issues linked to EHR accessibility and confidentiality.
  • Clinical Text Generation: 4.76 BLEU and 0.3306 ROUGE-2 were reported for a Transformer generating MIMIC-III discharge summaries from patient and ICU-stay information.The study compared the vanilla Transformer with GPT-2.
  • EHR Generation: 0.1 maximum F1 improvement over medGAN was reported by medWGAN and medBGAN for synthetic EHR generation, with medBGAN performing best.The models generate high-dimensional multi-label discrete EHR variables such as medications, diagnoses, and procedures.
  • Clinical Text Generation: 64.2% gain on selected clinical metrics followed reinforcement learning with entity-consistency and inference-consistency rewards for radiology report generation.
  • Summarization: Extractive summarization selects source-text chunks, whereas abstractive summarization generates clear language capturing salient concepts or ideas.EHR summarization is intended to condense documents into brief, readable summaries and reduce workload.
  • Abstractive Summarization: Roughly 3-4 ROUGE-point improvement over a pointer-generator baseline resulted when reinforcement learning optimized factual accuracy, linguistic likelihood, and target-summary overlap.The factual score compares generated summaries with variables extracted by the CheXpert labeler.

7 Other Topics

Other EHR NLP topics include question answering, phenotyping, multilingual processing, and retrieval. These studies adapt language models, reformulate tasks, or learn structured representations to handle clinical vocabulary, limited data, and rare phenotypes.

  • Question Answering: Biomedical QA underperforms open-domain QA partly because general-corpus models struggle with highly specific clinical vocabulary and large-scale training data remain limited.
  • Question Answering: 86.40 average ROUGE-1 was achieved by a pointer-generator QA model extracting medication dosage and frequency from medical conversations.The task generates template-based questions such as asking for a medication’s dosage or frequency.
  • Question Answering: 98.6% accuracy was achieved by logistic regression on domain-specific datasets in recognizing question entailment, while neural networks performed best on general-domain datasets.Question entailment relates specific questions to more general answered questions.
  • Phenotyping: Granite produces more distinct phenotypes and better captures rare phenotypes than previous tensor-factorization methods.Its average non-zero diagnosis and medication entries per phenotype were 110 versus 4 for comparators.
  • Multilinguality: 0.983 accuracy versus 0.968 for mBERT was achieved by XLM on Italian ICD-10 classification of short medical notes.
  • Other Applications: A weighted hierarchical rank-fusion approach combining lexical, semantic, BERT, and relevance-feedback runs achieved a 9.2% mean-average-precision gain in challenge rounds 4 and 5.

8 Conclusion and Future Direction

The survey synthesizes neural NLP research for unstructured-text EHR tasks and identifies resources and directions for future work. It highlights limited biomedical text availability and interpretability as continuing challenges.

  • The survey reviews deep NLP studies for downstream EHR tasks, emphasizing unstructured text and covering classification, representation learning, extraction, generation, and related topics.It also discusses question answering, phenotyping, knowledge graphs, and other EHR-NLP areas.
  • Limited domain-specific textual data and poor interpretability remain major challenges for applying deep learning to biomedical NLP.
  • Future directions include mining more knowledge from unstructured data, combining structured and unstructured data, and using transfer or unsupervised learning to address scarce annotations.The survey presents these approaches as potential ways to improve decision making, interpretability, or compensation for limited annotated text.

9 Appendix: Datasets and Tools

The appendix catalogs datasets and software resources supporting biomedical and clinical NLP, spanning EHR text, imaging, question answering, evaluation, and language processing tools. The resources vary in scale, task coverage, language, and modality.

  • Datasets: MIMIC-III and MIMIC-CXR provide de-identified critical-care data, with clinical notes in MIMIC-III and chest-X-ray images paired with 14-label free-text classifications in MIMIC-CXR.MIMIC-III includes notes from over 40,000 patients, while MIMIC-CXR contains over 377,000 radiology images.
  • Datasets: Biomedical language resources include Spanish reports with negation and uncertainty annotations, acronym datasets, MEDLINE, PubMed, and sentence-similarity benchmarks.These resources support domain text processing, terminology handling, literature analysis, and biomedical similarity estimation.
  • Datasets: Benchmark and challenge resources cover biomedical NLP tasks including question answering, semantic indexing, entity linking, abbreviation disambiguation, paraphrase detection, and clinical NLP competitions.Examples include BioASQ, WikiMed, PubMedDS, the Clinical Abbreviation Sense Inventory, CLINIQPARA, and i2b2/n2c2.
  • Datasets: BLUE centralizes ten datasets spanning five biomedical NLP tasks, including sentence similarity, named entity recognition, relation extraction, document classification, and natural language inference.
  • Datasets: Clinical and multimodal datasets support NLI, medical question answering, visual question answering, figure retrieval, and figure-to-text alignment.MedNLI contains 14,049 sentence pairs, MedQuAD contains 47,457 question-answer pairs, and MedICaT contains more than 217,000 medical images with captions and references.
  • Tools: The appendix lists libraries and frameworks for machine learning, NLP modeling, sequence generation, semantic modeling, linguistic preprocessing, and clinical information extraction.Examples include PyTorch, scikit-learn, TensorFlow, AllenNLP, Fairseq, Gensim, NLTK, PyText, SpaCy, Stanford CoreNLP, and Criteria2Query.
Loading 2107.02975v1…