Source-linked AI summary
Structured prediction models for RNN based sequence labeling in clinical text
Abhyuday Jagannatha, Hong Yu
TL;DR
Clinical notes contain medically relevant information, but extracting entities requires structured sequence labeling. This paper evaluates RNN-based CRF models with neural pairwise potentials and approximate skip-chain inference, finding improved performance for the modified models.
Problem
Clinical natural-language records contain medically relevant entities, motivating sequence-labeling methods for extracting information from clinical notes.
Method
The study uses bidirectional RNN feature extractors with CRF structured learning, neural pairwise potentials, and approximate skip-chain inference.
Results
Bi-LSTM-CRF-pair and Approx-Skip-Chain CRF achieve better F-scores than Bi-LSTM and Bi-LSTM-CRF in strict and relaxed evaluations.
Takeaways & Limitations
Pairwise modeling and approximate skip-chain inference improve the performance of Bi-LSTM-CRF models for medical-entity extraction.
Takeaways & Limitations
Approximate inference does not guarantee equal partition functions across marginals of the same sentence.
Abstract
from arXiv · showhide
Sequence labeling is a widely used method for named entity recognition and information extraction from unstructured natural language data. In clinical domain one major application of sequence labeling involves extraction of medical entities such as medication, indication, and side-effects from Electronic Health Record narratives. Sequence labeling in this domain, presents its own set of challenges and objectives. In this work we experimented with various CRF based structured learning models with Recurrent Neural Networks. We extend the previously studied LSTM-CRF models with explicit modeling of pairwise potentials. We also propose an approximate version of skip-chain CRF inference with RNN potentials. We use these methodologies for structured prediction in order to improve the exact phrase detection of various medical entities.
1 Introduction
Clinical natural-language records contain medically relevant information, but extracting exact medical phrases requires models that combine input-pattern recognition with structured label dependencies. This work applies RNN-based structured learning to clinical entity extraction, including explicit pairwise and long-term dependency modeling.
- Clinical notes contain information about adverse drug events, medications, and diagnoses useful for drug efficacy analysis and adverse-effect surveillance.
- RNNs recognize patterns in surrounding input features, whereas CRFs jointly model neighboring label predictions as structured sequences.
- Clinical extraction emphasizes exact medical phrase boundaries because medical entities often have complex nomenclature and rare forms.
- Long-term label dependencies can link an entity’s label to a distant label, which local pairwise constraints in linear-chain CRFs may not adequately model.
- The work evaluates RNN-based structured learning with neural pairwise CRF potentials and approximate skip-chain inference for medical-entity extraction.
2 Related Work
Prior work combines neural networks and structured inference for sequence labeling, including CRF models with neural unary potentials. Biomedical studies address entity extraction using specialized corpora and methods, while this work uses a real-world clinical EHR corpus annotated with medical entities.
- CRFs and LSTMs have established roles in sequence labeling, and recent NLP systems combine them by modeling CRF unary potentials with neural networks.
- Biomedical named-entity studies have used CRFs, LSTMs, and dictionaries to extract protein, gene, adverse-effect, and disease entities.
- This work distinguishes itself by using a real-world Electronic Health Record corpus annotated with multiple medical entity types.
3 Methods
The methods use Bi-RNN feature extractors within baseline, linear-chain CRF, neural pairwise-potential, and approximate skip-chain models. Recurrent computation approximates grouped messages for long-range dependencies, while approximate normalization can yield position-specific partition functions.
- Bi-RNN baseline: The study evaluates three structured-learning methodologies using Bi-RNN feature extractors, with a bidirectional recurrent network as the baseline.The baseline uses no structured inference and predicts label sequences through a softmax output layer.
- Bi-LSTM CRF: The Bi-LSTM-CRF model combines bidirectional RNN representations with linear-chain CRF inference.RNN outputs model unary potentials, while a label-transition matrix models pairwise potentials between adjacent positions.
- Neural pairwise potentials: The pairwise-potential model replaces fixed transition scores with a context-dependent nonlinear neural network.This design conditions pairwise potentials on the current word and context, while retaining the Bi-LSTM-CRF unary-potential calculation.
- Approximate skip-chain CRF: Skip-chain CRFs represent dependencies between nonadjacent labels, but their loopy graphs make exact inference intractable.The proposed approximation groups incoming factor-to-variable messages from the left and right and estimates them with bidirectional LSTM computation.
- Approximate skip-chain CRF: The approximation uses recurrent states to aggregate unary and binary potentials across the sequence, allowing skip edges beyond a fixed local neighborhood.Only one message-passing iteration is used, and LSTM states are maintained throughout the sentence.
- Approximate skip-chain CRF: Approximate inference does not guarantee equal partition functions across different sentence positions.The model therefore normalizes each marginal using its position-specific partition function Z_t and trains with cross-entropy loss.
4 Dataset
The dataset comprises annotated English Electronic Health Records from cancer patients, covering medical events and attributes with context-dependent categories.
- The corpus contains 1,154 English Electronic Health Records from cancer patients annotated by two annotators.
- Medical Events: Medical events include ADE, Drugname, Indication, and Other SSD categories.
- Medical Events: ADE, Indication, and Other SSD share vocabulary and are differentiated by clinical context.
- Attributes: Attributes include Severity, Route, Frequency, Duration, and Dosage, describing symptoms or medication properties.
5 Experiments
The experiments compare a bidirectional RNN baseline and structured-learning variants under controlled model sizes, using exact phrase and word-based evaluation with cross-validation.
- Architecture: All models use 200-dimensional pretrained word embeddings initialized from PubMed, Wikipedia, and unlabeled EHR text.
- Architecture: The bidirectional LSTM hidden size ranges from 200 to 250 to keep trainable parameters comparable across models.
- Evaluation: Figure 1 plots Recall, Precision, and F-score for all four methods, with Strict metrics using phrase-based evaluation.
- Evaluation: Relaxed metrics use word-based evaluation, while exact phrase evaluation requires matching the reference phrase boundary and label.
- Evaluation: Results use ten-fold cross-validation with 20% of each training fold's sentences reserved for validation and early stopping.
6 Results
The results favor structured models that incorporate pairwise or skip-chain dependencies, although exact CRF inference improves precision while reducing recall relative to the Bi-LSTM baseline.
- 0.8210 strict F-score and 0.8632 relaxed F-score were achieved by Skip-Chain CRF, the best-performing model.
- Exact CRF inference improves strict-evaluation precision by 2 to 5 percentage points.
- Bi-LSTM CRF-pair achieved the highest precision for exact-match evaluation.
- Recall for exact CRF models is lower than Bi-LSTM, with a smaller reduction for the Bi-LSTM-pair model.
- Bi-LSTM-CRF-pair and Approx-Skip-Chain improvements over baseline were statistically significant with p < 0.01.
7 Discussion
Structured models improve F-scores across strict and relaxed evaluations, while label rarity and phrase complexity shape recall differences among models.
- Overall Results: Bi-LSTM-CRF-pair and Approx-Skip-Chain CRF achieve better F-scores than Bi-LSTM and Bi-LSTM-CRF in both evaluations.
- Overall Results: Skip-Chain CRF recall is almost equal to or better than the other models, consistent with modeling dependencies between distant labels.
- Label Frequency: Exact CRF models have somewhat lower recall for rare labels, although Route, Indication, and Severity are exceptions.
- Label Complexity: Exact CRF models mislabel around 40% of Indication words as Other SSD, compared with 20% for the baseline.
- Pairwise Modeling: Bi-LSTM-CRF-pair has much better recall than Bi-LSTM-CRF even for sparse labels.
8 Conclusion
Modeling pairwise potentials and approximate skip-chain inference improves Bi-LSTM-CRF performance, supporting structured prediction as a direction for more accurate medical-entity phrase extraction.
- Pairwise-potential modeling and approximate skip-chain inference increase Bi-LSTM-CRF performance.
- The conclusion links improved structured modeling to the paper’s goal of detecting complete medical-entity phrases.
- These structured prediction models are promising directions for improving exact phrase extraction of medical entities.