Source-linked AI summary
Bidirectional Recurrent Neural Networks for Medical Event Detection in Electronic Health Records
Abhyuday Jagannatha, Hong Yu
TL;DR
Extracting medical events and attributes from noisy EHR notes is important for pharmacovigilance and drug surveillance, but fixed-context CRF models are limited by variable contextual dependencies. The paper evaluates LSTM and GRU recurrent networks for this sequence-labeling task and reports significant improvements over CRF baselines, while noting that output-label constraints remain unenforced.
Problem
EHR event extraction supports pharmacovigilance and drug surveillance, but CRF models using fixed context windows cannot readily capture surrounding dependencies that vary across event types.
Method
The study evaluates LSTM and GRU recurrent neural networks for sequence labeling of medication, diagnosis, and adverse drug events in EHR notes.
Results
All RNN models significantly outperform the CRF-context baseline; GRU-document achieves recall 0.8126, precision 0.7938, and F-score 0.8031.
Takeaways & Limitations
RNNs’ ability to learn dependencies over variable context ranges supports extracting relevant medical patterns from noisy EHR text.
Takeaways & Limitations
RNNs do not explicitly enforce constraints or dependencies over output labels, so adding a probabilistic graphical model remains future work.
Abstract
from arXiv · showhide
Sequence labeling for extraction of medical events and their attributes from unstructured text in Electronic Health Record (EHR) notes is a key step towards semantic understanding of EHRs. It has important applications in health informatics including pharmacovigilance and drug surveillance. The state of the art supervised machine learning models in this domain are based on Conditional Random Fields (CRFs) with features calculated from fixed context windows. In this application, we explored various recurrent neural network frameworks and show that they significantly outperformed the CRF models.
1 Introduction
The paper frames clinical event detection in EHR notes as sequence labeling for clinically relevant events and attributes. It motivates recurrent neural networks as a way to address noisy text, variable event patterns, and the fixed-context limitations of conventional models.
- EHR event detection targets medication, diagnosis, and adverse drug events for applications including drug surveillance and efficacy monitoring.
- EHR sequence labeling is difficult because notes are noisy, abbreviated, medically specialized, and contain events with different patterns and prevalences.
- CRF-style models depend on immediate neighborhoods, requiring manually designed context features to capture surrounding dependencies.
- Fixed context windows create a trade-off: short windows omit information, whereas large windows mix vital information with irrelevant words.
- Timely identification of new drug toxicities is presented as an unresolved clinical and public health problem.
- The study evaluates LSTM and GRU recurrent networks on EHR notes for detecting medication, diagnosis, and adverse drug events.
2 Related Work
Prior clinical NLP work includes rule-based, supervised, hybrid, and neural approaches for extracting medical concepts and events. This paper differs by applying recurrent sequence models to actual medical notes while detecting additional events and attributes.
- Existing EHR extraction approaches include domain rules, supervised sequence labeling, and hybrid mappings to external knowledge resources.
- Unlike cited work using abstracts or concept mapping, this study uses an actual medical-note corpus and detects additional events and attributes.
- Clinical named entity recognition has been advanced through shared-task benchmarks and systems using CRFs or structured support vector machines.
- Earlier biomedical sequence-labeling research used convolutional and bidirectional LSTM neural networks for tasks such as part-of-speech tagging and named entity recognition.
3 Dataset
The dataset comprises annotated English EHR notes from cancer patients with hematological malignancies. Its labels cover medication events and disease-related events, whose attributes and distinctions depend on clinical context.
- The corpus contains 780 English EHR notes, 613,593 word tokens, and an average of 786 words per note from cancer patients with hematological malignancy.
- Each note was annotated by at least two annotators, achieving an inter-annotator agreement of 0.93 kappa.
- Medication labels include Drugname, Dosage, Frequency, Duration, and Route, with attributes occurring less frequently than Drugname.
- Disease labels include ADE, Indication, Other SSD, and Severity.
- ADE denotes a drug side effect, Indication a currently diagnosed condition for which medication is prescribed, and Other SSD other signs, symptoms, or diseases.
- Preprocessing removes non-punctuation special characters and splits each note into sentences using regular expressions.
4 Methods
The paper compares recurrent neural network architectures, including bidirectional LSTM and GRU models, with CRF baselines for sequence labeling. The neural models use recurrent context and gated state mechanisms to capture dependencies beyond fixed local windows.
- Recurrent architectures: RNNs feed previous outputs back into the network, making current outputs depend on prior sequence states.This recurrent structure helps retain information from earlier positions for sequence labeling.
- Long Short Term Memory: LSTM uses input, forget, and output gates to regulate cell-state updates and expose selected information as output.The cell state is modified additively through the input and forget gates, supporting long-term dependency learning.
- Long Short Term Memory: The bidirectional LSTM concatenates forward and backward chain outputs before a Softmax layer predicts label probabilities.Words are first mapped to dense vectors, then represented using context from both directions.
- Gated Recurrent Units: GRU updates its activation through a candidate state and update gate, without an output gate controlling exposure.The update gate determines the contributions of the candidate activation and previous cell state.
- The Baseline System: CRF-context adds bag-of-words vectors for preceding and following sentence sections to provide a fixed-window context baseline.The CRF models use BIO tagging, while the RNN comparison uses shared skip-gram embeddings trained on a large biomedical corpus.
5 Experiments and Evaluation Metrics
The experiments compare CRF and recurrent models under sentence-level and document-level settings using note-level cross-validation. Performance is measured with exact phrase matching and micro-averaged precision, recall, and F-score.
- Experimental setup: CRF models operate in no-context and context modes, with the latter adding bag-of-words features from preceding and following sentence sections.The context mode is intended to approximate surrounding context available to bidirectional RNNs.
- Experimental setup: Sentence-level RNNs preserve states within one sentence, whereas document-level RNNs process whole documents to learn cues across sentence boundaries.Document-level states are not re-initialized at each sentence boundary.
- Experimental setup: The models use bidirectional recurrent chains with 100 hidden units per direction and a Softmax output over nine medical labels plus Outside.Both RNN frameworks use tanh hidden activations and categorical cross entropy.
- Evaluation: Micro-averaged precision, recall, and F-score are computed using exact phrase matching of boundaries and all phrase labels.A predicted phrase counts as correct only when its complete span and labels match the reference.
- Evaluation: Confusion-matrix heat maps use reference labels as rows, predicted labels as columns, and cell percentages for words in each reference label.The figure supports comparison of class-specific error patterns across methods.
6 Results
RNN models consistently outperform CRF-context on medical event detection, with GRU-document achieving the strongest overall scores and sentence-level models excelling on selected labels. RNNs also retain an advantage when training data is reduced.
- All RNN models significantly outperform the CRF-context baseline; GRU-document improves recall to 0.8126, precision to 0.7938, and F-score to 0.8031.Relative to the baseline, these correspond to gains of 19%, 2%, and 11%, respectively.
- GRU-document achieves the highest gain for six of nine tags, including indication or diagnosis, route, duration, severity, drug name, and other SSD.Its indication gain is about 0.19, nearly a 50% increase over baseline.
- Sentence-level models outperform document-level RNNs for ADE and Dosage, while GRU-sentence has the highest ADE F-score gain.
- RNN confusion matrices show increased diagonal entries across all ten labels, indicating higher overall system accuracy than the baseline.The Outside column remains the most densely populated and represents words erroneously labeled Outside.
- RNN models perform better than traditional CRF models even with smaller training-data sizes.The evaluation keeps training, development, and test setup fixed while randomly down-sampling only training data.
7 Discussion
RNN gains are associated with improved recall and fewer confusions involving Outside and semantically related medical labels. Context range, model granularity, and dataset size help explain where different architectures perform best.
- Improved recall appears to be the major reason for higher RNN F-scores, accompanied by significant decreases in Outside-column confusions.
- CRF errors primarily involve confusing medical words with Outside and confusing ADE, Indication, and Other SSD labels.A 1% confusion in the Outside row represents about 5000 words.
- RNNs reduce confusion among ADE, Indication, and Other SSD by learning variable-range contextual dependencies that distinguish shared vocabularies.Surrounding context provides discriminative cues when the underlying word or phrase alone is insufficient.
- GRU-document confuses Frequency with an unlabeled word 6.1% of the time, versus 31% for CRF-nocontext and 19% for CRF-context.
- Document models benefit from cross-sentence context for Indication, whereas sentence models perform better on ADE because local cues such as “secondary to” and “caused by” are informative.The document-level setting is difficult with only 905 ADE annotations.
- LSTM-document does not improve over sentence models as much as GRU-document, and reducing its hidden-layer size produces no significant improvement.The authors suggest the 780-document dataset may favor GRU’s smaller parameter count, while noting larger data could change the comparison.
- LSTM and GRU show little performance difference across training-data sizes.
- RNNs outperform CRFs on smaller datasets despite having more parameters, possibly because skip-gram initialization provides a strong embedding estimate.
8 Conclusion
Gated RNNs such as LSTM and GRU support extracting medical events and attributes from noisy EHR text by retaining information across varying dependency ranges. Their output-label constraints remain unenforced, motivating future integration with probabilistic graphical models.
- LSTM and GRU are valuable tools for extracting medical events and attributes from noisy EHR notes.
- Gated RNNs can remember information across different dependency ranges, which suits labels with different contextual dependencies.
- CRF models using hand-crafted features and fixed context windows lose information during medical event extraction.
- RNNs do not explicitly enforce constraints or dependencies over output labels.The paper proposes adding a probabilistic graphical model for structured output prediction as future work.