Source-linked AI summary
De-identification of Patient Notes with Recurrent Neural Networks
Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, Peter Szolovits
TL;DR
De-identifying patient notes is necessary to protect confidentiality but manual annotation is costly and error-prone. This paper introduces an artificial-neural-network system requiring no handcrafted features and reports state-of-the-art performance across two datasets, supporting ANN-based de-identification.
Problem
Patient notes contain investigation-critical information but must have protected health information removed, while manual de-identification is restricted, costly, and error-prone.
Method
The paper introduces an artificial-neural-network de-identification system that learns from token and character embeddings without handcrafted features or rules.
Results
The ANN model outperforms state-of-the-art systems on both the i2b2 2014 and MIMIC de-identification datasets.
Takeaways & Limitations
These findings support using ANNs for patient-note de-identification because they achieve strong performance without handcrafted features.
Takeaways & Limitations
Performance for the sensitive NAME category falls just below 98%, and the authors suggest institution-specific gazetteers as a possible improvement.
Abstract
from arXiv · showhide
Objective: Patient notes in electronic health records (EHRs) may contain critical information for medical investigations. However, the vast majority of medical investigators can only access de-identified notes, in order to protect the confidentiality of patients. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) defines 18 types of protected health information (PHI) that needs to be removed to de-identify patient notes. Manual de-identification is impractical given the size of EHR databases, the limited number of researchers with access to the non-de-identified notes, and the frequent mistakes of human annotators. A reliable automated de-identification system would consequently be of high value. Materials and Methods: We introduce the first de-identification system based on artificial neural networks (ANNs), which requires no handcrafted features or rules, unlike existing systems. We compare the performance of the system with state-of-the-art systems on two datasets: the i2b2 2014 de-identification challenge dataset, which is the largest publicly available de-identification dataset, and the MIMIC de-identification dataset, which we assembled and is twice as large as the i2b2 2014 dataset. Results: Our ANN model outperforms the state-of-the-art systems. It yields an F1-score of 97.85 on the i2b2 2014 dataset, with a recall 97.38 and a precision of 97.32, and an F1-score of 99.23 on the MIMIC de-identification dataset, with a recall 99.25 and a precision of 99.06. Conclusion: Our findings support the use of ANNs for de-identification of patient notes, as they show better performance than previously published systems while requiring no feature engineering.
1 Introduction and related work
Patient notes contain investigation-relevant information but must be de-identified to remove HIPAA-defined protected health information before sharing. The paper introduces an ANN-based system that avoids manually curated features and achieves state-of-the-art performance on the i2b2 2014 and MIMIC datasets.
- Motivation: Patient notes contain critical information for medical investigations that may not appear elsewhere in electronic health records.
- Problem: Before notes can be shared, de-identification must remove HIPAA-defined protected health information to preserve patient confidentiality.HIPAA defines 18 types of protected health information in the United States.
- Problem: Manual de-identification is limited because access restrictions prevent crowdsourcing and human annotators are prone to mistakes.The task requires labeling PHI in identified patient notes, which restricts who can perform it.
- Related work: Existing automated systems include human-authored rule-based methods and supervised machine-learning methods such as conditional random fields.Rule-based systems commonly use regular expressions and gazetteers, while statistical systems classify words as PHI or non-PHI, sometimes by PHI type.
- Contribution: ANNs automatically learn effective features from token representations, eliminating the need for handcrafted rules or features.Token embeddings can be learned jointly with the ANN parameters and initialized randomly or pretrained from large unlabeled datasets.
- Contribution: ANNs achieve state-of-the-art de-identification results on the i2b2 2014 challenge and MIMIC patient-note datasets without manually curated features.Unlike other machine-learning systems, the approach does not rely on features based on regular expressions or gazetteers.
2 Methods and materials
The study develops a CRF de-identifier as a challenging baseline and an ANN de-identifier built around LSTM-based recurrent neural networks. The ANN combines character-enhanced token embeddings with label prediction and sequence optimization to assign PHI or non-PHI labels.
- 2.1 CRF model: The CRF baseline tokenizes patient notes, extracts n-gram, morphological, orthographic, and gazetteer features, and uses four surrounding tokens as context.Its parameters are optimized to maximize the likelihood of gold-standard labels before predicting labels during testing.
- 2.2 ANN model: The ANN de-identifier uses LSTM recurrent neural networks and comprises character-enhanced token embedding, label prediction, and label sequence optimization layers.The embedding layer represents tokens, the prediction layer produces label probabilities, and the optimization layer outputs the most likely label sequence.
- 2.2.2 Character-enhanced token embedding: The character-enhanced embedding concatenates a direct token embedding with a character-based embedding generated by a bidirectional LSTM.Character-based embeddings capture sub-token patterns such as morphemes and roots, helping represent out-of-vocabulary tokens and different surface forms.
- 2.2.3 Label prediction layer: The label prediction layer applies a bidirectional LSTM and feed-forward network to produce a probability vector for each token.Labels correspond to PHI types or non-PHI; predicting all 18 HIPAA-defined PHI types therefore uses 19 labels.
- 2.2.4 Label sequence optimization layer: The label sequence optimization layer converts the token-level probability vectors into an output sequence of assigned labels.This layer selects the label sequence after the prediction layer has produced probabilities for the tokens.
3 Experiments and results
The ANN de-identification model was evaluated against CRF and challenge systems on i2b2 2014 and MIMIC datasets using precision, recall, and F1-score. It generally outperformed CRF, with gains depending on PHI category and training-set size, while character-enhanced embeddings supported generalization without handcrafted features.
- Datasets and evaluation: The experiments used i2b2 2014 for comparison with state-of-the-art systems and a newly assembled MIMIC de-identification dataset, with official i2b2 and randomly selected MIMIC test sets.The official i2b2 test set comprised 40% of that dataset, while 20% of MIMIC was randomly selected for testing.
- Main results: The ANN model achieved higher F1-score and recall than the CRF model on both datasets and exceeded the best i2b2 challenge system on i2b2.Results used the official i2b2 evaluation script and binary token-based metrics for HIPAA-defined PHI.
- Main results: On MIMIC, combining ANN and CRF improved recall but reduced precision and did not improve F1-score, while MIST was more competitive than on i2b2.Statistical testing found significant CRF–ANN differences for MIMIC recall and F1-score, but not precision; on i2b2, only recall was significant.
- Category-level results: ANN outperformed CRF across all PHI categories on both datasets except i2b2 ID, where CRF benefited from regular-expression features tailored to ID patterns.ANN significantly outperformed CRF for PROFESSION, while manually assembled gazetteers performed worse than ANN.
- Training-set size: ANN initially lagged CRF with very limited training data but increasingly outperformed it as training-set size grew.Combining ANN and CRF improved F1-score only for small training sets and became less competitive than ANN alone with more data.
- Ablation study: Removing the label-sequence optimization layer, pre-trained token embeddings, or token embeddings slightly reduced performance, whereas character embeddings alone still produced strong results.The ablation study evaluated four model variations by eliminating elements one at a time.
4 Conclusions
The proposed ANN system is the first neural approach for patient-note de-identification and outperforms CRF-based state-of-the-art systems on two datasets without handcrafted features. Its deployment results are encouraging, although NAME-category performance remains just below 98% F1 and may benefit from institution-specific gazetteers.
- 4 Conclusions: The ANN system is the first neural patient-note de-identification system, outperforming CRF-based state-of-the-art systems on two datasets without handcrafted features.It automatically learns effective features using token and character embeddings.
- 4 Conclusions: Over 99% recall on MIMIC discharge summaries supports the ANN system’s promise as an off-the-shelf de-identification system.However, its NAME-category F1-score falls just below 98%.
- 4 Conclusions: Adding gazetteer features from local patient and staff censuses is proposed as future work to improve NAME-category performance.The proposed gazetteers would use the deploying institution’s local census data.
Funding
The project was supported by Philips Research, although the content reflects the authors’ responsibility and not necessarily Philips Research’s official views.
- Funding: The project was supported by Philips Research, with the authors solely responsible for the content.The authors note that the content does not necessarily represent Philips Research’s official views.