Source-linked AI summary
Transfer Learning for Named-Entity Recognition with Neural Networks
Ji Young Lee, Franck Dernoncourt, Peter Szolovits
TL;DR
Patient-note de-identification requires labeled data, but access restrictions make annotations slow and expensive. The paper transfers an ANN trained on MIMIC to smaller i2b2 datasets and reports improved performance over state-of-the-art results on both.
Problem
Access restrictions prevent crowd-sourcing patient-note annotations, making large labeled corpora slow and expensive to create.
Method
The approach trains an ANN on the large MIMIC dataset, then fine-tunes the same model on smaller i2b2 2014 or 2016 datasets.
Results
Transfer learning improves performance over state-of-the-art results on both i2b2 2014 and i2b2 2016.
Takeaways & Limitations
Transfer learning may be especially beneficial when the target dataset has few labels.
Abstract
from arXiv · showhide
Recent approaches based on artificial neural networks (ANNs) have shown promising results for named-entity recognition (NER). In order to achieve high performances, ANNs need to be trained on a large labeled dataset. However, labels might be difficult to obtain for the dataset on which the user wants to perform NER: label scarcity is particularly pronounced for patient note de-identification, which is an instance of NER. In this work, we analyze to what extent transfer learning may address this issue. In particular, we demonstrate that transferring an ANN model trained on a large labeled dataset to another dataset with a limited number of labels improves upon the state-of-the-art results on two different datasets for patient note de-identification.
1 Introduction
Patient-note de-identification requires recognizing protected health information, but obtaining annotations is slow and expensive because access to original notes is restricted. The paper studies transfer learning from a large labeled dataset to improve de-identification on smaller datasets.
- 1 Introduction: De-identification removes protected health information from patient notes by treating PHI recognition as a named-entity recognition task.PHI includes names, identification numbers, addresses, and phone numbers.
- 1 Introduction: Restricted access to original patient notes makes large annotated corpora slow and expensive to create, limiting practical ANN deployment.Patient-note annotation cannot be crowd-sourced because only authorized individuals may access the notes.
- 1 Introduction: The paper evaluates whether transfer learning improves de-identification on datasets with limited labels.The model is trained on MIMIC and transferred to the smaller i2b2 2014 and i2b2 2016 datasets.
- 1 Introduction: Transferring an ANN trained on MIMIC to i2b2 2014 and i2b2 2016 outperforms state-of-the-art results on both datasets.
2 Related Work
Prior transfer-learning research spans multiple fields and often initializes target models with source-learned parameters. However, the paper identifies a lack of studies applying ANN transfer learning to named-entity recognition.
- 2 Related Work: Transfer learning uses knowledge learned from a source dataset to perform a task on a target dataset.
- 2 Related Work: In prior ANN studies, parameters learned on the source dataset initialize corresponding parameters in the target model, especially in computer vision.
- 2 Related Work: Before this work, transfer learning for ANN-based natural-language-processing models had received fewer studies, with no identified study focused on NER.Earlier NLP work cited by the paper focused on sentence classification rather than named-entity recognition.
3 Model
The paper uses an LSTM-based ANN for NER that combines token and character representations with sequence-level label optimization. Transfer learning initializes the target model with all or some parameters learned from a source dataset.
- 3 Model: The NER model combines token embeddings, character embeddings, recurrent LSTM layers, a fully connected label-scoring layer, and sequence optimization.The sequence optimizer selects labels using unigram scores and bigram label-transition scores.
- 3 Model: The character LSTM summarizes each token’s character sequence, while the token LSTM processes vectors formed from token and character representations.
- 3 Model: The fully connected layer produces per-token label scores, and sequence optimization outputs the most likely label sequence.
- 3 Model: All layers are jointly learned with stochastic gradient descent, with dropout and development-set early stopping used for regularization.Early stopping uses a patience of 10 epochs.
- 3 Model: Transfer learning trains model parameters on a source dataset, then transfers all or some parameters to initialize training on a target dataset.
4 Experiments
Experiments use MIMIC as the largest labeled source dataset and i2b2 2014 and 2016 as target datasets. They vary target training-set size and the subset of ANN parameters transferred to assess transfer effectiveness and layer importance.
- 4.1 Datasets: The experiments use three de-identification datasets: MIMIC, i2b2 2014, and i2b2 2016.MIMIC serves as the source dataset because it has the most labels, while either i2b2 dataset serves as the target.
- 4.2 Transfer learning: The first experiment varies target train-set size to measure how many labels are needed with and without transfer learning.
- 4.2 Transfer learning: The code extends the NeuroNER NER library used for the experiments.
- 4.2 Transfer learning: The second experiment transfers different parameter combinations to identify which ANN components are most important for transfer learning.It compares transfers beginning with lower layers because lower layers tend to represent more task-independent features.
5 Results
Transfer learning improves de-identification F1-scores, especially when the target dataset has few training samples. Transferring lower ANN layers captures most of the benefit, while transferring all layers does not hurt performance.
- Experiment 1: Transfer learning improves F1-scores over target-only training, with larger gains when the target train set is small.The comparison uses target-only training versus source-dataset pretraining followed by target fine-tuning.
- Experiment 1: 3.1 percentage points: on i2b2 2014 with 5% of the dataset for training, transfer learning raises F1-score from 90.12 to 93.21.The 5% train set contains around 2k PHI tokens out of 50k tokens.
- Experiment 1: 0.17 percentage point: with the full i2b2 2014 train set, transfer learning still raises F1-score from 97.80 to 97.97.
- Experiment 2: Transferring a few lower ANN layers is almost as efficient as transferring all layers, with improvements leveling off after the token or character LSTM.For i2b2 2014, gains are strongest up to the token LSTM; for i2b2 2016, they are strongest up to the character LSTM.
- Experiment 2: Lower-layer parameters contain much of the information relevant to de-identification, while adding topmost layers does not hurt performance.This supports the view that higher ANN layers are more specific to the source task and dataset.
6 Conclusion
The paper studies transferring ANN parameters from a large labeled dataset to smaller de-identification datasets. It reports improved performance over state-of-the-art results on two datasets, with particular benefit when target labels are scarce.
- Transfer learning improves performance over state-of-the-art results on two patient note de-identification datasets.
- Transfer learning may be especially beneficial when the target dataset has a small number of labels.