Source-linked AI summary
Med7: a transferable clinical natural language processing model for electronic health records
Andrey Kormilitzin, Nemanja Vaci, Qiang Liu, Alejo Nevado-Holgado
TL;DR
Clinical NLP must extract structured medical concepts from abundant but difficult-to-annotate EHR free text. Med7 combines self-supervised pre-training with noisy-label augmentation, active learning, and target-domain fine-tuning; it achieved strong source-domain NER performance, while transfer to CRIS required adaptation.
Problem
Extracting medical concepts from free-text EHRs is important, but high-quality clinical annotations are scarce and models may not generalise across domains.
Method
Med7 uses self-supervised pre-training, weakly supervised noisy labels, active learning, and a spaCy-based NER model for seven medication-related categories.
Results
Med7 achieved micro-averaged F1=0.957 (0.893) for lenient (strict) estimates across seven categories, while transfer evaluation showed the importance of adapting to the target domain.
Takeaways & Limitations
The results support transfer learning and target-domain fine-tuning as important for applying clinical NER across different EHR domains.
Takeaways & Limitations
Duration entities were underrepresented, and the small number of manually annotated OxCRIS examples limited rigorous transferability evaluation across all seven categories.
Abstract
from arXiv · showhide
The field of clinical natural language processing has been advanced significantly since the introduction of deep learning models. The self-supervised representation learning and the transfer learning paradigm became the methods of choice in many natural language processing application, in particular in the settings with the dearth of high quality manually annotated data. Electronic health record systems are ubiquitous and the majority of patients' data are now being collected electronically and in particular in the form of free text. Identification of medical concepts and information extraction is a challenging task, yet important ingredient for parsing unstructured data into structured and tabulated format for downstream analytical tasks. In this work we introduced a named-entity recognition model for clinical natural language processing. The model is trained to recognise seven categories: drug names, route, frequency, dosage, strength, form, duration. The model was first self-supervisedly pre-trained by predicting the next word, using a collection of 2 million free-text patients' records from MIMIC-III corpora and then fine-tuned on the named-entity recognition task. The model achieved a lenient (strict) micro-averaged F1 score of 0.957 (0.893) across all seven categories. Additionally, we evaluated the transferability of the developed model using the data from the Intensive Care Unit in the US to secondary care mental health records (CRIS) in the UK. A direct application of the trained NER model to CRIS data resulted in reduced performance of F1=0.762, however after fine-tuning on a small sample from CRIS, the model achieved a reasonable performance of F1=0.944. This demonstrated that despite a close similarity between the data sets and the NER tasks, it is essential to fine-tune on the target domain data in order to achieve more accurate results.
1. Introduction
Clinical records contain valuable but difficult-to-process free text, motivating robust named-entity recognition and domain adaptation for medical information extraction.
- Clinical free text contains nuanced patient information, but manual processing is expensive and limits its use for data mining.
- NER classifies words into predefined categories and provides a foundational component for extracting medical concepts from clinical records.
- Models trained on one EHR domain may underperform on a related target domain because supervised learning assumes shared source and target distributions.
- Med7 combines self-supervised pre-training on more than 2 million MIMIC-III documents, weakly supervised noisy labels, and active learning with human review.
- Direct transfer from US intensive-care MIMIC data to UK secondary-care mental-health CRIS data failed to generalise well, whereas domain adaptation enabled adequate transfer.
2. Related work
Clinical information extraction has progressed from rule-based and lexicon-driven systems toward statistical, embedding-based, and deep-learning approaches.
- Clinical NLP methods evolved from lookup tables, semantic lexicons, and rule-based systems to hidden Markov models, conditional random fields, and deep learning.
- UMLS and SNOMED CT support transforming free text into coded clinical representations for electronic health records.
- Word embeddings were introduced as numerical representations of textual data and became inputs for later neural NLP methods.
3. Materials and Methods
The study evaluates medication-focused clinical NER across MIMIC-III, n2c2, and UK-CRIS, using self-supervised pre-training, noisy-label augmentation, and standardized metrics.
- 3.1. Data: The evaluation used MIMIC-III and n2c2 medication-extraction data, with UK-CRIS providing an additional secondary-care mental-health domain.UK-CRIS notes underwent cleaning and annotation-offset adjustment before comparison.
- 3.2.2. Self-supervised learning: Self-supervised language-model pre-training initialized the NER model before downstream training, using cloze-style word reconstruction and convolutional neural network layers.The pre-trained model predicted GloVe word vectors with cosine loss and transferred its weights to the NER model.
- 3.2.3. Named entity recognition model: The model extracted medication concepts from clinical text using spaCy’s convolutional NER architecture with hashed Bloom embeddings.The target concepts include drug names, dosages, and administration frequencies.
- 3.2.4. Model training augmentation with bootstrapped noisy labels: Noisy-label augmentation created silver training data from labelling functions based on rules, keyword patterns, and sense2vec-derived terms.Gold and silver examples were combined during model training.
- Evaluation: Performance was measured with strict and lenient precision, recall, and micro- and macro-averaged F1 scores under the SemEval’13 evaluation schema.Strict matching required exact surface strings and labels, whereas lenient matching allowed partial matches.
4. Results
Results show that model accuracy improved as more training data were used, while evaluation focused on performance and transferability across clinical datasets.
- Named-entity recognition model: Accuracy improved steadily as the training fraction increased from 25% to 100%, with the strongest improvement in the final fraction.The authors interpret this trend as indicating advantages from collecting additional data.
- Named-entity recognition model: The model was trained using 1212 documents combining silver examples, n2c2 gold data, and 606 manually annotated MIMIC-III documents.Manual annotation used Prodigy with an active-learning procedure.
- Translation to UK-CRIS data: The study examined generalisability beyond the training distribution by applying the model to records from another clinical setting.The motivation was the documented difficulty of generalising clinical information-extraction systems across medical centres and datasets.
- Translation to UK-CRIS data: F1=0.762 versus F1=0.944 demonstrates reduced performance when the Med7 model was transferred from MIMIC-III to CRIS.The comparison concerns application to CRIS data from Oxford.
5. Discussion
Med7 recognizes seven medication-related clinical concepts and achieves strong overall performance, while transfer to a different clinical domain requires target-domain fine-tuning. The model is interoperable, CPU-compatible, and constrained by underrepresented categories and limited OxCRIS annotations.
- 5. Discussion: F1=0.957 (0.893) across seven categories, with Precision=0.982 (0.916) and Recall=0.933 (0.871) under lenient (strict) estimates.Duration and Frequency performed worse because of sparse, ambiguous, and inconsistently annotated examples.
- 5. Discussion: F1=0.893 for the larger architecture versus F1=0.884 for the default architecture, while larger models increased training time, disk size, and memory consumption.The larger model showed especially visible gains for Duration and Strength.
- 5. Discussion: F1=0.762 without fine-tuning on OxCRIS, compared with F1=0.944 after fine-tuning on a small target-domain sample.The evaluation transferred Med7 from US intensive-care records to UK secondary-care mental-health records.
- 5. Discussion: Med7 interoperates with HuggingFace, Thinc, and spaCy pipelines and runs on standard CPUs rather than requiring expensive GPUs.The model can be integrated with additional components and the Prodigy annotation tool.
- 5. Discussion: Duration entities are intrinsically skewed in medical records, and the small number of manually annotated OxCRIS examples limits rigorous transferability evaluation.The Duration reporting pattern persists in both MIMIC and OxCRIS data.
- 5. Discussion: Future work targets deployment and transferability in the UK-CRIS Network Trust members, as well as structuring free-text patient data for downstream analysis.
6. Conclusion
The conclusion presents Med7 as a self-supervised, actively developed clinical NER model for free-text EHRs that transfers across domains after adaptation and runs on standard CPUs. It also identifies underrepresented concepts and concept normalization as future priorities.
- 6. Conclusion: Med7 combines self-supervised pre-training, n2c2 annotations, noisy labels, and active learning to recognize clinical concepts in free-text EHRs.
- 6. Conclusion: Transfer learning is essential for applying Med7 robustly across different clinical domains, while the model can run on standard CPU-based machines.
- 6. Conclusion: Future releases will address naturally underrepresented concepts, extracted-concept normalization, and UMLS linkage.
Appendix A. The evaluation schema for extracted concepts
The appendix defines a SemEval’13-based evaluation schema that classifies NER errors and computes precision and recall from those error categories. It distinguishes complete, incorrect, partial, missing, and spurious predictions.
- Appendix A. The evaluation schema for extracted concepts: The SemEval’13 schema evaluates NER output using defined error categories and associated performance metrics.
- Appendix A. The evaluation schema for extracted concepts: Correct matches require both annotation-boundary and entity-type agreement, whereas partial matches overlap the gold span without matching it exactly.
- Appendix A. The evaluation schema for extracted concepts: Possible entities equal COR + INC + PAR + MIS, while Actual entities equal COR + INC + PAR + SPU.
- Appendix A. The evaluation schema for extracted concepts: The appendix includes a token-level confusion matrix comparing predicted entity errors with ground-truth labels.
Appendix B. Inter-annotator agreement analysis
The appendix assesses agreement between two annotators and the n2c2 gold annotations using ten randomly sampled documents from the 202-document test set. It reports pairwise comparisons and examines disagreements in concept labeling.
- Appendix B. Inter-annotator agreement analysis: Agreement was estimated from ten randomly sampled documents drawn from the 202-document test set.
- Appendix B. Inter-annotator agreement analysis: Token-level confusion matrices summarize annotator 1 and annotator 2 predictions against the gold-standard annotations.
- Appendix B. Inter-annotator agreement analysis: The analysis compares each annotator with the n2c2 gold annotations and compares the two annotators directly.
- Appendix B. Inter-annotator agreement analysis: The study examined cases where both annotators labeled concepts differently from the n2c2 gold-annotated dataset.
Appendix C. Fine-tuning on UK-CRIS
Appendix C presents token-level confusion matrices for Med7 on 134 manually annotated OxCRIS documents, comparing predictions before and after fine-tuning on OxCRIS.
- The reported comparison contrasts Med7-predicted categories before fine-tuning on OxCRIS with predictions after fine-tuning on OxCRIS.
- The appendix evaluates Med7 on 134 manually annotated documents from OxCRIS, the Oxford instance of the UK-CRIS electronic medical records Network.
- Tables C.14 and C.15 provide token-level confusion matrices for the model trained on MIMIC-III and evaluated on OxCRIS.