Source-linked AI summary
Multi-domain Clinical Natural Language Processing with MedCAT: the Medical Concept Annotation Toolkit
Zeljko Kraljevic, Thomas Searle, Anthony Shek, Lukasz Roguski, Kawsar Noor, Daniel Bean, Aurelie Mascio, Leilei Zhu, Amos A Folarin, Angus Roberts, Rebecca Bendayan, Mark P Richardson, Robert Stewart, Anoop D Shah, Wai Keong Wong, Zina Ibrahim, James T Teo, Richard JB Dobson
TL;DR
Unstructured EHR text is difficult to analyse, while biomedical concept extraction faces scarce labelled data and challenges with ambiguity and contextualisation. MedCAT addresses these needs with an open-source, self-supervised and customisable toolkit integrated with CogStack. The paper reports effective information extraction, deployment across hospitals, and support for clinical research and care-delivery use cases, while noting dependence on concept-database quality and context similarity.
Problem
Biomedical EHR extraction is challenging because clinical information is often unstructured, labelled data are scarce, and concepts require disambiguation and contextualisation.
Method
MedCAT combines self-supervised and supervised concept extraction with ontology linking, clinician annotation and customisation, contextualisation, and CogStack integration.
Results
MedCAT demonstrated improved performance over prior solutions on open-access datasets and was deployed across NHS hospitals for clinical information-extraction use cases.
Takeaways & Limitations
MedCAT makes information in existing clinical text accessible for clinical research and potential care-delivery improvements, including alerting, stratification, trial recruitment, and coding.
Takeaways & Limitations
The approach depends strongly on concept-database quality, and disambiguation can be difficult when different concepts have similar contexts.
Abstract
from arXiv · showhide
Electronic health records (EHR) contain large volumes of unstructured text, requiring the application of Information Extraction (IE) technologies to enable clinical analysis. We present the open-source Medical Concept Annotation Toolkit (MedCAT) that provides: a) a novel self-supervised machine learning algorithm for extracting concepts using any concept vocabulary including UMLS/SNOMED-CT; b) a feature-rich annotation interface for customising and training IE models; and c) integrations to the broader CogStack ecosystem for vendor-agnostic health system deployment. We show improved performance in extracting UMLS concepts from open datasets (F1:0.448-0.738 vs 0.429-0.650). Further real-world validation demonstrates SNOMED-CT extraction at 3 large London hospitals with self-supervised training over ~8.8B words from ~17M clinical records and further fine-tuning with ~6K clinician annotated examples. We show strong transferability (F1 > 0.94) between hospitals, datasets, and concept types indicating cross-domain EHR-agnostic utility for accelerated clinical and research use cases.
1. Introduction
MedCAT addresses the difficulty of extracting and contextualising biomedical concepts from largely unstructured EHR text, where labelled data are scarce and existing tools leave important requirements unmet. It combines open-source extraction, linking, contextualisation, clinician training, and health-system integration.
- EHR systems store substantial clinical information in unstructured or proprietary formats, making it difficult to manipulate, extract, and analyse.
- MedCAT is an open-source NER+L and contextualisation library with an annotation and online-learning interface plus CogStack integration for health-system deployment.
- 1.1. Problem Definition: Deep-learning NER models improve accuracy but require supervised training, while biomedical annotation is costly because clinical text is sensitive and expert-labelled data are scarce.
- 1.2. NER +L in a Biomedical Context: Dictionary-based biomedical extraction avoids manual annotations but must handle spelling variation, form variability, and ambiguous abbreviations or concepts.
- 1.3. Existing Biomedical NER +L Tools: Prior biomedical tools differ in supervision, linking, contextualisation, ontology support, customisation, and deployment constraints.
- 1.3. Existing Biomedical NER +L Tools: MedCAT supports clinician-driven contextualisation of linked concepts, addressing shortcomings that prior tools only partially or do not address.
2. Methods
The methods describe MedCAT as a modular platform for developing information-extraction pipelines through self-supervised and supervised learning, with vocabulary resources, concept databases, and trainable disambiguation. Its workflow combines terminology selection, self-supervised training, clinician annotation, and deployment-oriented integration.
- MedCAT provides decoupled technologies for developing information-extraction pipelines across varied health-informatics use cases.
- The platform’s methods cover biomedical concept extraction and contextualisation through self-supervised and supervised learning, alongside CogStack integration and real-world evaluation.
- MedCAT NER+L relies on a Vocabulary for document words and a Concept Database containing biomedical concept IDs and names.
- The Concept Database can represent one concept ID through multiple synonymous names, such as heart failure, myocardial failure, weak heart, and cardiac failure.
- The initial pipeline cleans and spell-checks text, then tokenizes and lemmatizes it before dictionary-based candidate detection with a moving expanding window.
- Spell correction checks words against the Vocabulary but corrects them only against the Concept Database, while abbreviations are never corrected.
1. Given a document d1
MedCAT identifies concept candidates with an expanding dictionary-matching window, then learns and applies contextual representations to recognize and disambiguate biomedical concepts.
- 1. Given a document d1: The expanding window scans token sequences, marking dictionary concepts, continuing longer matches, or resetting and advancing when no match is found.MedCAT can ignore token order for up to two tokens, excluding stopwords from that limit.
- 1. Given a document d1: Steps 3 and 4 address overlapping entities by expanding candidate windows and evaluating successive matches.
- 2.2. Self-Supervised Training Procedure: MedCAT initially learns context from unambiguous concept mentions, then compares ambiguous candidates with learned context embeddings for annotation and linking.Similarity above a threshold triggers annotation and linking, while similarity also provides an annotation confidence score.
- 2.2. Self-Supervised Training Procedure: Self-supervised training updates concept embeddings toward observed contexts, while cosine-similarity scaling favors newer contexts over frequently observed past contexts.Negative sampling uses K=2s randomly selected vocabulary words to reduce domination by frequent words.
- 2.2. Self-Supervised Training Procedure: Supervised training instead updates the concept embedding using human-provided correct concepts, removing the requirement for a unique concept name.
3. Results
MedCAT outperformed prior tools on open UMLS concept-extraction benchmarks and showed strong transferability across hospitals, datasets, concepts, and clinical domains. Its performance benefited from domain-specific self-supervised and supervised training, while contextualisation results were difficult to compare across sites because tasks and concepts differed.
- Entity Extraction and Linking: MedCAT improved performance over prior tools across tested metrics, except precision against supervised ScispaCy/CLAMP models.The comparison used consistent precision, recall, and F1 calculations for UMLS concept extraction.
- Entity Extraction and Linking: On ShARe/CLEF, MedCAT improved F1 by 9 percentage points over the next-best system.ShARe/CLEF produced the best performance across the evaluated tools.
- Entity Extraction and Linking: MedCAT’s Word2Vec configuration performed better on average than its Bio ClinicalBERT embedding configuration.The BERT variant used pretrained static word embeddings rather than the full BERT model for context representations.
- Entity Extraction and Linking Model Transferability: Model transfer from KCH initially reduced F1 by 0.044 at UCLH and 0.062 at SLaM, but subsequent self-supervised and supervised training largely restored KCH-level performance.At UCLH, 499 supervised annotations delivered comparable performance to the KCH-trained model; at SLaM, successive training improved all measures.
- Contextualisation Model Performance: 0.892-0.977 weighted F1 and 0.841-0.860 macro F1 were achieved across clinical contextualisation tasks and sites.The reported macro and weighted scores reflect variation from unbalanced datasets across most tasks.
- Entity Extraction and Linking Model Transferability: Cross-concept transfer achieved 0.82-0.85 performance, while cross-site fine-tuning improved Status classification by +0.08 macro and +0.09 weighted F1.The cross-concept result suggests disorder-independent representations for the Diagnosis classification boundary.
4. Discussion
MedCAT improves biomedical concept extraction through self-supervised learning, transfers across hospitals and clinical domains, and supports locally tunable deployment. Its effectiveness depends on vocabulary quality, contextual disambiguation, annotation consistency, and substantial training time.
- Evaluation: F1=0.495 vs 0.464 for disorder detection and F1=0.448 vs. 0.429 for general concept detection on MedMentions using only self-supervised learning.The evaluation compared MedCAT with existing tools handling large biomedical databases.
- Evaluation: F1=0.71 with supervised training exceeded BioBERT+'s F1=0.56 on MedMentions general concept detection.The authors attribute the improvement to MedMentions' long tail of entities with insufficient training data for some methods.
- Scope: MedCAT supports languages beyond tested English when suitable concept dictionaries exist, although this expectation is based on likely rather than demonstrated results.SNOMED-CT has translated vocabularies for Spanish, Dutch, Swedish, and Danish.
- Transferability: Models trained at KCH transferred to UCLH and SLaM with average F1 drops of 0.044 and 0.062, then reached comparable performance after site-specific training.The reported post-training differences were -0.021 at UCLH and -0.002 at SLaM.
- Clinical use: Macro F1 ranged from 0.841-0.860 and weighted F1 from 0.892-0.977 across sites and contextualisation tasks.The datasets were often highly unbalanced, explaining differences between macro and weighted averages; expected generalization was between them.
- Limitations: MedCAT's main limitation is dependence on the concept database when names are missing or rarely appear in text.Large databases such as UMLS can mitigate this through synonyms, acronyms, and variant forms.
- Limitations: Disambiguation becomes difficult when concepts share similar contexts, while supervised contextualisation requires consistently labelled data and detailed annotation guidelines.The paper also reports that at least 30 concept occurrences were needed for disambiguation in its experiments.
- Future work: Self-supervised training over approximately 20 years of KCH EHR data took over two weeks to complete.The authors suggest parallelization and model sharing as future improvements.
5. Conclusions
The paper presents MedCAT as an open-source, ontology-flexible clinical NLP toolkit embedded in the CogStack ecosystem. It combines concept extraction, contextualisation, annotation, reuse, and deployment support for clinical research and care-related applications.
- Contribution: MedCAT is a multi-domain clinical NLP toolkit within the open-source CogStack ecosystem.The toolkit is designed for clinical information extraction across health informatics use cases.
- Method: MedCAT leverages biomedical concept databases such as UMLS and SNOMED-CT, including equivalent names, abbreviations, and synonyms.Its method relies on at least one concept name being unique in a sufficiently large corpus.
- Results: MedCAT demonstrates improved performance over prior solutions on open-access biomedical datasets.The broader ecosystem supports transfer, validation, reuse, and fine-tuning across sites, domains, and concept vocabularies.
- Applications: MedCAT is deployed in UK hospitals for real-time alerting, patient stratification, clinical trial recruitment, and clinical coding.The paper positions these deployments as ways to leverage information captured in existing clinical text.
Data Availability
The paper provides reproduction materials for open-access experiments and code for MedCAT and its CogStack integrations. Patient-level clinical records remain unavailable because of confidentiality and legal constraints.
- Data: Reproduction data for core NER+L experiments include MedMentions, ShARe/CLEF 2014 Task 2, and MIMIC-III.These datasets are freely available through their data owners.
- Access constraints: Patient-level free-text data cannot be made available because of confidentiality and reidentification risks.De-identified aggregate access may be discussed subject to legal permissions.
- Code: Code is available for the experiments, toolkit, and wider CogStack deployment integrations.The listed resources include MedCAT tutorials, MedCATtrainer, MedCATservice, and CogStack repositories.
- Disclosure: The paper reports research funding, industry support, and shareholdings for JTHT.The disclosure names InnovateUK, Bristol-Myers-Squibb, iRhythm Technologies, Glaxo Smithkline, and Biogen.
Appendix C.1. Self-Supervised Training Configuration
MedCAT's self-supervised configuration combines spelling correction, long- and short-context embeddings, and a similarity threshold for recognition and linking.
- Preprocessing: Misspelled words are corrected only within one edit for words under six characters and two edits for longer words.This configuration limits automatic correction by word length and edit distance.
- Embeddings: Each concept uses the average of long and short embeddings, covering 9 and 2 words on each side respectively.The context-window sizes were selected by testing combinations from 0 to 10.
- Recognition: A concept candidate is recognized and linked when its Equation 2 similarity exceeds 0.3 by default.The threshold applies to a candidate concept or sequence of words unless otherwise specified.
Appendix C.2. Qualitative Analysis Training Configura-
MedCAT was trained self-supervised on the full UMLS vocabulary and 2.4 million MIMIC-III clinical notes using a small one-core server.
- 3.82 million concepts from 207 vocabularies were used to train MedCAT self-supervised on MIMIC-III.The training used the entirety of UMLS.
- 2.4 million clinical notes were processed, including nursing notes, clinician notes, and discharge reports.
- Approximately 30 hours of training were completed on a small one-core server.
Appendix D. Contextualisation Task Results Per Site
This appendix reports contextualisation results at KCH and describes the meta-annotation tasks used to evaluate extracted concepts. Figure B.4 provides three MedMentions text extracts for disambiguation training and testing.
- KCH: KCH aggregate performance was evaluated over all extracted concepts listed in Appendix A.The appendix defines multiple meta-annotation tasks for contextualising concepts.
- KCH: Presence classifies each concept as affirmed, negated, or hypothetical.
- KCH: Experiencer classifies whether the concept is experienced by the patient, family, or another person.
- KCH: Temporality classifies whether the concept refers to the past, recent period, or future.
- Disambiguation examples: Figure B.4 shows three MedMentions extracts used to assess training-sample requirements for disambiguation.The extracts include two training cases and one disambiguation-performance test case.
Appendix D.2. Meta Annotation Results Breakdown for SLaM
The SLaM appendix aggregates meta-annotation performance across all extracted concepts and defines status and diagnosis tasks.
- SLaM performance is aggregated over all extracted concepts listed in Appendix A.
- Status classifies whether a concept is affirmed to affect the patient or affects someone else, with an NA category.
- Diagnosis classifies whether a concept is a diagnosis related to the patient.
Appendix D.3. Meta Annotation Results Breakdown for UCLH
The UCLH appendix aggregates meta-annotation performance across extracted concepts and defines five contextual classification tasks: negation, experiencer, problem temporality, certainty, and irrelevance.
- UCLH performance is aggregated over all extracted concepts listed in Appendix A.
- Negation classifies whether a concept is negated, using Yes and No values.
- Experiencer classifies whether the concept is experienced by the patient or someone else.
- Problem Temporality distinguishes historical mentions from current problems.
- Certainty classifies whether a concept is confirmed to be present or suspected.
- Irrelevant classifies whether a concept is relevant.