Source-linked AI summary
Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Review
Seyedmostafa Sheikhalishahi, Riccardo Miotto, Joel T Dudley, Alberto Lavelli, Fabio Rinaldi, Venet Osmani
TL;DR
Chronic-disease clinical records contain valuable information, much of it locked in difficult-to-model free-text narratives, while prior reviews had not focused specifically on chronic diseases. This systematic review analyzed clinical NLP literature across 43 chronic diseases and found increasing use of machine learning, continued reliance on relatively simple methods, and scarce public data.
Problem
Clinical records are difficult to represent because they are high-dimensional, noisy, heterogeneous, sparse, incomplete, and biased, while clinical history is often locked in free-text narratives; prior reviews had not focused specifically on chronic diseases.
Method
The review followed PRISMA guidelines and searched several databases for English-language journal articles on clinical notes, NLP or information extraction, and chronic diseases published from January 1, 2007, to February 6, 2018.
Results
106 articles covering 43 chronic diseases were included; machine-learning approaches were frequently used, 16 papers employed hybrid methods, and most studies used nonpublic datasets.
Takeaways & Limitations
Future work should move clinical NLP from extraction toward understanding, recognize relations and temporal events, exploit alternative knowledge sources, and develop large-scale de-identified clinical corpora.
Takeaways & Limitations
The review was limited to English-language journal articles and excluded clinical articles from non-EHR systems, potentially omitting relevant results and introducing domain-related method bias.
Abstract
from arXiv · showhide
Of the 2652 articles considered, 106 met the inclusion criteria. Review of the included papers resulted in identification of 43 chronic diseases, which were then further classified into 10 disease categories using ICD-10. The majority of studies focused on diseases of the circulatory system (n=38) while endocrine and metabolic diseases were fewest (n=14). This was due to the structure of clinical records related to metabolic diseases, which typically contain much more structured data, compared with medical records for diseases of the circulatory system, which focus more on unstructured data and consequently have seen a stronger focus of NLP. The review has shown that there is a significant increase in the use of machine learning methods compared to rule-based approaches; however, deep learning methods remain emergent (n=3). Consequently, the majority of works focus on classification of disease phenotype with only a handful of papers addressing extraction of comorbidities from the free text or integration of clinical notes with structured data. There is a notable use of relatively simple methods, such as shallow classifiers (or combination with rule-based methods), due to the interpretability of predictions, which still represents a significant issue for more complex methods. Finally, scarcity of publicly available data may also have contributed to insufficient development of more advanced methods, such as extraction of word embeddings from clinical notes. Further efforts are still required to improve (1) progression of clinical NLP methods from extraction toward understanding; (2) recognition of relations among entities rather than entities in isolation; (3) temporal extraction to understand past, current, and future clinical events; (4) exploitation of alternative sources of clinical knowledge; and (5) availability of large-scale, de-identified clinical corpora.
Overview
Chronic diseases create a major healthcare challenge, motivating secondary use of EHRs and NLP to transform narrative records into structured, analyzable data. This review addresses a gap in chronic-disease-specific clinical NLP evidence and future directions.
- Overview: Chronic diseases impose substantial healthcare and societal burdens, including effects on quality of life and care costs.
- Overview: EHR analysis can improve understanding of clinical trajectories, patient stratification, risk prediction, and clinical decision making.
- Overview: NLP is needed because clinically important information is often locked in free-text narratives amid noisy, heterogeneous, and incomplete EHR data.
- Overview: Manual review of clinical notes creates scalability and cost problems, especially when notes dominate structured data for chronic diseases.
- Overview: This review specifically examines NLP challenges across 43 chronic diseases and proposes directions including understanding, relation recognition, temporal extraction, alternative knowledge sources, and larger corpora.
- Overview: The search covered English-language journal articles from January 1, 2007, to February 6, 2018, across several bibliographic databases.
Article Selection
The review narrowed an initially broad search through staged screening, retaining 106 articles and organizing chronic diseases into ICD-10 categories. Circulatory diseases were most represented, whereas metabolic diseases were least represented, reflecting differences in EHR structure.
- Article Selection: 2652 initially retrieved articles were narrowed to 478 after refining the search strategy to focus on the review topic.
- Article Selection: 46 duplicates were removed, leaving 432 articles for title and abstract screening; 159 proceeded to further analysis.
- Article Selection: 106 full-text articles fulfilled the inclusion criteria after independent review by five authors.
- Article Selection: 43 chronic diseases were grouped into 10 categories using ICD-10 to assess NLP coverage across disease types.
- Disease Categorization: 38 papers addressed circulatory diseases, 34 addressed neoplasms, and 14 addressed endocrine, nutritional, and metabolic diseases.
- Disease Categorization: Circulatory diseases were represented more often than metabolic diseases despite the latter’s higher general-population incidence, plausibly because circulatory records contain more unstructured data.
Disease Groups
The reviewed disease-group studies applied NLP to risk estimation, disease identification, clinical indicators, comorbidities, therapies, and heart-failure characterization. Methods included hybrid pipelines, rule-based tools, and standard machine-learning classifiers across clinical notes and EHRs.
- Disease Groups: Heart-disease studies primarily estimated risk using clinical notes, EHRs, and hybrid machine-learning and rule-based pipelines.
- Disease Groups: Stroke and major-bleeding risk in atrial fibrillation was predicted from structured data and clinical notes, while heart failure was identified from clinical notes alone.
- Disease Groups: NLP studies extracted peripheral arterial disease and critical limb ischemia cases, clinical entities, and drug-safety-related complications from clinical notes.
- Disease Groups: Hypertension studies extracted indicators, comorbidities, drug therapies, and numerical blood-pressure values from clinical narratives.
- Disease Groups: Heart-failure work identified Framingham signs and symptoms, extracted left ventricular ejection fraction, and distinguished left- from right-sided failure.
- Disease Groups: Heart-failure NLP also examined ineffective self-management, quality of care, and signs and symptoms in primary-care EHRs.
Neoplasms
Neoplasms were studied through NLP applications spanning cancer detection, tumor-information extraction, disease trajectories, recurrence, staging, and pathology or radiology analysis.
- Neoplasms: Cancer-related studies addressed detection, tumor characteristics, disease trajectories, recurrence, and cancer stage.
- Neoplasms: Methods extracted tumor characteristics, translated pathology sentences into graph representations, detected negation, and recognized entities in pathology and radiology reports.
- Neoplasms: Breast cancer had 8 papers, colorectal cancer had 7, and prostate cancer had 4, making them the three most common cancer types reviewed.
- Neoplasms: Studies used NLP and EHRs to identify colorectal cancer screening needs, colonoscopy concepts and temporal information, and positive prostate-biopsy diagnoses.
- Neoplasms: Additional work extracted predefined concepts from liver-cancer reports and identified confirmed intraductal papillary mucinous neoplasm diagnoses.
Endocrine, Nutritional, and Metabolic Diseases
NLP applications in endocrine, nutritional, and metabolic diseases covered disease identification, negation and family-history detection, temporal tagging, and complication-related findings.
- NLP studies addressed negation detection, family-history mentions, temporal tagging, obesity and diabetes identification, and diabetes complications.
- Two SVMs identified obesity types by extracting obesity- and diabetes-related concepts from clinical text.
- An SVM system identified EHR progress notes pertaining to diabetes, while foot examination findings were used to predict quality of life.
Other Disease Categories
Across other disease categories, reviewed studies applied SVMs, CRFs, lexicons, ontologies, rules, and neural networks to classification, extraction, comorbidity analysis, and concept similarity.
- Method comparisons: 16 papers used hybrid approaches combining rule-based and machine learning methods, including CRFs, SVMs, and manually built lexicons.
- Method comparisons: 24 papers compared rule-based and machine learning methods, typically using rule-based systems as baselines for machine learning algorithms.
- Rule-based methods: Rule-based approaches included dictionary lookup, ontology-based terminology identification, manually defined rules, and regular-expression patterns.
- Machine learning methods: SVMs were the most widely used machine learning approach, followed by naïve Bayes, CRFs, and random forests.
- Deep learning: Only 3 papers used deep learning, applying neural networks to patient representation, psychiatric phenotyping, and disease risk assessment.
Natural Language Processing Tasks, Methods, and Datasets
The reviewed NLP literature was dominated by text classification and entity recognition, and most studies relied on clinical datasets that were not publicly available.
- Tasks: Text classification and entity recognition were the most frequently described NLP tasks.
- Methods: Standard classifiers dominated text classification, including SVM (n=12) and naïve Bayes (n=4).
- Datasets: Most studies used datasets that were not publicly available, while 12 of 16 papers using public corpora exploited i2b2 datasets.
Comparisons to Other Systematic Reviews
This review targets chronic diseases specifically and broadens prior systematic reviews by covering clinical-note methodologies alongside information extraction applications.
- Prior reviews: Earlier reviews covered case detection, clinical NLP tasks, patient-cohort identification, psychiatry, cancer, radiology, or information extraction rather than chronic diseases specifically.
- Prior reviews: The review by Pons et al analyzed practical NLP applications in radiology across five application categories.
- Scope of this review: Compared with previous reviews, this review includes methodologies for analyzing clinical notes and provides broader, more recent coverage of chronic diseases.
Publication Venues
The review’s 106 articles appeared across 50 venues, with publication activity increasing over time and most studies published in medical informatics journals. Findings also show a transition toward machine learning, while advanced methods and broader clinical-text modeling remained limited.
- Publication venues: 106 articles were published in 50 unique venues, most commonly medical informatics journals.Venues were manually grouped into clinical medicine, medical informatics, and computer science.
- Publication trends: Publication numbers increased over the years, except in 2018 because retrieval covered only part of that year.
- Methods: Only 3 papers used deep classifiers or general deep learning methods, while most studies relied on shallow classifiers.The review attributes this partly to deep learning’s emergence and the journal-focused scope of the review.
- Applications: Most papers identified disease risk factors or classified disease phenotypes, whereas few extracted comorbidities or integrated notes with structured data.The review also notes that simple methods retain an interpretability advantage over more complex approaches.
- Advanced methods: Few studies extracted word embeddings from clinical notes, potentially reflecting insufficient training data and the recent development of embedding methods.The review suggests transfer learning and disease-specific precomputed embeddings as possible responses.
- Data availability: Public datasets remained scarce, and reviewed studies typically came from institutions with in-house NLP teams and clinical-data access.The review links this scarcity to the sensitivity of clinical data and associated legal and regulatory issues.
Limitations
The review’s scope was constrained by its English-language, journal-article focus and exclusion of clinical articles from non-EHR systems. It also identifies a need for clinical NLP to move beyond concept extraction toward understanding.
- Scope limitations: The review included only English-language journal articles and excluded papers using clinical articles from non-EHR systems.These scope choices may have omitted additional findings, especially from clinical narratives in other languages.
- Scope limitations: Focusing on the clinical domain may have biased the reviewed method distribution because rule-based methods are more prevalent there than in other domains.
- Future direction: Clinical NLP should progress from extracting clinical concepts toward understanding relationships and incorporating clinical, domain, and general knowledge.
Multimedia Appendix 1
The multimedia appendix provides supplementary materials accompanying the review, including reviewed-paper data, classifications, algorithms, venues, and excluded papers.
- Supplementary material: Multimedia Appendix 1 is provided as a PDF file accompanying the article.
- Contents: The supplementary material contains the complete list of reviewed papers, chronic diseases and classifications, algorithms used, publication venues, and excluded papers.
- Supplementary material: The article also provides a supplementary XLSX file.
- Publication record: The article was published in JMIR Medical Informatics on April 27, 2019, after peer review and revision.
- License: The article is distributed under a Creative Commons Attribution License requiring attribution of the original publication.