Source-linked AI summary
Large Language Models to Identify Social Determinants of Health in Electronic Health Records
Marco Guevara, Shan Chen, Spencer Thomas, Tafadzwa L. Chaunzwa, Idalid Franco, Benjamin Kann, Shalini Moningi, Jack Qian, Madeleine Goldstein, Susan Harper, Hugo JWL Aerts, Guergana K. Savova, Raymond H. Mak, Danielle S. Bitterman
TL;DR
SDoH are important to patient outcomes but incompletely captured in structured EHR data, with relevant information commonly documented in free-text clinic notes. The study evaluates language models and synthetic clinical text for extracting six SDoH categories and assesses algorithmic bias. Fine-tuned Flan-T5 models outperformed zero- and few-shot ChatGPT-family models and identified substantially more patients with adverse SDoH than structured diagnostic codes.
Problem
SDoH are important to patient outcomes but rarely documented comprehensively in structured EHR data, limiting their availability for research and care.
Method
The study annotated 800 clinical notes, trained multilabel BERT and Flan-T5 classifiers, evaluated GPT3.5 and GPT4 with synthetic text, and assessed demographic sensitivity.
Results
Fine-tuned Flan-T5 models outperformed zero- and few-shot ChatGPT-family models and identified 91.8% more patients with adverse SDoH than structured diagnostic codes.
Takeaways & Limitations
These models could improve real-world SDoH evidence and help flag patients who may benefit from proactive resource and social work referral.
Takeaways & Limitations
The predominantly white Boston-based training and validation populations limit generalizability, and synthetic-data generation methods were not exhaustively assessed.
Abstract
from arXiv · showhide
Social determinants of health (SDoH) have an important impact on patient outcomes but are incompletely collected from the electronic health records (EHR). This study researched the ability of large language models to extract SDoH from free text in EHRs, where they are most commonly documented, and explored the role of synthetic clinical text for improving the extraction of these scarcely documented, yet extremely valuable, clinical data. 800 patient notes were annotated for SDoH categories, and several transformer-based models were evaluated. The study also experimented with synthetic data generation and assessed for algorithmic bias. Our best-performing models were fine-tuned Flan-T5 XL (macro-F1 0.71) for any SDoH, and Flan-T5 XXL (macro-F1 0.70). The benefit of augmenting fine-tuning with synthetic data varied across model architecture and size, with smaller Flan-T5 models (base and large) showing the greatest improvements in performance (delta F1 +0.12 to +0.23). Model performance was similar on the in-hospital system dataset but worse on the MIMIC-III dataset. Our best-performing fine-tuned models outperformed zero- and few-shot performance of ChatGPT-family models for both tasks. These fine-tuned models were less likely than ChatGPT to change their prediction when race/ethnicity and gender descriptors were added to the text, suggesting less algorithmic bias (p<0.05). At the patient-level, our models identified 93.8% of patients with adverse SDoH, while ICD-10 codes captured 2.0%. Our method can effectively extracted SDoH information from clinic notes, performing better compare to GPT zero- and few-shot settings. These models could enhance real-world evidence on SDoH and aid in identifying patients needing social support.
INTRODUCTION
SDoH affect health outcomes but are incompletely represented in structured EHR data, with relevant information concentrated in clinic-note free text. The study evaluates large language models, synthetic clinical text, and prediction bias for extracting six SDoH categories.
- SDoH influence access to and quality of medical care and may contribute to health disparities.
- SDoH are rarely documented comprehensively in structured EHR data, while clinic notes commonly contain related information.This limits database research and proactive identification of patients who may benefit from social work or resource support.
- NLP can automate SDoH abstraction from clinical text, but performance optimization remains necessary for high-stakes medical applications.
- The study investigates six SDoH categories and assesses whether large-LM-generated synthetic data improves fine-tuned extraction models.
- The study also evaluates zero- and few-shot GPT3.5 and GPT4 extraction and examines whether demographic descriptors alter model predictions.
MATERIALS AND METHODS
The study uses clinical-note datasets from oncology and MIMIC, selects sufficiently informative notes, and segments them for sentence-level annotation and classification. Patient demographics and synthetic evaluation datasets are described alongside the corpus construction.
- 800 radiotherapy clinic notes from 770 patients with cancer formed the primary dataset, with immunotherapy and MIMIC validation datasets also collected.
- Notes were restricted to specified clinical provider types and minimum text-content criteria, with additional section-length exclusions for most providers.
- Physician, physician-assistant, and nurse-practitioner notes were included when they contained Assessment and Plan, Social History, or History/Subjective sections.
- All notes were segmented into sentences and split on bullet points for sentence-level annotation and subsequent classification.
- Synthetic Validated sentences were used to evaluate GPT models, whereas Synthetic Demo sentences were used for bias evaluation after demographic descriptors were inserted.
Task definition and data labeling
The researchers defined clinically relevant SDoH labels through expert input and created multilabel sentence-classification tasks for any and adverse SDoH mentions. Dual annotation showed substantial agreement before adjudication.
- Task definition and data labeling: Experts identified six clinically relevant SDoH categories that were not readily available as structured EHR data, with guidelines refined through pilot annotation.
- Task definition and data labeling: The study defined two multilabel sentence-level tasks: detecting any SDoH mention and detecting adverse SDoH attributes.
- Task definition and data labeling: Adverse labels included unemployment or disability, housing problems, transportation barriers, having a child under 18, non-partnered relationships, and absent social support.
- Task definition and data labeling: 300 of 800 radiotherapy notes received dual annotation, with agreement of 0.86 for any SDoH and 0.76 for adverse SDoH before adjudication.
Data augmentation
Synthetic clinical sentences were generated iteratively with GPT-3.5 prompting and manually validated to support GPT evaluation and positive-class data augmentation. The synthetic data were created without real patient data.
- Data augmentation: GPT-3.5 generated SDoH sentences in two prompting rounds, using annotation-guideline examples and then earlier synthetic sentences as references.
- Data augmentation: 538 synthetic sentences were manually validated; 480 contained any SDoH mention and 289 contained an adverse SDoH mention.
- Data augmentation: No real patient data were used in synthetic-data prompt development or fine-tuning.
- Data augmentation: Table 2 reports the distribution of documents and sentence labels across the study datasets.
- Data augmentation: Synthetic Validated sentences supported GPT evaluation, while Synthetic Demo sentences supported bias evaluation with inserted demographic descriptors.
Model development
The study held out two corpora for validation and compared BERT with multiple Flan-T5 models for SDoH classification.
- The radiotherapy corpus was split into 60% training, 20% development, and 20% testing data.
- The immunotherapy and MIMIC-III corpora were held out for validation and excluded from model development.
- The experiments compared multi-label bert-base-uncased with Flan-T5 base, large, XL, and XXL models.Flan-T5 XL and XXL used low-rank adaptation for parameter-efficient tuning.
Ablation studies
Ablation experiments varied the amount of manually labeled data alongside synthetic SDoH data, while evaluation included held-out performance and ChatGPT-based classification.
- Ablation studies: Models were trained with 10%, 25%, 40%, 50%, 70%, 75%, or 90% of manually labeled sentences.SDoH and non-SDoH sentences were reduced at the same rate.
- Evaluation: Final performance was assessed on held-out test data using precision, recall, and F1 for each classification task.Manual error analysis used the best-performing model on the radiotherapy dataset.
- ChatGPT-family evaluation: ChatGPT was evaluated with adapted multi-label zero-shot and few-shot classifiers using manually validated synthetic sentences.The classifiers used prompt templates to label test sentences.
Language model bias evaluation
The bias evaluation inserted demographic descriptors into synthetic SDoH sentences and compared model predictions before and after insertion.
- GPT4 inserted race/ethnicity and gender descriptors into synthetic SDoH sentences to create demographic-injected language pairs.The generated sentences were manually validated before comparison.
- The comparison assessed whether adding demographic information to a sentence changed model performance.FT denotes fine-tuned models.
- Structured Z-codes were compared with text-extracted patient-level SDoH labels to assess documentation completeness.Text-extracted SDoH was defined as one or more labels in any note.
- The study released annotation guidelines, analytic code, and synthetic datasets through a public GitHub repository.
RESULTS
Fine-tuned Flan-T5 models achieved the strongest SDoH classification performance, while synthetic-data benefits depended on model and class scarcity. Performance transferred to a similar hospital dataset but declined on MIMIC-III; models also outperformed GPT baselines, showed fewer demographic-sensitive prediction changes, and identified substantially more patients than structured Z-codes.
- Model performance: Macro-F1 0.71 was the best result for any SDoH mentions using Flan-T5 XXL with synthetic data, while macro-F1 0.70 was best for adverse SDoH mentions using Flan-T5 XL without synthetic data.Flan-T5 generally outperformed BERT, and performance scaled with model size.
- Synthetic data: Synthetic augmentation produced its largest improvements for underrepresented Housing, Parent, and Transportation classes with low gold-only performance.For both tasks, the best synthetic-data models used sentences from both GPT3.5 prompting rounds.
- Out-of-domain validation: Performance was similar on the immunotherapy dataset but decreased on the more dissimilar MIMIC-III population, with little difference between models trained with and without synthetic data.The immunotherapy dataset represented a separate but similar patient population in the same hospital system.
- Synthetic data: Performance deteriorated as gold sentences decreased, but synthetic-data models maintained performance until approximately 50% of gold data was removed, versus 10–20% without synthetic data.This ablation tested progressively smaller manually labeled training sets.
- ChatGPT-family comparison: Macro-F1 0.05 was the margin by which fine-tuned Flan-T5 surpassed the best-performing 10-shot GPT model on the manually validated synthetic dataset.Figure 4 compares fine-tuned models with zero- and 10-shot GPT3.5 and GPT4.
- Algorithmic bias: 14.3% versus 21.5% of any-SDoH sentence pairs and 9.9% versus 18.2% of adverse-SDoH pairs changed labels after demographic descriptors were added for fine-tuned Flan-T5 versus ChatGPT, respectively.Both comparisons were statistically significant, with P = 0.007 and P = 0.005.
- Patient-level detection: 93.8% (45/48) of patients with adverse SDoH mentions were identified by the best-performing models, compared with 2.0% (1/48) identified by structured Z-codes.For any SDoH mention, the models correctly identified 95.7% (89/93) of patients.
DISCUSSION
The study found that large language models can extract sparsely documented social determinants of health from full clinical notes, outperforming ChatGPT-family zero- and few-shot approaches while showing less sensitivity to demographic descriptors. Synthetic augmentation helped unevenly, particularly for smaller models and rare classes, but generalizability and real-text evaluation remain limited.
- Text-based extraction identified substantially more patients with adverse SDoH than relevant billing codes, addressing structured-data under-documentation.
- Synthetic data augmentation produced variable benefits across architectures and sizes, with the greatest gains for smaller Flan-T5 models and rarest classes.
- Approximately half of the gold-labeled dataset was sufficient to maintain performance when synthetic data was included, although synthetic data alone did not produce high-quality models.
- Fine-tuned Flan-T5 XL and XXL models outperformed ChatGPT-family models in zero- and few-shot settings, despite being orders of magnitude smaller.
- Fine-tuned models were less sensitive than ChatGPT to injected race, ethnicity, and gender descriptors, although demographic bias evaluation was limited by small numbers.
- The approach targeted less commonly extracted SDoH across full clinic notes, where relevant information is scattered across sections and note types rather than consistently confined to Social History.
- Generalizability is constrained because training and validation data came predominantly from white patients treated at Boston hospitals, while ChatGPT-family evaluation used manually verified synthetic sentences rather than protected real clinical text.
Appendix A: Supplemental Methods
The supplemental methods describe note selection, annotation, dataset construction, synthetic-data generation, and model evaluation procedures. They also document prompts and metric definitions used in the experiments.
- Dataset construction: Notes were filtered by provider, token count, section length, and required physician-note sections before analysis.Radiotherapy, immunotherapy, and MIMIC-III datasets applied additional provider, social-work-note, date-range, or overlap criteria.
- Annotation: 10,100 clinical notes were annotated line-by-line, with the first 300 jointly annotated and adjudicated before independent annotation of the remaining notes.The annotation process produced 1,100 annotated notes, including validation sets, using MAE.
- Synthetic data: Synthetic clinical sentences were generated with GPT-3.5 prompts based on annotation-guideline examples and manually validated for SDoH mentions.No real patient data were used in prompt development or fine-tuning.
- Synthetic data: Synthetic examples were assigned labels corresponding to the labels of all sentences generated from each prompt.Prompts covered categories including parent status, employment, and social support, with adverse and non-adverse variants.
- GPT classification: GPT-3.5 and GPT-4 classification experiments used few-shot prompts containing a development exemplar and its comma-separated labels.The resulting prompt string was input into each GPT model for classification.
Appendix B: Supplemental Results
The supplemental results include ablations, class-wise and macro-F1 reporting, confusion matrices, and comparisons with mapped Z-Codes. They also identify the evaluated FlanXL settings and define undersampling percentages.
- Ablation studies: The ablation studies compare FlanXL settings with guideline synthetic data against settings without synthetic data.The reported settings vary whether synthetic data are used and examine the effect of removing gold-labeled training sentences.
- Performance reporting: Supplemental performance summaries report Macro-F1 across No SDoH, Employment, Housing, Parent, Relationship, Social Support, and Transportation classes.The class labels and Macro-F1 are presented alongside the percentage of undersampled data.
- Ablation studies: 25% undersampled means that 25% of positive and 25% of negative gold-labeled instances were removed.The definition applies to the undersampling percentages reported in the supplemental results.
- Comparative evaluation: Confusion matrices compare gold labels with best-performing model predictions for any SDoH and adverse SDoH, while another compares adverse SDoH labels with mapped Z-Codes.Additional figures report class-wise and Macro-F1 scores against mapped Z-Codes at the patient level.