Source-linked AI summary
Patient-Reported Survey Data Improve Prediction of Opioid Use Disorder
Xiyue Jiang, Zihan Ding, Grace Han, Yinan Liu, Richard N. Rosenthal, Fusheng Wang
TL;DR
EHRs may incompletely capture factors associated with OUD, leaving the added predictive value of participant-reported information uncertain. This study evaluated survey augmentation of EHR-based OUD prediction and found consistent improvement across modeling approaches and look-back windows, while noting that survey availability differed by OUD status.
Problem
EHRs may incompletely represent social and other factors associated with OUD, and whether survey information adds predictive value for OUD beyond longitudinal EHR data remained unknown.
Source: Introduction · p. 1Method
The study evaluated participant-reported survey information as an addition to longitudinal EHR data for predicting a first recorded OUD diagnosis.
Source: Discussion · p. 9, Conclusion · p. 9Results
Survey augmentation consistently improved OUD prediction beyond structured EHR data across eight modeling approaches and three look-back windows.
Source: Conclusion · p. 9, Discussion · p. 9Takeaways & Limitations
Survey data provide complementary predictive signals, but their contribution depends on response content and when and for whom surveys are available.
Source: Discussion · p. 9, Conclusion · p. 10OUD status based on recorded diagnostic codes may miss unrecorded cases, and survey participation was incomplete and differed by outcome status.
Source: Discussion · p. 9
Abstract
from arXiv · showhide
Electronic health records (EHRs) may incompletely capture patient-reported factors associated with opioid use disorder (OUD). We evaluated whether survey data improve prediction of a first recorded OUD diagnosis among 267,747 All of Us participants with documented opioid exposure, including 15,287 OUD cases. We compared EHR-only and EHR+survey models across 6-, 12-, and 24-month look-back windows using logistic regression, random forest, XGBoost, LightGBM, multilayer perceptron, LSTM, GRU, and Transformer. Survey augmentation improved PR-AUC across all 24 model-window combinations by 0.0087-0.0505; the best 24-month LightGBM model improved from 0.6219 to 0.6603. Survey coverage increased with longer windows and differed by OUD status (24 months: 21.7% OUD-positive vs. 60.7% OUD-negative). Permutation analysis ranked survey features as the second most important information domain at 24 months in both evaluated models. Patient-reported data provide complementary predictive signals beyond structured EHRs while highlighting the importance of survey availability.
Abstract
EHRs may incompletely represent social and other factors associated with OUD, and the incremental value of survey data for OUD prediction had remained unknown. This study examines that value and the information domains contributing to prediction.
Abstract
- Patient-reported information may capture OUD-related factors underrepresented in clinical data, including social conditions that EHRs may record inconsistently.Prior work on survey-based prediction concerned persistent opioid use after surgery, distinct from OUD.Source: Introduction · pp. 1–2
- The study evaluates whether surveys improve prediction of a first recorded OUD diagnosis beyond EHR data across machine-learning approaches and look-back windows.It also examines which information domains and survey questions contribute most strongly.Source: Introduction · p. 2
Methods
The study uses the All of Us Research Program repository to assemble a large cohort of participants with documented opioid medication exposure and classify them by recorded OUD diagnosis.
Methods
- 267,747 participants with documented opioid exposure formed the cohort, including 15,287 OUD-positive cases and 252,460 OUD-negative controls.The case-to-control ratio was approximately 1:16.5.Source: Data source and study cohort · p. 2
- Data came from the All of Us Research Program Curated Data Repository and were available through October 1, 2023.Source: Data source and study cohort · p. 2
- Table 1 summarizes baseline characteristics for the cohort overall and by OUD status.Source: Table 1 · p. 2
Variable Overall OUD-positive OUD-negative
The supplied material identifies a Gender subsection as continuing a table and describes the temporal design used to extract histories before each participant’s index date.
Gender
- The Gender subsection marks a table continuation from the preceding page.Source: Gender · p. 3
Temporal design
- Cases were indexed at their first recorded OUD diagnosis, while controls were indexed at their most recent medical encounter; features came from the preceding 6, 12, or 24 months.Data on or after the index date were excluded to prevent temporal leakage.Source: Temporal design · p. 3, Figure 1 · p. 3
EHR-Derived Features
EHR-derived features span demographic, clinical, medication, measurement, and observation domains, with low-frequency concepts filtered within each look-back window.
EHR-Derived Features
- Concepts observed in fewer than 0.5% of participants per look-back window were excluded, using frequency alone before data splitting.The filtering did not use OUD status.Source: EHR-Derived Features · p. 3, Table 2 · p. 4
- Demographic features included age, gender, race, ethnicity, sex at birth, and a composite race/ethnicity variable.Source: EHR-Derived Features · pp. 3–4
- Condition features included occurrence counts for diagnoses such as chronic pain, substance abuse, depression, anxiety, and posttraumatic stress disorder.Participants without a recorded occurrence received a count of zero.Source: EHR-Derived Features · p. 4
- Medication features included opioid and non-opioid analgesics, muscle relaxants, benzodiazepines, and antidepressants, represented as occurrence counts.Source: EHR-Derived Features · p. 4
- Laboratory and physical measurements were converted to common units and summarized by count, mean, and most recent value.When unavailable, the count was zero and the mean and most recent values remained missing.Source: EHR-Derived Features · p. 4
- Numeric clinical observations used count, mean, and most recent value, while occurrence-based observations used counts with zero for no recorded occurrence.Source: EHR-Derived Features · p. 4
Participant-reported survey data
The study combines four All of Us surveys covering demographic, behavioral, health, and social information. It retains eligible pre-index responses, encodes response options, and measures survey availability separately from the amount and recency of responses.
Survey Sources and Content
- Four All of Us surveys capture demographics and socioeconomic conditions, health behaviors, general health, and social and environmental factors.The Social Determinants of Health survey follows the baseline surveys.Source: Participant-reported survey data · p. 4
Survey Availability
- Survey coverage counts participants with at least one eligible pre-index survey interaction, including an answered item or an explicitly skipped or declined response.Among respondents, the analysis separately summarizes answered and skipped questions and the interval since the most recent response.Source: Participant-reported survey data · pp. 4–5
Survey-Derived Features
- Questions with response rates below 50% among survey contributors were excluded, and retained response options were binary-encoded.Single-response questions used one-hot encoding, while multiple-response questions used multi-hot encoding; missing indicated no usable response.Source: Participant-reported survey data · p. 5
- The analysis retained 30 survey features at 6 and 12 months and 108 at 24 months.Questions were retained even when their constructs, such as pain or substance use, also appeared in EHR data.Source: Participant-reported survey data · p. 5
Model development
The study compares patient-level tabular models with sequence models that use the temporal ordering of clinical events. Paired EHR-only and EHR+survey comparisons use consistent patient splits, while sequential models combine temporal and static representations.
Modeling approaches
- Tabular models summarize patient features over each look-back window, while sequential models use the temporal ordering of clinical events.The evaluated tabular models were logistic regression, random forest, XGBoost, LightGBM, and MLP; sequential models were LSTM, GRU, and Transformer.Source: Model development · p. 5
Cohort splitting and preprocessing
- The same patient-level 68% training, 12% validation, and 20% test splits were used for paired EHR-only and EHR+survey models.Preprocessing parameters were estimated on training data and applied unchanged to validation and test data.Source: Model development · p. 5
Class imbalance
- Unadjusted training achieved the highest validation PR-AUC, 0.6533, so final models used no class-imbalance adjustment.The comparison used 24-month XGBoost with EHR and survey features; class weighting, random undersampling, and SMOTE achieved 0.6475, 0.5906, and 0.4075, respectively.Source: Model development · p. 5
Sequential models
- Sequential models combine a temporal representation with static features, adding the same survey-derived features to the static vector for EHR+survey models.LSTM and GRU use the final hidden state; the Transformer uses masked mean pooling, and sequences are padded or truncated to the most recent K activity dates.Source: Model development · pp. 5–6
Evaluation
Performance is evaluated on held-out test data using PR-AUC as the primary metric, with thresholds selected on validation data. Permutation importance measures the PR-AUC decrease when information domains or survey questions are permuted.
Performance evaluation
- PR-AUC was the primary test metric; thresholds were selected on validation data to maximize positive-class F1 across 181 values from 0.05 to 0.95.The selected threshold was applied unchanged to the test set; F1+ denotes positive-class F1.Source: Evaluation · p. 6
Feature importance
- Permutation importance was the mean PR-AUC decrease across five repetitions after permuting a domain or survey question on held-out test data.The analysis covered LightGBM and GRU across all windows, jointly permuting encoded response categories and whole patient trajectories for time-varying sequential inputs.Source: Evaluation · p. 6
Results
Survey coverage rose with longer look-back windows but remained lower among OUD-positive participants. At 24 months, OUD-negative respondents contributed more answered questions, and the latest eligible response was farther from the index date than at six months.
Survey Availability
- At 24 months, OUD-negative respondents answered a median of 77 questions versus 28 among OUD-positive respondents.Median answered-question counts were similar between groups at 6 and 12 months.Source: Survey Availability · p. 6
- The median interval from the latest eligible survey response to the index date increased from 51 and 63 days at 6 months to 217 and 270 days at 24 months for OUD-positive and OUD-negative participants, respectively.Explicitly skipped or declined questions were uncommon, with median values of 0–1 per respondent across windows.Source: Survey Availability · pp. 6–7
- Coverage rose from 9.5% to 21.7% among OUD-positive participants and from 21.6% to 60.7% among OUD-negative participants between 6 and 24 months.Table 3 reports coverage and completion behavior by OUD status and look-back window; the analyzed groups included 15,287 OUD-positive and 252,460 OUD-negative participants.Source: Survey Availability · pp. 6–7, Table 3 · p. 7
Model Evaluation
Survey augmentation improved PR-AUC across all evaluated model-window combinations, while LightGBM and XGBoost achieved the strongest overall performance. PR-AUC increased with longer look-back windows, although survey-feature gains did not rise monotonically for every model.
Model Evaluation
- PR-AUC increased with longer look-back windows across all models, while LSTM, GRU, and Transformer generally underperformed gradient-boosting models, particularly at longer windows.Survey features nevertheless improved performance for neural and sequential models, indicating their contribution was not dependent on a specific architecture.Source: Model Evaluation · p. 7
- At 24 months, LightGBM achieved the highest PR-AUC with survey features (0.6603), followed by XGBoost (0.6576).Across evaluated models, ROC-AUC ranged from 0.8180 to 0.9430 and PR-AUC from 0.3239 to 0.6603.Source: Table 4 · p. 7, Model Evaluation · p. 7
- 0.0087–0.0505 absolute PR-AUC gains followed survey augmentation across all 24 model-window combinations.The largest gains for all eight models occurred at 24 months, ranging from 0.0181 to 0.0505, although gains did not increase monotonically across windows for every model.Source: Model Evaluation · p. 7, Table 5 · p. 8
- At 24 months, survey features ranked second behind laboratory measurements in both models; permutation decreased PR-AUC by 0.172 in LightGBM and 0.234 in GRU.Survey-feature permutation decreases also rose across the 6-, 12-, and 24-month windows in both models.Source: Model Evaluation · p. 8
- Highest-ranked survey questions spanned behavioral, functional, socioeconomic, and general-health domains, with recreational drug use, concentration difficulty, independent errands, and employment recurring.The most influential individual questions varied across models and look-back windows.Source: Model Evaluation · p. 8
Discussion
Survey information improved OUD prediction across modeling approaches and observation windows, while survey availability and retained-feature counts varied with the look-back period. The findings also highlight limitations in survey availability and evaluation that require further study.
Discussion
- Survey augmentation improved prediction across all modeling approaches and observation windows, supporting patient-reported information as a complement to EHR-based OUD prediction.The gains were modest but consistent.Source: Discussion · p. 9
- At 24 months, survey-derived features ranked among the most important information sources in both evaluated model architectures.Influential questions covered substance use, functioning, employment, income, pain, and general health.Source: Discussion · p. 9
- LSTM, GRU, and Transformer models generally did not outperform gradient-boosting approaches despite modeling event sequences.Summarized laboratory and physical-measurement values may have limited sequential models’ ability to capture changes over time.Source: Discussion · p. 9
- At 24 months, 108 survey features were retained, versus 30 at the 6- and 12-month windows, so larger gains may reflect both longer histories and greater survey availability.The optimal observation horizon may depend on the timing of prior information and the likelihood that each data source is available.Source: Discussion · p. 9
- OUD-negative participants had greater survey coverage across all three windows, raising the possibility that missing survey data are informative and that availability patterns contribute to observed gains.Future ablations separating availability features from survey responses may clarify their respective contributions.Source: Discussion · p. 9
- OUD labels relied on recorded diagnoses, survey participation differed by outcome, and evaluation used an internal held-out test set focused primarily on discrimination.External validation, calibration, subgroup performance, and prospective evaluation are needed before clinical use.Source: Discussion · p. 9
- Integrating patient-reported information with longitudinal EHR data adds predictive signal, but its contribution depends on response content and when and for whom data are available.Future work should assess external generalizability and whether these data improve clinically meaningful OUD screening and risk-assessment workflows.Source: Discussion · p. 9
Conclusion
Patient-reported survey information consistently improved OUD prediction beyond structured EHR data across eight modeling approaches and three look-back windows. It may add signals not fully represented in clinical records, although further validation is needed for earlier risk assessment and prevention-oriented screening.
Conclusion
- Patient-reported survey information consistently improved OUD prediction beyond structured EHR data across eight modeling approaches and three look-back windows.Source: Conclusion · p. 9
- Behavioral, social, functional, and self-reported health information captures predictive signals not fully represented in routine clinical records.Source: Conclusion · p. 10
- With further validation, integrating patient-reported data with longitudinal EHRs may support earlier and more comprehensive OUD risk assessment and prevention-oriented screening.Source: Conclusion · p. 10