Source-linked AI summary
Building a COVID-19 Vulnerability Index
Dave DeCaprio, Joseph Gartner, Thadeus Burgess, Kristian Garcia, Sarthak Kothari, Shaayan Sayed, Carol J. McCall
TL;DR
The paper addresses the difficulty of identifying who is most vulnerable when simple risk criteria miss differences among people with the same chronic condition. It presents three C-19 Index implementations; all had ROC AUC 0.87, while validation among admitted COVID-19 patients yielded ROC AUC 0.68.
Problem
Simple risk criteria do not capture differences among people with the same chronic condition or account for frailty and multiple comorbidities, complicating vulnerability identification.
Source: §1.3 · p. 2Method
The paper presents three C-19 Index implementations with different feature sets and trade-offs between predictive accuracy and ease of implementation.
Source: §2.3 · p. 4Results
All three models had ROC AUC 0.87; in validation, average scores were 2.4% for admitted patients who survived and 3.3% for those who died, with ROC AUC 0.68.
Source: §3 · p. 7, §3.1 · p. 7Takeaways & Limitations
The authors provide implementations and access routes for healthcare data science users, hoping broader adoption will support more effective intervention strategies.
Source: §4 · p. 9No actual COVID-19 cases were used to train the models, so their usefulness depends on COVID-19 outcomes matching the proxy outcome.
Source: §3.3 · p. 9
Abstract
from arXiv · showhide
COVID-19 is an acute respiratory disease that has been classified as a pandemic by the World Health Organization. Characterization of this disease is still in its early stages. However, it is known to have high mortality rates, particularly among individuals with preexisting medical conditions. Creating models to identify individuals who are at the greatest risk for severe complications due to COVID-19 will be useful for outreach campaigns to help mitigate the disease's worst effects. While information specific to COVID-19 is limited, a model using complications due to other upper respiratory infections can be used as a proxy to help identify those individuals who are at the greatest risk. We present the results for three models predicting such complications, with each model increasing predictive effectiveness at the expense of ease of implementation.
1. Introduction
COVID-19 can cause severe illness and death, with risk elevated among older people and those with certain conditions. Because simple risk criteria miss differences among people, identifying vulnerability is challenging.
1.1. COVID-19 Illness
- COVID-19 is a contagious respiratory infection that can progress to pneumonia, organ failure, cardiac arrest, or death.The passage says experts were still learning about the disease’s complications and treatments.Source: §1.1 · p. 1
1.2. Flattening the Curve
- COVID-19 mitigation matters because U.S. health systems operate close to capacity, making each preventable case consequential.The passage notes that experts agree mitigation is needed to slow spread and prevent health-system collapse.Source: §1.2 · p. 2
1.3. Identifying Vulnerable People
- Simple CDC-style rules cannot capture differences in risk among people with the same chronic condition or account for frailty and multiple comorbidities.More than 55% of Medicare beneficiaries meet at least one CDC risk criterion.Source: §1.3 · p. 2
2.1. Proxy Outcome
The proxy outcome captures near-term risk of severe respiratory complications, using inpatient or observation diagnoses. The prediction window is three months to support immediate targeting.
2.1. Proxy Outcome
- The index measures near-term risk of severe complications from respiratory infections, using ARDS and four related respiratory diagnosis categories as proxies.The proxy event is defined from diagnosis codes on claims associated with an inpatient visit or observation stay.Source: §2.1 · p. 2
- The models predict whether a patient will have an inpatient stay for one of the proxy conditions within the next three months.The three-month window was chosen for immediate COVID-19 targeting rather than the longer horizons typical of claims-based models.Source: §2.1 · pp. 2–3
2.2. Datasets
The study combined CMS and Healthfirst claims populations to approximate the overall U.S. population. The cohorts used different eligibility and prediction-date rules before being split and sampled for modeling.
2.2. Datasets
- CMS and Healthfirst claims data represented different population segments and were combined to create a population with a demographic profile consistent with the overall U.S. population.CMS data were predominantly Medicare beneficiaries over 65 or disabled; Healthfirst primarily represented Medicaid members, including healthier adults.Source: §2.2 · p. 3
2.2.1 CMS Data Preparation
- The CMS cohort included living members over 65 who had fee-for-service coverage and sufficient continuous eligibility and follow-up.The cohort used the latest available prediction date to minimize ICD-9 history and excluded under-65 members because Healthfirst better represented that population.Source: §2.2.1 · p. 3
2.2.2 Healthfirst Data Preparation
- Healthfirst members were evaluated at each eligible month as a prediction date, unlike CMS members, who had a single prediction date.Healthfirst members had to be at least 18 and have three months of eligibility afterward.Source: §2.2.2 · pp. 3–4, §2.2.1 · p. 3
2.2.3 Combined Population
- Each dataset was split by person 80%/20%; training negatives were sampled to match the U.S. adult population’s 21% age-65-or-older share while preserving Healthfirst’s 3.9X age-group prevalence difference.The combined test population used the full CMS test set and a random 20% sample of the Healthfirst test set.Source: §2.2.3 · p. 4
2.3. Models
The paper presents three implementations of the C-19 Index with different feature sets and implementation demands. Their designs range from a questionnaire-based model to claims-based models that use more complex features.
2.3. Models
- Three models were built and tested on the same dataset to balance predictive accuracy against ease of implementation.The paper describes them as the Survey, Open source, and Full models.Source: §2.3 · p. 4
2.3.1 Survey Model
- The Survey model maps medical-history questions to CCSR diagnosis categories and combines those indicators with age, gender, and prior inpatient or emergency visits in logistic regression.It is designed for a simple health-risk questionnaire and requires no technical implementation.Source: §2.3 · p. 4, §2.3.1 · p. 5
2.3.2 Open Source Model
- The Open source model uses gradient-boosted trees with 559 CCSR indicator features and delays claims use by three months to simulate processing lag.The approach permits full diagnosis histories and provides scripts to prepare model features from CSV claims files.Source: §2.3.2 · p. 5
2.3.3 Full Model
- The Full model uses the ClosedLoop platform to incorporate engineered features such as social determinants of health and the Charlson Comorbidity Index, which the authors say improve accuracy.The platform handles feature computation and data linking that were difficult to extract into an open-source format.Source: §2.3.3 · pp. 5, 7
3. Results and Model Interpretation
All three models had ROC AUC 0.87, and models built for the proxy outcome were more accurate than CCI; validation among COVID-19 admissions found higher average scores among those who died. The approach remains dependent on the proxy outcome, claims data, and an adult population.
3.1. Validation
- Scores averaged 3.3% for admitted patients who died versus 2.4% for survivors, a 38% relative difference; the COVID-19 admission validation ROC AUC was 0.68.The authors expected lower ROC AUC than in testing because this cohort included only people already admitted, making prediction more difficult.Source: §3.1 · p. 7
- Table 6 organizes adult Medicaid and Medicare admissions by vulnerability-index decile, showing death rates and lift relative to each population’s overall death rate.The passage describes the table’s layout but does not report its decile values.Source: §3.1 · p. 7, Table 6 · p. 8
3. Results and Model Interpretation
- ROC AUC was 0.87 for all three models, and proxy-outcome models were more accurate than CCI; Figure 1 plots sensitivity through 20% alert rates.The paper identifies sensitivity at low alert rates as more appropriate for targeting highly vulnerable populations.Source: §3 · p. 7, Figure 1 · p. 8
3.2. Accessing Models
- The models are offered through a web survey, an open-source GitHub implementation, and a hosted full model.The open-source implementation is in Python; the hosted platform is offered free of charge.Source: §3.2 · p. 9
3.3. Limitations
- No actual COVID-19 cases were used in training, so the index’s usefulness depends on whether COVID-19 outcomes match the respiratory-infection proxy.The authors planned to validate the proxy as COVID-19 case data became available and eventually build models directly on COVID-19 vulnerability.Source: §3.3 · p. 9
- Claims data lack clinical detail such as lab values, so the authors identify population health with claims-only data—not inpatient settings—as the index’s useful context.Source: §3.3 · p. 9
- Pediatric populations were excluded because the authors could not confidently establish that the proxy outcome was appropriate for people under 18.Source: §3.3 · p. 9
4. Conclusions
The conclusion presents the C-19 Index as a tool intended to inform intervention policy under constrained healthcare resources, with multiple implementations for users of varying technical expertise.
4. Conclusions
- The authors frame better information as important for intervention policy while healthcare resources are constrained.Source: §4 · p. 9
- The paper provides several implementations and access routes for people with varying levels of technical expertise.Source: §4 · p. 9
- The authors hope widespread adoption will lead to more effective intervention strategies and help curtail the pandemic’s worst effects.Source: §4 · p. 9
5. Addendum
The addendum describes organizations using the C-19 Index to prioritize proactive COVID-19 outreach, including by redirecting care-management teams or creating messaging campaigns.
5. Addendum
- Organizations applied the index to prioritize proactive outreach toward their most vulnerable members.Source: §5 · pp. 9–10
- Some organizations redirected existing care-management teams’ prioritization and intervention focus toward COVID-19.Source: §5 · pp. 9–10
- Other organizations developed COVID-19 phone or text campaigns using the index as a prioritization mechanism.Source: §5 · p. 10
6. Ethics
The ethics statement describes data-use and business-associate agreements, HIPAA-compliant storage for identifiable Healthfirst data, and compliance-team review.
6. Ethics
- For CMS LDS data, the researchers submitted a proposal describing their predictive-modeling plans as part of the data-use agreement.Source: §6 · p. 10
- Identifiable Healthfirst data were covered by a Business Associate Agreement and stored in ClosedLoop’s HIPAA-compliant cloud storage.Source: §6 · p. 10
- The project was part of ongoing Healthfirst quality improvement, did not require IRB preapproval, and received review and approval from Healthfirst’s compliance team.Source: §6 · p. 10
8. Appendix: Full Feature List
The Full model’s listed features were chosen for use across both datasets and rely only on medical claims data. Most indicate whether a medical event occurred 15 months before prediction, excluding the most recent three months.
8. Appendix: Full Feature List
- The Full model’s feature set was designed to work across both datasets using only medical claims data.Although the platform can incorporate pharmacy claims, electronic health records, patient-reported data, and linked census data, those sources were excluded here because of CMS data-use constraints.Source: §8 · p. 12
- Most features are binary indicators of medical events during the 15-month period before prediction, excluding the three most recent months.Source: §8 · p. 12