PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-assisted cohort selection with bias analysis for generating large-scale cohorts from the EHR for oncology research

Benjamin Birnbaum, Nathan Nussbaum, Katharina Seidl-Rathkopf, Monica Agrawal, Melissa Estevez, Evan Estola, Joshua Haimson, Lucy He, Peter Larson, Paul Richardson

arXiv:2001.09765v1cs.CYcs.LG

TL;DR

Selecting oncology cohorts from unstructured EHR data can require reviewing all potentially eligible patients. MACS uses machine learning to reduce chart review by 78% while missing 4% of cohort-eligible patients, with bias analysis assessing cohort preservation.

  • Problem

    Selecting cohorts from unstructured EHR records can require reviewing all potentially cohort-eligible patients.

  • Method

    MACS is a framework that uses machine learning to filter patients during EHR cohort selection and includes bias analysis.

  • Results

    The method reduced charts needing human review by 78% while missing only 4% of cohort-eligible patients.

  • Takeaways & Limitations

    EHR cohort selection can be performed more efficiently while still preserving the cohort.

  • Takeaways & Limitations

    Bias analysis does not fully preclude the possibility of residual bias.

Abstract

from arXiv · show

Objective Electronic health records (EHRs) are a promising source of data for health outcomes research in oncology. A challenge in using EHR data is that selecting cohorts of patients often requires information in unstructured parts of the record. Machine learning has been used to address this, but even high-performing algorithms may select patients in a non-random manner and bias the resulting cohort. To improve the efficiency of cohort selection while measuring potential bias, we introduce a technique called Model-Assisted Cohort Selection (MACS) with Bias Analysis and apply it to the selection of metastatic breast cancer (mBC) patients. Materials and Methods We trained a model on 17,263 patients using term-frequency inverse-document-frequency (TF-IDF) and logistic regression. We used a test set of 17,292 patients to measure algorithm performance and perform Bias Analysis. We compared the cohort generated by MACS to the cohort that would have been generated without MACS as reference standard, first by comparing distributions of an extensive set of clinical and demographic variables and then by comparing the results of two analyses addressing existing example research questions. Results Our algorithm had an area under the curve (AUC) of 0.976, a sensitivity of 96.0%, and an abstraction efficiency gain of 77.9%. During Bias Analysis, we found no large differences in baseline characteristics and no differences in the example analyses. Conclusion MACS with bias analysis can significantly improve the efficiency of cohort selection on EHR data while instilling confidence that outcomes research performed on the resulting cohort will not be biased.

OBJECTIVE … INTRODUCTION

MACS addresses the difficulty of selecting oncology cohorts from EHRs when eligibility information is embedded in unstructured data and automated selection may introduce bias. Applied to metastatic breast cancer, it improved selection efficiency while Bias Analysis found no large differences in baseline characteristics or example analyses.

  • MATERIALS AND METHODS: 17,263 patients trained a TF-IDF and logistic-regression model, while 17,292 patients formed the test set for performance measurement and Bias Analysis.The MACS cohort was compared with the no-MACS reference cohort using clinical and demographic distributions and two example research-question analyses.
  • RESULTS: 0.976 AUC, 96.0% sensitivity, and 77.9% abstraction efficiency gain characterized algorithm performance.Bias Analysis found no large differences in baseline characteristics and no differences in the example analyses.
  • CONCLUSION: MACS with Bias Analysis can significantly improve EHR cohort-selection efficiency while instilling confidence that outcomes research on the resulting cohort will not be biased.This conclusion follows the reported efficiency gain and Bias Analysis findings.
  • INTRODUCTION: EHRs provide longitudinal, recent, generalizable, and clinically deep data for oncology research, but important eligibility information is often stored in unstructured notes and pathology reports.These characteristics create both opportunities for research and challenges for cohort construction.
  • INTRODUCTION: Manual review remains slow and costly, while structured-data-only cohort selection can miss patients and still requires reviewing all potentially eligible patients.The paper therefore identifies a need for more scalable cohort-selection approaches.
  • INTRODUCTION: High-performing machine-learning algorithms may select patients non-randomly, potentially biasing cohorts and producing erroneous research findings.MACS was introduced to improve efficiency while assessing the resulting cohort’s suitability for research.
  • INTRODUCTION: MACS applies a two-step filter in which a model predicts cohort eligibility and human abstractors assess patients deemed eligible by the model.Bias Analysis compares the MACS cohort with a reference cohort whose inclusion and exclusion decisions were all made by a human abstractor.
  • OBJECTIVE: The framework was illustrated by selecting metastatic breast cancer patients, for whom accurately identifying metastatic disease requires unstructured data because ICD codes lack sensitivity.This application creates an opportunity for substantial automation-related efficiency improvement.

MATERIALS AND METHODS · Data Source and Processing

The study used the Flatiron Health longitudinal EHR database, with patient-level demographic, clinical, and outcomes data processed from structured and unstructured records. Data were harmonized across clinics, while documents, mortality, and treatment lines were standardized using defined abstraction and construction methods.

  • Data Source and Processing: More than two million active patients from over 265 cancer clinics and approximately 800 United States care sites were available for analysis.The majority of patients were seen in community oncology settings.
  • Data Source and Processing: Institutional Review Board approval was obtained before study conduct, and informed patient consent was addressed.
  • Data Source and Processing: Patient-level demographic, clinical, and outcomes data were extracted from the source EHR system, including structured and unstructured clinical-document elements.
  • Data Source and Processing: Structured data were aggregated, normalized, and harmonized across clinics.
  • Data Source and Processing: Documents were classified into 24 standard categories, and unstructured elements were extracted using technology-enabled abstraction.
  • Data Source and Processing: Dates of death came from a composite mortality variable, while lines of therapy were constructed from drug orders and administrations using oncologist-defined rules.

Labeled Data Preparation · Modeling approach · Labels

The study assembled and labeled an EHR cohort for mBC model development, performance assessment, and bias evaluation. Patients were identified through recent activity and breast cancer codes, reviewed for metastatic disease, and split into training and test sets.

  • Labeled Data Preparation: Patients with at least two office visits on separate dates after January 1, 2011 were selected for labeled-data generation.
  • Labeled Data Preparation: Oncology nurses and tumor registrars reviewed a random subset and abstracted mBC from unstructured EHR data.mBC was defined as Stage IV breast cancer or recurrence of early-stage breast cancer.
  • Labeled Data Preparation: Patients with metastatic diagnosis before January 1, 2011 were removed to avoid immortal time bias from requiring a post-date visit.
  • Labeled Data Preparation: 34,555 patients were identified, with 17,263 assigned to model training and 17,292 reserved for testing and bias evaluation.
  • Labels: Each patient received a binary label: positive indicated an mBC diagnosis, whereas negative indicated no mBC diagnosis.Because all patients had breast cancer ICD codes, most negative patients had non-metastatic breast cancer.
  • Labels: Approximately 8.5%–9% of cases were positive in both the training and test sets.

Feature generation … Analysis

The study transformed selected unstructured EHR documents into TF-IDF features, trained a regularized logistic regression model, and chose a threshold targeting 95% sensitivity. It evaluated model performance and potential MACS bias by comparing MACS-selected patients with a manually abstracted reference cohort across characteristics and example survival analyses.

  • Feature generation: Patient documents were chronologically concatenated, normalized, and represented with unigram and bigram TF-IDF features, retaining the 100,000 most common features.Documents included visit notes, pathology reports, procedure and operative reports, and radiology reports.
  • Feature generation: Cross-validation experiments found that including more than the retained feature set did not noticeably improve performance.
  • Training: An L2-regularized logistic regression model was trained on a random 80% subset, with its regularization constant selected to maximize cross-validated ROC AUC.Feature generation and training used scikit-learn.
  • Choosing a score threshold: 95% sensitivity was the threshold target, using the largest score x for which at least 95% of validation positives scored above x.The high target sensitivity was selected to reduce potential bias.
  • Analysis: Model evaluation measured ROC AUC, sensitivity, and abstraction efficiency gain on the test set, with confidence intervals based on 1000 bootstrap samples.Abstraction efficiency gain was defined as the fraction excluded from manual abstraction because scores fell below threshold.
  • Analysis: The Reference Standard comprised all test-set patients manually determined positive for metastasis, whereas the MACS Cohort comprised those classified positive by the model.
  • Analysis: Bias analysis compared baseline demographic and clinical distributions between cohorts, estimating differences with 95% empirical confidence intervals from 1000 bootstrap samples.Continuous variables were summarized with medians and IQRs, and categorical variables with percentages.
  • Analysis: Example analyses compared median overall survival and survival by HR/HER2 status between cohorts using Kaplan-Meier methods, median-OS differences, and log-rank tests.The comparisons included HR+/HER2− versus triple-negative subpopulations within each cohort.

RESULTS … Distribution of demographic and clinical characteristics

MACS accurately identified metastatic breast cancer patients and substantially reduced manual chart review. Its selected cohort closely matched the reference standard across demographic and clinical characteristics, with only small observed differences.

  • Model performance: 0.976 AUC (95% CI: 0.973, 0.979) was achieved on held-out test data for predicting metastatic breast cancer diagnosis.The model used logistic regression.
  • Model performance: 96.0% sensitivity (95% CI: 94.9, 96.9) in the test set indicated that the training-selected threshold generalized well.The threshold chosen during training was 0.047.
  • Model performance: 77.9% abstraction efficiency gain (95% CI: 77.2, 78.5) reduced charts needing human review by 78% while missing 4% of positive patients.This was observed at the chosen threshold.
  • Distribution of demographic and clinical characteristics: Differences between the Reference Standard and MACS Cohort were small across demographic and clinical baseline characteristics.The comparison assessed whether missing 4% of patients through MACS could meaningfully change downstream analyses.
  • Distribution of demographic and clinical characteristics: The largest qualitative differences involved earlier metastatic diagnosis dates and a shift toward Stage IV at primary breast cancer diagnosis in the MACS Cohort.These were described as small shifts relative to the Reference Standard.

Clinical characteristics

Clinical characteristics were closely aligned across the compared cohorts, including stage, receptor and BRCA status, healthcare utilization, and follow-up measures. The largest reported category was unknown/not assessed BRCA status: 1161 [78.18] versus 1112 [78.04].

  • Stage at primary breast cancer diagnosis: 31.04% versus 31.72% had stage IV disease at primary breast cancer diagnosis, with stage I, II, and III distributions also similar.Stage I: 136 [9.16] versus 127 [8.91]; stage II: 356 [23.97] versus 334 [23.44]; stage III: 311 [20.94] versus 297 [20.84].
  • HR status at mBC diagnosis: 75.22% versus 75.30% were HR-positive at mBC diagnosis, while 20.88% versus 20.77% were HR-negative.Results unknown/not assessed were 58 [3.91] versus 56 [3.93].
  • HER2 status at mBC diagnosis: 71.78% versus 71.72% were HER2-negative at mBC diagnosis, compared with 16.09% versus 16.14% HER2-positive.Equivocal results were 90 [6.06] versus 86 [6.04], and unknown/not assessed results were 90 [6.06] versus 87 [6.11].
  • BRCA status (germline): 78.18% versus 78.04% had unknown/not assessed germline BRCA results; positive results were 45 [3.03] versus 44 [3.09].Negative results were 270 [18.18] versus 260 [18.25].
  • Healthcare utilization and follow-up: 37 [15, 72] versus 37 [15, 72] was the median number of visits, while median therapy lines were 1 versus 2.Median follow-up from mBC diagnosis was 16.23 [6.07, 34.07] versus 16.77 [6.07, 34.07] months.

Effect on example research questions

MACS produced overall survival estimates closely matching the Reference Standard. This agreement also held in HR+/HER2- and triple-negative subgroups, including the expected significant survival difference.

  • Effect on example research questions: 751 Reference Standard patients and 731 MACS Cohort patients had recorded all-cause death events, among cohorts of 1485 and 1425 patients, respectively.The first analysis assessed overall survival from metastatic breast cancer diagnosis.
  • Effect on example research questions: 2.71 years was the Reference Standard median OS versus 2.72 years in the MACS cohort, a difference of -0.01 years.The 95% CIs were 2.51–2.93 for the Reference Standard and 2.52–2.96 for the MACS cohort; the difference CI was -0.31–0.30.
  • Effect on example research questions: 3.03 years versus 1.16 years was the Reference Standard median OS for HR+/HER2- versus triple-negative patients, respectively, with log-rank p-value < 0.001.This subgroup difference was statistically significant and aligned with clinical expectations.
  • Effect on example research questions: 3.08 years versus 1.17 years was the MACS Cohort median OS for HR+/HER2- versus triple-negative patients, with log-rank p-value < 0.001.Compared with the Reference Standard, median OS differences were -0.04 years for HR+/HER2- patients and -0.02 years for triple-negative patients.

DISCUSSION

The discussion presents MACS as a scalable way to reduce manual EHR cohort-selection effort while examining whether model-based selection introduces clinically meaningful bias. In metastatic breast cancer, MACS reduced chart review substantially without materially changing cohort characteristics or example research findings, but still depended on human curation.

  • Selecting oncology cohorts from unstructured EHR data traditionally requires labor-intensive human chart review, limiting studies in large populations.
  • MACS uses machine learning to filter patients unlikely to belong to a cohort, improving efficiency for deidentified EHR-based research datasets.
  • 78% fewer charts required human review while missing only 4% of cohort-eligible patients in the metastatic breast cancer application.
  • Bias analysis found no large baseline clinical or demographic differences, and example research findings did not depend on cohort selection.
  • MACS with Bias Analysis still requires manual abstraction to create training and testing labels and extract variables for cohort comparisons and example analyses.
  • MACS can reduce time and cost, allowing resources to support other abstraction tasks and expand research datasets across disease areas.

Limitations · Future work

The bias analysis cannot fully exclude bias, and reference-standard construction is inefficient for rare populations. Future work includes longitudinal registry use, broader cohort applications, and models that better represent evolving EHR data and temporal information.

  • Limitations: Bias analysis cannot assess variables absent from the dataset or every joint distribution and research question, and clinical judgment determines which checks to perform.These limitations constrain confidence in the resulting cohort’s representativeness.
  • Limitations: Using the reference-standard cohort for comparison may mask bias already present in that reference standard.
  • Limitations: 9% prevalence in mBC required abstracting over 15,000 patients to identify 1485 reference-standard patients, making the approach inefficient for rarer populations.
  • Limitations: MACS had decreased sensitivity for patients diagnosed at an early stage or becoming metastatic recently, possibly because its linear model omitted EHR longitudinality.Future work could explore nonlinear models or model time more directly.
  • Future work: MACS can support longitudinal EHR-based registries, but evolving patient data, population distributions, documentation, and care standards require continuously refreshed training and test sets and periodic model deployment.
  • Future work: MACS can identify oncology cohorts defined by biomarkers, therapies, or advanced disease, and similar approaches can generate cohorts across diverse disease states.
  • Future work: MACS improves efficiency when unstructured data are required and eligible patients are relatively rare, provided sufficient labeled data and ongoing performance and bias monitoring.

CONCLUSION · FUNDING AND COMPETING INTERESTS · FIGURES

MACS with Bias Analysis combines automated cohort selection, human validation, and rigorous comparison to improve EHR research efficiency while preserving confidence in data quality. The study was sponsored by Flatiron Health, and its figures depict the selection workflow, model performance, and survival analyses.

  • CONCLUSION: MACS with Bias Analysis combines automated machine learning, technology-enabled abstraction, and rigorous validation to address challenges in large-scale EHR research.
  • CONCLUSION: MACS enables more efficient EHR cohort selection while preserving confidence in data quality and accelerating outcomes research.
  • FUNDING AND COMPETING INTERESTS: The study was sponsored by Flatiron Health Inc., an independent subsidiary of the Roche group.
  • FUNDING AND COMPETING INTERESTS: During the study period, all authors reported employment and equity ownership in Flatiron Health and stock ownership in Roche.
  • FIGURES: Figure 1 depicts MACS as a two-step process: model-based eligibility prediction followed by manual assessment by human abstractors.
  • FIGURES: Figure 1 also shows Bias Analysis comparing the MACS cohort with a reference cohort whose inclusion and exclusion decisions were all made by a human abstractor.
  • FIGURES: Figure 2 presents the logistic regression model’s ROC curve with an AUC of 0.976.
  • FIGURES: Figures 3 and 4 present Kaplan-Meier estimates of overall survival for the reference standard and MACS cohort, including hormone receptor and HER2 subgroups.
Loading 2001.09765v1…