Source-linked AI summary
Deep Learning Predicts Hip Fracture using Confounding Patient and Healthcare Variables
Marcus A. Badgeley, John R. Zech, Luke Oakden-Rayner, Benjamin S. Glicksberg, Manway Liu, William Gale, Michael V. McConnell, Beth Percha, Thomas M. Snyder, Joel T. Dudley
TL;DR
Hip-fracture CAD models may learn patient and hospital-process signals alongside disease-related image features, complicating interpretation of radiologist-facing risk scores. This study evaluates those signals across radiographs, incorporates covariates into multimodal models, and tests whether matching reveals their contribution.
Problem
Hip fractures are a major cause of death and disability, while radiographs can miss fractures and deep-learning CAD features are difficult to understand.
Method
The study trains deep-learning models to predict fracture, patient traits, and hospital-process variables, then uses multimodal modeling and matched cohorts to assess their contributions.
Results
All 20 variables were predictable from radiographs; fracture prediction improved from image-only AUC=0.78 to AUC=0.91 with patient and hospital-process features, while matching patient and acquisition variables reduced performance to AUC=0.52.
Takeaways & Limitations
Radiographs contain non-biological signals associated with fracture, and directly adding known covariates can improve performance but complicates interpretation in radiologist-facing CAD.
Takeaways & Limitations
The study lacked gold-standard images and used automatically inferred fracture labels and imputed covariates with limited accuracy.
Abstract
from arXiv · showhide
Hip fractures are a leading cause of death and disability among older adults. Hip fractures are also the most commonly missed diagnosis on pelvic radiographs. Computer-Aided Diagnosis (CAD) algorithms have shown promise for helping radiologists detect fractures, but the image features underpinning their predictions are notoriously difficult to understand. In this study, we trained deep learning models on 17,587 radiographs to classify fracture, five patient traits, and 14 hospital process variables. All 20 variables could be predicted from a radiograph (p < 0.05), with the best performances on scanner model (AUC=1.00), scanner brand (AUC=0.98), and whether the order was marked "priority" (AUC=0.79). Fracture was predicted moderately well from the image (AUC=0.78) and better when combining image features with patient data (AUC=0.86, p=2e-9) or patient data plus hospital process features (AUC=0.91, p=1e-21). The model performance on a test set with matched patient variables was significantly lower than a random test set (AUC=0.67, p=0.003); and when the test set was matched on patient and image acquisition variables, the model performed randomly (AUC=0.52, 95% CI 0.46-0.58), indicating that these variables were the main source of the model's predictive ability overall. We also used Naive Bayes to combine evidence from image models with patient and hospital data and found their inclusion improved performance, but that this approach was nevertheless inferior to directly modeling all variables. If CAD algorithms are inexplicably leveraging patient and process variables in their predictions, it is unclear how radiologists should interpret their predictions in the context of other known patient data. Further research is needed to illuminate deep learning decision processes so that computers and clinicians can effectively cooperate.
Introduction
Hip fractures are clinically consequential yet frequently missed on radiographs, motivating computer-aided detection. Existing studies largely focused on image-only prediction, leaving patient and hospital-process confounding and clinical augmentation insufficiently examined.
- Clinical motivation: 1.3 million hip fractures occur annually and are associated with 740,000 deaths and 1.75 million disability adjusted life-years.Three-month mortality increases fivefold for women and eightfold for men relative to age- and sex-matched controls.
- Clinical motivation: Radiographs are the recommended first-line test for suspected hip fracture, but indeterminate or negative studies may require follow-up pelvic MRI.Fractures are the most commonly missed diagnosis on radiographs of the spine and extremities, with most errors being perceptual.
- Prior work: Machine-learning and deep-learning studies have detected fractures, but most evaluated algorithm-only systems or isolated image-model performance.Prior work included hand-engineered features, hierarchical pixel patterns, and a reported hip-fracture DL detector with AUC 0.994.
- Research gap: Existing image-based fracture studies did not consider patient and hospital covariates or how algorithms could augment human decision processes.This leaves the role of non-image variables in fracture predictions insufficiently characterized.
- Confounding concern: Observational medical datasets may contain patient and healthcare-process biases that statistical learning can exploit instead of biological signal.Prior medical-record research found hospital process variables more predictive of mortality than biological signal.
- Study aim: This study analyzes whether DL can detect disease, patient traits, and hospital processes from radiographs and whether these variables contribute to fracture detection.The authors develop multimodal models, matched patient cohorts, and Naive Bayes ensembles to examine prediction and radiologist-support implications.
Results
Deep learning models extracted radiograph features that predicted fracture, patient traits, and hospital-process variables, but fracture performance declined as test cohorts were matched on confounders. Directly combining image, patient, and hospital-process data performed best among the evaluated predictor sets.
- Dataset and feature extraction: 23,602 hip radiographs were collected, and 23,557 were used to train and test CNNs after preprocessing and a 3:1 split.The study separated explanatory variables into disease, image, patient, and hospital-process categories known at image acquisition.
- Dataset and feature extraction: Scanner identity explained the greatest radiograph variation, with projection view forming additional clusters within scanners.Inception-v3 features were represented as 2048-dimensional vectors before clustering analysis.
- Predicting fracture and covariates: All 20 image models outperformed random; fracture reached AUC 0.78, while scanner device, manufacturer, and priority status reached AUC 1.00, 0.98, and 0.79, respectively.Most patient and hospital-process factors were also associated with fracture, and scan year was the best-predicted continuous variable at R2 = .39.
- Predicting fracture and covariates: Hospital-process features predicted fracture better than image or patient features, while the full IMG+PT+HP set achieved AUC 0.91.Adding image features to hospital-process features alone did not improve performance, but directly modeling all three feature groups produced the best result.
- Matched-cohort evaluation: Matching progressively reduced fracture-associated covariates, and image-only performance fell from AUC 0.78 to 0.67 after matching patient variables.Random subsampling and demographic matching did not significantly change AUC.
- Matched-cohort evaluation: After matching all covariates, the fracture detector performed randomly at AUC 0.52, whereas Gale et al.’s model remained at AUC 0.99 in a fully matched cohort.The comparison indicates that model robustness differed across the two fracture detectors under matched testing conditions.
- Secondary evidence integration: Naive Bayes integration improved over clinical data alone but remained inferior to directly combining image, patient, and hospital-process data.The directly modeled IMG+PT and IMG+PT+HP systems achieved AUC 0.86 and 0.91, respectively.
Discussion
CNNs can encode fracture-related patient and hospital process variables from radiographs, and these covariates substantially shape fracture predictions. Matching on richer patient and process variables removes predictive performance, raising interpretability and generalization concerns for CAD systems.
- Discussion: CNNs encode disease, patient, and hospital process variables from radiograph pixels, while directly adding known covariates further improves prediction.The study reports that secondary combination is less beneficial than directly modeling all variables.
- Discussion: Matching radiographs on patient and process factors eliminated fracture-prediction performance, indicating indirect prediction through associated variables.The authors distinguish this from directly measuring fracture-specific image features.
- Discussion: Demographic matching alone was insufficient; symptoms and hospital process variables were additionally required to reveal the model’s dependence on confounding variables.The richer matching set also equated fracture odds on the two leading devices after symptoms were added.
- Discussion: The covariate dependence differed from a prior hip-fracture model, potentially because the studies used different strategies, populations, metadata, labels, and image resolutions.The prior model used region-of-interest localization and 1024 × 1024 inputs, whereas these experiments used broader data and 299 × 299 images.
- Discussion: If radiologists receive image risk scores alongside documented patient information, hidden use of those variables makes the score’s new versus redundant evidence unclear.The authors note that autonomous systems may have a different rationale for incorporating patient and process variables.
- Discussion: Better documentation of clinician-considered risk factors could enable more image-recognition studies to incorporate multimodal predictor sets.
- Discussion: The study is limited by non-gold-standard imaging, automatically inferred labels, imputed covariates, reduced image resolution, and constrained feature dimensionality.The simplification prioritized comparing many models rather than training a best-performing model.
- Discussion: Further research should examine sampling bias and generalization, while localization, complete metadata, and methods such as adversarial or domain-separation networks may mitigate undesired signal.The authors note that public radiology datasets often lack image-acquisition and hospital-process annotations.
Datasets
The study assembled retrospective hip-radiograph data from Mount Sinai and processed images, labels, covariates, and missing values for model development, matching, and evaluation.
- Datasets: Separate site-specific models predicted fracture on internal test images, with matching and evaluation procedures applied to both sites.The dataset combined newly collected Mount Sinai data with previously published University of Adelaide data.
- Datasets: 23,557 radiographs from 9,024 patients were included after retrieving 23,602 studies from PACS.Images were collected from 2008 to 2016 across several clinical sites and 12 devices.
- Datasets: Images were standardized, downsampled, padded to 299×299 pixels, and normalized per image.
- Datasets: Labels combined DICOM acquisition specifications with clinical-note demographics, interpretation times, symptoms, and radiologists’ impressions.Regular expressions abstracted pain and falls, while a word2vector algorithm inferred fracture from physician interpretations.
- Datasets: Feasibility filters replaced out-of-range values with NA and removed implausible latency and BMI entries before modeling.Latency values below one minute or above one day were removed, as were BMI values above 60.
- Datasets: Continuous and categorical variables were converted to binary representations for logistic regression.Continuous variables were split at the median, while categorical variables used abstractions or the two most common levels.
- Datasets: Missing categorical values received an explicit “(Missing)” category, while BMI was imputed by regression and other continuous variables by medians.
Model Architecture and Training
The study extracts abstract radiograph features with a pre-trained CNN, reduces them for modeling, and fine-tunes image-only or combined image-metadata predictors. It also ensembles image and patient-information predictions with Naive Bayes.
- Image feature extraction: A pre-trained inception-v3 CNN encodes radiographs, after which the final layer is retrained using transfer learning.The model was initialized with parameters optimized for natural object recognition in ImageNet.
- Image feature extraction: The removed classification layer yields 2048 image feature scores per radiograph for subsequent unsupervised analyses.
- Image feature visualization: t-SNE projects image feature vectors into a two-dimensional plane to visualize image variation, comparing randomly initialized and ImageNet-pre-trained models.The projection uses an initial PCA reduction to 50 dimensions, perplexity 30, and theta 0.5.
- Predictive model training: Logistic regression predicts binary outcomes by maximizing AUC, whereas linear regression predicts continuous outcomes by minimizing RMSE.
- Predictive model training: Image-only models use 10 principal components of the 2048-dimensional vectors, while combined models concatenate these components with scalar metadata.The 10 components describe 68% of image variation in the 2048-dimensional feature space.
- Prediction ensembling: Naive Bayes combines predictions from a patient-demographics-and-symptoms model with an image-only model under an independence assumption.The prior probability is estimated using a kernel based on 10-fold cross-validation of the training partition.
Statistical Methods
The analyses use dimensionality reduction for image-feature visualization, bootstrap-based ROC evaluation and comparison, and Fisher’s exact tests for variable–fracture associations.
- Dimensionality reduction: t-SNE projects image feature vectors into a two-dimensional plane after initial PCA reduction to visualize image variation.The analysis uses the Rtsne package with perplexity 30 and theta 0.5.
- ROC analysis: AUCs are reported with 95% confidence intervals from 2000 stratified bootstrap replicates controlling case-control frequency.
- ROC analysis: ROC-curve comparisons use two-sided tests based on 2000 stratified bootstrap replicates for unmatched test cohorts.
- Association testing: Associations between binarized covariates and fracture are tested with two-sided Fisher’s exact tests for count data.