Source-linked AI summary
Multimodal Machine Learning-based Knee Osteoarthritis Progression Prediction from Plain Radiographs and Clinical Data
Aleksei Tiulpin, Stefan Klein, Sita M. A. Bierma-Zeinstra, Jérôme Thevenot, Esa Rahtu, Joyce van Meurs, Edwin H. G. Oei, Simo Saarakkala
TL;DR
Knee OA progression is difficult to predict, despite its clinical and societal burden and the limited prognostic capability of routine diagnostic tools. The paper develops a multimodal machine-learning model combining raw radiographs with clinical and patient-history data. It reports better progression identification than previously used OA-literature methods and presents the approach as relevant to drug-trial recruitment and personalized treatment planning.
Problem
Routine OA diagnostic modalities do not allow effective prognosis assessment, while direct raw-radiograph prediction of structural progression had not been studied.
Method
A machine-learning model combines raw knee radiographs with physical examination, medical history, anthropometric data, and optionally KL grade to predict progression.
Results
The proposed model identified progressor cases on average 6% more accurately than methods previously used in the OA literature.
Takeaways & Limitations
The approach could support subject selection for OA disease-modifying drug trials and development of personalized treatment plans.
Takeaways & Limitations
The model was tested only on United States populations and standardized radiographs acquired with a positioning frame, limiting demonstrated generalizability.
Abstract
from arXiv · showhide
Knee osteoarthritis (OA) is the most common musculoskeletal disease without a cure, and current treatment options are limited to symptomatic relief. Prediction of OA progression is a very challenging and timely issue, and it could, if resolved, accelerate the disease modifying drug development and ultimately help to prevent millions of total joint replacement surgeries performed annually. Here, we present a multi-modal machine learning-based OA progression prediction model that utilizes raw radiographic data, clinical examination results and previous medical history of the patient. We validated this approach on an independent test set of 3,918 knee images from 2,129 subjects. Our method yielded area under the ROC curve (AUC) of 0.79 (0.78-0.81) and Average Precision (AP) of 0.68 (0.66-0.70). In contrast, a reference approach, based on logistic regression, yielded AUC of 0.75 (0.74-0.77) and AP of 0.62 (0.60-0.64). The proposed method could significantly improve the subject selection process for OA drug-development trials and help the development of personalized therapeutic plans.
Introduction
Knee OA is a disabling, costly disease whose progression cannot be effectively assessed with current routine diagnostic tools. The section motivates direct, multimodal prediction from radiographs and patient data as an alternative to subjective or impractical approaches.
- Motivation: Knee OA causes substantial disability worldwide and imposes major healthcare and societal costs, with total knee replacement as the invasive end-stage treatment.In the United States, about 12% of the population suffers from OA, and annual TKR rates among 45–64-year-olds have doubled since 2000.
- Clinical gap: Routine diagnosis combines clinical assessment, physical examination, and radiography but does not support effective prognosis assessment for treatment planning or drug-trial recruitment.MRI could improve assessment but is costly, time-consuming, limited in availability, and unsuitable for broad use.
- Existing approaches: Deep Learning can automate radiographic KL-grade assessment, but existing diagnostic tools remain imperfect and inconsistent because practitioner judgments are subjective.
- Research gap: No prior study had directly used raw radiographs to predict structural knee OA progression instead of radiologist-defined KL grades.
- Existing approaches: Existing progression models combine imaging texture descriptors, KL grade, and clinical or anthropometric data, but their performance and generalizability are difficult to assess.Texture descriptors may be sensitive to acquisition settings, limiting usable sample sizes, while only a few studies used external datasets.
Results
The study evaluated reference models and a multimodal CNN–GBM approach for predicting knee OA progression using radiographic, clinical, and patient-history information. On independent testing, multimodal models outperformed the strongest reference model, with performance further improving when a radiologist-provided KL grade was included.
- Reference methods: 0.76 AUC and 0.63 AP were achieved by the strongest GBM reference model using clinical variables and KL grade.This model outperformed the corresponding logistic-regression reference model.
- Predicting progression from raw image data: 0.79 AUC and 0.68 AP were achieved by the CNN using baseline knee radiographs on the independent test set.The CNN directly used raw knee DICOM images and jointly predicted progression and current KL grade.
- Predicting progression from raw image data: GradCAM attention maps highlighted image zones used by the CNN, but the authors cautioned that these maps do not theoretically establish causal relationships.The paper also reported cases where attention focused on the compartment opposite the later visible degenerative change.
- Predicting progression from raw image data: 0.79 AUC and 0.68 AP were obtained by the fully automatic multimodal model without a KL-grade input.This model combined CNN predictions with clinical measures through stacked GBM generalization.
- Predicting progression from raw image data: 0.78 and 0.80 AUC were achieved by models 6 and 7, respectively, for knees classified as KL-0 or KL-1 at baseline.Their corresponding AP values were 0.58 and 0.62.
Discussion
The study presents a patient-specific method that predicts structural knee OA progression from a single clinical visit using raw radiographs and patient data. It reports stronger performance than conventional reference methods, while identifying limitations in population, imaging, outcome definition, and interpretability.
- The method predicts structural knee OA progression from a single clinical visit using raw knee images rather than observer-derived measures such as JSW, KL, or texture descriptors.
- The proposed model combines plain radiographs with symptomatic assessment and injury or surgery history, and the combined data produced more accurate predictions.
- Machine-learning predictions are less transparent than conventional statistical models, and GradCAM attention maps lack a theoretical guarantee of identifying causal image features.
- Validation was limited to United States populations and standardized radiographs using a positioning frame, while progression was defined only through KL grading and excluded symptomatic progression.
- The authors report that the model identifies progressors on average 6% more accurately than methods previously used in OA research.
- The approach could support OA disease-modifying drug-development trials and personalized treatment plans, with potential for low-cost primary-healthcare use.
Methods
The study used longitudinal OAI and MOST cohorts to identify knee OA progressors and non-progressors among knees without end-stage baseline disease. Clinical and anthropometric variables were selected while accounting for missing data in reference-model training and testing.
- The experiments used OAI and MOST follow-up cohorts containing clinical and imaging data from subjects at risk of developing OA.
- The analysis included knees with baseline KL grades 0, 1, 2, or 3 and excluded knees with TKA, end-stage OA, or missing baseline KL data.
- Knees with any KL-grade increase during follow-up were assigned progression according to the earliest observed increase, while non-progressors had to remain progression-free through the final examination.
- The clinical variables included age, sex, BMI, injury history, surgery history, and total WOMAC score.
- LR training excluded knees with missing values, whereas missing test-set variables were imputed for LR evaluation.
Image pre-processing
Knee DICOM images were standardized through region-of-interest extraction, landmark-based alignment, intensity conversion, resizing, and left-right mirroring. Model selection used subject-wise cross-validation, with AP guiding hyperparameter and checkpoint selection.
- Each knee image was standardized by extracting a 140×140 mm region of interest and aligning images using BoneFinder landmark localization.
- All experiments used the same stratified 5-fold subject-wise cross-validation on OAI data for training and validation splits.
- GBM hyperparameters were optimized with 500 Bayesian trials that maximized AP on cross-validation.
- Five CNN models were built through cross-validation, retaining the validation checkpoint with the maximum AP in each split.
Deep neural network’s implementation details
The model used a multi-task CNN with shared convolutional features and separate outputs for OA progression and baseline KL grade. Transfer learning, staged unfreezing, optimization controls, and image augmentation were used during training.
- The multi-task CNN contained convolutional and two fully connected blocks, with separate outputs for three progression classes and five baseline KL grades.
- The convolutional design was based on se-resnext50 32x4d, initialized from ImageNet through transfer learning, while the fully connected layers started from random noise.
- Convolutional weights were frozen for the first 2 epochs, then all CNN layers were trained for 20 epochs.
- Training used a 1e−3 learning rate, batch size 64, weight decay 1e−4, and Adam optimization.
- On-the-fly augmentation added noise, rotations up to ±5 degrees, random crops, and gamma correction to improve invariance to acquisition parameters.
Inference pipeline
At test time, predictions are stabilized by averaging across cross-validation models and five spatial crops. Progression probability is evaluated as the combined probability of the two progression classes.
- Five-crop test-time augmentation averages predictions from four corner crops and one center crop.Each crop is 300×300 pixels, and the five crop predictions are averaged.
- Cross-validation model outputs are averaged after test-time augmentation to produce the final prediction.
- P(prog|x) is computed as P(y = 1|x) + P(y = 2|x) during CNN-only evaluation.This replaces the fine-grained progression-class division with a single progression probability.
Interpreting neural network’s decisions
The study adds an interpretability procedure to inspect which radiological features the network detects. A TTA-compatible GradCAM produces attention maps that identify image regions positively correlated with the network output.
- A modified GradCAM approach examines the radiological features detected by the progression network.
- The GradCAM output is an attention map showing image regions positively correlated with the network output.
- Because the TTA operations and summed progression probabilities are differentiable, GradCAM can be applied to the TTA pipeline.
Model stacking: fusing heterogeneous data using tree gradient boosting
The model fuses neural-network predictions with clinical and demographic measures using stacked generalization and gradient boosting. Performance is assessed with PR and ROC metrics, with bootstrap confidence intervals and DeLong significance testing.
- Model stacking: fusing heterogeneous data using tree gradient boosting: Neural-network KL-grade and progression probabilities are fused with age, sex, BMI, injury history, WOMAC, and optionally KL grade.The fusion uses stacked generalization because combining these inputs is challenging and prone to overfitting.
- Model stacking: fusing heterogeneous data using tree gradient boosting: A second-level GBM uses cross-validation CNN predictions for each knee joint together with clinical measures as input features.
- Evaluation: Average Precision summarizes the precision-recall curve and reflects average positive predictive value for identifying progressors.
- Evaluation: Stratified bootstrapping with 2,000 iterations estimates AUC and AP confidence intervals, while DeLong’s test assesses model differences.
Data Availability Statement
The study uses publicly available OAI and MOST datasets and releases its implementation resources online. These resources include the Dockerfile, source code, pretrained models, and related data.
- OAI and MOST are publicly available datasets that can be requested through their stated websites.
- The Dockerfile, source code, pretrained models, and other relevant data are publicly available in the project repository.
Supplementary data
The supplementary material documents the datasets, cohort characteristics, KL-grade organization, and GradCAM attention-map examples used across training, testing, and progression categories.
- Dataset characteristics: MOST test participants had mean age 62.50±8.11 years and mean BMI 30.74±5.97, including 1,303 females and 826 males.BMI denotes Body Mass Index.
- Dataset organization: Dataset tables organize knee subsets by KL grade, with categories ranging from grade 0 through grade 4.The supplied table heading lists KL-grade categories 0, 1, 2, 3, and 4.
- Data selection: The OAI dataset was used to train the model, while the MOST dataset was used to test it.Figures 1 and 2 present the respective data-selection flowcharts.
- Attention-map examples: GradCAM attention maps illustrate knees progressing from no osteoarthritis to osteoarthritis, with fine-grained progression subtypes specified.The examples are presented at 140×140 mm.
- Attention-map examples: Additional GradCAM examples cover baseline osteoarthritis knees that later progressed, baseline non-osteoarthritic knees that progressed within seven years, and early-OA knees that did not progress within seven years.Baseline KL grades are specified for the latter two example groups.