Source-linked AI summary
Automatic Detection of Knee Joints and Quantification of Knee Osteoarthritis Severity using Convolutional Neural Networks
Joseph Antony, Kevin McGuinness, Kieran Moran, Noel E O'Connor
TL;DR
The paper addresses automatic quantification of knee osteoarthritis severity from X-ray images, requiring both joint localization and severity assessment. It introduces FCN-based localization and CNNs trained from scratch with combined classification and regression objectives, reporting improved overall quantification and simultaneous outputs. The approach is evaluated on OAI and MOST data, while grade 1 remains difficult to classify and KL categories are criticized as nonequidistant.
Problem
Automatic knee OA quantification requires reliable localization followed by classification, while prior approaches have limited localization accuracy and severity assessment has challenged consecutive grades.
Method
The paper combines an FCN for knee-joint localization with CNNs trained from scratch using a weighted ratio of categorical cross-entropy and mean-squared-error losses.
Results
The jointly trained classification-regression network improves overall quantification, and automatically localized joints achieve classification results comparable to manually segmented joints.
Takeaways & Limitations
The pipeline provides simultaneous multi-class classification and regression outputs for automatic knee OA severity quantification from localized joints.
Takeaways & Limitations
Classification involving KL grade 1 is challenging, and KL categories are not equidistant; OARSI readings might provide better results.
Abstract
from arXiv · showhide
This paper introduces a new approach to automatically quantify the severity of knee OA using X-ray images. Automatically quantifying knee OA severity involves two steps: first, automatically localizing the knee joints; next, classifying the localized knee joint images. We introduce a new approach to automatically detect the knee joints using a fully convolutional neural network (FCN). We train convolutional neural networks (CNN) from scratch to automatically quantify the knee OA severity optimizing a weighted ratio of two loss functions: categorical cross-entropy and mean-squared loss. This joint training further improves the overall quantification of knee OA severity, with the added benefit of naturally producing simultaneous multi-class classification and regression outputs. Two public datasets are used to evaluate our approach, the Osteoarthritis Initiative (OAI) and the Multicenter Osteoarthritis Study (MOST), with extremely promising results that outperform existing approaches.
1 Introduction
The paper presents an end-to-end pipeline that localizes knee joints and quantifies osteoarthritis severity from X-ray images. It combines FCN-based localization with CNN classification and joint classification-regression training.
- Motivation: Knee OA severity assessment is clinically important because severe disease causes pain and may lead to total joint arthroplasty, while X-ray remains the accessible preliminary diagnostic standard.Early diagnosis is described as crucial for clinical treatment and pathology.
- Pipeline: The proposed pipeline first detects and extracts knee-joint regions of interest, then classifies the localized knee-joint images.
- Localization: An FCN automatically localizes knee joints through end-to-end pixel-wise predictions and is designed to fit into the severity-quantification network.
- Classification: The classification component trains CNNs from scratch for multi-class classification and for a weighted combination of categorical cross-entropy and mean-squared-error losses.
- Contribution: Joint multi-objective training improves overall quantification while producing simultaneous multi-class classification and regression outputs.
2 Related Work
Earlier approaches used handcrafted features, template matching, SVMs, transfer learning, or regression to assess knee OA. The paper motivates CNN-based learning and improved automatic localization as alternatives to these limitations.
- Prior assessment methods: Earlier knee OA assessment methods treated severity primarily as an image-classification problem using approaches such as WND-CHARM and artificial neural networks.
- CNN approaches: CNNs have outperformed many handcrafted-feature methods in computer-vision tasks, and pretrained networks have been fine-tuned to classify knee OA images through transfer learning.
- Regression: Regression of continuous grades was argued to reduce mean-squared error and improve overall quantification compared with binary or multi-class accuracy alone.
- Automatic localization: Template matching was slow on large datasets, while the prior SVM detector had limited accuracy and precision and could compromise the aspect ratio of extracted knee joints.
3 Data
The study evaluates bilateral PA fixed-flexion knee X-rays from the OAI and MOST datasets using KL grades as the radiographic ground truth. KL grades span five severity categories from normal to severe.
- Datasets: Experiments use bilateral PA fixed-flexion knee X-ray images from the OAI and MOST datasets.Both are described as standard datasets for knee osteoarthritis studies.
- Ground truth: KL grading provides five radiographic severity categories: grade 0 normal, grade 1 doubtful, grade 2 minimal, grade 3 moderate, and grade 4 severe.
- OAI: The OAI selection contains 8,892 knee images distributed across grades 0 through 4 as 3,433, 1,589, 2,353, 1,222, and 295 images, respectively.
- Ground truth: Figure 1 presents the KL grading system used to assess knee OA severity.
4 Methods
The method uses a two-stage pipeline that localizes knee joints in X-rays with an FCN, then quantifies OA severity using CNN classification and regression. Jointly optimizing categorical cross-entropy and mean-squared error supports simultaneous categorical and continuous outputs.
- The pipeline first detects and extracts knee-joint regions of interest, then classifies the localized images for OA severity.
- Automatically Localizing Knee Joints using a FCN: The FCN predicts knee-joint regions from radiographs using pixel-wise outputs, enabling end-to-end localization.It uses convolutional stages, max pooling, up-sampling, and a final sigmoid layer producing an output with the input image size.
- Automatically Localizing Knee Joints using a FCN: Binary masks from manual annotations supervise FCN training, and contour detection converts predictions into bounding boxes for extracting the joints.Bounding boxes are upscaled to the original radiograph size to preserve the extracted joints' aspect ratio.
- Quantifying knee OA severity using CNNs: A CNN trained from scratch classifies localized knee images using a lightweight architecture with convolutional and fully connected layers.The classifier uses batch normalization, ReLU activations, max pooling, dropout, L2 regularization, and categorical cross-entropy loss.
- Quantifying knee OA severity using CNNs: The joint model optimizes a weighted ratio of categorical cross-entropy and mean-squared error for simultaneous classification and regression.Cross-entropy supplies grade-quantization information, while mean-squared error supplies information about grade ordering.
5 Experiments and Results
The experiments evaluate FCN-based knee-joint localization and CNN-based OA severity quantification, including classification-only and joint classification–regression training. Automatic localization is highly accurate, joint training improves classification and regression metrics, and grade 1 remains especially difficult to classify.
- 5.1 Localizing the Knee Joints using a FCN: The evaluation used OAI and MOST datasets, with training and test splits and Jaccard-index comparisons against manual annotations.The combined evaluation included 4,400 knee joints for the reported detection results.
- 5.1 Localizing the Knee Joints using a FCN: 99.9% of knee joints achieved J≥0.5, while 91.4% achieved J≥0.75, demonstrating highly accurate FCN-based automatic detection.These figures correspond to 4,396/4,400 and 4,020/4,400 joints, respectively.
- 5.2 Classification of Knee OA Images using a CNN: The CNN trained from scratch outperformed WND-CHARM and improved over earlier transfer-learning approaches for knee OA image classification.The authors attribute the improvement to a lightweight architecture with 5.4 million free parameters versus 62 million in BVLC CaffeNet, trained on roughly 10,000 samples.
- 5.3 Jointly trained CNN for Classification and Regression: Classification and regression results after automatic knee-joint localization were comparable to results after manual localization.This supports using the automatic localization stage within the full quantification pipeline.
- 5.3 Jointly trained CNN for Classification and Regression: Joint classification and regression training achieved 63.4% multi-class accuracy and 0.661 mean-squared error, versus 60.3% and 0.898 for classification-only training.The joint model optimizes a weighted ratio of categorical cross-entropy and mean-squared error, with regression weight fixed at 0.5.
- 5.3 Jointly trained CNN for Classification and Regression: Successive KL grades were challenging to distinguish, with grade 1 having lower classification metrics than the other grades.Misclassifications involved grade 1 predicted as grades 0, 2, or 3 and other grades predicted as grade 1; some images showed minimal variation in joint-space width and osteophytes.
6 Conclusion
The paper presents fully convolutional localization and jointly trained classification–regression networks for automatic knee OA quantification. The joint approach improves classification metrics, while fine-grained distinctions around KL grades 0–2 remain challenging.
- Fully convolutional networks automatically localize knee joints, while CNNs jointly classify and regress localized knee images; both networks are trained from scratch.
- The FCN-based localization method is highly accurate compared with previous methods, and classification using automatically localized joints is comparable with manually segmented joints.
- Joint classification and regression improve multi-class classification accuracy, precision, recall, and F1 score compared with the previous method.
- Classifying images conditioned on KL grade 1 is challenging because consecutive grades, particularly grades 0–2, have small variations.
- Future work will integrate localization and classification into an end-to-end network and compare automatic quantification with human-level accuracy to improve fine-grained classification.