Source-linked AI summary
Medical Image Retrieval using Deep Convolutional Neural Network
Adnan Qayyum, Syed Muhammad Anwar, Muhammad Awais, Muhammad Majid
TL;DR
The paper addresses the difficulty of retrieving relevant images from rapidly growing medical repositories and the semantic gap between low-level visual features and human-perceived meaning. It trains a deep CNN on multimodal medical-image classes, then uses learned features and class predictions for retrieval, achieving 99.77% average classification accuracy and 0.69 mean average precision.
Problem
Rapidly growing medical-image repositories and the semantic gap between device-captured features and human-perceived semantics complicate content-based medical image retrieval.
Method
A deep CNN is trained end-to-end to classify 2D medical images by body-part or organ classes and provide learned features for retrieval.
Results
99.77% average classification accuracy and 0.69 mean average precision are achieved, with retrieval improving when predicted class information limits the search.
Takeaways & Limitations
The framework is designed to retrieve multimodal medical images across different body organs using learned representations and class-based search.
Abstract
from arXiv · showhide
With a widespread use of digital imaging data in hospitals, the size of medical image repositories is increasing rapidly. This causes difficulty in managing and querying these large databases leading to the need of content based medical image retrieval (CBMIR) systems. A major challenge in CBMIR systems is the semantic gap that exists between the low level visual information captured by imaging devices and high level semantic information perceived by human. The efficacy of such systems is more crucial in terms of feature representations that can characterize the high-level information completely. In this paper, we propose a framework of deep learning for CBMIR system by using deep Convolutional Neural Network (CNN) that is trained for classification of medical images. An intermodal dataset that contains twenty four classes and five modalities is used to train the network. The learned features and the classification results are used to retrieve medical images. For retrieval, best results are achieved when class based predictions are used. An average classification accuracy of 99.77% and a mean average precision of 0.69 is achieved for retrieval task. The proposed method is best suited to retrieve multimodal medical images for different body organs.
1. Introduction
Medical image repositories are expanding rapidly, creating a need for retrieval systems that help clinicians manage and search large datasets. The paper investigates deep CNNs for learning medical-image representations that reduce the semantic gap while requiring less annotation effort.
- Motivation: Rapid growth in hospital imaging data increases the need for effective medical image retrieval systems.Such systems are intended to aid clinicians in browsing large medical-image datasets.
- CBIR challenge: CBIR searches large databases using image-derived features and similarity measurements in a feature space.Feature representation and similarity measurement are identified as critical components.
- CBIR challenge: The semantic gap separates imaging-device visual information from the higher-level information perceived by human vision.The paper describes machine learning and domain knowledge as routes for reducing this gap.
- Deep learning motivation: Deep learning models high-level data abstractions through multiple nonlinear transformations and can learn features automatically at multiple abstraction levels.This motivates its application to medical image retrieval.
- Proposed direction: The proposed CNN learns representations for different imaging modalities and body organs using weak global body-part labels for 2D slices.This labeling strategy reduces the annotation effort required for training.
- Contributions: The paper contributes a multimodal dataset, a trained deep-learning framework, and a retrieval system based on learned features for large medical-image collections.The stated contributions cover dataset construction, model training, and retrieval.
2. Related Work
Prior CBIR systems rely on feature extraction and similarity search, while medical retrieval faces growing databases and clinical-use demands. Related work spans global, local, handcrafted, and deep-learning representations, motivating the proposed DCNN framework.
- CBIR systems: Generic CBIR systems extract features offline and compare query features with database features online using similarity or distance.Images with high similarity or low distance are presented as retrieval results.
- Feature descriptors: CBIR feature representations include global shape, color, and texture descriptors alongside local SIFT, SURF, and Bag of Words methods.These descriptors have been used in image-retrieval studies.
- Medical retrieval: Medical image collections are expanding with hospital PACS, increasing the need for retrieval systems that can support clinical browsing and decision-making.The related-work discussion connects retrieval with similar images and case histories.
- Medical retrieval methods: Earlier medical retrieval approaches include SIFT-based Bag of Visual Words for brain MRI and filtering, similarity fusion, and SVM classification for skin lesions.Class prediction can reduce the search area by eliminating irrelevant images.
- Deep learning: Deep learning uses multiple processing layers and linear or nonlinear transformations to model high-level abstractions in data.Its recent progress is associated with improved computation, deeper nonlinear networks, and better initialization.
- Medical deep learning: Deep-learning methods have been applied to medical classification, including a CNN system that characterized lung patterns with 85.5% classification performance.The cited example used seven classes, including six interstitial lung-disease patterns and healthy tissue.
- Proposed framework: The proposed framework is presented as a DCNN-based medical retrieval system with convolutional, pooling, and fully connected layers.Its block diagram is identified as the proposed framework for content-based medical image retrieval.
3. Methodology
The methodology uses a supervised deep convolutional neural network to classify medical images and support retrieval through learned feature representations and predicted class labels. Training combines convolutional, pooling, and fully connected layers with backpropagation and stochastic gradient descent.
- 3.1 Phase 1: Classification: The proposed framework trains a deep convolutional neural network to classify medical images using supervised learning.The network learns feature hierarchies and the classifier end to end from training images.
- 3.1.1 The DCNN Model Architecture: The DCNN contains five convolutional and three fully connected layers, with FCL3 producing 24 class probabilities through softmax.It accepts grayscale 224 × 224 images.
- 3.1.1 The DCNN Model Architecture: Pooling layers downsample convolutional feature maps, while dropout after the first two fully connected layers is used to reduce overfitting.Pooling follows the first, second, and fifth convolutional layers.
- 3.1.2 Training Details: Training uses stochastic gradient descent with backpropagation, a learning rate of 0.0001, up to 30 epochs, and negative log likelihood as the objective.Weights are initialized from a Gaussian distribution with mean zero and standard deviation 0.01.
- 3.1.2 Training Details: Backpropagation computes forward activations, backward loss gradients, and parameter derivatives so SGD can update model weights and biases.The forward pass computes z values, while the backward pass recursively computes δ values.
- 3.2 Phase 2: Features Extraction for CBMIR: For retrieval, features from the last three fully connected layers are compared with database features using Euclidean distance, while predicted class labels restrict the search area.Lower distances correspond to higher similarity in the displayed retrieval results.
4. Experimental Results
The experiments evaluate a 24-class multimodal medical-image dataset, DCNN classification, and retrieval with or without predicted class labels. Class prediction improves retrieval quality, while the dataset includes interclass and intra-class variation.
- Dataset: The dataset contains 24 medical-image classes collected from public databases, with 7,200 images split into 5,040 training and 2,160 testing images.Each class contributes 300 randomly selected images, and the training and testing sets contain no identical images.
- Classification Performance: The classification framework was evaluated using average precision, average recall, accuracy, and F1 measure.The reported definitions use TP, FP, TN, and FN across N = 24 classes.
- Classification Performance: 99.76% average precision, 99.77% average recall, and 99.76% average F1 measure were obtained on the classification evaluation.Each class achieved 100% average accuracy except stomach, liver, and bladder, which achieved 98.9%, 97.7%, and 98.9%, respectively.
- Retrieval Performance: Features from FCL1, FCL2, and FCL3 were evaluated for retrieval using precision–recall analysis with and without class prediction.Retrieved images for chest and renal queries were shown in ranked order under both retrieval strategies.
- Retrieval Performance: 0.53 mean average precision without class predictions and 0.69 with class prediction were achieved for retrieval.With class prediction, retrieved images belong only to the class predicted by the classification framework.
- Comparison: The comparison uses classification metrics and mean average precision because no standard medical dataset was available for direct retrieval benchmarking.A cited higher-mAP system used a single modality, whereas the proposed system handled multimodal data.
5. Conclusion
The paper presents a DCNN-based CBMIR framework that learns discriminative features through medical-image classification. It reports high classification accuracy and improved multimodal retrieval when class prediction is used.
- Conclusion: The framework trains a deep convolutional neural network to classify medical images and uses the learned features for content-based retrieval.The last three fully connected layers provide features for retrieval, with class prediction used as one retrieval strategy.
- Conclusion: 0.69 mean average precision was achieved for multimodal retrieval with class prediction.The conclusion identifies class-prediction retrieval as the reported multimodal retrieval result.