PaperlayerView exact revision on arXiv

Source-linked AI summary

From Hand-Crafted to Deep Learning-based Cancer Radiomics: Challenges and Opportunities

Parnian Afshar, Arash Mohammadi, Konstantinos N. Plataniotis, Anastasia Oikonomou, Habib Benali

arXiv:1808.07954v3cs.CV

TL;DR

Radiomics seeks quantitative or semi-quantitative imaging features for cancer diagnosis, prognosis, and personalized treatment, but biopsy and radiomics workflows face important limitations. The paper presents an integrated interdisciplinary sketch spanning hand-crafted, deep, and hybrid radiomics, while highlighting data, standardization, explainability, and computational challenges.

  • Problem

    Biopsy cannot capture all information from spatially and temporally heterogeneous tumors, motivating imaging-based cancer assessment and prognostic modeling.

  • Method

    The paper synthesizes hand-crafted, deep, and hybrid Radiomics approaches alongside processing modules and supporting image, clinical, and genomic data sources.

  • Results

    The review presents practical application examples and an integrated account of Radiomics resources, workflows, and signal-processing solutions.

  • Takeaways & Limitations

    Radiomics can assist cancer diagnosis and prognosis, while combining imaging with genomic biomarkers can improve prediction accuracy and survival-curve representation.

  • Takeaways & Limitations

    Deep Radiomics requires substantial training data and computational resources, while patient-data access is constrained by ethical regulations.

Abstract

from arXiv · show

Recent advancements in signal processing and machine learning coupled with developments of electronic medical record keeping in hospitals and the availability of extensive set of medical images through internal/external communication systems, have resulted in a recent surge of significant interest in "Radiomics". Radiomics is an emerging and relatively new research field, which refers to extracting semi-quantitative and/or quantitative features from medical images with the goal of developing predictive and/or prognostic models, and is expected to become a critical component for integration of image-derived information for personalized treatment in the near future. The conventional Radiomics workflow is typically based on extracting pre-designed features (also referred to as hand-crafted or engineered features) from a segmented region of interest. Nevertheless, recent advancements in deep learning have caused trends towards deep learning-based Radiomics (also referred to as discovery Radiomics). Considering the advantages of these two approaches, there are also hybrid solutions developed to exploit the potentials of multiple data sources. Considering the variety of approaches to Radiomics, further improvements require a comprehensive and integrated sketch, which is the goal of this article. This manuscript provides a unique interdisciplinary perspective on Radiomics by discussing state-of-the-art signal processing solutions in the context of Radiomics.

I. INTRODUCTION

Radiomics extracts quantitative and semi-quantitative information from medical images to support predictive, prognostic, and diagnostic modeling. This article surveys hand-crafted, deep learning-based, and hybrid approaches from a signal processing perspective, alongside applications, challenges, and opportunities.

  • Unlike invasive, focal biopsy, imaging can provide noninvasive information relevant to heterogeneous tumors, although interpretation may be time-consuming and experience-dependent.
  • Radiomics extracts semi-quantitative and quantitative image features to develop predictive and prognostic models and assess tumor phenotype and heterogeneity.
  • The field has expanded beyond traditional computer-aided diagnosis by extracting hundreds to thousands of features for diagnostic, predictive, and prognostic applications.
  • Radiomics includes hand-crafted pipelines, deep learning-based feature discovery, and hybrid schemes that combine multiple feature or data sources.
  • The article provides an integrated signal processing perspective covering applications, hand-crafted methods, deep architectures, hybrid solutions, and open challenges.
  • CT and PET protocols introduce standardization challenges because feature properties depend on acquisition parameters, scanner characteristics, calibration, and patient conditions.

II. APPLICATIONS

Radiomics has been applied across cancer-related tasks including diagnosis, detection, classification, treatment response, survival, recurrence, and risk prediction. The reviewed studies also compare automated systems with experts and combine imaging with clinical or genomic information.

  • Automatic diagnosis systems have potential to achieve human-level performance and serve as one of two experts, while raising patient-privacy concerns requiring strict regulation.
  • Radiomics studies use modalities including CT, PET, MRI, and mammography across lung, breast, prostate, brain, bladder, kidney, and liver cancer tasks.
  • Applications can combine hand-crafted imaging features with clinical or genomic biomarkers for tasks such as lung-cancer recurrence prediction.
  • The field spans oncology as well as cardiology and neurology, although cancer-related applications have received particular research interest.
  • Radiomics applications include cancer diagnosis, tumor detection, classification, treatment-response prediction, survival estimation, recurrence prediction, and cancer-risk assessment.
  • Radiomics may support early diagnosis and targeted treatment by characterizing malignant or heterogeneous tumor regions and locating abnormalities precisely.

III. STATE-OF-THE-ART IN HAND-CRAFTED RADIOMICS

Hand-crafted Radiomics (HCR) combines image processing, segmentation, feature extraction, and statistical analysis to derive clinically relevant information from medical images. Its clinical use is motivated by biopsy limitations and requires accurate, reproducible processing stages.

  • Biopsy is invasive, focal, and limited by spatial and temporal tumor heterogeneity, so it may not capture information needed for inclusive cancer decisions.
  • Radiomics can complement biopsy non-invasively by capturing intra-tumoral heterogeneity and identifying suspicious locations for imaging-guided biopsy.
  • Hand-crafted Radiomics studies typically follow sequential steps beginning with pre-processing and segmentation before feature extraction and statistical analysis.
  • Segmentation is critical because HCR features are extracted from segmented regions, while manual delineation is time-consuming and sensitive to inter-observer variability.
  • Automatic and semi-automatic segmentation methods may use predefined features or deep-learning techniques to classify tumorous and non-tumorous image regions.

A. Radiomics Feature Extraction

Radiomics feature extraction represents segmented regions through intensity, shape, texture, and higher-order descriptors. These feature families capture complementary properties, but some depend strongly on acquisition settings or parameter choices.

  • HCR features are commonly organized into first-order intensity and shape features, second-order texture features, and higher-order features.
  • Intensity-based Features: Intensity-based features summarize ROI pixel-intensity histograms using measures such as energy, entropy, kurtosis, and skewness.
  • Intensity-based Features: Intensity features are sensitive to image acquisition parameters such as slice thickness, and histogram-bin selection can make them unreliable representatives.
  • Shape-based Features: Shape-based features quantify 2D or 3D tumor geometry, including volume, surface, sphericity, compactness, diameter, and flatness.
  • Texture-based Features: Texture-based features capture spatial relationships among neighboring pixels and are especially informative when tissue heterogeneity matters.
  • Texture-based Features: Common texture matrices include GLCM, GLRLM, NGTDM, and GLZLM, which respectively characterize intensity pairs, equal-intensity runs, neighborhood differences, and connected intensity zones.
  • Higher Order Radiomics Features: Higher-order wavelet and Fourier features capture imaging biomarkers across frequencies, with wavelet coefficients representing texture and gradient information.
  • Minkowski functionals are computed from thresholded binary ROIs, with area and perimeter derived from white pixels and edges.

B. Radiomics Feature Reduction Techniques

Feature reduction is necessary because Radiomics produces many correlated, irrelevant, or overfitting-prone variables. Supervised and unsupervised methods balance relevance, redundancy, reproducibility, and generalization differently.

  • Feature reduction addresses high-dimensional Radiomics feature sets in which variables may be correlated, irrelevant, or liable to make models sensitive to noise.
  • Feature-selection objectives include reproducibility, informativeness and relevancy, and low redundancy among selected features.
  • Supervised Feature Selection Methodologies: Filtering methods evaluate each feature’s relationship with the class label independently, whereas wrapper methods assess the combined predictive performance of feature subsets.
  • Supervised Feature Selection Methodologies: Wrapper methods combine relevancy and redundancy in their score, but computational difficulty prevents testing every possible feature subset.
  • Supervised Feature Selection Methodologies: Supervised feature-selection methods use class labels but can overfit and may not transfer easily across applications after training on a given feature set.
  • Unsupervised Feature Selection Methodologies: Unsupervised methods remove redundant features without labels and are less prone to overfitting, but they do not guarantee an optimum feature space.

C. Radiomics Statistical Analysis

Radiomics statistical analysis applies extracted features to classification, regression, and survival tasks. Common survival tools include Kaplan-Meier curves, Cox proportional hazards regression, and the Log-Rank test, while model and metric choices affect evaluation.

  • Radiomics statistical analysis uses extracted features for tasks including cancer diagnosis, tumor-stage classification, and survivability analysis.
  • Classification and Regression: Classification and regression models predict discrete or continuous outcomes, with Random Forest, Support Vector Machine, and Neural Network among commonly used techniques.
  • Survivability Analysis: Survivability analysis estimates survival time or survival probability from prior-patient data and can help assess treatment effectiveness.
  • Kaplan-Meier Survival Curve: The Kaplan-Meier curve represents survival probability over time and can compare groups formed by thresholds on a selected Radiomics feature.
  • Cox Proportional Hazards Model: The Cox proportional hazards model predicts risk of dying at time t from Radiomics covariates, with coefficients representing predictor impacts and h0(t) the baseline hazard.
  • Deep Learning Risk Models: Radiomics risk can be modeled as a learned nonlinear function through deep-learning architectures rather than only as a linear combination of features.
  • Log-Rank Test: The Log-Rank test compares survival curves between samples, but group size can influence results and motivates larger, more evenly sized groups.
  • Evaluation: Accuracy alone is inadequate when false-positive and false-negative errors have unequal clinical consequences, so ROC area and confusion matrices are useful evaluation measures.

D. Radiomics Stability

Radiomics stability measures whether extracted features remain consistent across repeated imaging and observer delineations. The section presents ICC and Friedman testing, while IBSI addresses reproducibility across the broader pipeline.

  • Stability assessment: Radiomics stability quantifies feature dependency on preprocessing steps and is evaluated through test-retest or inter-observer reliability experiments.Test-retest uses repeated imaging examinations, whereas inter-observer analysis uses ROI delineations by multiple observers.
  • ICC: The intra-class correlation coefficient measures feature reliability from 0, indicating no reliability, to 1, indicating complete reliability.ICC uses mean-square terms from ANOVA; repeated examinations define the test-retest setting, while residual mean squares and observer count define the inter-observer setting.
  • Friedman test: The Friedman test is a nonparametric repeated-measures test for differences among multiple observations in inter-observer settings.It does not require a Gaussian population, and features ranked 1 are considered the most stable.
  • Implications: Higher-stability features have been reported to show greater prognostic performance, so stability analysis can also function as feature reduction.The cited feature groups include Laplacian of Gaussian, intensity-based, and texture features.
  • Reproducibility: IBSI provides standardized definitions, guidelines, and steps spanning acquisition, preprocessing, segmentation, and feature calculation to address reproducibility.The initiative is described as an international collaboration responding to reproducibility challenges in Radiomics.

E. Radiogenomics

Radiogenomics links imaging characteristics with genomic properties to support noninvasive biological investigation and clinical decision-making. The section also surveys deep Radiomics inputs, architectures, training choices, and segmentation trade-offs.

  • Radiogenomics: Radiogenomics relates imaging characteristics to gene expression, mutations, and other genome-related characteristics, potentially supporting treatment-response prediction without biopsy.One cited study found that 28 Radiomics features reconstructed 78% of global gene expressions in human liver cancer cells.
  • GSEA: Genes are ranked by Spearman correlation with a Radiomics feature, and GSEA tests whether biologically related gene sets cluster near the top or bottom of that ranking.The resulting heat map represents associations between gene sets and Radiomics features.
  • Clinical data gaps: Radiogenomics can leverage datasets lacking clinical outcomes by using prior knowledge about relationships among genomics and clinical outcomes.This enables analysis when outcomes are difficult or time-consuming to collect, within the scope described by the paper.
  • Deep Radiomics: Deep Radiomics extracts task-relevant image features through architectures such as CNNs or auto-encoders for diagnosis, cancer-type prediction, or survival prediction.The network input may be the original image, segmented ROI, or both, and features may feed the network or an external decision model.
  • Segmentation: Eliminating segmentation reduces delineation time and cost, while manual annotations are observer-dependent and automatic segmentation remains error-prone.Deep networks can alternatively combine original, segmented, preprocessed, or differently oriented images as multi-channel inputs.
  • DLR design choices: DLR studies vary by input hierarchy, model initialization, and network architecture, using slices, volumes, or patient examinations as inputs.Variable slice counts can motivate architectures such as recurrent neural networks, while models may be fine-tuned or trained from scratch.

A. Input Hierarchy

DLR inputs are organized from independent slices to complete volumes and patient examinations, while training strategies address limited data and class imbalance. These choices define how features are extracted and used for prediction.

  • Input hierarchy: DLR input hierarchy comprises slice-level, volume-level, and patient-level analysis.These correspond respectively to individual slices, all slices from one visit, and all volumes from a patient.
  • Input hierarchy: Slice-level classification analyzes images independently, but it may not adequately support decisions based on an entire volume of interest.This limitation motivates volume-level approaches.
  • Training constraints: Training from scratch is problem-specific but can be limited by overfitting and class imbalance when privacy and expert labeling constrain medical datasets.Positive labels are commonly fewer than negative labels, which can bias classifiers toward the negative class.
  • Feature pathways: Deep Radiomics may use the original image, segmented ROI, or both, and extracted features can continue through the network or feed an external decision model.This describes alternative feature-extraction pathways rather than a single fixed architecture.
  • Input hierarchy: Volume-level processing must handle patients having different numbers of slices, for which recurrent neural networks are one possible variable-size architecture.Patient-level analysis supplies all examination-associated volumes as one input.
  • Mitigation strategies: Data augmentation, multitask training, loss-function modification, and transfer learning are described as responses to inadequate data or class imbalance.Transfer learning uses natural-image pretraining followed by fine-tuning on the target medical dataset.

C. Deep Learning Architectures in Radiomics

Deep learning-based Radiomics extracts learned features through discriminative and generative architectures, including CNNs, RNNs, auto-encoders, and Capsule networks. These architectures support multiscale inputs, sequential analysis, reconstruction-based representation learning, and transformation-aware classification.

  • Architectural categories: Deep Radiomics uses discriminative networks to learn class-distinguishing features and generative networks to learn representations without class labels.CNNs and RNNs are reviewed as discriminative architectures, while auto-encoders and deep belief networks provide generative alternatives.
  • Convolutional Neural Networks: CNNs automatically learn convolutional kernels rather than relying on pre-designed features, allowing feature extraction without prior domain knowledge.Fully connected outputs or features from layers before classification can serve as DLR features for classification or the remainder of the Radiomics pipeline.
  • Convolutional Neural Networks: Multiple CNNs incorporate patches across modalities, angles, and scales; one lung-tumor system used sagittal, coronal, and axial patches at different scales.Scale selection depends on the distinction being made, such as separating tumors from vessels or solid from non-solid tumors.
  • Convolutional Neural Networks: A Multi-view CNN used 7 resized patches at different scales and achieved higher feature separability than a single-scale CNN for ternary lung-tumor classification.The study classified benign, primary malignant, and metastatic malignant tumors, with deeper layers learning features of higher separability.
  • Capsule Networks: Capsule networks use routing by agreement to preserve spatial relations, while capsule-vector lengths represent the probability that an instance is present.Capsule predictions are coupled according to agreement, supporting robustness to affine transformations and rotations.
  • Capsule Networks: CapsNet outperformed CNN classification for both original brain images and tumor masks, with mask inputs performing better because backgrounds distracted feature extraction.The cited CNN baseline achieved 78% accuracy for tumor images; CapsNet’s proposed advantages include handling small datasets and transformation robustness.
  • Recurrent Neural Networks: LSTMs address the vanishing-gradient problem by learning what to store and forget, and sequential ultrasound classification achieved higher predictive accuracy than independent-image decisions.Shared weights reduce the training time and cost associated with recurrent architectures.
  • Generative architectures: Auto-encoders learn latent variables that reconstruct input images, allowing the encoded representations to function as Radiomics features.Denoising auto-encoders add input noise to capture more robust features, and these features can be supplied to an SVM for lung-tumor classification.

D. Explainability of Deep Learning-based Radiomics

Explainability methods aim to reveal why deep Radiomics models make decisions and to connect learned features with clinically recognizable image landmarks. The reviewed approaches include expert checking, feature visualization, sensitivity analysis, feature-space projection, and CapsNet reconstruction.

  • Motivation: Deep Radiomics requires interpretability because physicians need learned features related to image landmarks, while explanations can expose model failure points.The paper frames explainability as relevant both to improving models and to reducing risks from incorrect medical decisions.
  • Expert validation: Expert double-checking can validate automatic predictions, but it is time-consuming and requires complete supervision.The paper presents this trade-off as conflicting with Radiomics’ aim of automatized and personalized treatment.
  • Feature visualization: Feature visualization examines filters or feature maps across model layers to reveal what kinds of image features the network is seeking.Features learned in later layers are described as more informative because those layers are most responsible for the network output.
  • Sensitivity analysis: Sensitivity analysis produces heat maps of image regions influencing predictions by measuring the effect of changing individual input pixels.Brighter heat-map areas indicate regions with greater influence on the prediction.
  • Feature-space projection: t-SNE projection can reveal clearly separated lung-tumor clusters, verifying discriminating feature learning without identifying the exact decision rationale.The method reduces the deep network’s high-dimensional feature space to a two-dimensional plane.
  • Capsule reconstruction: CapsNet improves interpretability by reconstructing the input image from the final features used for classification.Feature-vector values can be perturbed before reconstruction to visualize how learned features affect the reproduced image.

V. HYBRID SOLUTIONS TO RADIOMICS

Hybrid Radiomics combines hand-crafted and deep features or integrates Radiomics with other data sources when either category alone may be insufficient. The paper compares the two approaches across data requirements, processing, interpretability, and applications before reviewing hybrid strategies.

  • Workflow comparison: Hand-crafted features typically come from segmented regions and are followed by feature selection, whereas deep Radiomics does not necessarily require segmented inputs or feature selection.The comparison is summarized in Table VII.
  • Data requirements: Hand-crafted features do not require big datasets because they are defined independently of data, while deep Radiomics requires huge datasets to learn features.The table contrasts predefined feature construction with data-dependent feature learning.
  • Computational cost: Hand-crafted Radiomics usually has insignificant processing time, whereas deep Radiomics can incur high computational cost depending on architecture and dataset size.This contrast is listed as a comparison between the two feature categories.
  • Interpretability: Hand-crafted features are tangible because they are pre-designed, but the logic behind deep features and decisions remains a black box.The comparison distinguishes feature transparency from decision transparency.
  • Publication trends: Figure 9 compares publication popularity for hand-crafted and deep-learning Radiomics across four distinct applications using Google Scholar data.The figure provides an application-level comparison of research activity rather than a model-performance evaluation.
  • Comparative findings: Hand-crafted and deep Radiomics are reported to be on a par in tumor classification and detection, while deep-learning publications are fewer for survival and recurrence prediction.The paper suggests domain knowledge may be less accessible in deep-learning methods for those prediction tasks.
  • Hybrid strategies: Hybrid strategies combine Radiomics with other data sources or jointly combine hand-crafted and deep features.They are proposed for scenarios where neither approach alone provides sufficiently informative Radiomics features.

A. Combination of Radiomics and Other Data Sources

Radiomics can be combined with complementary imaging, clinical, staging, and genomic information to improve cancer prediction and prognosis.

  • Combining CT and PET Radiomics features in a classifier produced higher lung cancer survival-prediction accuracy than either modality separately.
  • Fusing multiple MRI-derived Radiomics representations outperformed a single-modal classifier for brain tumor classification.
  • Radiomics combined with lung cancer staging information improved prognostic performance compared with Radiomics or staging alone.
  • Radiomics integrated with clinical risk factors such as age can form nomograms for visually calculating brain cancer survival probability.
  • Averaging classifiers trained on hand-crafted Radiomics and genomic biomarkers increased lung cancer recurrence-prediction accuracy and more closely resembled the survival Kaplan-Meier plot.

B. Fusion of HCR with DLR (i.e., Engineered Features Coupled with Deep Features)

Hybrid Radiomics combines hand-crafted and deep features through decision-level or feature-level fusion, seeking to exploit their complementary strengths while managing high dimensionality.

  • Hybrid Radiomics is motivated by the distinct advantages and disadvantages of engineered and deep features.
  • Decision-level Fusion: Soft voting averages classifier probability outputs, and combined hand-crafted/deep models achieved higher breast cancer prediction accuracy across mammography, ultrasound, and MRI.
  • Decision-level Fusion: Adaptive voting learns contribution weights because classifiers may not contribute equally to the final decision.
  • Feature-level Fusion: Feature-level fusion concatenates deep and hand-crafted feature vectors before classification and achieved the highest lung cancer survival-prediction performance in one study.
  • Feature-level Fusion: Concatenating many feature types can cause over-fitting when training samples are relatively fewer than features, motivating stability, informativeness, redundancy analysis, and dimensionality reduction.

A. Challenges of Hand-Crafted Radiomics

Hand-crafted Radiomics faces feature-design, segmentation, data, standardization, annotation, and fusion challenges that constrain reliable clinical development.

  • Hand-crafted feature extraction is difficult because cancerous tissues often exhibit limited distinct visual variation, encouraging broad low-level and high-level feature extraction.
  • Hand-crafted Radiomics commonly requires segmented regions of interest, but expert annotations are costly and time-ineffective.
  • Deep Radiomics still lacks a systematic strategy for selecting optimal architectures and adapting natural-image models to medical-image signals.
  • Deep Radiomics requires large training datasets whose size increases with model complexity, while patient-data access is restricted by ethical regulations.
  • Cropping medical images to reduce computational cost can remove size information that discriminates normal from abnormal tissues.
  • Scanner protocols, image contrast, resolution, expert annotations, and class balance vary across institutions, complicating standardized comparison and reliable modeling.
  • More rigorous fusion rules are needed because hybrid Radiomics has shown promising initial results but remains in its infancy.
Loading 1808.07954v3…