PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Learning Feature Fusion Maps from PET-CT Images of Lung Cancer

Ashnil Kumar, Michael Fulham, Dagan Feng, Jinman Kim

arXiv:1810.02492v2cs.CV

TL;DR

PET-CT analysis must combine PET’s functional sensitivity with CT’s anatomical information, but existing approaches do not account for spatially varying modality priorities. The proposed supervised co-learning CNN derives explicit fusion maps from modality-specific features and significantly improves region detection and segmentation over baseline methods.

  • Problem

    Existing PET-CT methods process modalities separately or fuse them without accounting for how their visual information varies in importance across anatomical locations.

  • Method

    A supervised CNN encodes PET and CT features separately, derives spatially varying fusion maps, and uses them to weight modality-specific features before reconstruction and analysis.

  • Results

    The co-learning method significantly improved foreground detection and outperformed baseline CNN-based fusion and tumor-segmentation methods in PET-CT lung-cancer experiments.

  • Takeaways & Limitations

    Explicit spatially varying fusion maps provide a supported approach for integrating complementary functional and anatomical features across locations in multi-modality medical image analysis.

  • Takeaways & Limitations

    Experiments used 50 PET-CT images, and rescaling modalities to a common 256×256 resolution could lose CT boundary details or introduce PET interpolation noise.

Abstract

from arXiv · show

The analysis of multi-modality positron emission tomography and computed tomography (PET-CT) images for computer aided diagnosis applications requires combining the sensitivity of PET to detect abnormal regions with anatomical localization from CT. Current methods for PET-CT image analysis either process the modalities separately or fuse information from each modality based on knowledge about the image analysis task. These methods generally do not consider the spatially varying visual characteristics that encode different information across the different modalities, which have different priorities at different locations. For example, a high abnormal PET uptake in the lungs is more meaningful for tumor detection than physiological PET uptake in the heart. Our aim is to improve fusion of the complementary information in multi-modality PET-CT with a new supervised convolutional neural network (CNN) that learns to fuse complementary information for multi-modality medical image analysis. Our CNN first encodes modality-specific features and then uses them to derive a spatially varying fusion map that quantifies the relative importance of each modality's features across different spatial locations. These fusion maps are then multiplied with the modality-specific feature maps to obtain a representation of the complementary multi-modality information at different locations, which can then be used for image analysis. We evaluated the ability of our CNN to detect and segment multiple regions with different fusion requirements using a dataset of PET-CT images of lung cancer. We compared our method to baseline techniques for multi-modality image fusion and segmentation. Our findings show that our CNN had a significantly higher foreground detection accuracy (99.29%, p < 0.05) than the fusion baselines and a significantly higher Dice score (63.85%) than recent PET-CT tumor segmentation methods.

I. INTRODUCTION

PET-CT combines PET’s functional sensitivity with CT’s anatomical localization, but existing approaches do not adapt modality importance to spatially varying visual information. The paper introduces a supervised CNN that explicitly learns such fusion for multi-region lung-cancer image analysis.

  • Motivation: PET detects abnormal function while CT localizes it anatomically, making their combination useful for cancer diagnosis and staging.PET has lower spatial resolution, whereas CT adds precision to interpreting abnormal FDG uptake.
  • Limitations of existing approaches: Separate-modality methods can confuse collapsed lung tissue with tumors, while physiological cardiac PET uptake can resemble disease-related uptake.These examples motivate integrating functional and anatomical evidence rather than relying on either modality alone.
  • Limitations of existing approaches: Prior PET-CT tumor methods learned from tumor-centered patches without modeling feature variation across tumors in different anatomical locations.This limits their consideration of spatially differing PET and CT characteristics.
  • Contribution: The proposed CNN derives explicit spatially varying fusion maps from modality-specific features to quantify each modality’s importance at each location.This differs from implicit multi-channel fusion or later concatenation of modality-specific branches.
  • Evaluation: The study evaluates the CNN against baseline fusion and tumor-segmentation methods on PET-CT lung-cancer images.The intended framework supports tasks including visualization, classification, and segmentation.

II. METHODS

The method uses separate PET and CT encoders, co-learning, and reconstruction components to process paired image slices and produce region predictions. Training addresses multimodal feature-distribution changes with normalization, Leaky ReLU activations, augmentation, and cross-validation.

  • Dataset: The dataset contains 50 biopsy-proven NSCLC PET-CT scans selected to represent solitary tumors and varying hilar, mediastinal, and chest-wall involvement.The study formed 855 paired thorax slices containing all target regions for balanced CNN training.
  • Preprocessing: Both modalities were rescaled to 256 × 256 pixels so their spatial coordinates aligned for multimodal processing.PET was normalized using standard uptake values before thorax-slice selection.
  • Architecture: The CNN contains PET and CT encoders, a co-learning and fusion component, and a reconstruction component operating on axial 2D slices.The encoders extract modality-relevant features before fusion and reconstruction produce image-scale predictions.
  • Architecture: Each encoder uses four blocks with convolutional feature extraction and max pooling, while batch normalization reduces downstream distribution shifts.The design addresses cascading changes from the two modality-specific branches that could slow convergence.
  • Optimization: Leaky ReLU uses a nonzero negative-side gradient controlled by 0 < α < 1 to avoid dead neurons during training.This activation is applied after feature-map normalization.

D. Multi-modality Feature Co-Learning and Fusion

The co-learning component derives spatially varying fusion maps from stacked PET and CT feature maps, then uses those maps to weight modality-specific features. This enables location-dependent integration rather than a single global fusion ratio.

  • Co-learning unit: The co-learning unit receives PET and CT feature maps, stacks them, and applies a learnable 3D convolution across spatial and modality dimensions.The stacked tensor has dimensions w × h × m × c with m = 2 modalities.
  • Fusion operation: Element-wise multiplication of Ffusion with stacked modality-specific maps produces a fused co-learned feature map whose values are weighted modality features.The operation separately indicates CT and PET feature weighting in the conceptual fusion diagram.
  • Fusion-map derivation: The fusion map controls the importance assigned to PET and CT information at each location, unlike global PET-CT pixel-intermixing ratios.The authors omit normalization of this output so the learned map directly affects subsequent layer inputs.
  • Fusion-map derivation: The resulting fusion map reflects spatial integration of features from both modalities through the 3D neighborhood.The modality dimension is included in the convolution so PET and CT jointly determine the map.
  • Fusion operation: The network generates four fused feature maps, one for each pair of encoder blocks, and passes them to reconstruction.Stacking is denoted by ⊕ and element-wise multiplication by ⊗.

E. Reconstruction

The reconstruction component combines co-learned features across scales, upsamples them to input resolution, and converts per-pixel feature vectors into probability maps for target regions and an ‘other’ class.

  • Multi-scale reconstruction: Reconstruction blocks merge co-learned features with outputs from prior blocks, then upsample and refine them using convolutional layers.Nearest-neighbor upsampling doubles spatial dimensions before feature refinement.
  • Pixel-wise prediction: The final reconstruction output matches the input image dimensions and represents each pixel with a 64-channel feature vector.A 1×1 convolution maps these vectors to R + 1 region-class feature maps.
  • Pixel-wise prediction: The additional ‘other’ class captures pixels outside the target ROIs and makes the per-pixel class probabilities sum to 1.The experiments use lung fields, mediastinum, tumors, and ‘other’ as the four output classes.
  • Output visualization: Probability-map intensity encodes region membership confidence, with brighter pixels indicating higher probability for the displayed class.The example maps cover lung fields, mediastinum, tumors, and ‘other’ regions.

F. Network Training

The CNN was trained with a modified categorical cross-entropy objective, class-specific scaling, L2 regularization, data augmentation, and empirically selected parameters.

  • Training used stochastic mini-batch gradient descent with momentum, random cropping, and flipping to improve robustness and reduce overfitting.
  • Loss Function: The loss function extends categorical cross-entropy for pixel observations assigned to one of R + 1 ROI classes.
  • Loss Function: Class-specific scaling compensates for variation in the number of pixels belonging to each ROI class.
  • Loss Function: L2 regularization prioritizes lower weights across the CNN’s parameter set W.
  • Parameter Selection: Training parameters were empirically derived using two-fold cross-validation on the training data.

G. Experimental Design

The study evaluated co-learned PET-CT fusion for ROI detection and segmentation using cross-validation and comparisons with multiple fusion and tumor-segmentation baselines.

  • Experiments used five-fold cross-validation with identical training and test datasets for the proposed method and all baselines, and significance was assessed with two-sample t-tests.
  • The evaluation targeted detection and segmentation of lung, mediastinum, tumor, and other regions in lung-cancer PET-CT images.
  • Comparison with Fusion Baselines: Fusion comparisons replaced the co-learning component while retaining a similar architecture across multi-branch, multi-channel, and fused-input CNN baselines.
  • Comparison with Lung Tumor Segmentation Baseline: Tumor-segmentation comparisons included a co-segmentation method and a variational method, evaluated with tumor Dice scores.
  • Comparison with Lung Tumor Segmentation Baseline: Tumor baselines were tested with both tumor-centered patches and full 256 × 256-pixel PET-CT slices.
  • Evaluation of the Fusion Effects: Fusion effects were examined through learned fusion maps and visual comparisons between spatially varying and uniform fusion.

III. RESULTS

Across detection and segmentation experiments, the co-learning CNN outperformed fusion and tumor-segmentation baselines and produced spatially varying fusion behavior.

  • The co-learning CNN significantly exceeded all baseline fusion CNNs in Dice score for individual and collective foreground ROIs (p < 0.05).
  • The method improved upon all fusion baselines in 15 of 16 detection metrics, with 14 improvements statistically significant against all baselines.
  • The method consistently detected regions similar in size to ground truth, whereas MC under-detected and MB and FS over-detected regions.
  • The co-learning CNN achieved significantly higher tumor Dice scores than two recent PET-CT segmentation baselines (p < 0.05).
  • Visual segmentation comparisons showed three tumors detected across lung and mediastinal locations, with slight over-segmentation relative to ground truth.
  • Fusion maps assigned different priorities to image information across feature-map channels, with higher absolute intensity indicating greater fusion importance.
  • Uniform fusion was sensitive to the fusion ratio and image, whereas co-learned fusion provided consistent tumor detection across the compared studies.

IV. DISCUSSION

The co-learning method improved detection consistency and segmentation performance, which the authors attribute to spatially varying fusion maps integrating functional and anatomical features across locations.

  • The authors attribute improved detection and segmentation to spatially varying fusion maps that integrate functional and anatomical features across PET-CT locations.

A. Comparison with Baseline Fusion CNNs

The co-learning CNN outperformed fusion and segmentation baselines by preserving and spatially balancing complementary PET-CT information. It detected and segmented regions more consistently across full image slices, although lung Dice remained below competition state of the art.

  • A. Comparison with Baseline Fusion CNNs: The co-learning CNN achieved higher detection precision, sensitivity, specificity, and accuracy than the MB CNN across foreground ROIs individually and collectively.The authors attribute this to multi-scale feature fusion through multiple co-learning units, which limits information loss from standard pooling.
  • A. Comparison with Baseline Fusion CNNs: Early fusion in the MC CNN prioritized one modality over the other, detecting mainly the tumor portion with peak SUV while missing less subtle regions.The MC CNN combines PET and CT in its first convolutional layer without deriving modality-specific feature maps.
  • A. Comparison with Baseline Fusion CNNs: 79.11% precision for lungs but much lower precision for non-lung ROIs showed that the FS baseline’s pre-fusion was uneven across regions.Its detected tumor and mediastinum regions were larger than the ground truth, indicating more false positives.
  • A. Comparison with Baseline Fusion CNNs: The co-learning CNN produced more consistent ROI detections than FS, whose performance was sensitive to the uniform fusion-ratio parameter.This supports the potential robustness of learned spatially varying fusion over a predetermined fusion setting.
  • A. Comparison with Baseline Fusion CNNs: The co-learning CNN obtained a significantly higher Dice score than baseline fusion methods and baseline tumor-segmentation methods.The paper reports p < 0.05 for both comparisons, while full-slice tumor segmentation remained challenging for hilar and mediastinal involvement.
  • A. Comparison with Baseline Fusion CNNs: Approximately 91% lung Dice was below the approximately 95% state-of-the-art competition result because the model targeted general fusion rather than segmentation-optimized boundaries.The authors suggest segmentation-specific loss and multi-scale refinement as possible improvements.
  • A. Comparison with Baseline Fusion CNNs: Unlike the baselines, the CNN consistently segmented tumors from full PET-CT slices by balancing modality-specific information through fusion maps.It could segment tumors across different anatomical locations without cropping patches around the tumor.

C. Fusion Map Analysis

The co-learning CNN explicitly weights modality-specific features with spatially varying fusion maps, allowing different image regions to prioritize different modalities. Fusion-map analysis supports this design while identifying dataset, preprocessing, and architectural boundaries.

  • Fusion mechanism: The CNN multiplies modality-specific feature maps by image-specific fusion maps before subsequent convolutions, unlike baselines that merge modalities without spatial relevance.This prioritization occurs after the fusion map is derived.
  • Spatial analysis: Fusion maps prioritize CT information in lung fields relative to tumor-containing areas, showing distinct fusion requirements across structures.The model can derive these maps end to end without separate tasks for each region of interest.
  • Spatial analysis: Visually similar fusion channels can contain subtly different weight distributions, supporting finer integration than uniform fusion for heterogeneous tumors.The apparent redundancy is attributed to display normalization of 32-bit floating-point images.
  • Experimental design: The baseline comparison kept CNN architectures similar so that the principal experimental difference was the presence of the co-learning component.This design limited confounding architectural differences between methods.
  • Scope: The method requires manually selected thorax extent and common coordinate space, while rescaling can lose CT boundary detail or add PET interpolation noise.The dataset contained 50 PET-CT images, and larger datasets may provide more opportunities to learn fusion characteristics.
  • Experimental design: The experiments used grayscale PET inputs, excluding color lookup tables because colorization might bias functional information.The authors identify color information as a future source of additional visual features.
  • Scope: The lung-cancer experiments do not establish performance across whole-body diseases with more varied anatomy and unknown region-specific fusion requirements.The authors identify lymphoma and metastatic cancer as examples requiring further evaluation.

V. CONCLUSION

The paper concludes that explicit fusion maps improve multi-modality PET-CT analysis and may generalize to related medical imaging applications. Cross-validation was used to check architecture and training parameters for the dataset.

  • Conclusion: The supervised CNN significantly improved region detection and segmentation over several baseline CNN-based multi-modality methods on lung-cancer PET-CT images.The conclusion reports statistical significance at p < 0.05.
  • Conclusion: The proposed architectural component explicitly derives fusion maps that quantify modality-feature importance across spatial locations.The approach is presented as potentially useful for applications such as PET-CT and PET-MR analysis.
  • Validation: Cross-validation used the 40 studies in Fold 1, split into two sets of 20 studies without placing slices from one study in both sets.This verification assessed whether the architecture and training parameters were appropriate for the dataset.

CT PET

The supplementary analyses examine detailed visual and histogram evidence for fusion maps, including all 128 maps from the first co-learning unit and tumor-region distributions. These materials support inspection of channel-specific behavior rather than introducing a separate performance result.

  • Fusion-map visualization: Supplementary Figure S5 presents all 128 fusion maps from the first co-learning unit at larger scale.It is an expanded version of Figure 8 from the main text.
  • Tumor-region analysis: Figure S7 compares tumor-region pixel distributions in the PET image with distributions from three generated fusion-map channels.The analysis uses the tumor region within the PET image as the region of examination.
  • Tumor-region analysis: The analysis computes intensity histograms within the ground-truth tumor region because high tumor intensity makes visual differences among fusion maps difficult to ascertain.The same tumor-region mask is used for PET and fusion-map pixels.
  • Tumor-region analysis: The tumor PET distribution is heterogeneous, with an approximate maximum SUV of 20, a mode near SUV 10, and a mean SUV of 12.The distribution is skewed toward lower SUV values.
  • Tumor-region analysis: Fusion Map A has a relatively homogeneous tumor-region weight distribution, with minimum and maximum weights within two standard deviations of the mean.The passage offers this as a potential interpretation of how the map differentiates tumor from surrounding areas.

SIV. DATASET CREATION PROCESS

The dataset pipeline constructs aligned, SUV-normalized PET-CT inputs and region ground truths from 50 lung-cancer studies. It combines DICOM-derived preprocessing with thresholding and manual refinement steps.

  • Pipeline overview: Figure S8 provides a diagrammatic illustration of the image-preprocessing and ground-truth-creation pipeline.The supplementary material describes this as the dataset creation process.
  • Data acquisition: The process begins with 50 PET-CT lung-cancer studies and corresponding clinical reports retrieved from hospital PACS and RIS systems.The studies provide the imaging and report information used throughout dataset creation.
  • PET preprocessing: Study-specific SUV coefficients are calculated from DICOM headers, and PET volumes are transformed into SUV-normalized form.The clinical reports also provide each study’s peak tumor SUV.
  • Image alignment: CT and PET volumes are rescaled into a shared 256 × 256-pixel coordinate space using DICOM pixel-spacing information.This alignment enables the modalities to be processed together by the CNN.
  • Ground-truth creation: Adaptive thresholding extracts lung regions from the CT thorax subvolume, with parameters potentially requiring manual adjustment and subsequent mask refinement.Connected thresholding is also used to extract mediastinal regions.
Loading 1810.02492v2…