PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Spatial Attention Module for Targeting Multimodal PET-CT Lung Tumor Segmentation

Xiaohang Fu, Lei Bi, Ashnil Kumar, Michael Fulham, Jinman Kim

arXiv:2007.14728v2eess.IVcs.CVcs.LG

TL;DR

PET-CT segmentation remains challenging because PET’s high tumor sensitivity is accompanied by limited spatial resolution and physiologic high-uptake regions, while automated methods have not fully exploited PET to guide segmentation. The paper introduces MSAM, which learns PET-derived spatial attention for a CNN, and reports a 71.44% mean DSC, 7.59% above the previous state of the art for lung tumor segmentation.

  • Problem

    PET-CT segmentation is challenging because PET has limited spatial resolution and benign high-uptake regions can resemble tumors, while existing automated methods do not fully exploit PET’s sensitivity to guide segmentation.

    Source: §I · pp. 1–2

  • Method

    MSAM learns PET-derived spatial attention to emphasize tumor regions and suppress benign high-uptake areas while guiding a CNN for PET-CT segmentation.

    Source: §I · p. 2

  • Results

    71.44% mean DSC with MSAM and U-Net, 7.59% higher than the previous state of the art for lung tumor segmentation.

    Source: C. Comparison Against the… · p. 7

  • Takeaways & Limitations

    MSAM consistently improves segmentation across backbone CNNs and surpassed previous state-of-the-art methods for PET-CT lung tumor segmentation.

    Source: §IV · p. 7, §V · p. 9

    The attention architecture was not optimized, and the elementwise multiplication used to apply its map to CT features may be improved.

    Source: D. Future Work · p. 9

Abstract

from arXiv · show

Multimodal positron emission tomography-computed tomography (PET-CT) is used routinely in the assessment of cancer. PET-CT combines the high sensitivity for tumor detection with PET and anatomical information from CT. Tumor segmentation is a critical element of PET-CT but at present, there is not an accurate automated segmentation method. Segmentation tends to be done manually by different imaging experts and it is labor-intensive and prone to errors and inconsistency. Previous automated segmentation methods largely focused on fusing information that is extracted separately from the PET and CT modalities, with the underlying assumption that each modality contains complementary information. However, these methods do not fully exploit the high PET tumor sensitivity that can guide the segmentation. We introduce a multimodal spatial attention module (MSAM) that automatically learns to emphasize regions (spatial areas) related to tumors and suppress normal regions with physiologic high-uptake. The resulting spatial attention maps are subsequently employed to target a convolutional neural network (CNN) for segmentation of areas with higher tumor likelihood. Our MSAM can be applied to common backbone architectures and trained end-to-end. Our experimental results on two clinical PET-CT datasets of non-small cell lung cancer (NSCLC) and soft tissue sarcoma (STS) validate the effectiveness of the MSAM in these different cancer types. We show that our MSAM, with a conventional U-Net backbone, surpasses the state-of-the-art lung tumor segmentation approach by a margin of 7.6% in Dice similarity coefficient (DSC).

I. INTRODUCTION

PET-CT combines PET’s tumor sensitivity with CT anatomy, but segmentation remains challenging because PET has limited spatial resolution and benign high-uptake regions can resemble tumors. The paper introduces MSAM to use PET-derived spatial attention to guide segmentation of PET-CT tumors.

I. INTRODUCTION

  • PET-CT combines PET’s sensitivity for detecting abnormal tumor metabolism with CT anatomical information and is increasingly used for cancers including NSCLC.Source: §I · p. 1
  • PET-CT segmentation remains challenging because PET has limited spatial resolution, benign processes can show increased uptake, and interpretation still relies on specialist expertise.That reliance carries high costs and inter- and intra-observer inconsistencies.Source: §I · p. 1
  • SUV thresholding can fail because benign processes may have high FDG uptake while some primary tumors have SUV<2.5.Threshold choice is also affected by scanner and acquisition factors and requires specialist knowledge.Source: §I · pp. 1–2
  • Earlier PET-guided approaches used PET intensities to constrain CT segmentation, but did not account for PET spatial and contextual characteristics and depended heavily on SUVs.Source: §I · p. 2
  • MSAM uses PET-derived attention maps to direct a segmentation CNN toward CT regions with greater tumor likelihood.It is designed to distinguish normal high-uptake regions from abnormal tumor regions and can be trained end-to-end with a backbone CNN.Source: §I · p. 2

II. METHODS

The experiments use PET-CT datasets from patients with NSCLC and soft tissue sarcoma, with tumor annotations serving as evaluation ground truth. Images are rescaled and tumor-free slices are excluded.

A. Materials

  • The study used a 50-patient NSCLC dataset and a public 51-patient soft tissue sarcoma dataset.The NSCLC patients had pathologically proven disease; the STS patients had histologically proven disease.Source: A. Materials · p. 2
  • NSCLC tumor regions were annotated through specialist localization, connected thresholding, and minor manual adjustments, while STS contours were manually delineated by an expert radiation oncologist.The resulting NSCLC annotations were used as evaluation ground truth.Source: A. Materials · pp. 2–3
  • Images were rescaled to 256 × 256, PET intensities converted to SUVs, and CT intensities converted to Hounsfield units; slices without tumor pixels were excluded.Source: A. Materials · p. 3

B. Overview of our Proposed Method

The method combines a PET-driven spatial attention subnetwork with a CT-processing encoder-decoder backbone. The MSAM learns a spatial attention map from PET and applies it to CT features to focus segmentation on likely tumor regions.

B. Overview of our Proposed Method

  • The model uses a PET-processing MSAM to infer spatial attention and a backbone CNN to extract tumors from CT data.The PET-derived map is applied to CT feature maps at multiple backbone scales to produce the final segmentation.Source: B. Overview of our Proposed Method · p. 3

C. Multimodal Spatial Attention Module (MSAM) Design

D. Backbone Integration and Segmentation

The MSAM map is resampled and multiplied with backbone skip features to form gated connections for the decoder. The complete network produces tumor and background probabilities and trains the MSAM jointly with the backbone.

D. Backbone Integration and Segmentation

  • The PET attention map is resampled to the skip-feature resolution and multiplied elementwise with skip features to produce gated skip connections.The map is broadcast channel-wise; bilinear interpolation is used when downsampling is needed.Source: D. Backbone Integration and Segmentation · p. 3
  • Gated skip features are concatenated with corresponding decoder features produced by transposed convolution or upsampling.Source: D. Backbone Integration and Segmentation · p. 4
  • The network processes scans slice-wise and outputs per-pixel tumor and background probabilities, while MSAM parameters train jointly with the backbone using the segmentation loss.No auxiliary loss is needed to optimize the module.Source: D. Backbone Integration and Segmentation · p. 4

E. Implementation Details

The experiments use consistent training settings and compare MSAM across input configurations, backbones, and attention methods. Five-fold cross-validation and pixel-wise tumor metrics are used for evaluation.

E. Implementation Details

  • Networks were trained end-to-end for 100 epochs with batch size 4, Adam, mean per-pixel cross-entropy, and a fixed learning rate of 0.0001.The first and second moment estimates were 0.9 and 0.999; dropout was not used.Source: E. Implementation Details · p. 4
  • Training images were mean-subtracted, unit-variance normalized, and augmented with random flips or 90-, 180-, or 270-degree rotations.Training took 2 hours for the lung cancer dataset and 7 hours for the STS dataset.Source: E. Implementation Details · p. 4

F. Evaluation Setup

  • U-Net experiments compare CT, PET, and concatenated PET-CT inputs with and without MSAM, including alternative modality assignments to the backbone and MSAM.The evaluation also tests MSAM with ResNet-50 and compares it with ARL, CBAM, and attention gates.Source: F. Evaluation Setup · p. 4
  • All methods were evaluated using five-fold cross-validation with identical patient splits and no patient shared between a fold’s training and test sets.The split was 80/20, with 40 patients for training and 10 for testing; one STS scan was excluded to equalize fold sizes.Source: F. Evaluation Setup · p. 4
  • Tumor-region DSC was the main metric, alongside pixel-wise precision, sensitivity, and specificity.DSC combines precision and sensitivity via a harmonic mean.Source: F. Evaluation Setup · p. 4

III. RESULTS

Across both datasets, segmentation improved with MSAM, and the best-performing configuration used CT in the backbone with MSAM. Qualitative outputs also show fewer errors than baseline U-Net predictions.

A. MSAM Contribution Analysis

B. Comparison Against Current Attention Methods

MSAM outperformed other attention methods across reported segmentation metrics. Its maps emphasized tumor regions and suppressed irrelevant bright areas, unlike the compared CBAM and AG maps.

B. Comparison Against Current Attention Methods

C. Comparison Against the State-of-the-Art

The proposed MSAM with U-Net exceeded prior PET-CT lung tumor segmentation methods in mean DSC. Its predictions also better captured tumor shape nuances than the compared methods.

C. Comparison Against the State-of-the-Art

  • 71.44% mean DSC was achieved by MSAM with U-Net, 7.59% higher than the previous state of the art.The comparison is presented for PET-CT lung tumor segmentation.Source: Table III · p. 7, C. Comparison Against the… · p. 7
  • Table III compares the proposed method against the state of the art for PET-CT lung tumor segmentation.Source: Table III · p. 7
  • Li et al. and Zhong et al. tended to under-segment tumors, whereas Kumar et al. tended to over-segment them.The proposed method better captured shape nuances and produced more accurate segmentations.Source: C. Comparison Against the… · p. 7

IV. DISCUSSION

MSAM improved segmentation across backbone CNNs and outperformed existing attention approaches and prior lung tumor methods. The discussion attributes these gains to PET-guided attention that retains complementary CT information, while noting backbone and dataset differences.

IV. DISCUSSION

  • MSAM consistently improved backbone CNNs, outperformed existing image-attention approaches, and enabled U-Net+MSAM to surpass prior state-of-the-art lung segmentation methods.The authors report this as their main finding.Source: §IV · p. 7

A. MSAM

  • Channel-concatenated PET-CT produced false positives or false negatives, while CT-only input performed poorly and PET offered the greatest single-modality utility.The authors attribute the concatenation errors to difficulty distinguishing benign from cancerous high uptake.Source: A. MSAM · p. 7
  • With PET-only input, MSAM improved segmentation without adding input information, isolating the attention configuration's contribution.Source: A. MSAM · p. 8
  • PET-derived maps suppress misleading heart and bladder hotspots, boost subtle tumor regions, and fill PET-region holes while unweighted CT features preserve morphological guidance.The authors describe this PET-attention and CT-feature configuration as using complementary information from both modalities.Source: A. MSAM · p. 8
  • MSAM improved segmentation with ResNet-50 despite its initial 7 × 7 convolution losing resolution and preventing use of the full-resolution attention map.The attention maps were still applied at lower resolutions.Source: A. MSAM · p. 8
  • STS segmentation was more challenging than lung segmentation, with more heterogeneous tumor shapes, sizes, anatomical locations, and visual features.Source: A. MSAM · p. 8

B. Comparison Against Current Attention Methods

MSAM's attention maps better highlighted tumors and suppressed benign high-intensity regions than existing spatial attention methods. The state-of-the-art comparisons also describe under-segmentation by some prior methods and coarse over-segmentation by another.

B. Comparison Against Current Attention Methods

  • MSAM better suppressed benign high-intensity regions and highlighted tumors than existing spatial attention methods, whose segmentation effects varied across datasets.Source: B. Comparison Against Current… · p. 8
  • CBAM emphasized already-bright input regions, while AG produced random maps and could not reliably distinguish tumors from hotspots.The authors say these methods did not leverage PET's spatial sensitivity and multimodal information as MSAM did.Source: B. Comparison Against Current… · p. 8

C. Comparison Against the State-of-the-Art

  • Li et al. and Zhong et al. tended to under-segment or miss tumors, particularly hilar and mediastinal lymph-node tumors with challenging adjacent structures.Their methods required centered tumor-region crops, whereas MSAM did not require predefined tumor boundaries or initialization seeds.Source: C. Comparison Against the… · p. 8
  • Kumar et al. detected tumors more successfully but tended toward over-segmentation and coarse predictions, emphasizing fusion over fine morphological detail.Source: C. Comparison Against the… · p. 8

D. Future Work

The conclusion reports that MSAM enhances CNN segmentation of multimodal PET-CT tumors and outperforms existing spatial-attention methods and prior lung methods. The authors identify manual slice selection, 2D processing, and unoptimized module design as areas for further work.

D. Future Work

  • Experiments used tumor-positive 2D slices selected through a preprocessing step requiring manual input; the authors suggest automatic tumor detection and 3D extension for future work.They note 3D CNNs may improve performance but carry greater computational expense, slower speed, and more parameters that increase overfitting risk.Source: D. Future Work · p. 9
  • Evaluation covered two independent datasets totaling 100 PET-CT scans across two cancer types, with further evaluation on other cancer datasets planned.Source: D. Future Work · p. 9

V. CONCLUSION

  • MSAM automatically emphasizes tumor regions and suppresses benign areas, enhancing backbone CNN segmentation and surpassing prior PET-CT lung tumor methods.Source: §V · p. 9
Loading 2007.14728v2…