PaperlayerView exact revision on arXiv

Source-linked AI summary

CS2-Net: Deep Learning Segmentation of Curvilinear Structures in Medical Imaging

Lei Mou, Yitian Zhao, Huazhu Fu, Yonghuai Liu, Jun Cheng, Yalin Zheng, Pan Su, Jianlong Yang, Li Chen, Alejandro F Frang, Masahiro Akiba, Jiang Liu

arXiv:2010.07486v2eess.IVcs.CV

TL;DR

Curvilinear-structure segmentation is needed for quantitative medical-image analysis, but existing methods face anatomical variability, imaging degradation, and costly manual refinement. CS2-Net combines encoder-decoder processing with channel and spatial self-attention and boundary-focused convolutions for unified 2D and 3D segmentation. Across nine datasets, the authors report improved and overall state-of-the-art segmentation performance, while noting over-segmentation and substantial 3D computational demands.

  • Problem

    Existing curvilinear-structure segmentation methods struggle with anatomical variability, changing scales, noise, poor contrast, low resolution, and costly manual annotation or refinement.

  • Method

    CS2-Net uses an encoder-decoder network with channel and spatial self-attention plus 1×3 and 3×1 convolutions for boundary features.

  • Results

    Experiments on nine datasets across six 2D and three 3D datasets show overall state-of-the-art performance and improved segmentation results.

  • Takeaways & Limitations

    CS2-Net has reported potential for computer-aided diagnosis and automated biological image interpretation across medical imaging modalities.

  • Takeaways & Limitations

    Diseased cells resembling curvilinear structures can cause over-segmentation, while 3D volumetric segmentation requires considerable GPU resources.

Abstract

from arXiv · show

Automated detection of curvilinear structures, e.g., blood vessels or nerve fibres, from medical and biomedical images is a crucial early step in automatic image interpretation associated to the management of many diseases. Precise measurement of the morphological changes of these curvilinear organ structures informs clinicians for understanding the mechanism, diagnosis, and treatment of e.g. cardiovascular, kidney, eye, lung, and neurological conditions. In this work, we propose a generic and unified convolution neural network for the segmentation of curvilinear structures and illustrate in several 2D/3D medical imaging modalities. We introduce a new curvilinear structure segmentation network (CS2-Net), which includes a self-attention mechanism in the encoder and decoder to learn rich hierarchical representations of curvilinear structures. Two types of attention modules - spatial attention and channel attention - are utilized to enhance the inter-class discrimination and intra-class responsiveness, to further integrate local features with their global dependencies and normalization, adaptively. Furthermore, to facilitate the segmentation of curvilinear structures in medical images, we employ a 1x3 and a 3x1 convolutional kernel to capture boundary features. ...

1. Introduction

Curvilinear structures are clinically relevant across diverse medical imaging modalities, but their variable appearance and costly manual annotation motivate unified automated segmentation. CS2-Net addresses this need with attention-based features and boundary-focused convolutions for 2D and 3D images.

  • Geometrical and topological changes in vessels, airways, and nerve fibres are linked to diseases including diabetes, stroke, hypertension, and keratitis.
  • Medical imaging modalities include 2D colour fundus, OCTA, fluorescence angiography, confocal microscopy, and 3D MRA or CTA.
  • Accurate extraction supports quantitative analysis and computer-aided diagnosis, whereas manual annotation is time-consuming, error-prone, and impractical for high-throughput settings.
  • Existing methods struggle with anatomical variability, multiple structure scales, noise, poor contrast, and low resolution, while OCTA retinal-vessel segmentation remains underdeveloped.
  • CS2-Net uses channel and spatial self-attention with 1×3 and 3×1 convolutions to extract curvilinear structures across imaging modalities.
  • Experiments on nine datasets—six 2D and three 3D—report overall state-of-the-art quantitative and qualitative performance.

2. Related Works

Prior work spans filtering, convolutional, recurrent, context-aware, and volumetric learning methods for curvilinear segmentation. Persistent challenges include manual tuning, difficult 2D variability, and methods that do not explicitly model tubular structure.

  • 2.1. 2D Curvilinear Segmentation: Traditional 2D approaches use Hessian, matched, multi-oriented, symmetry, and Gabor filters to model vessel or fibre appearance.
  • 2.1. 2D Curvilinear Segmentation: Deep 2D methods include CNNs, conditional random fields, recurrent U-shaped networks, oriented-cylinder detectors, context-aware networks, and U-Net variants.
  • 2.2. 3D Curvilinear Segmentation: 3D methods exploit volumetric depth information for vascular, kidney, tubular-organ, aneurysm, and vessel-centerline segmentation.
  • 2.2. 3D Curvilinear Segmentation: Many 3D filter-based methods rely on manual tuning, while some learning methods omit dedicated modules for tubular structure.

3.1. Network Architecture

CS2-Net combines an encoder, parallel channel and spatial attention blocks, and a decoder for unified curvilinear-structure segmentation in 2D and 3D medical images.

  • CS2-Net contains an encoder, channel and spatial attention module, and decoder, with separate architectures illustrated for 2D images and 3D volumes.
  • The encoder extracts input features before feeding them into parallel channel-attention and spatial-attention blocks.
  • The decoder restores high-level semantic features to the original dimensions layer by layer instead of directly up-sampling attention features.

3.2. 2D Attention Network

The 2D CS2-Net combines an encoder-decoder backbone with spatial and channel attention to model long-range feature dependencies and improve curvilinear-structure representation.

  • The 2D network uses an encoder, CSAM, and decoder to extract global features, enhance feature expression, and reconstruct curvilinear features.
  • 2D Spatial Attention Module: Spatial attention models relationships between any two pixels using features captured by 3 × 1 and 1 × 3 convolutions.These directional convolutions generate vertical and horizontal curvilinear-structure features before spatial correlation modeling.
  • 2D Spatial Attention Module: The spatial correlation matrix promotes similar spatial points and suppresses different ones through softmax attention.This operation helps the network learn curvilinear structures across different spatial locations.
  • 2D Channel Attention Module: Channel attention models dependencies among feature channels and applies softmax to enhance discrimination between curvilinear structures and background.Similar channels promote one another, whereas different channels inhibit one another, producing attention-enhanced features through residual addition.
  • The attention operations enhance contrast between class-dependent features and improve their expressiveness.
  • Objective Function: The 2D model is trained with pixel-wise binary cross-entropy because segmentation is treated as curvilinear-structure-versus-background classification.The loss compares ground-truth labels with predicted pixel probabilities.

3.3. 3D Attention Network

The 3D CS2-Net extends the attention architecture to volumetric data by replacing 2D operations with 3D ones and modeling dependencies across depth, while using a combined weighted-cross-entropy and Dice loss.

  • The 3D extension addresses limited learning-based 3D segmentation methods and the inability of 2D attention to integrate depth-direction features.
  • The 3D architecture contains an encoder, 3D CSAM, and decoder, with 3D operations replacing the corresponding 2D operations.The diagram identifies spatial and channel attention blocks within the 3D CSAM.
  • 3D Spatial Attention Module: The 3D spatial attention block uses 1 × 3 × 1 and 3 × 1 × 1 convolutions to capture tubular boundary features before forming voxel-level attention.It reshapes feature maps using N = H × W × D and combines attention-enhanced features with the input.
  • 3D Spatial Attention Module: Unlike the 2D block, the 3D spatial attention block maps features across width, height, and depth directions.The authors state that this increases the network’s feature expression ability.
  • 3D Channel Attention Module: The 3D channel attention block produces voxel-level channel-wise attention, which is combined with the spatial block and input by voxel-level matrix summation.
  • Loss Function: For sparse 3D cerebrovascular annotations, training uses weighted cross-entropy to adjust vascularity-background bias and Dice loss to support microcerebrovascular segmentation.
  • Loss Function: The combined 3D optimization loss balances weighted cross-entropy and Dice loss with α empirically set to 0.6.

4. Experimental Results over 2D Images

The 2D evaluation uses established and in-house datasets spanning retinal vessels and corneal nerve fibres, with cross-validation for some datasets. Across these tests, CS2-Net generally outperforms competing methods, including in low-contrast, small-vessel, and continuity-preservation cases.

  • Materials: The evaluation covers six 2D datasets, including retinal fundus, corneal nerve fibre, OCTA, and OCT RPE images.The datasets include DRIVE, STARE, IOSTAR, CORN-1, OCTA, and OCT RPE; the section states that six 2D datasets are used.
  • Materials: The study uses k-fold cross-validation for STARE, IOSTAR, and the in-house datasets where specified.STARE uses k=4 cross-validation, IOSTAR uses k=5, and the general section states a k=5 procedure for dividing training and testing data.
  • Color fundus images: CS2-Net outperforms competing methods on ACC and AUC across three retinal fundus datasets, with all pairwise differences statistically significant at p<0.05.Visual comparisons also show more vessel extraction in low-contrast regions containing multiple vessel scales.
  • In-house OCTA images: On OCTA images, CS2-Net detects small vessels more effectively and achieves the highest performance across all reported metrics.The authors attribute this to adaptive integration of local features with global dependencies and normalization; pairwise p-values are all below 0.05.
  • CORN-1 images: On CORN-1 images, CS2-Net preserves continuous nerve-fibre tracing and achieves the best performance in both SE and FDR.Other methods falsely detect portions of K-structures as nerve fibres, while CS2-Net maintains continuity.
  • Cross-dataset comparison: Across DRIVE, STARE, IOSTAR, CORN-1, OCT-A, and OCT RPE, CS2-Net generally outperforms state-of-the-art methods despite modality-dependent structure, contrast, and noise variation.The comparison is presented through ROC curves at selected true-positive and false-positive rates.

5. Experimental Results over 3D Volumes

The 3D CS2-Net is evaluated on brain MRA and synthetic vascular volumes using metrics suited to segmentation quality and sparse annotations. Across these datasets, it generally outperforms competing methods, particularly in boundary discrimination, tiny-vessel extraction, and background suppression.

  • Datasets and setup: The evaluation covers one brain MRA dataset and two synthetic 3D vascular datasets, with comparisons against several established segmentation methods.The synthetic experiments use four-fold cross-validation, while VascuSynth volumes also include simulated imaging noise.
  • Metrics: Because MIDAS annotations are sparse, the study uses TPR, FNR, FPR, OR, and UR rather than relying on Dice or IoU alone.OR and UR quantify over- and under-segmentation relative to the annotated and predicted voxel sets.
  • MRA results: CS2-Net reduces the FPR of six comparison methods by 0.0644, 0.0598, 0.0587, 0.0548, 0.0049, and 0.0005, respectively.The authors interpret this as stronger discrimination of cerebrovasculature from complex MRA background artefacts.
  • MRA results: 0.9706 TPR is achieved by CS2-Net on cerebral vasculatures, while its UR is 0.0291, 0.0102 lower than 3D U-Net.The authors also report p < 0.001 versus selected comparison methods.
  • Qualitative results: In MRA visualizations, CS2-Net extracts tiny vessels more effectively and produces vessel thicknesses more similar to the ground truth than 3D U-Net.The comparison is consistent with the reported OR and UR differences.
  • Synthetic results: On the Synthetic dataset, CS2-Net outperforms 3D U-Net across TPR, FNR, FPR, and DC, with DC increasing by 0.0807.Visual results also show better boundary discrimination than 3D U-Net.

6. Discussions

The discussion attributes CS2-Net’s performance to spatial and channel attention combined with directional boundary convolutions. Attention visualizations and ablations indicate complementary roles for feature normalization, spatial contrast, long-range dependencies, and background suppression.

  • Architecture: CS2-Net uses spatial and channel attention to capture tree-like structural information in horizontal and vertical directions, respectively.The network also adds 1 × 3 and 3 × 1 convolutions to capture boundary features.
  • Attention visualization: Attention maps become brighter and smoother from D1 to D4, indicating progressively refined focus on curvilinear structures during decoding.The channel attention module emphasizes classified curvilinear regions, while spatial attention highlights locations across the image.
  • Attention visualization: Compared with DANet and U-Net, CS2-Net shows stronger curvilinear-structure responses and greater suppression of background interference in intermediate attention maps.The comparison includes vessels and nerve fibres across multiple datasets.
  • Ablation study: The spatial attention branch provides larger gains than channel attention alone in the reported MIDAS ablation, while CSAM combines both advantages.The backbone’s TPR changes from 0.9517 without attention to 0.9663 with channel attention alone, whereas the authors report better performance with spatial attention alone.
  • Ablation study: The ablation attributes channel attention to feature-space normalization and spatial attention to image-wide contrast enhancement for inter-class discrimination.This division of roles explains why the combined CSAM is used in the final model.
  • CSAM visualization: CSAM output visualizations apply up-sampling and sigmoid normalization so voxel probabilities can be compared before and after attention processing.The authors report better clustering of curvilinear-structure probabilities after CSAM.

7. Conclusion and Future Works

CS2-Net is presented as a unified curvilinear-structure segmentation network for 2D images and 3D volumes, with improved segmentation results across 9 datasets and 6 imaging modalities. The authors identify over-segmentation and computational demands as priorities for future work.

  • Conclusion: CS2-Net applies self-attention in channel and spatial dimensions to improve inter-class discrimination and intra-class aggregation for curvilinear structures.The network is designed for both 2D images and 3D volumes.
  • Results: Improved segmentation results were demonstrated across 9 datasets spanning 6 imaging modalities.
  • Future evaluation: The work takes a first step toward evaluating AI models across different images and applications to demonstrate their effectiveness and applicability.
  • Limitations and future work: Diseased cells with features similar to curvilinear structures can cause over-segmentation, motivating local-neighbourhood and continuity constraints.
  • Limitations and future work: Three-dimensional volumetric segmentation requires considerable GPU resources, increasing training demands and motivating architectural simplification.
Loading 2010.07486v2…