PaperlayerView exact revision on arXiv

Source-linked AI summary

CTNet: Context-based Tandem Network for Semantic Segmentation

Zechao Li, Yanpeng Sun, Jinhui Tang

arXiv:2104.09805v1cs.CV

TL;DR

Semantic segmentation is challenged by visually similar categories, motivating methods that use contextual information without losing category-specific cues. CTNet interactively combines channel and spatial context, achieving strong results across three benchmarks without pre-training.

  • Problem

    Visually similar categories are difficult to distinguish from appearance alone, while existing spatial- and channel-context methods lack effective communication between these dimensions.

  • Method

    CTNet connects a Channel Contextual Module and a Spatial Contextual Module in tandem, using CCM-learned semantic features as prior knowledge to guide SCM’s spatial-dependency learning.

  • Results

    CTNet achieves 55.5% mIoU on PASCAL-Context, 45.94% mIoU on ADE20K, and 85.3% mIoU on PASCAL VOC2012 without pre-training.

  • Takeaways & Limitations

    Interactive joint learning of long-range spatial and semantic channel dependencies improves the learned representations for semantic segmentation.

  • Takeaways & Limitations

    Prior spatial-context methods can introduce category information absent from an image, producing errors such as recognizing ground pixels as sky.

Abstract

from arXiv · show

Contextual information has been shown to be powerful for semantic segmentation. This work proposes a novel Context-based Tandem Network (CTNet) by interactively exploring the spatial contextual information and the channel contextual information, which can discover the semantic context for semantic segmentation. Specifically, the Spatial Contextual Module (SCM) is leveraged to uncover the spatial contextual dependency between pixels by exploring the correlation between pixels and categories. Meanwhile, the Channel Contextual Module (CCM) is introduced to learn the semantic features including the semantic feature maps and class-specific features by modeling the long-term semantic dependence between channels. The learned semantic features are utilized as the prior knowledge to guide the learning of SCM, which can make SCM obtain more accurate long-range spatial dependency. Finally, to further improve the performance of the learned representations for semantic segmentation, the results of the two context modules are adaptively integrated to achieve better results. Extensive experiments are conducted on three widely-used datasets, i.e., PASCAL-Context, ADE20K and PASCAL VOC2012. The results demonstrate the superior performance of the proposed CTNet by comparison with several state-of-the-art methods.

1 INTRODUCTION

Semantic segmentation struggles with visually similar categories, motivating contextual reasoning across spatial and channel dimensions. CTNet links these dimensions so channel-derived semantic features guide spatial dependency learning, with experiments reporting strong benchmark performance.

  • Motivation: Visually similar categories make pixel-level semantic segmentation difficult when appearance alone cannot distinguish regions.Examples include chairs versus sofas and water versus ground.
  • Limitations of Existing Context: Spatial-context methods can miss category information, causing pixels to be assigned labels that do not appear in the image.The paper calls this problem category field deficiency.
  • Motivation: Channel context is useful because feature-map channels correspond to semantic responses that are correlated and can be enhanced or suppressed for the task.Existing approaches include SENet, SKNet and EncNet.
  • Proposed CTNet: CTNet interactively explores spatial pixel dependencies and channel semantic dependencies, using CCM features as prior knowledge to guide SCM learning.The design jointly captures long-range spatial and semantic dependencies.
  • Channel Contextual Module: CCM uses the MCE block and a class probability loss to learn multi-scale channel context, middle features and class-specific features.These representations encode category probabilities and semantic responses for use by SCM.
  • Spatial Contextual Module: SCM models global spatial dependency through pixel-category correlations and a self-attention scheme designed to reduce computational complexity without reducing performance.The resulting spatial representation aggregates category features according to pixel-category correlations.

2 RELATE WORK

Prior segmentation methods model spatial and channel context separately or independently, leaving their complementarity underused. CTNet addresses this gap by exchanging knowledge between the two contextual dimensions while using attention to capture long dependencies.

  • Spatial and Channel Context: Spatial-context methods expand receptive fields through dilated convolutions or multi-scale pooling, but mainly model local feature relationships.Examples include Deeplabv3, Deeplabv3+ and PSPNet.
  • Spatial and Channel Context: Channel-context methods use adaptive operations or attention to obtain richer channel representations from feature maps.The related-work discussion cites channel-context approaches such as SENet and SKNet.
  • Existing Joint Methods: Methods such as DANet and CFNet capture spatial and channel dependencies in separate branches without knowledge communication between them.Their parallel or independent designs do not model complementarity interactively.
  • CTNet: CTNet interactively explores pixel-level spatial dependencies and channel-level semantic dependencies, exchanging the knowledge uncovered by both contexts.This interaction is proposed to address limitations of independent context modeling.
  • Self-Attention: Self-attention models global dependencies but can incur high computation and memory costs, motivating sparse and other efficient alternatives.The cited alternatives seek to reduce these costs while preserving performance.

3 THE PROPOSED APPROACH

CTNet jointly explores channel and spatial context through communicating CCM and SCM modules, using channel-derived semantic features to guide spatial-context learning. The network combines these representations to improve semantic segmentation features while reducing spatial-context computation.

  • Overall architecture: CTNet uses a tandem CCM–SCM architecture whose modules communicate and whose outputs are aggregated for semantic segmentation.CCM learns channel context, SCM learns spatial context, and their features are finally integrated.
  • Channel Context Module: CCM learns channel importance and class-specific features from channel dependencies, using multi-local channel contexts captured at multiple scales.The MCE block applies multiple 1-D convolutions with different kernels before aggregating the resulting local channel-context maps.
  • Interaction between modules: CCM produces a middle feature map and class feature matrix that serve as prior knowledge for SCM and help it capture spatial context more concisely.The middle feature and class features are input to SCM, linking channel semantics with spatial-context learning.
  • Channel Context Module: A Class Probability Loss regularizes CCM to predict category occurrence probabilities, helping channel context encode the categories present in each image.The target probability is calculated from ground-truth annotations and category pixel occurrence frequency.
  • Spatial Context Module: SCM describes global spatial context through pixel–category matching rather than direct pixel-to-pixel correlations, reducing computation from O(HW × HW × C) to O(HW × N × C).Because N << HW, the category-based formulation reduces the computational burden relative to the non-local block while producing spatially updated features.

4 EXPERIMENTAL

Experiments evaluate CTNet through ablations and comparisons on three widely used semantic-segmentation datasets, without COCO pre-training.

  • Experiments evaluate CTNet on PASCAL-Context, ADE20K, and PASCAL VOC2012.All reported results use the proposed CTNet without COCO pre-training.

4.1 Datasets and Evaluation Metrics

The evaluation uses three semantic-segmentation datasets with different category counts and dataset sizes, reporting mean intersection over union as the principal metric.

  • Datasets: PASCAL-Context contains 4,998 training images, 5,105 test images, and 60 categories.
  • Datasets: ADE20K contains 20,000 training images, 2,000 validation images, 3,000 test images, and 150 semantic categories.
  • Datasets: PASCAL VOC2012 contains 21 categories, and the augmented annotations expand training data to 10,582 images.
  • Evaluation Metrics: Multi-scale testing averages predictions from images resized across scales 0.5, 0.75, 1.0, 1.25, 1.5, and 1.75.
  • Evaluation Metrics: Performance is evaluated with mIoU, the mean of class-wise intersection over union.

4.2 Implementation Details

CTNet uses ImageNet-pretrained ResNet backbones, JPU for efficiency, and SGD with a polynomial learning-rate schedule; ablations vary pooling and channel-context settings.

  • Implementation Details: ResNet-50 is used for ablations, while ResNet-101 is used for other experiments.JPU replaces dilated convolution to address memory consumption and improve segmentation efficiency.
  • Ablation Configuration: The implementation evaluates different local channel-context scales and global pooling strategies in CCM.
  • Implementation Details: SGD uses momentum 0.9, weight decay 0.0001, and a polynomial learning-rate policy with power 0.9.

4.3 Ablation Study

Ablations show that CTNet benefits from multi-scale channel context, spatial context, tandem interaction, and class probability loss, while excessive channel-context weighting or fusion can reduce performance.

  • 4.3.1 For CCM: 52.9% mIoU is achieved with MCE scale set K = [9, 17, 33, 65].Performance decreases when additional local contexts create redundant channel-context information.
  • 4.3.1 For CCM: 80.4% accuracy and 53.3% mIoU are achieved with global average pooling on PASCAL-Context using ResNet-50.
  • 4.3.2 For SCM: 7.11MB parameters make SCM a lightweight self-attention module on PASCAL-Context.OSCM and CTNet have the same computational cost because CCM adds little resource consumption.
  • 4.3.2 For SCM: 54.6% mIoU is obtained by OSCM on PASCAL-Context, while CTNet reaches 56.1% and 45.94% mIoU on PASCAL-Context and ADE20K.
  • 4.3.3 For single module and connection mode: CTNet and PANet outperform DANet, while tandem CTNet improves segmentation over parallel PANet.The comparison attributes tandem benefits to interactive learning between channel and spatial contexts.
  • 4.3.4 For Class Probability Loss (CP-Loss): When β = 0, CTNet has its worst mIoU, and the best results occur at α = 0.3 and β = 0.1.Increasing CP-Loss weight beyond the optimum first improves and then reduces performance by disrupting channel-spatial balance.

4.4 State-of-the-Art Comparisons

CTNet outperforms compared methods across PASCAL-Context, ADE20K, and PASCAL VOC2012, while qualitative results show improved handling of confusing categories.

  • PASCAL-Context: 55.5% mIoU is achieved by CTNet on the PASCAL-Context test set, surpassing all compared methods without COCO pre-training.With ResNet-50, CTNet also outperforms RefineNet and MSCI despite their deeper networks.
  • PASCAL-Context: CTNet distinguishes confusing categories more accurately than EncNet and FastFCN in qualitative PASCAL-Context examples.EncNet and FastFCN misclassify grass as tree in a visually similar region, whereas CTNet produces better segmentation.
  • ADE20K: CTNet more accurately labels challenging ADE20K regions than EncNet and FastFCN in the illustrated segmentation examples.CTNet distinguishes cabinet and bed regions that EncNet incorrectly labels.
  • PASCAL VOC2012: 85.3% mIoU is achieved by CTNet on PASCAL VOC2012 without COCO pre-training.The model is first trained on an enhanced dataset and then fine-tuned on the original dataset before official-server evaluation.

5 CONCLUSION

CTNet combines channel and spatial contextual modeling through tandem interactive training for semantic segmentation. It reports superior performance on three datasets against state-of-the-art methods.

  • Conclusion: CTNet combines Channel Contextual Module and Spatial Contextual Module to model channel and spatial context interactively.CCM features guide SCM learning as prior knowledge, while the two modules communicate during tandem training.
  • Conclusion: CTNet achieves 55.5% mIoU on PASCAL-Context, 45.94% mIoU on ADE20K, and 85.3% mIoU on PASCAL VOC2012.These results are reported as superior to state-of-the-art methods.
Loading 2104.09805v1…