PaperlayerView exact revision on arXiv

Source-linked AI summary

SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentation

Zhengze Xu, Dongyue Wu, Changqian Yu, Xiangxiang Chu, Nong Sang, Changxin Gao

arXiv:2312.17071v2cs.CV

TL;DR

Real-time semantic segmentation seeks rich long-range context without the computational overhead of an additional semantic branch. SCTNet transfers semantic information from a training-only transformer to a lightweight CNN using CFBlock and SIAM, and reports new state-of-the-art results across three datasets.

  • Problem

    Additional semantic branches improve contextual representation but incur computational overhead and slow real-time inference.

  • Method

    SCTNet trains a single-branch CNN with a transformer semantic branch, CFBlock, and SIAM to align transformer and CNN semantic information before inference.

  • Results

    SCTNet achieves new state-of-the-art speed-accuracy trade-offs and outperforms existing methods on Cityscapes, ADE20K, and COCO-Stuff-10K.

  • Takeaways & Limitations

    The paper presents training-only semantic alignment as a way to obtain transformer-derived long-range context without extra inference computation cost.

  • Takeaways & Limitations

    The paper does not report CPU latency on mobile devices and leaves CPU-specific modifications for future work.

Abstract

from arXiv · show

Recent real-time semantic segmentation methods usually adopt an additional semantic branch to pursue rich long-range context. However, the additional branch incurs undesirable computational overhead and slows inference speed. To eliminate this dilemma, we propose SCTNet, a single branch CNN with transformer semantic information for real-time segmentation. SCTNet enjoys the rich semantic representations of an inference-free semantic branch while retaining the high efficiency of lightweight single branch CNN. SCTNet utilizes a transformer as the training-only semantic branch considering its superb ability to extract long-range context. With the help of the proposed transformer-like CNN block CFBlock and the semantic information alignment module, SCTNet could capture the rich semantic information from the transformer branch in training. During the inference, only the single branch CNN needs to be deployed. We conduct extensive experiments on Cityscapes, ADE20K, and COCO-Stuff-10K, and the results show that our method achieves the new state-of-the-art performance. The code and model is available at https://github.com/xzz777/SCTNet

1 Introduction

Real-time segmentation must balance rich long-range semantic context against inference efficiency. SCTNet addresses this trade-off with a single CNN branch trained using transformer semantic information and feature alignment.

  • Motivation: Self-attention methods can have square computation complexity with image resolution, increasing latency on high-resolution images.
  • Motivation: Recent real-time methods use bilateral architectures to separate spatial detail and contextual information, but added branches and fusion increase computational cost.Feature-sharing designs move the separation deeper to balance speed and accuracy.
  • SCTNet: SCTNet learns long-range context from a training-only transformer branch while deploying only a lightweight single-branch CNN at inference.The approach jointly encodes semantic information and spatial details after alignment.
  • SCTNet: CFBlock captures long-range context using convolutional operations, while SIAM aligns transformer and CNN features to reduce their semantic gap.SIAM is described as the semantic information alignment mechanism used alongside CFBlock.
  • Results: SCTNet is reported to outperform existing state-of-the-art real-time segmentation methods on Cityscapes, ADE20K, and COCO-Stuff-10K.The paper presents this as a speed-and-performance improvement for real-time segmentation.

2 Related Work

Prior semantic segmentation work uses CNNs, multi-scale context, attention, and bilateral architectures to improve semantic representations. These approaches can introduce latency through costly operations or additional processing structures.

  • Semantic Segmentation: CNN-based segmentation methods enlarge receptive fields with dilated convolution or capture multi-scale context by fusing features from different levels.
  • Real-Time Segmentation: SCTNet uses a single hierarchy branch with a semantic extraction transformer, avoiding the extra branch and costly fusion module during inference.The architecture is presented as an alternative real-time segmentation paradigm.
  • Attention-Based Methods: Attention mechanisms for real-time segmentation target low latency and strong semantic extraction, but shift, reshape, large-kernel, or multi-scale operations can add latency.

3 Methodology

SCTNet addresses the speed–accuracy tension in real-time segmentation by training a transformer semantic branch to transfer long-range information into a deployable single-branch CNN. Its CFBlock and alignment design support this transfer while keeping inference focused on the lightweight CNN.

  • SCTNet Architecture: SCTNet uses a training-only transformer branch to enrich a single CNN branch with semantic information without retaining the transformer during inference.The architecture simplifies two-branch inference into one CNN branch, using the transformer only for semantic alignment during training.
  • Conv-Former Block: CFBlock simulates a transformer block with convolution-only attention to help CNN features learn semantic information from transformer features.Its transformer-like structure is intended to reduce the semantic gap between the two feature types while using efficient convolution operations.
  • Convolutional Attention: The proposed convolutional attention replaces GFA matrix multiplication with pixel-wise convolution, preserving spatial structure without flattening and reshaping operations.The design also enlarges learnable vectors into kernels, allowing patch-level similarity and retaining local spatial information to some extent.
  • Convolutional Attention: Stripe convolution approximates each k × k convolution with sequential 1 × k and k × 1 convolutions to reduce computational cost.The figure identifies GDN as Grouped Double Normalization, ⊗ as convolution, ⊕ as addition, and k as kernel size.
  • Semantic Information Alignment: SIAM aligns CNN and transformer representations through backbone feature alignment and shared decoder head alignment during training.The method uses CWD Loss to focus alignment on semantic information by converting feature activations into channel-wise probability distributions; temperature controls distribution softness.

4 Experiments

SCTNet is evaluated against real-time segmentation methods across Cityscapes, ADE20K, and COCO-Stuff-10K, with experiments covering benchmark performance, block choices, alignment, and visual quality.

  • Benchmark Results: SCTNet-B-Seg100 achieves 80.5% mIoU at 62.8 FPS on Cityscapes.
  • Benchmark Results: SCTNet-B-Seg75 reaches 79.8% mIoU and is twice as fast as RTFormer-B and DDRNet-23.
  • Benchmark Results: SCTNet achieves the best accuracy with the fastest speed on ADE20K, while SCTNet-B reaches 43.0% mIoU.
  • Benchmark Results: 35.9% mIoU at 141.5 FPS makes SCTNet-B 0.6% higher than RTFormer-B and about 1.6 times faster on COCO-Stuff-10K.
  • Ablation Studies: CFBlock outperforms ResBlock and SegFormer Block in mIoU while using fewer parameters and faster speed than MSCANBlock.
  • Ablation Studies: SIAM consistently improves SegFormer, SegNext, SeaFormer, and DDRNet, although gains are relatively slim for SeaFormer and DDRNet.
  • Visualization Results: SCTNet produces finer details and more accurate large-area predictions than DDRNet-23 and RTFormer-B in Cityscapes visualizations.

5 Conclusion

The paper concludes that SCTNet extracts high-quality long-range context in a single-branch architecture without extra inference computation. Experiments report new state-of-the-art results and motivate using transformer knowledge rather than deploying an additional branch.

  • Extensive experiments demonstrate that SCTNet achieves new state-of-the-art results.
  • SCTNet suggests boosting real-time segmentation by using transformer knowledge without necessarily adopting a deployed transformer branch.

A More Implementation Details

The implementation uses ImageNet-pretrained hierarchical CNN variants, training-only SegFormer teachers, specified architectural hyperparameters, and a combined CE-CWD loss.

  • SCTNet-B and SCTNet-S use ImageNet-1K-pretrained CNN backbones before fine-tuning on segmentation datasets.
  • Cityscapes training uses AdamW, 160k iterations, batch size 16, and TensorRT v8.2.3 for TensorRT speed measurements.
  • SCTNet-B uses C = {64, 128, 256, 512}, whereas SCTNet-S uses C = {32, 64, 128, 256}; both use layer numbers {2, 2, 3, 2}, N = 64, and k = 7.
  • The overall training loss combines CE loss and CWD loss, with λmain = 1.0, λaux = 0.4, and λcwd = [3, 15, 15, 15].
  • SegFormer-B3 serves as the training transformer teacher for SCTNet-B, while SegFormer-B2 serves SCTNet-S.

Ablation Studies on the design of CFBlock

The CFBlock’s convolutional attention is evaluated with different kernel implementations and sizes, balancing segmentation accuracy against computational efficiency.

  • Stripe convolution approximates a 7×7 convolution using sequential 1×7 and 7×1 convolutions for efficiency.
  • The experiments set the learnable-kernel size to 7 because standard 7×7 convolution is slower and achieves lower mIoU than stripe convolutions.

Ablation Studies on Semantic Alignment Loss

The study evaluates semantic alignment choices for transferring transformer context to the CNN. Alignment location and loss weighting affect mIoU without changing inference FPS, while SegFormer is selected as the training semantic branch.

  • Alignment loss: CWD loss improves alignment performance, whereas L2 decreases accuracy and KL remains nearly unchanged before alignment.CWD aligns channel-wise probability distributions, while L2 and KL directly align feature maps element-wise.
  • Alignment location: Applying alignment at stage3&4, the decoder, and output logits gives the best mIoU performance.The alignment-loss location does not affect inference FPS.
  • Alignment weight: Feature alignment weight 15 and output-logit weight 3 are the most suitable settings in the reported ablation.Weights that are too small underuse transformer semantics, while very large weights compete with CE loss and can destabilize training.
  • Semantic branch: SegFormer and InternImage provide the most promising improvement among the tested training-only semantic branches, and SegFormer requires less training time and memory.The consistent improvement across OCRNet, InternImage, Swin, and SegFormer supports SIAM; SegFormer is selected for training.

C Comparison with More Methods

SCTNet is compared across challenging datasets, devices, and real-time architectures. The reported results show competitive or superior speed-accuracy trade-offs while retaining a single-branch CNN at inference.

  • Comparison on Cityscapes: SCTNet establishes a new state-of-the-art speed-accuracy trade-off on Cityscapes using one NVIDIA RTX 2080Ti.Figure 8 presents SCTNet methods in red and other methods in blue.
  • Comparison on ADE20K and COCO-Stuff-10K: SCTNet outperforms transformer-based and CNN-based methods on ADE20K and COCO-Stuff-10K by a significant margin.These datasets contain numerous images with over 150 classes, challenging lightweight CNN-based models.
  • Comparison on Cityscapes: On the same RTX 2080Ti, SCTNet-B outperforms existing bilateral real-time networks by a large margin, while SCTNet-S maintains a competitive trade-off.RTFormer speed is reported from its original paper because TensorRT does not support some of its operations and acceleration details are unavailable.
  • Single- and multi-branch methods: Lightweight single-branch methods often lack long-range semantics, while heavier encoder, convolution, or decoder designs weaken their speed advantage.Multi-branch designs can improve performance but incur higher computational costs and slower inference than two-branch methods.
  • CFBlock comparison: CFBlock achieves similar performance to MSCAN with fewer parameters and faster speed.The comparison is reported in Table 4 of the main paper.
  • Distillation perspective: SCTNet uses transformer semantic information to assist a single-branch CNN without requiring joint training, resulting in a smaller training cost.The combined use of SIAM and CFBlock also offers insights for heterogeneous transformer-to-CNN distillation.

Limitations and Future work

The paper identifies deployment scope and model scaling as directions for future work. Its current evaluation emphasizes GPU and TensorRT settings, while CPU mobile latency remains unreported.

  • Limitations: CPU latency on mobile devices is not reported because a consistent benchmark and broad method evaluation are beyond the paper’s scope.SCTNet also lacks CPU-specific modifications such as depthwise separable convolutions or ReLU6.
  • Future work: The authors plan to modify SCTNet for CPU usage in future work.The current demonstrations cover various GPU devices and datasets using torch and TensorRT models.
  • Future work: SCTNet-L is proposed as a future scaling direction because SCTNet-B exceeds the 30 FPS real-time standard by 32.8 FPS.The authors state that scaling could further improve the accuracy-speed trade-off.
  • Visualization evidence: On ADE20K and Cityscapes, SCTNet-B produces finer details and more accurate large-area predictions than the cited comparison methods.The ADE20K comparison includes SegFormer-B0, RTFormer-B, TopFormer-B, and SeaFormer-B; the Cityscapes comparison includes SeaFormer-B and STDC2.

E More Visualization Results

Additional visualizations illustrate SCTNet’s segmentation quality on ADE20K and Cityscapes. The examples emphasize class discrimination, large-area consistency, and boundaries of thin objects.

  • ADE20K: On ADE20K, SCTNet-B improves internal consistency, boundaries, object masks, and suppression of wrong masks across the shown examples.The examples cover buildings, trees, curtains, cushions, armchairs, paintings, fences, and poles.
  • ADE20K: Compared with SegFormer-B0, RTFormer-B, TopFormer-B, and SeaFormer-B, SCTNet-B better distinguishes different classes in the visualized ADE20K cases.The comparison is based on Figure 9 visualizations.
  • Cityscapes: On Cityscapes, SCTNet improves large-area classes and boundaries for small or thin objects compared with SeaFormer and STDC.The cited objects include roads, sidewalks, poles, traffic lights, traffic signs, and cars.
  • Cityscapes: The Cityscapes visualizations indicate that SCTNet combines long-range context with preservation of fine details.These additional results are presented in Figure 10.
Loading 2312.17071v2…