Source-linked AI summary
SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation
Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, Shi-Min Hu
TL;DR
Semantic segmentation needs contextual representations while handling dense, multi-scale prediction efficiently, motivating alternatives to transformer-dominated designs. SegNeXt rethinks convolutional attention with multi-scale convolutional features, a convolutional encoder, and a lightweight global-context decoder. It reports substantial gains over transformer-based methods across multiple benchmarks, including a Cityscapes efficiency comparison, while acknowledging limits in scaling beyond 100M parameters and transferring to other tasks.
Problem
Successful semantic segmentation requires strong encoders, multi-scale interaction, spatial attention, and low computational complexity, especially for high-resolution imagery.
Method
SegNeXt uses multi-scale convolutional features for spatial attention, a mostly convolutional encoder, and a lightweight Hamburger decoder for global-context extraction.
Results
SegNeXt surpasses transformer-based state-of-the-art methods across ADE20K, Cityscapes, COCO-Stuff, Pascal VOC, Pascal Context, and iSAID, while SegNeXt-S reaches 81.3% versus 81.0% for SegFormer-B2 at 1/6 the computation.
Takeaways & Limitations
The results support CNN-based semantic segmentation with properly designed convolutional attention as an efficient alternative to transformer-based architectures.
Takeaways & Limitations
The authors identify extending the method to models with 100M+ parameters and evaluating it on other vision or NLP tasks as limitations for future work.
Abstract
from arXiv · showhide
We present SegNeXt, a simple convolutional network architecture for semantic segmentation. Recent transformer-based models have dominated the field of semantic segmentation due to the efficiency of self-attention in encoding spatial information. In this paper, we show that convolutional attention is a more efficient and effective way to encode contextual information than the self-attention mechanism in transformers. By re-examining the characteristics owned by successful segmentation models, we discover several key components leading to the performance improvement of segmentation models. This motivates us to design a novel convolutional attention network that uses cheap convolutional operations. Without bells and whistles, our SegNeXt significantly improves the performance of previous state-of-the-art methods on popular benchmarks, including ADE20K, Cityscapes, COCO-Stuff, Pascal VOC, Pascal Context, and iSAID. Notably, SegNeXt outperforms EfficientNet-L2 w/ NAS-FPN and achieves 90.6% mIoU on the Pascal VOC 2012 test leaderboard using only 1/10 parameters of it. On average, SegNeXt achieves about 2.0% mIoU improvements compared to the state-of-the-art methods on the ADE20K datasets with the same or fewer computations. Code is available at https://github.com/uyzhang/JSeg (Jittor) and https://github.com/Visual-Attention-Network/SegNeXt (Pytorch).
1 Introduction
SegNeXt is motivated by four properties of successful segmentation models and introduces a convolutional attention encoder-decoder designed to combine strong representation, multi-scale interaction, spatial attention, and low computation. It reports broad benchmark gains and strong efficiency against transformer-based methods.
- Motivation: Successful segmentation models need strong encoders, multi-scale information interaction, spatial attention, and low computational complexity.The latter is especially important for high-resolution remote-sensing and urban-scene images.
- Method: SegNeXt uses multi-scale convolutional features and element-wise multiplication to evoke spatial attention in an efficient encoder-decoder architecture.Its decoder aggregates multi-level features and uses a Hamburger module to extract global context.
- Results: SegNeXt-S reaches 81.3% versus SegFormer-B2's 81.0% on Cityscapes using 124.6G versus 717.1G computation and 13.9M versus 27.6M parameters.The comparison concerns high-resolution urban-scene segmentation.
- Conclusion: SegNeXt is presented as a tailored CNN-style architecture whose encoder uses simple, cheap convolutions while outperforming vision transformers at lower computational cost.The claimed advantage is especially associated with processing object details.
- Results: The method improves state-of-the-art semantic segmentation performance across ADE20K, Cityscapes, COCO-Stuff, Pascal VOC, Pascal Context, and iSAID.The paper presents this as a broad benchmark improvement rather than a result on one dataset.
2 Related Work
Prior segmentation models use CNN or transformer architectures, while SegNeXt emphasizes multi-scale feature interaction, efficient attention, and cheaper large-kernel convolutions.
- Transformer-based methods have recently outperformed CNN-based methods in semantic segmentation.
- Successful segmentation models require strong encoders and multi-scale information interaction because segmentation differs from single-object image classification.
- Earlier multi-scale architectures extract features through multi-branch processing or aggregate high- and low-resolution representations.
- SegNeXt combines multi-scale encoder features with efficient attention and cheaper, larger-kernel convolutions.
- Attention mechanisms adaptively select important content through spatial or channel attention, while vision transformers generally omit channel-dimension adaptability.
- Unlike VAN, SegNeXt addresses multi-scale feature aggregation, which the paper identifies as crucial for segmentation-like tasks.
3 Method
SegNeXt uses a hierarchical convolutional encoder with multi-scale convolutional attention and a lightweight global-context decoder, favoring efficient feature processing.
- SegNeXt follows an encoder-decoder architecture and replaces self-attention with a multi-scale convolutional attention module.
- MSCA combines depth-wise local aggregation, multi-branch strip convolutions for multi-scale context, and 1 × 1 convolution for channel relationships.
- MSCA reweights input features by element-wise multiplying them with attention maps derived from multi-scale convolutional features.
- Strip-convolution branches use kernel sizes 7, 11, and 21 to approximate large depth-wise kernels with lightweight operations.
- MSCAN stacks these blocks in four hierarchical stages with decreasing spatial resolutions, using stride-2 convolutions for downsampling.
- The model family contains four sizes: MSCAN-T, MSCAN-S, MSCAN-B, and MSCAN-L, paired with corresponding SegNeXt variants.
- The SegNeXt decoder aggregates the last three encoder stages and applies a lightweight Hamburger module to model global context.
- SegNeXt excludes Stage 1 features because they contain excessive low-level information and add computational overhead.
4 Experiments
Experiments evaluate SegNeXt across classification and segmentation benchmarks, ablations, efficiency trade-offs, and comparisons with CNN- and transformer-based methods. The results show strong performance with efficient computation and favorable decoder and multi-scale attention designs.
- Datasets and implementation: SegNeXt is evaluated on ImageNet-1K, ADE20K, Cityscapes, Pascal VOC, Pascal Context, COCO-Stuff, and iSAID.ImageNet-1K is used for encoder pretraining, while the other datasets support segmentation evaluation.
- Encoder performance: MSCAN outperforms ConvNeXt, Swin Transformer, and MiT on ImageNet classification.The comparison evaluates SegNeXt’s encoder against recent CNN-based and transformer-based classification models.
- Ablation studies: Each component of MSCA contributes to final performance on ImageNet and ADE20K.The ablation includes multi-scale convolution branches, channel mixing, and element-wise attention.
- Ablation studies: SegNeXt’s decoder achieves the best performance with low computational cost among the evaluated decoder structures.The comparison covers three decoder designs for producing high-resolution segmentation outputs.
- Comparison with state-of-the-art methods: 48.5 mIoU versus 46.5 for SegFormer-B2 is achieved by SegNeXt-B using 56% of the computations on ADE20K.On Cityscapes, SegNeXt-B gains 1.6 mIoU over SegFormer-B2 with 40% less computation.
- Comparison with state-of-the-art methods: SegNeXt-L surpasses Mask2Former with Swin-T by 3.3 mIoU on ADE20K with similar parameters and computational cost.It also outperforms HRNet with fewer parameters and computations, and SegNeXt-T reaches 25 FPS on Cityscapes using one RTX-3090 GPU.
5 Conclusions and Discussion
The paper concludes that convolutional attention and a CNN-style architecture can outperform transformer-based segmentation methods. It also identifies limits in extending the approach to larger models and other vision or NLP tasks.
- Conclusions: SegNeXt combines the MSCA convolutional attention module with a CNN-style network after analyzing successful segmentation models.The design captures multi-scale features and uses convolutional attention for segmentation.
- Conclusions: SegNeXt surpasses current state-of-the-art transformer-based methods by a considerable margin.The conclusion summarizes the experimental comparison across segmentation benchmarks.
- Limitations and future work: The paper identifies extending SegNeXt to models with 100M+ parameters and evaluating it on other vision or NLP tasks as limitations.These directions are left for future work.