Source-linked AI summary
Dual Attention Network for Scene Segmentation
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, Hanqing Lu
TL;DR
Scene segmentation must distinguish confusing categories and objects that vary in appearance, scale, lighting, and viewpoint. DANet uses spatial and channel self-attention to integrate local features with global dependencies, achieving state-of-the-art performance across three challenging benchmarks, including 39.7% Mean IoU on COCO Stuff.
Problem
Scene segmentation requires discriminating confusing categories and objects with varied appearance, scale, lighting, and viewpoint, while local convolutions can produce inconsistent same-label features.
Method
DANet adds parallel position and channel self-attention modules to a dilated FCN, selectively aggregating global spatial and channel dependencies.
Results
DANet achieves state-of-the-art results on Cityscapes, PASCAL Context, and COCO Stuff, reaching 39.7% Mean IoU on COCO Stuff.
Takeaways & Limitations
Across four scene-segmentation datasets, dual attention captures long-range contextual information and produces more precise segmentation results.
Abstract
from arXiv · showhide
In this paper, we address the scene segmentation task by capturing rich contextual dependencies based on the selfattention mechanism. Unlike previous works that capture contexts by multi-scale features fusion, we propose a Dual Attention Networks (DANet) to adaptively integrate local features with their global dependencies. Specifically, we append two types of attention modules on top of traditional dilated FCN, which model the semantic interdependencies in spatial and channel dimensions respectively. The position attention module selectively aggregates the features at each position by a weighted sum of the features at all positions. Similar features would be related to each other regardless of their distances. Meanwhile, the channel attention module selectively emphasizes interdependent channel maps by integrating associated features among all channel maps. We sum the outputs of the two attention modules to further improve feature representation which contributes to more precise segmentation results. We achieve new state-of-the-art segmentation performance on three challenging scene segmentation datasets, i.e., Cityscapes, PASCAL Context and COCO Stuff dataset. In particular, a Mean IoU score of 81.5% on Cityscapes test set is achieved without using coarse data. We make the code and trained model publicly available at https://github.com/junfu1115/DANet
1 Yong Li · 1. Introduction
Scene segmentation must distinguish semantically confusing regions and diverse objects under varying scales, occlusion, and illumination while modeling global relationships. DANet addresses these challenges with parallel spatial and channel self-attention modules, achieving state-of-the-art results on three benchmarks.
- 1. Introduction: Scene segmentation parses each pixel into semantic regions covering stuff such as sky, road, and grass and objects such as people, cars, and bicycles.The task supports applications including automatic driving, robot sensing, and image editing.
- 1. Introduction: Confusing categories and object variation from scale, occlusion, and illumination make effective scene segmentation challenging.Field and grass can be difficult to distinguish, while cars may be affected by changing visual conditions.
- 1. Introduction: Multi-scale context fusion and encoder-decoder FCN methods capture varied object scales but cannot leverage global relationships between objects or stuff.Recurrent methods model long-range dependencies, but capture global relationships implicitly through recurrent neural networks.
- 1. Introduction: DANet appends parallel position and channel attention modules to a dilated FCN to model feature dependencies in spatial and channel dimensions.The framework uses self-attention to capture these dependencies explicitly.
- 1. Introduction: DANet adaptively integrates similar features across scales from a global view, helping represent inconspicuous or incomplete objects without allowing salient-object context to dominate.The motivation includes treating features at different scales equally when they represent the same semantics.
- 1. Introduction: The position attention module learns spatial interdependencies, while the channel attention module models channel interdependencies over local features.Together, they improve feature discriminability and segmentation by modeling rich contextual dependencies.
- 1. Introduction: DANet achieves new state-of-the-art results on Cityscapes, PASCAL Context, and COCO Stuff.These are identified as three popular scene-segmentation benchmarks.
2. Related Work
Prior semantic-segmentation methods enhance contextual aggregation through multi-scale pooling or dilated convolutions, while related work applies self-attention to image generation and non-local operations to videos and images. This work extends self-attention to scene segmentation with two attention modules for richer contextual relationships and improved feature representations.
- Semantic Segmentation: Semantic-segmentation models enhance contextual aggregation using atrous spatial pyramid pooling or pyramid pooling modules.Deeplabv2 and Deeplabv3 use parallel dilated convolutions with different dilation rates, while PSPNet collects contextual priors through pyramid pooling.
- Self-Attention: Related self-attention work targets image generation, while non-local operations model relationships across spacetime dimensions in videos and images.
- Our Approach: The proposed method extends self-attention to scene segmentation through two attention modules that capture rich contextual relationships and improve feature representations with intra-class compactness.Comprehensive empirical results verify the effectiveness of the proposed method.
3. Dual Attention Network
The Dual Attention Network augments a dilated residual backbone with position and channel attention modules that model long-range contextual dependencies over local features, then fuses their outputs for pixel-level prediction.
- Feature Fusion: The outputs of the position and channel modules are transformed by convolution, summed element-wise, and passed through a final convolution to generate the prediction map.The modules are directly insertable, and cascading is avoided because it requires more GPU memory.
- Motivation: DANet addresses intra-class inconsistency from local receptive fields by adaptively aggregating global contextual information among features.Scene content varies across scale, lighting, and view, causing same-label pixels to differ and harming recognition accuracy.
- Network Framework: The network uses a pretrained dilated residual backbone, removing downsampling and applying dilated convolutions in the final two ResNet blocks to produce a feature map at 1/8 input size.Two attention modules draw global context over the resulting local features for pixel-level prediction.
- Position Attention Module: The position attention module computes an N×N spatial attention map and represents each position as a weighted sum of features across all positions and the original features.Similar semantic features reinforce one another, improving intra-class compactness and semantic consistency.
- Channel Attention Module: The channel attention module models interdependencies among C channel maps, producing weighted combinations of all channels and original features to boost feature discriminability.It preserves channel relationships by avoiding feature-embedding convolutions and uses spatial information at all corresponding positions rather than global pooling or encoding.
4. Experiments
The experiments evaluate DANet on four scene-segmentation datasets and report state-of-the-art performance on three of them. The section covers datasets and implementation details, Cityscapes ablations, and results on PASCAL VOC 2012 and PASCAL Context.
- Datasets: DANet is evaluated on Cityscapes, PASCAL VOC2012, PASCAL Context, and COCO Stuff.The experiments use four datasets to assess the proposed method.
- Overall results: DANet achieves state-of-the-art performance on three datasets.The passage summarizes the experimental results as state of the art across three datasets.
- Experimental organization: The experiments first introduce datasets and implementation details, then conduct Cityscapes ablations, and finally report results on PASCAL VOC 2012 and PASCAL Context.This sequence describes the organization of the experiment subsections.
4.1. Datasets and Implementation Details
The experiments use Cityscapes, PASCAL VOC 2012, and COCO Stuff with specified dataset splits and semantic categories. The method is implemented in PyTorch with a polynomial learning-rate schedule and synchronized batch normalization.
- Datasets: Cityscapes contains 5,000 images from 50 cities, with 19 semantic classes and 2,979 training, 500 validation, and 1,525 test images.Images are 2048 × 1024 pixels, and coarse data is not used.
- Datasets: PASCAL VOC 2012 contains 10,582 training, 1,449 validation, and 1,456 testing images covering 20 foreground classes and one background class.
- Datasets: COCO Stuff includes 9,000 training and 1,000 testing images, with results reported on 171 categories comprising 80 objects and 91 stuff classes.
- Implementation Details: The method is implemented in PyTorch using a polynomial learning-rate policy, synchronized batch normalization, momentum 0.9, weight decay 0.0001, and batch sizes of 8 for Cityscapes and 16 otherwise.The Cityscapes base learning rate is 0.01.
4.2. Results on Cityscapes Dataset
On Cityscapes, dual attention substantially improves segmentation over the FCN baseline, with position and channel modules contributing complementary gains. Combining attention with augmentation, multi-grid features, and multi-scale inference reaches 81.50% Mean IoU on validation and 81.5% on the test set.
- Ablation study: 75.74% Mean IoU from position attention improves the ResNet-50 FCN baseline by 5.71%, while channel attention alone improves it by 4.25%.Combining both modules further raises performance to 76.34%.
- Attention visualization: Position attention produces clearer details and object boundaries, including poles and sidewalks, by selectively fusing local features.The visualizations show clearer semantic similarity and long-range relationships in position-attention maps.
- Attention visualization: Channel attention corrects some misclassified categories and enhances semantic responses, with attended channels specializing in car and vegetation regions.The eleventh channel responds to cars, while the fourth responds to vegetation across the examples.
- Strategy comparison: 81.50% performance is achieved after random-scale augmentation, Multi-Grid, and multi-scale probability fusion, surpassing Deeplabv3’s 79.30% by 2.20%.Random-scale augmentation contributes almost 1.26%, and Multi-Grid adds a further 1.11%.
- Testing-set comparison: 81.5% Mean IoU on the Cityscapes testing set enables DANet to outperform existing approaches using only fine annotated data.The comparison uses DANet-101 with a ResNet-101 backbone and reports advantages over PSANet and DenseASPP.
4.3. Results on PASCAL VOC 2012 Dataset
On PASCAL VOC 2012, DANet improves substantially over the baseline, and a deeper ResNet-101 backbone achieves a Mean IoU of 80.4%. The best model is further fine-tuned on the trainval set for test-set evaluation.
4.4. Results on PASCAL Context Dataset
On PASCAL Context, DANet improves Mean IoU over the Dilated FCN-50 baseline, reaching higher performance with ResNet101 pretraining. Experiments use the same training and testing settings as PASCAL VOC 2012.
- Results on PASCAL Context Dataset: 52.6% Mean IoU is achieved by DANet with a pretrained ResNet101 on PASCAL Context.This result is reported as outperforming previous methods.
- Results on PASCAL Context Dataset: 50.1% Mean IoU is achieved by DANet-50, compared with 44.3% for the Dilated FCN-50 baseline.DANet-50 boosts the baseline performance by 5.8 percentage points.
- Results on PASCAL Context Dataset: The PASCAL Context experiments adopt the same training and testing settings as the PASCAL VOC 2012 dataset.Quantitative results are presented in Table 6.
4.5. Results on COCO Stuff Dataset
On the COCO Stuff testing set, DANet achieves 39.7% Mean IoU, substantially outperforming previous state-of-the-art methods. The comparison includes methods modeling spatial dependencies with chain-RNNs and improving inconspicuous-object and background-stuff segmentation with decoder gating.
- Results on COCO Stuff Dataset: 39.7% Mean IoU is achieved by the proposed model on the COCO Stuff testing set.These experiments evaluate the network’s generalization on COCO Stuff.
- Results on COCO Stuff Dataset: The model outperforms previous state-of-the-art methods by a large margin.The comparisons are reported in Table 7.
- Results on COCO Stuff Dataset: DAG-RNN models rich spatial dependencies in 2D images with chain-RNNs, while Ding et al. improve inconspicuous-object and background-stuff segmentation through decoder-stage gating.These are methods included among the comparisons.
5. Conclusion
The paper presents DANet for scene segmentation, adaptively integrating local semantic features through self-attention. Its position and channel attention modules capture global dependencies, while ablations show improved long-range context modeling and segmentation precision.
- DANet adaptively integrates local semantic features using the self-attention mechanism for scene segmentation.
- The position and channel attention modules capture global dependencies in spatial and channel dimensions, respectively.
- Ablation experiments show that the dual attention modules effectively capture long-range contextual information and produce more precise segmentation results.