Source-linked AI summary
Dual Graph Convolutional Network for Semantic Segmentation
Li Zhang, Xiangtai Li, Anurag Arnab, Kuiyuan Yang, Yunhai Tong, Philip H. S. Torr
TL;DR
Semantic segmentation needs long-range context because isolated pixels are ambiguous and conventional approaches have limited or inefficient contextual modeling. DGCNet uses complementary spatial and channel graph reasoning through projected feature spaces, achieving state-of-the-art results on Cityscapes and Pascal Context.
Problem
Semantic segmentation requires long-range contextual information because local pixel evidence is ambiguous and conventional fully convolutional networks have limited receptive fields.
Method
DGCNet models global context with orthogonal coordinate-space and feature-space graphs, projecting features into lower-dimensional spaces for efficient relational reasoning before reprojection.
Results
82.0% mean IoU on Cityscapes and 53.7% mean IoU on Pascal Context are reported as state-of-the-art results.
Takeaways & Limitations
The two-branch graph-convolutional module provides significant improvements over a strong baseline for semantic segmentation.
Abstract
from arXiv · showhide
Exploiting long-range contextual information is key for pixel-wise prediction tasks such as semantic segmentation. In contrast to previous work that uses multi-scale feature fusion or dilated convolutions, we propose a novel graph-convolutional network (GCN) to address this problem. Our Dual Graph Convolutional Network (DGCNet) models the global context of the input feature by modelling two orthogonal graphs in a single framework. The first component models spatial relationships between pixels in the image, whilst the second models interdependencies along the channel dimensions of the network's feature map. This is done efficiently by projecting the feature into a new, lower-dimensional space where all pairwise interactions can be modelled, before reprojecting into the original space. Our simple method provides substantial benefits over a strong baseline and achieves state-of-the-art results on both Cityscapes (82.0% mean IoU) and Pascal Context (53.7% mean IoU) datasets. Code and models are made available to foster any further research (\url{https://github.com/lxtGH/GALD-DGCNet}).
1 Introduction
Semantic segmentation requires long-range contextual reasoning because isolated pixels provide ambiguous evidence. DGCNet addresses this with complementary spatial and channel graph reasoning, achieving state-of-the-art results on Cityscapes and Pascal Context.
- Semantic segmentation assigns an object class label to every pixel and supports applications including autonomous driving, augmented reality, virtual reality, and medical diagnosis.
- Local pixel evidence is ambiguous and noisy, so contextual information is needed to reason about occlusions, small objects, and object co-occurrences.
- FCNs have limited receptive fields for long-range pixel relationships, while dilated convolutions can produce representations dominated by large objects and poor small-object performance.
- DGCNet combines a coordinate-space GCN for pixel relationships with a feature-space GCN for channel interdependencies and distributes both relation-aware features back to the original space.
- 82.0% mean IoU on Cityscapes and 53.7% mean IoU on Pascal Context establish new state-of-the-art results for the proposed approach.
Related Work
Prior segmentation methods expand context through dilated or multiscale convolutions, recurrent propagation, or attention-based relations. DGCNet instead models global context with two orthogonal graphs in one framework and projects features into spaces where relational reasoning is efficient.
- Dilated convolutions, pyramid pooling, and dense multiscale connections increase the effective receptive field of fully convolutional networks.
- DAG-RNN and PSANet explicitly model pixel relationships, whereas EncNet and DFN use channel attention to represent global class co-occurrences.
- Non-local operators propagate information across the whole image but can incur high memory costs because their affinity matrices grow quadratically with pixel count.
- DGCNet models global context through two orthogonal graphs, projecting features into coordinate and feature spaces for efficient relational reasoning before reprojection and addition to the original feature.
3 Methodology
DGCNet uses two graph-convolutional branches to model contextual relationships in coordinate and feature spaces, then combines their reprojections with the original feature. The design projects features into lower-dimensional spaces for efficient global reasoning before returning them to the CNN feature space.
- Dual graph framework: DGCNet builds coordinate-space and feature-space GCNs to capture spatial relationships and channel-wise feature interdependencies.The coordinate branch reasons over image locations, while the feature branch captures correlations among abstract features such as object parts.
- Coordinate space: The coordinate branch downsamples the input feature into a coarser spatial grid before constructing a lightweight fully connected graph.Average pooling or chained depth-wise convolutions can perform the spatial projection, and graph nodes aggregate information from pixel clusters.
- Coordinate space: Global relational reasoning applies three learnable transformations to the downsampled coordinate features to produce graph messages.The resulting messages are reprojected by nearest-neighbor upsampling and a 1×1 convolution to restore the original spatial and channel dimensions.
- Feature space: The feature branch projects the input into a lower-dimensional feature space whose fully connected graph models relationships among feature descriptors.The projection uses 1×1 convolutions, and the feature-space graph uses trainable edge weights with Laplacian smoothing and a residual identity connection.
- DGCNet architecture: The refined feature is the point-wise sum of the original feature and the coordinate- and feature-space outputs, allowing insertion into CNN backbones.The implementation inserts the module between two 3×3 convolutions near the end of an FCN using an ImageNet-pretrained ResNet-101 backbone.
4 Experiments
Experiments on Cityscapes and Pascal Context evaluate DGCNet through ablations, computational analysis, and comparisons with existing methods. The method improves strong baselines and achieves state-of-the-art segmentation results on both datasets.
- Ablation studies: 80.5% mean IoU results from combining Spatial-GCN and Channel-GCN, exceeding the dilated-convolution baseline by 4.1 percentage points for the strongest single module.Channel-GCN alone reaches 79.3% with a 4.1% improvement, while Spatial-GCN alone improves the baseline by 3.6%.
- Ablation studies: Consistent improvements from additional training and inference strategies show that the proposed GCN module is complementary to these strategies.The evaluated strategies include OHEM, Multi-Grid, and multi-scale ensembling.
- Computational cost and mapping strategies: Fewer floating point operations than DANet accompany higher performance, while average pooling and strided convolution provide similar results across downsampling ratios.Average pooling slightly outperforms strided convolution at d = 4, and both strategies remain robust for d = 4, 8, and 16.
- Comparisons with state-of-the-art: 82.0% mean IoU on Cityscapes is achieved when training uses both train-fine and val-fine sets, establishing state-of-the-art performance.Using only the training set yields 80.7%, outperforming PSANet by 2.1% and OC-Net by 0.8%; the method obtains the highest IoU in 16 of 19 classes.
- Pascal Context: 53.7% mean IoU on Pascal Context with multiscale inference surpasses the previous state-of-the-art by a large margin.With ResNet-50, DGCNet reaches 50.1% under single-scale evaluation; with ResNet-101, it reaches 53.0% before multiscale inference.
5 Conclusion
The paper proposes a graph-convolutional module for contextual relationships in semantic segmentation, using separate spatial and channel-context branches. It reports improvements over a strong baseline and state-of-the-art results on Cityscapes and Pascal Context.
- The proposed graph-convolutional module models contextual relationships in images for dense prediction tasks such as semantic segmentation.
- DGCNet uses one branch to capture spatial context and another to capture channel context in convolutional feature maps.
- The approach improves substantially over a strong baseline and achieves state-of-the-art results on Cityscapes and Pascal Context.
Appendix
The appendix compares Cityscapes results with a Dilated FCN ResNet101 baseline and Pascal Context results with EncNet. The reported Cityscapes testing results reach 82.0% mean IoU and lead in 16 of 19 classes.
- Cityscapes results are compared with the Dilated FCN ResNet101 baseline.
- 82.0% Mean IoU is achieved on the Cityscapes testing set, with the highest IoU in 16 of 19 classes.
- Pascal Context results are compared with the state-of-the-art EncNet method, with more consistent results and fewer artifacts reported.