Source-linked AI summary
Spatial Pyramid Based Graph Reasoning for Semantic Segmentation
Xia Li, Yibo Yang, Qijie Zhao, Tiancheng Shen, Zhouchen Lin, Hong Liu
TL;DR
Limited CNN receptive fields make global context difficult to model efficiently in semantic segmentation. The paper introduces SpyGR, which applies graph reasoning with a data-dependent Laplacian and attention metric directly to spatial-pyramid features. Experiments across four datasets report effective segmentation with computational and memory overhead advantages.
Problem
CNN convolution captures mainly local information because of limited receptive fields, while semantic segmentation requires long-range context and pixel-level interactions.
Method
SpyGR performs graph reasoning directly in original CNN feature space organized as a spatial pyramid, using a data-dependent Laplacian and an attention diagonal matrix.
Results
Experiments on Cityscapes, COCO Stuff, PASCAL Context, and PASCAL VOC show effective segmentation with advantages in computational and memory overhead.
Takeaways & Limitations
Direct graph reasoning preserves spatial relationships and enables multiple long-range contextual patterns from different scales without introducing more computational or memory consumption.
Takeaways & Limitations
The propagation rule is considered under the assumption that it is applied to CNN features represented as H × W × C tensors.
Abstract
from arXiv · showhide
The convolution operation suffers from a limited receptive filed, while global modeling is fundamental to dense prediction tasks, such as semantic segmentation. In this paper, we apply graph convolution into the semantic segmentation task and propose an improved Laplacian. The graph reasoning is directly performed in the original feature space organized as a spatial pyramid. Different from existing methods, our Laplacian is data-dependent and we introduce an attention diagonal matrix to learn a better distance metric. It gets rid of projecting and re-projecting processes, which makes our proposed method a light-weight module that can be easily plugged into current computer vision architectures. More importantly, performing graph reasoning directly in the feature space retains spatial relationships and makes spatial pyramid possible to explore multiple long-range contextual patterns from different scales. Experiments on Cityscapes, COCO Stuff, PASCAL Context and PASCAL VOC demonstrate the effectiveness of our proposed methods on semantic segmentation. We achieve comparable performance with advantages in computational and memory overhead.
1. Introduction
The paper addresses limited CNN receptive fields in semantic segmentation by introducing graph reasoning directly in CNN feature space across a spatial pyramid. Its data-dependent Laplacian and attention metric retain spatial relationships, reduce overhead, and support long-range context from multiple scales.
- Motivation: CNNs mainly capture local information because convolutional operations have limited receptive fields, making deep stacking necessary for global context.The paper identifies this stacking strategy as inefficient for aggregating global information.
- Motivation: Long-range interactions are important for semantic segmentation because distant pixels from the same object are difficult to connect, while local visual similarity can cause misclassification.The problem is especially relevant to pixel-level classification in complex scenes.
- Method: The proposed method performs graph reasoning directly in original CNN features organized as a spatial pyramid, avoiding projection and re-projection.This design is named the Spatial Pyramid Based Graph Reasoning (SpyGR) layer.
- Method: The improved Laplacian is data-dependent and includes a diagonal matrix with position-agnostic attention on inner products to learn a better distance metric.This formulation differs from methods using a learnable data-independent Laplacian.
- Method: Direct reasoning in feature space preserves spatial relationships and lets the spatial pyramid capture long-range contextual patterns from finer and coarser representations.Finer features provide more detailed long-range context, while coarser features provide more global relationships.
- Results: Experiments on PASCAL Context, PASCAL VOC, Cityscapes, and COCO Stuff show top performance with computational and memory overhead advantages.The SpyGR layer is described as lightweight and easy to plug into CNN architectures.
2. Related Work
Prior semantic-segmentation methods enhance resolution, multi-scale context, or global dependencies through pyramids, attention, advanced convolutions, CRFs, and RNNs. SpyGR instead performs graph reasoning on each downsampled scale of the final predicting feature and aggregates the upsampled outputs while retaining spatial relationships.
- Semantic segmentation: Semantic-segmentation architectures use deconvolution, encoder-decoder designs, skip connections, and multi-path refinement to preserve spatial resolution and object details.These methods build on fully convolutional networks.
- Multi-scale context: Multi-scale context methods include Laplacian-pyramid inputs, image-level features, atrous spatial pyramid pooling, and spatial pyramid pooling.These approaches aggregate contextual information across image or feature scales.
- Global context: Global-context methods use advanced convolutions, attention, CRFs, and RNN variants to build long-range dependencies, but efficient explicit semantic reasoning remains an open need.The related work frames graph convolution as another route to global information.
- Graph reasoning: SpyGR performs graph reasoning directly on original feature space, sparing projection and re-projection while retaining spatial relationships.Its learnable Laplacian is data dependent rather than a fixed data-independent matrix.
- Feature pyramid: The method builds a spatial pyramid by downsampling and upsampling the final predicting feature, then aggregates graph reasoning outputs from each scale.This differs from using multi-scale features from the CNN backbone.
3. Our Methods
The method performs data-dependent graph reasoning directly on CNN feature maps organized as a spatial pyramid. It combines an improved Laplacian, efficient computation, and multiscale propagation to capture long-range context without projection and re-projection.
- 3.1. Graph Reasoning on Graph Structures: Graph convolution models signals on graph-structured data through spectral filtering, with Chebyshev polynomial approximations avoiding explicit Laplacian eigendecomposition.The common simplified formulation limits the polynomial order to K = 1 and approximates the largest Laplacian eigenvalue by 2.
- 3.2. Graph Reasoning on Spatial Features: The method applies graph reasoning directly to the original CNN feature space, treating the grid as a low-dimensional graph and avoiding projection and re-projection.The graph Laplacian performs the global left multiplication that projection-based methods use to establish interactions across spatial locations.
- 3.2. Graph Reasoning on Spatial Features: The improved Laplacian is data-dependent, using a symmetric normalized similarity matrix whose structure varies with the input feature.The similarity matrix uses dot-product distance, while the diagonal attention matrix reweights the embedded feature dimensions to learn a better distance metric.
- 3.3. Graph Reasoning on Spatial Pyramid: Spatial-pyramid graph reasoning performs propagation independently at downsampled scales and combines the outputs through upsampling to capture long-range patterns at multiple resolutions.The implementation uses max-pooling with stride 2 for downsampling and bilinear interpolation for upsampling, operating on the final predicting feature.
- 3.4. Complexity Analysis: The efficient computation strategy avoids forming quadratic spatial interactions, and four-scale reasoning adds no drastic computational or memory increase in the reported setting.The method computes degree and Laplacian products through vector multiplications and circumvents O(H^2W^2) computation on spatial locations; the analysis uses C = 512, M = 64, and H = W = 97.
4. Experiments
Experiments across several semantic-segmentation benchmarks show that SpyGR improves or matches competing methods while maintaining relatively low computational and memory overhead. Ablations and visualizations further examine its components and multi-scale contextual behavior.
- Benchmark comparisons: SpyGR outperforms GloRe by 0.7 in mIoU on Cityscapes and exceeds DANet despite lower computation and memory requirements.The Cityscapes comparison uses ResNet-101 with output stride 8, fine annotations, multi-scale inference, and flipping.
- Benchmark comparisons: SpyGR consistently gains at least 1 mIoU over DeepLabV3 across Cityscapes and PASCAL VOC comparisons.Its advantage over DeepLabV3+ is reported as more significant on PASCAL VOC than on Cityscapes.
- Benchmark comparisons: On COCO Stuff, SpyGR outperforms other methods, matches DANet comparably, and shows significant superiority over SGR.Training uses output stride 8, 30K iterations, and multi-scale input with flipping for testing.
- Benchmark comparisons: On PASCAL Context, SpyGR with ResNet-50 matches SGR on ResNet-101, surpasses MSCI on ResNet-152, and exceeds SGR+ with ResNet-101.The reported comparisons use multi-scale and flipping augmentations; SpyGR also outperforms DANet by a small margin with lower overhead and shorter training.
- Ablation studies: Ablations show gains from the simplest GCN, learned diagonal attention, data dependence, identity connection, and spatial-pyramid organization.The simplest GCN increases mIoU by 1.64; a data-independent diagonal matrix adds 0.60 over it, while data-dependent attention adds another 0.47 over the data-independent case.
- Visualization analysis: Similarity visualizations show that different spatial-pyramid scales capture distinct long-range contextual patterns, aggregated at the finest level.The visualization compares four pyramid scales from coarsest to finest for randomly sampled image locations.
5. Conclusion
The paper introduces SpyGR to model long-range context for semantic segmentation through graph reasoning directly in a spatial-pyramid feature space. Its data-dependent formulation and efficient computation improve performance without additional computational or memory consumption.
- SpyGR models long-range context for semantic segmentation using graph reasoning directly in the original feature space organized as a spatial pyramid.
- The method uses a data-dependent improved Laplacian and an attention diagonal matrix to learn a better distance metric.
- Removing projection and re-projection preserves spatial relationships and enables spatial-pyramid reasoning across contextual scales.
- The proposed computing scheme significantly reduces computational overhead while avoiding additional computational or memory consumption.