Source-linked AI summary
ISNet: Integrate Image-Level and Semantic-Level Context for Semantic Segmentation
Zhenchao Jin, Bin Liu, Qi Chu, Nenghai Yu
TL;DR
Image-level context aggregation can introduce information from other categories, while giving less significance to representations of the same category. ISNet combines image-level and semantic-level context to augment pixel representations. It reports state-of-the-art accuracy on ADE20K, LIP, COCOStuff, and Cityscapes.
Problem
Image-level context methods can mix information from other categories and weaken the significance of pixel representations from the same category.
Source: §1 · p. 2Method
ISNet extracts whole-image and same-category context, then weights both contexts when augmenting pixel representations using their similarities to each pixel.
Source: §1 · p. 2Results
ISNet reports state-of-the-art accuracy on ADE20K, LIP, COCOStuff, and Cityscapes.
Source: §1 · p. 2Takeaways & Limitations
The paper presents integrating image-level and semantic-level context as an approach to improve pixel representations for semantic segmentation.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Co-occurrent visual pattern makes aggregating contextual information a common paradigm to enhance the pixel representation for semantic image segmentation. The existing approaches focus on modeling the context from the perspective of the whole image, i.e., aggregating the image-level contextual information. Despite impressive, these methods weaken the significance of the pixel representations of the same category, i.e., the semantic-level contextual information. To address this, this paper proposes to augment the pixel representations by aggregating the image-level and semantic-level contextual information, respectively. First, an image-level context module is designed to capture the contextual information for each pixel in the whole image. Second, we aggregate the representations of the same category for each pixel where the category regions are learned under the supervision of the ground-truth segmentation. Third, we compute the similarities between each pixel representation and the image-level contextual information, the semantic-level contextual information, respectively. At last, a pixel representation is augmented by weighted aggregating both the image-level contextual information and the semantic-level contextual information with the similarities as the weights. Integrating the image-level and semantic-level context allows this paper to report state-of-the-art accuracy on four benchmarks, i.e., ADE20K, LIP, COCOStuff and Cityscapes.
1. Introduction
Semantic segmentation benefits from context aggregation, but image-level methods can mix information from different categories and weaken same-category pixel representations. ISNet combines image-level and semantic-level context, reporting state-of-the-art accuracy on four benchmarks.
1. Introduction
- The paper identifies context modeling as a way to encode contextual information into pixel features, alongside improving encoder structures.It situates this focus among segmentation methods built on FCN’s encoder-decoder structure.Source: §1 · p. 1
- Image-level context methods can bring in information from other categories, especially for boundary and small-object pixels, risking mislabeling.These approaches may capture context unevenly from a pixel’s own category region and regions of other categories.Source: §1 · p. 2
- ISNet aggregates whole-image context and same-category context, then uses pixel-to-context similarities to weight both in augmenting pixel representations.Semantic category regions are learned under ground-truth segmentation supervision.Source: §1 · p. 2
- The paper presents ISNet as a framework for leveraging ILCM and SLCM to improve semantic image segmentation.It describes this as the first exploration of augmenting pixel representations with both image-level and semantic-level context.Source: §1 · p. 2
- ISNet reports state-of-the-art accuracy on ADE20K, LIP, COCOStuff, and Cityscapes.The framework combines image-level and semantic-level context modules.Source: §1 · p. 2
2. Related Work
Prior segmentation work improves pixel representations through backbones and contextual information, including multi-scale and relational context aggregation. ISNet’s semantic-level module differs from region-based methods by using only the representation of a pixel’s own category.
2. Related Work
- Segmentation research has improved pixel representations through backbone design and contextual-information aggregation, including multi-scale cues, attention, and image or feature pyramids.Examples include HRNet and ResNeSt for backbones, and atrous convolution, pooling, and neural attention for context.Source: §2 · p. 2
- ISNet combines semantic-level and image-level contextual information, with similarities between pixel representations and each context guiding augmentation.The framework overview names SLCM and ILCM as the modules extracting those contexts.Source: Figure 2 · p. 3
- ISNet augments a pixel using the region representation of its own category, unlike region-based methods that aggregate representations from all regions.The paper notes that its semantic-level context module resembles region-based approaches but differs in this category-matched aggregation.Source: §2 · p. 3
3. Methodology
ISNet augments pixel representations with both whole-image and same-category context. Its formulation computes similarities to each context and combines them with the original representations for pixel classification.
3.1. Formulation
- ISNet forms image-level and semantic-level contexts for each pixel, computes similarities to both, and fuses the weighted contexts with the original representation before classification.The augmented representation is upsampled through a classification head to predict pixel labels.Source: §3.1 · pp. 3–4
3.2. Image-Level Context Module
- Channel-wise spatial means provide global context, which is repeated and fused with pixel representations through a transform function implemented as a 1 × 1 convolution.The paper notes that ILCM can be replaced by existing modules such as ASPP [4] or PPM [35].Source: §3.2 · p. 4
3.3. Semantic-Level Context Module
- SLCM groups pixel representations by predicted category and computes a weighted composite representation for each category to provide semantic-level context.The category assignment uses the argmax of the predicted distribution, and the category representations are weighted using class probabilities.Source: §3.3 · p. 4
3.4. Loss Function
- The loss jointly trains category-distribution predictions and final pixel predictions with cross-entropy terms, weighting the auxiliary distribution loss by α = 0.4 by default.The model parameters are learned jointly through backpropagation.Source: §3.4 · p. 5
4. Experiments
Experiments evaluate ISNet on four segmentation benchmarks, test the contributions of its context modules, and compare it with existing methods. The reported results include state-of-the-art performance across the benchmarks, with a configuration change for LIP.
4.1. Experimental Setup
- Experiments cover ADE20K, COCOStuff, LIP, and Cityscapes, use mIoU for evaluation, and describe benchmark-specific training and inference settings.The default testing procedure uses no multi-scale or flipping tricks; some state-of-the-art comparisons separately use them for fair comparison.Source: §4.1 · p. 5
4.2. Ablation Study
- On ADE20K, ILCM, SLCM, and their combination improved the base framework by 5.54%, 5.93%, and 7.13% mIoU, respectively.The combined gain exceeded either module alone; qualitative examples also show improved segmentation after adding SLCM, including a partly visible painting whose ground truth was wrong.Source: §4.2 · pp. 5–6, Table 1 · p. 6, Figure 3 · p. 6
4.3. Comparison with State-of-the-Art
- 47.31% mIoU with ResNet-101 exceeded ADE20K’s previous best by 1.41%, while ResNeSt-101 reached 47.55%.The ADE20K comparison uses multi-scale and flipping testing.Source: Table 4 · p. 7, §4.3 · p. 7
- 42.08% mIoU with ResNeSt-101 established a new COCOStuff test-set state of the art.With ResNet-101, ISNet also surpassed OCRNet, EMANet, DANet, and ACNet by 2.10%, 1.70%, 1.90%, and 1.50% mIoU, respectively.Source: Table 5 · p. 7, §4.3 · p. 8
4.2. Ablation Study
- On ADE20K, ISNet scored 44.09% mIoU versus 42.50% for its ILCM-only variant, while the proposed scheme used the least parameters and FLOPs among the compared context schemes.The comparison used the validation set; complexity was measured during inference on a single NVIDIA Tesla V100 GPU.Source: Table 3 · p. 6, Table 2 · p. 6, §4.2 · p. 7
4.3. Comparison with State-of-the-Art
- On LIP, ISNet‡ with ASPP exceeded OCRNet with HRNetV2-W48 by 0.31% mIoU, while ASPP alone scored 55.34% mIoU.The LIP comparison uses flipping testing; ‡ denotes ASPP as the image-level context module.Source: Table 6 · p. 7, §4.3 · p. 8
- On the Cityscapes validation set, the method reported a new state-of-the-art mIoU of 81.10% under single-scale testing, supporting aggregation of image-level and semantic-level context to augment each pixel representation.Source: Table 7 · p. 7, §4.3 · p. 8
5. Conclusion
ISNet integrates image-level and semantic-level context to augment pixel representations for semantic segmentation. The paper reports new state-of-the-art results on four benchmarks.
5. Conclusion
- ISNet augments pixel representations by weighted aggregation of image-level and semantic-level contextual information.The method first captures global semantic structure, then aggregates semantic-level context for each pixel.Source: §5 · p. 8
- The authors report new state-of-the-art performance on ADE20K, Cityscapes, LIP, and COCOStuff.Source: §5 · p. 8
- The paper addresses the risk that image-level context from other categories may lead to pixel mislabeling.Source: §5 · p. 8