Source-linked AI summary
Context Prior for Scene Segmentation
Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, Nong Sang
TL;DR
Existing segmentation methods often provide insufficient contextual information and rarely distinguish intra-class from inter-class dependencies. The paper introduces a supervised Context Prior and CPNet, achieving favorable state-of-the-art performance across ADE20K, PASCAL-Context, and Cityscapes.
Problem
Existing methods provide insufficient contextual information, while most approaches rarely distinguish intra-class and inter-class contextual dependencies.
Method
The method supervises a Context Prior with Affinity Loss and uses Context Prior Layers to selectively aggregate intra-class and inter-class context.
Results
46.3% mIoU on ADE20K, 53.9% mIoU on PASCAL-Context, and 81.3% mIoU on Cityscapes demonstrate favorable performance against state-of-the-art approaches.
Takeaways & Limitations
CPNet provides selectively aggregated contextual dependencies and robust feature representation for scene segmentation.
Abstract
from arXiv · showhide
Recent works have widely explored the contextual dependencies to achieve more accurate segmentation results. However, most approaches rarely distinguish different types of contextual dependencies, which may pollute the scene understanding. In this work, we directly supervise the feature aggregation to distinguish the intra-class and inter-class context clearly. Specifically, we develop a Context Prior with the supervision of the Affinity Loss. Given an input image and corresponding ground truth, Affinity Loss constructs an ideal affinity map to supervise the learning of Context Prior. The learned Context Prior extracts the pixels belonging to the same category, while the reversed prior focuses on the pixels of different classes. Embedded into a conventional deep CNN, the proposed Context Prior Layer can selectively capture the intra-class and inter-class contextual dependencies, leading to robust feature representation. To validate the effectiveness, we design an effective Context Prior Network (CPNet). Extensive quantitative and qualitative evaluations demonstrate that the proposed model performs favorably against state-of-the-art semantic segmentation approaches. More specifically, our algorithm achieves 46.3% mIoU on ADE20K, 53.9% mIoU on PASCAL-Context, and 81.3% mIoU on Cityscapes. Code is available at https://git.io/ContextPrior.
1. Introduction
Scene segmentation requires pixel-level category assignment for comprehensive scene understanding, but existing context aggregation can confuse categories. CPNet addresses this by explicitly modeling intra-class and inter-class dependencies with supervised Context Prior learning and achieves favorable benchmark performance.
- Scene segmentation assigns every pixel a category label and supports applications including augmented reality, autonomous driving, human-machine interaction, and video content analysis.
- Pyramid-based aggregation captures homogeneous context, while attention-based methods may aggregate undesirable relationships without explicit regularization.Confused categories can produce less reliable context and less correct predictions.
- Context Prior models same-category and different-category pixel relationships as prior knowledge to improve feature robustness and reduce the search space of possible categories.
- Affinity Loss supervises a Context Prior Layer that selectively aggregates intra-class context with the prior map and inter-class context with its reversed prior.The layer uses spatial aggregation to reason about contextual relationships.
- CPNet combines a backbone network with the Context Prior Layer, whose supervised prior map refines predictions through intra-class and inter-class context.
2. Related Work
Prior scene-segmentation methods aggregate contextual information through pyramid or attention mechanisms, but often lack explicit distinction between contextual relationships. The paper introduces supervised Context Prior learning to regularize intra-class and inter-class aggregation.
- Context Aggregation: Pyramid pooling and related methods partition feature maps into regions or scales and aggregate regional context.
- Attention Mechanism: Existing attention mechanisms may capture undesirable contextual dependencies because they do not explicitly distinguish different contextual relationships.
- Attention Mechanism: Attention-based methods use channel, spatial, or point-wise attention to selectively capture heterogeneous and long-range contextual information.
- Context Prior: The Context Prior Layer aggregates intra-class and inter-class context from the learned prior map and its reverse to support per-pixel prediction.
- Context Prior: The proposed Context Prior models relationships between same-category and different-category pixels using a Context Prior Layer supervised by Affinity Loss.
3. Context Prior
The Context Prior separates intra-class and inter-class contextual relationships under explicit Affinity Loss supervision. CPNet uses this prior with efficient spatial aggregation to refine scene segmentation features.
- Context Prior: Context Prior models same-category and different-category pixel relationships separately instead of mixing contextual dependencies.The prior and its reversed form target intra-class and inter-class context, respectively.
- Affinity Loss: The Ideal Affinity Map is constructed by one-hot encoding downsampled ground truth and computing A = b_Lb_L^T.The resulting N × N map indicates whether pixel pairs belong to the same category.
- Affinity Loss: Affinity Loss combines unary binary cross-entropy with a global term that jointly encodes intra-class and inter-class relationships.The global term uses row-level predictive value, true intra-class rate, and true inter-class rate.
- Context Prior Layer: The Context Prior Layer learns a prior map, aggregates intra-class features with P, inter-class features with 1 − P, and concatenates both with the original feature.This produces a representation containing both contextual types for prediction.
- Aggregation Module: The Aggregation Module uses asymmetric fully separable convolutions to preserve the standard convolution’s receptive field with less computation.Spatial factorization and depth-wise convolution reduce computation while retaining receptive-field size.
- Network Architecture: CPNet combines a dilated convolutional backbone, Context Prior Layer, segmentation supervision, auxiliary loss, and Affinity Loss.The network is a fully convolutional architecture whose Context Prior Layer captures intra-class and inter-class context.
4. Experimental Results
Experiments evaluate CPNet on three scene-segmentation benchmarks and through ADE20K ablations, generalization tests, and visualizations. Results show gains from Context Prior, Affinity Loss, suitable local spatial information, and multi-scale evaluation.
- Experimental setup: CPNet is evaluated on ADE20K, PASCAL-Context, and Cityscapes using standard segmentation metrics and multi-scale or flipped inference.The experiments compare the proposed model with state-of-the-art methods across three challenging datasets.
- ADE20K ablations: The ADE20K baseline improves from 34.38% to 36.24% mIoU with auxiliary loss, establishing the model used for subsequent comparisons.The corresponding pixAcc increases from 76.51% to 77.37%.
- ADE20K ablations: 43.92% mIoU and 80.77% pixAcc result from combining IntraPrior, InterPrior, and Affinity Loss; multi-scale and flipped testing raises performance to 44.46% mIoU and 81.38% pixAcc.With ResNet-101 and the testing strategy, CPNet reaches 46.27% mIoU and 81.85% pixAcc.
- ADE20K ablations: With filter size 11, Context Prior reaches 43.92% mIoU and a 2.41% relative gain; larger filters reduce both performance and improvement.The results indicate that Context Prior requires appropriate local spatial information to reason about relationships.
- Comparison with state-of-the-art: CPNet achieves 46.27% mIoU and 81.85% pixAcc on ADE20K, 53.9% mIoU on PASCAL-Context, and 81.3% mIoU on the Cityscapes test set.The reported comparisons place the model favorably against recent state-of-the-art methods on each benchmark.
5. Concluding Remarks
The paper presents CPNet, combining a backbone with a Context Prior Layer that uses an Affinity Loss to distinguish contextual dependencies. Quantitative and qualitative comparisons show favorable performance against recent state-of-the-art scene segmentation approaches.
- CPNet combines a backbone network with a Context Prior Layer for scene segmentation.The Context Prior Layer embeds the proposed Context Prior into the network.
- Affinity Loss supervises the Context Prior to distinguish different contextual dependencies.The loss supports modeling intra-class and inter-class relationships.
- The Aggregation Module aggregates spatial information to reason about contextual relationships.It is embedded into the Context Prior Layer.
- CPNet performs favorably against recent state-of-the-art scene segmentation approaches.The comparison includes extensive quantitative and qualitative evaluation.