Source-linked AI summary
Exploring Cross-Image Pixel Contrast for Semantic Segmentation
Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, Luc Van Gool
TL;DR
Semantic segmentation models typically exploit local within-image context while neglecting semantic relations among labeled pixels across images. The paper introduces supervised pixel-wise contrastive learning, including region memory and informative sampling, and reports consistent gains across datasets and architectures without extra inference burden.
Problem
Existing segmentation methods use local pixel dependencies but neglect global semantic relations among labeled pixels across different images.
Method
The paper combines pixel-wise cross-entropy with supervised pixel-wise contrastive learning, region memory, and informative sampling to structure pixel embeddings.
Results
The method consistently improves intersection-over-union scores across Cityscapes, PASCAL-Context, COCO-Stuff, and CamVid with several segmentation architectures and backbones.
Takeaways & Limitations
The results support exploring global pixel relationships and metric learning as part of fully supervised dense image prediction training.
Abstract
from arXiv · showhide
Current semantic segmentation methods focus only on mining "local" context, i.e., dependencies between pixels within individual images, by context-aggregation modules (e.g., dilated convolution, neural attention) or structure-aware optimization criteria (e.g., IoU-like loss). However, they ignore "global" context of the training data, i.e., rich semantic relations between pixels across different images. Inspired by the recent advance in unsupervised contrastive representation learning, we propose a pixel-wise contrastive framework for semantic segmentation in the fully supervised setting. The core idea is to enforce pixel embeddings belonging to a same semantic class to be more similar than embeddings from different classes. It raises a pixel-wise metric learning paradigm for semantic segmentation, by explicitly exploring the structures of labeled pixels, which were rarely explored before. Our method can be effortlessly incorporated into existing segmentation frameworks without extra overhead during testing. We experimentally show that, with famous segmentation models (i.e., DeepLabV3, HRNet, OCR) and backbones (i.e., ResNet, HR-Net), our method brings consistent performance improvements across diverse datasets (i.e., Cityscapes, PASCAL-Context, COCO-Stuff, CamVid). We expect this work will encourage our community to rethink the current de facto training paradigm in fully supervised semantic segmentation.
1. Introduction
The paper identifies a gap in segmentation training: models exploit within-image context but neglect semantic relations among labeled pixels across images. It proposes supervised pixel-wise contrastive learning, region memory, and informative sampling, reporting consistent improvements without inference changes.
- Motivation: Existing segmentation models primarily use local pixel dependencies within individual images, leaving global semantic relations across training images underexplored.A well-structured embedding space should support both pixel categorization and intra-class compactness with inter-class dispersion.
- Proposed paradigm: The proposed supervised pixel-wise contrastive loss pulls same-class pixel embeddings together and pushes different-class embeddings apart.It complements pixel-wise cross entropy by shaping the embedding space using labeled pixel structures.
- Proposed paradigm: A region memory stores pooled features from same-label regions, enabling pixel-to-region contrast alongside pixel-to-pixel contrast.This design targets representative samples in the large, structured pixel-training space.
- Training design: Informative example, negative, positive, and anchor sampling is proposed so the model focuses more on segmentation-hard pixels than random pixel sampling.The introduction emphasizes the importance of mining informative samples in dense metric learning.
- Results: The method integrates into existing networks without base-model changes or extra inference burden, while improving intersection-over-union scores across four datasets and multiple architectures.Reported settings include Cityscapes, PASCAL-Context, COCO-Stuff, CamVid, DeepLabV3, HRNet, OCR, ResNet, and HR-Net.
2. Related Work
Prior segmentation and metric-learning methods largely model local or within-image structure, whereas this work targets cross-image semantic relations through supervised pixel-wise contrast.
- Semantic segmentation: Semantic segmentation methods progressed from FCN but commonly addressed limited visual context through receptive fields and later context-aggregation or structure-aware objectives.The related work frames these approaches as improvements to within-image modeling and pixel-structure optimization.
- Semantic segmentation: Earlier approaches generally neglected semantic correlations among labeled pixels across different training images.The paper contrasts local pixel dependencies with global labeled-data context.
- Contrastive learning: The proposed method uses supervised pixel-to-pixel contrast as a training protocol that regularizes the segmentation embedding space with global pixel relations.Unlike concurrent dense contrastive methods described here, it is presented as a fully supervised segmentation-training approach rather than pre-training.
- Metric learning: Prior metric-learning methods focused on object or instance content, while the paper emphasizes inter-image pixel similarity without extra inference cost or base-network modification.The paper distinguishes its compact combination of cross-entropy and pixel-to-pixel contrast from clustering-based alternatives requiring extra labeled data at inference.
3. Methodology
The methodology introduces contrastive learning and memory-bank concepts before applying them to dense semantic segmentation, where pixels are represented and compared in an embedding space.
- Method setup: The method first reviews unsupervised contrastive learning and memory banks before detailing its supervised segmentation algorithm.This establishes the conceptual basis for the later pixel-level formulation.
- Contrastive learning: Unsupervised visual contrastive learning trains an encoder to map each image to a feature vector while distinguishing an augmented positive from dissimilar negative images.The formulation is described as learning to compare an anchor with positive and negative samples in an embedding space.
P LCE (Eq. 2)
The paper supplements pixel-wise cross-entropy with supervised pixel contrast, using global semantic relations to structure segmentation embeddings. Its region memory and sampling strategies address dense pixel data while emphasizing informative samples and hard pixels.
- P LCE (Eq. 2): Pixel-wise cross-entropy classifies each image pixel from dense encoder features and a segmentation head, but does not directly supervise the learned representations.The loss depends on logits produced from pixel embeddings and softmax scores.
- P LCE (Eq. 2): The supervised contrastive loss uses pixels as training samples, pulling same-class embeddings together and pushing different-class embeddings apart, including across images.Positive and negative samples are not restricted to the same image.
- P LCE (Eq. 2): The combined objective makes cross-entropy and contrastive learning complementary: classification remains discriminative while the embedding space gains intra-class compactness and inter-class separability.The paper reports that embeddings learned with LSEG become more compact and well separated.
- P LCE (Eq. 2): A region memory stores pooled features for semantic regions rather than individual pixels, reducing redundancy and enabling pixel-to-region contrast.Each region feature is obtained by average pooling embeddings of same-class pixels from one image; the memory size is |C|×N×D.
- P LCE (Eq. 2): Sampling strategies mine informative positives, negatives, and anchors, with semi-hard example sampling outperforming hardest-example sampling and hard-anchor sampling further improving segmentation performance.The method focuses contrastive learning on pixels that are difficult for classification.
- P LCE (Eq. 2): The architecture combines an FCN encoder, segmentation head, and project head, and supports ResNet or HRNet backbones with DeepLabV3, HRNet, or OCR heads.The project head maps high-dimensional pixel embeddings into a 256-dimensional representation.
4. Experiment
Experiments evaluate the method across four datasets, multiple segmentation architectures, and varied contrastive designs. Cross-image contrast and memory components improve segmentation results, including gains over strong baselines and more accurate qualitative segments.
- Experimental Setup: Experiments cover Cityscapes, PASCAL-Context, COCO-Stuff, and CamVid using DeepLabV3, HRNet, and OCR with ResNet or HRNet backbones.Cityscapes uses 5,000 finely annotated images; PASCAL-Context has 59 categories; COCO-Stuff contains 10,000 images and 80 object classes.
- Ablation Studies: Using both pixel-to-pixel and pixel-to-region memories achieves 81.0%, supporting the memory design and the need for both contrast types.The ablation reports that more pixel samples improve embeddings and that both relation types provide informative cues.
- Comparison to State-of-the-Arts: The method improves strong baselines across datasets, including HRNetV2 from 54.0 to 55.1 and OCR from 56.2 to 57.2 mIoU on PASCAL-Context.On COCO-Stuff, OCR+Ours reaches 41.0% mIoU, a 0.5% gain over OCR, while HRNetV2+Ours gains 0.6%.
- Comparison to State-of-the-Arts: On CamVid, the method improves both HRNetV2 and OCR, while qualitative comparisons show more accurate segments across challenging examples.The visual comparisons cover Cityscapes, PASCAL-Context, and COCO-Stuff.
5. Conclusion and Discussion
The paper presents supervised pixel-wise contrastive learning as a training paradigm that combines unary classification with structured metric learning. It reports improved segmentation performance while identifying several directions for future work.
- Conclusion: The proposed paradigm combines unary classification with structured metric learning through pixel-wise contrastive learning.It explores global semantic relations between training pixels to guide cross-image category-discriminative embeddings.
- Discussion: The method improves segmentation performance and may extend to dense prediction tasks such as pose estimation and medical image segmentation.The paper describes these applications as showing potential rather than reporting experiments for them.
- Discussion: The paper identifies smart sampling, metric-loss design, class rebalancing, and multi-layer feature contrast as challenges for future research.
A. Comparison to Other Losses
The contrastive loss is compared with standard and structure-aware segmentation losses under a common HRNetV2 training setup. It performs best and captures global dataset-level relations that existing losses define within individual images.
- Loss Comparison: On Cityscapes, the contrastive loss outperforms Lovász loss by 0.7% and RMI and AAF losses by 1.2% and 2.3%, respectively.All structure-aware losses outperform standard Cross-Entropy loss in the reported comparison.
- Loss Comparison: Existing losses model unary, adjacent-pixel, region-level, or pixel-clique relations within individual images, whereas the contrastive loss computes pairwise relations across the training dataset.The paper characterizes this distinction as local versus global context and pairwise versus higher-order structure.
B. Additional Quantitative Result
Additional Cityscapes validation results show consistent gains from cross-image pixel contrast across baseline models, with negligible training-speed impact and no extra inference overhead.
- Quantitative Results: Cross-image pixel contrast improves DeepLabV3, HRNetV2, and OCR by 1.2, 1.1, and 0.8 mIoU points, respectively.The models are trained on Cityscapes train for 80,000 iterations with mini-batch size 8.
- Efficiency: The contrastive loss causes negligible training-speed decrease and adds no inference overhead.
C. Additional Qualitative Result
Additional qualitative comparisons show HRNetV2+Ours improving segmentation on difficult regions across four benchmarks, including small objects and cluttered backgrounds.
- Qualitative comparisons: HRNetV2+Ours produces improved segmentation results over HRNetV2 across Cityscapes, PASCAL-Context, COCO-Stuff, and CamVid.The improved regions are marked by dashed boxes in the supplementary figures.
- Qualitative comparisons: The qualitative improvements particularly affect hard regions such as small objects and cluttered backgrounds.