Source-linked AI summary
Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank
Inigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, Ana C. Murillo
TL;DR
Semi-supervised semantic segmentation must learn with limited pixel-level labels while using unlabeled data effectively. The paper introduces a teacher-student, positive-only pixel-level contrastive module with a quality-filtered class-wise memory bank, and reports state-of-the-art results across semi-supervised segmentation and domain adaptation benchmarks, especially with less labeled data.
Problem
Supervised semantic segmentation requires expensive per-pixel annotations, while existing semi-supervised methods do not enforce dataset-wide structure that improves inter-class feature separability.
Method
A teacher updates a class-wise memory bank with high-quality labeled pixel features, and a student learns similar class-wise representations from labeled and unlabeled data using positive-only contrastive learning.
Results
The approach reaches state-of-the-art results on semi-supervised semantic segmentation and semi-supervised domain adaptation benchmarks, with larger improvements when less labeled data is available.
Takeaways & Limitations
Pixel-level contrastive learning with selected labeled features can boost semi-supervised segmentation and naturally extend to semi-supervised domain adaptation.
Takeaways & Limitations
The method is restricted by the number of semantic classes and the number of memory-bank entries stored per class.
Abstract
from arXiv · showhide
This work presents a novel approach for semi-supervised semantic segmentation. The key element of this approach is our contrastive learning module that enforces the segmentation network to yield similar pixel-level feature representations for same-class samples across the whole dataset. To achieve this, we maintain a memory bank continuously updated with relevant and high-quality feature vectors from labeled data. In an end-to-end training, the features from both labeled and unlabeled data are optimized to be similar to same-class samples from the memory bank. Our approach outperforms the current state-of-the-art for semi-supervised semantic segmentation and semi-supervised domain adaptation on well-known public benchmarks, with larger improvements on the most challenging scenarios, i.e., less available labeled data. https://github.com/Shathe/SemiSeg-Contrastive
1. Introduction
The paper addresses expensive pixel-level annotation by introducing a teacher-student semi-supervised segmentation framework that learns class-separable pixel features from labeled and unlabeled data. Its memory-bank contrastive module uses selected labeled features to guide feature learning and achieves state-of-the-art results across several settings.
- Motivation: Pixel-level semantic segmentation is important for scene understanding but requires expensive per-pixel annotations in supervised training.The annotation burden is a bottleneck for generalization when large labeled datasets are needed.
- Motivation: Semi-supervised learning uses a small labeled subset and extracts knowledge from unlabeled samples to address limited annotation availability.
- Research gap: Previous semi-supervised segmentation methods mainly use entropy minimization or consistency regularization without enforcing dataset-wide feature structure for inter-class separability.The authors hypothesize that addressing this limitation can improve feature learning, especially with little labeled data.
- Approach: The proposed teacher-student framework stores selected labeled features in a memory bank and trains the student to produce similar class-wise features from labeled and unlabeled data.The teacher generates feature candidates, while feature quality and learned relevance determine which vectors are retained and weighted.
- Contributions: The framework combines high-quality labeled pixel features, relevance-weighted positive-only contrastive learning, and alignment between labeled and unlabeled feature representations.
- Results: The method reaches state-of-the-art results on semi-supervised segmentation and domain adaptation benchmarks, with larger improvements as the percentage of unlabeled data increases.
2. Related Work
Prior work applies semi-supervised learning through pseudo-labeling, entropy minimization, consistency regularization, and teacher-student training, while contrastive learning has mostly been used for pre-training or other settings. This paper extends positive-only contrastive learning to semi-supervised segmentation by aligning class-wise pixel features through a quality-filtered memory bank.
- Semi-Supervised Learning: Related semi-supervised methods include pseudo-labeling, direct entropy minimization, and consistency regularization for learning from unlabeled data.
- Semi-Supervised Learning: Mean Teacher uses an exponential moving average of model parameters to produce more robust predictions for teacher-student consistency training.
- Semi-Supervised Learning: Other segmentation approaches use GAN-based confidence-map discrimination, two-branch training, or dynamic mutual training to exploit unlabeled predictions.
- Contrastive Learning: Contrastive learning traditionally attracts positive pairs and repels negative pairs, while newer methods use positive pairs only through similarity maximization with distillation or redundancy reduction.
- Contrastive Learning: In semantic segmentation, contrastive learning has mainly supported pre-training, with recent work also applying pixel-, region-, or saliency-based comparisons.
- Positioning: This work applies positive-only contrastive learning to align class-wise per-pixel features from labeled and unlabeled data with quality-selected labeled features stored in a memory bank.Unlike prior image-level memory banks, the proposed bank stores per-pixel features for semantic classes.
3. Method
The method combines teacher-generated pseudo-labels and supervised segmentation with pixel-level positive-only contrastive learning. A class-wise memory bank stores selected labeled features, while student features from labeled and unlabeled data are weighted and aligned with same-class memory entries.
- Teacher-student scheme: A teacher-student scheme uses labeled samples for supervised learning and unlabeled samples with teacher-generated pseudo-labels.The teacher is updated as an exponential moving average of the student, providing pseudo-labels for unlabeled inputs.
- Training objective: The overall objective combines supervised, pseudo-label, entropy-minimization, and positive-only contrastive losses.The supervised term uses weighted cross-entropy on labeled data, while pseudo-labels and entropy minimization regularize predictions on unlabeled data.
- Contrastive optimization: Student features are projected and predicted, grouped by semantic class, and assigned learned per-feature weights before contrastive alignment.Class-specific self-attention modules produce weights for student prediction vectors and teacher memory-bank projections; the weights are L1-normalized.
- Contrastive optimization: The contrastive loss uses cosine similarity to enforce similarity between weighted student predictions and same-class memory-bank projections.The memory-bank projections serve as the target vectors for the positive-only contrastive objective.
- Memory bank: A class-wise memory bank stores high-quality teacher projection vectors from labeled examples for contrastive optimization.Features are filtered by prediction confidence, grouped by class, ranked with class-specific attention, and updated with top-scoring vectors in a FIFO queue.
4. Experiments
The experiments evaluate the method across semi-supervised segmentation, domain adaptation, and ablations. Results show stronger gains when fewer labels are available, while component studies identify the contrastive module and quality-selected prototypes as important.
- Evaluation setup: The evaluation covers Cityscapes, Pascal VOC, and GTA5, including semi-supervised segmentation and semi-supervised domain adaptation.Cityscapes, Pascal VOC, and GTA5 provide urban, natural-scene, and synthetic urban-like benchmarks.
- Implementation details: The experiments use DeepLabv2 with ResNet101 for most benchmarks and ablations, with DeepLabv3+ and ResNet50 also evaluated.All experiments use 150K SGD iterations, while augmentation strength differs between labeled and unlabeled samples.
- Implementation details: The memory bank contains 256 vectors per class, and features are accepted using a confidence threshold of 0.95.The number of vectors added per image and class is max(1, ψ |X_l|).
- Semi-supervised semantic segmentation: Our approach outperforms the current state-of-the-art on Cityscapes, with increasing performance differences as less labeled data is available.The comparison includes labeled-unlabeled ratios of 1/30, 1/8, and 1/4, alongside a fully supervised reference.
- Semi-supervised semantic segmentation: On Pascal VOC, the method outperforms previous methods in most configurations and provides larger benefits when only a small fraction of data is labeled.The evaluated labeled-unlabeled ratios are 1/50, 1/20, and 1/8.
- Semi-supervised domain adaptation: In semi-supervised domain adaptation from GTA5 to Cityscapes, the method outperforms ASS across all evaluated setups, with larger improvements under lower label availability.The comparison uses GTA5 as labeled source data and Cityscapes as the target domain.
- Ablation experiments: The four losses are complementary, producing a 10 mIoU increase over the supervised-only baseline in the reported 1/30-label setting.The ablation also identifies generating and selecting high-quality prototypes as the most important factor.
- Ablation experiments: The contrastive module reaches 54.32 mIoU on Cityscapes with 1/30 labeled data even without the other complementary losses.Adding pseudo-labeling and entropy regularization produces further gains.
5. Conclusion
The proposed pixel-level contrastive learning module improves semi-supervised semantic segmentation and semi-supervised domain adaptation by aligning class-wise features through a quality- and relevance-filtered memory bank.
- The memory bank is continuously updated with selected teacher-produced features from labeled data.Selection is based on feature quality and relevance for contrastive learning.
- The student network learns similar class-wise features from both labeled and unlabeled data.
- Pixel-level positive-only contrastive learning boosts performance in semi-supervised semantic segmentation and semi-supervised domain adaptation.
- The approach outperforms state-of-the-art methods on several public benchmarks.
- Improvements are particularly significant when the amount of available labeled data is low.