Source-linked AI summary
Bootstrapping Semantic Segmentation with Regional Contrast
Shikun Liu, Shuaifeng Zhi, Edward Johns, Andrew J. Davison
TL;DR
Semantic segmentation depends on expensive pixel-level labels and often produces blurry boundaries or confusions among related classes. ReCo applies regional pixel-level contrastive learning with active sampling of hard queries and adaptive negatives, improving supervised and semi-supervised segmentation, especially with very few labels.
Problem
Pixel-level annotation is expensive, while segmentation models can produce blurry contours and confuse rare objects with related classes.
Method
ReCo is a regional pixel-level contrastive framework that pulls queries toward class means, pushes them from other classes, and actively samples low-confidence queries with adaptive negative keys.
Results
ReCo consistently improves supervised and semi-supervised segmentation across methods and datasets, with especially strong gains in few-label settings and up to 5 −10% relative improvement in the fewest-label full-label setting.
Takeaways & Limitations
ReCo enables high-accuracy segmentation with very few human annotations while producing sharper, more accurate object boundaries.
Abstract
from arXiv · showhide
We present ReCo, a contrastive learning framework designed at a regional level to assist learning in semantic segmentation. ReCo performs semi-supervised or supervised pixel-level contrastive learning on a sparse set of hard negative pixels, with minimal additional memory footprint. ReCo is easy to implement, being built on top of off-the-shelf segmentation networks, and consistently improves performance in both semi-supervised and supervised semantic segmentation methods, achieving smoother segmentation boundaries and faster convergence. The strongest effect is in semi-supervised learning with very few labels. With ReCo, we achieve high-quality semantic segmentation models, requiring only 5 examples of each semantic class. Code is available at https://github.com/lorenmt/reco.
1 Introduction
Semantic segmentation needs costly pixel-level annotations, while models tend to blur boundaries and confuse rare objects with a few related classes. ReCo addresses these issues with regional contrastive learning that actively samples difficult pixels and class-specific negatives.
- Pixel-level annotation is a practical bottleneck because labeling one CityScapes image can take more than 90 minutes.
- Negative sampling adapts to query-class relationships, such as sampling more bicycle representations when learning the person class.
- Smooth-labeling bias produces blurry contours and frequent mislabeling of rare objects, often between a small number of related classes.
- ReCo brings pixel representations toward their class mean while pushing them away from negative representations sampled from other classes.
- ReCo samples fewer than 5% of available pixels, selecting low-confidence queries and dynamically related negative keys to focus supervision on confusing pixels with minimal memory.
- Across supervised and semi-supervised settings, ReCo consistently improves performance, producing sharper boundaries and more accurate predictions with few annotations.
2 Related Work
Prior segmentation research has improved architectures, optimization, pseudo-labeling, augmentation, and contrastive pre-training, but dense contrastive methods can require substantial memory. ReCo extends model-agnostic optimization with active sampling for hard-pixel supervision using much less memory.
- Semantic Segmentation: Semantic segmentation research has advanced through convolutional architectures, dilated convolutions, encoder-decoder designs, and skip connections.
- Semantic Segmentation: Other optimization strategies address class imbalance or refine uncertain high-frequency pixels through specialized loss functions and point-wise rendering.
- Semi-supervised Classification and Segmentation: Semi-supervised segmentation commonly uses consistency, entropy minimization, adversarial learning, class activation maps, pseudo-labeling, and augmentation-based consistency.
- Contrastive Learning: Dense contrastive representations provide pixel-level correspondence and additional supervision for fine-grained prediction tasks.
- Contrastive Learning for Semantic Segmentation: Earlier contrastive segmentation approaches could require huge memory, whereas ReCo uses active sampling to perform contrastive learning with much less memory.
3 Regional Contrast (ReCo)
ReCo adds pixel-level contrastive learning to segmentation by sampling sparse, difficult queries and semantically related negative keys. It supports supervised and semi-supervised training through an auxiliary representation head and corresponding loss terms.
- Pixel-Level Contrastive Learning: ReCo applies pixel-level contrastive learning across semantic classes using dense representations, with class-specific query sets, negative keys, and a positive key equal to the class mean.The contrastive loss measures query-key similarity with a normalized dot product and uses a temperature parameter.
- Active Hard Sampling: ReCo samples fewer than 5% of available pixels, reducing the memory and computation required for high-resolution pixel-level contrastive learning.The method uses sparse queries and keys rather than processing every pixel.
- Active Hard Sampling: ReCo samples negative keys according to a dynamically updated pairwise class relationship graph, allocating more samples to confusing negative classes.The graph is based on normalized dot products between class-mean representations, followed by SoftMax normalization.
- Active Hard Sampling: ReCo samples hard queries whose predicted confidence is below a user-defined threshold, focusing supervision on less certain pixels rather than common easy pixels.The confidence threshold δs determines whether a query is treated as easy or hard; Figure 2 illustrates this selection on Cityscapes.
- Learning with ReCo: In supervised training, ReCo combines supervised cross-entropy with its contrastive loss; in semi-supervised training, it also uses pseudo-label supervision from a moving-average teacher.The semi-supervised framework applies ReCo to pixels from labelled and unlabelled images, while confidence filtering limits unreliable pseudo-labels.
4 Experiments
ReCo is evaluated across supervised and semi-supervised segmentation settings, including benchmarks with full labels and partial pixel labels. Across datasets, baselines, and sampling conditions, ReCo improves performance and produces sharper boundaries, while active sampling and sparse queries provide efficiency advantages.
- Experiment Setup: ReCo evaluates two semi-supervised settings: Partial Dataset Full Labels and Partial Labels Full Dataset, alongside supervised segmentation.The first uses a small subset of fully labelled images; the second provides sparse labels across all images and tests semantic class completion.
- Experiment Setup: ReCo is tested on Cityscapes and Pascal VOC for partial labels, and additionally on SUN RGB-D for full labels.The partial-label setting uses 1, 1%, 5%, and 25% labelled pixels per class per image.
- Full-Label Results: Across three datasets, ReCo improves mean IoU in both semi-supervised and supervised settings, with up to 5−10% relative improvement in the fewest-label cases.Table 1 reports means and ranges over three independent runs.
- Full-Label Results: On existing Pascal VOC benchmarks, ReCo reaches PseudoSeg’s performance while requiring only half the labelled data.The comparison uses the PseudoSeg data partition and training strategy, with all baselines re-implemented.
- Full-Label Results: With few labelled images, ReCo produces sharper boundaries for small objects and ambiguous classes than supervised and ClassMix baselines.Visualisations use 20 labelled Cityscapes images and 50 labelled SUN RGB-D images; reported examples include people, bicycles, lamps, and pillows.
- Partial-Label Results: In the partial-label setting, ReCo improves ClassMix by around 1−5% relatively, although very sparse ground-truth annotations can provide inaccurate supervision.Qualitative Pascal VOC results trained with 1 labelled pixel per class per image show smoother and more accurate boundaries.
- Ablative Analysis: Using 32 queries per class, less than 0.5% of available pixels, still yields a non-trivial improvement, while ReCo can use 50× less memory than a concurrent method.Increasing queries and keys helps until improvements become marginal; hard-query sampling is substantially more effective than easy-query sampling or random sampling.
- Ablative Analysis: ReCo surpasses the ClassMix baseline with only 10% unlabelled data and maintains similar gains across varied semi-supervised methods.Batch-wise sampling achieves 49.86 mIoU versus 49.34 mIoU for a feature-bank variant, while training is faster.
5 Visualisations and Interpretability of Class Relationships
The visualisations compare semantic class relationships learned with and without ReCo, showing feature disentanglement and interpretable groupings of semantically similar or ambiguous classes.
- Visualisation setup: ReCo compares semantic class relationship graphs and dendrograms using pixel embeddings averaged across validation-set classes.The analysis examines embeddings Z and R, where Z supports pixel classification and R is used for the ReCo loss and active sampling.
- CityScapes relationships: ReCo helps disentangle semantic features compared with supervised learning, where many class pairs remain similar.The ReCo dendrogram based on embedding Z also forms a more structured semantic tree grouping semantically similar classes.
- CityScapes relationships: The CityScapes dendrogram learned with ReCo groups semantically similar classes into a clearer and more interpretable hierarchy.
- SUN RGB-D relationships: The SUN RGB-D dendrogram identifies ambiguous pairs including night stand–dresser, table–desk, and floor–floormat.These relationships are consistent with the corresponding results shown in Fig. 5.
- Interpretability: The pair-wise relation graph clarifies how semantic class structure relates to incorrect predictions by exposing confusing class relationships.
6 Conclusion
ReCo is a pixel-level contrastive framework with active sampling for semantic segmentation. It improves supervised and semi-supervised methods with minimal additional memory, with its strongest effect when very few labels are available.
- ReCo improves supervised and semi-supervised semantic segmentation methods with minimal additional memory footprint.
- ReCo has its strongest effect in semi-supervised learning with very few labels, where it improves on the previous state of the art by a large margin.
- The framework uses pixel-level contrastive learning with active sampling and is designed specifically for semantic segmentation.
A Implementation Details
The implementation uses fixed optimisation, preprocessing, augmentation, and ReCo sampling settings across the evaluated datasets, with dataset-specific crop and batch configurations.
- Optimisation: All methods use SGD with learning rate 2.5 × 10^-3, momentum 0.9, weight decay 5 · 10^-4, polynomial annealing, and 40k iterations.The annealing power is 0.9.
- Dataset preprocessing: CityScapes images are downsampled to 512 × 1024, cropped to 512 × 512, and trained with batch size 2.
- Dataset preprocessing: Pascal VOC uses 321 × 321 crops, random scaling from 0.5 to 1.5, and batch size 10.
- Dataset preprocessing: SUN RGB-D uses 384 × 512 images, 321 × 321 crops, random scaling from 0.5 to 1.5, and batch size 5.Its training and validation split is reorganised to 9860 and 475 samples, respectively.
- Augmentation: All datasets use Gaussian blur, colour jittering, and random horizontal flipping as additional augmentation.
- ReCo settings: ReCo samples 256 queries and 512 keys per mini-batch with temperature τ = 0.5, representation dimension m = 256, and confidence thresholds δw = 0.7 and δs = 0.97.
B Results on Semi-supervised Segmentation Benchmarks
On semi-supervised Pascal VOC and CityScapes benchmarks, ReCo is evaluated with ClassMix across two DeepLab backbones and achieves the best performance in most settings.
- Benchmark design: ReCo is evaluated on semi-supervised CityScapes and Pascal VOC benchmarks relative to corresponding fully supervised performance gaps.
- Results: ReCo applied to ClassMix achieves the best performances in most cases across both datasets.
- Experimental coverage: The benchmark evaluates ReCo with both DeepLabv2 and DeepLabv3+ backbones.
- Results: The results indicate robustness to different backbone architectures and numbers of labelled training images.
C Visualisation on Pascal VOC (Trained with 60 Labelled Images)
ReCo + ClassMix predicts rare classes more accurately than Supervised and ClassMix in the full-label setting.
- ReCo + ClassMix predicts rare classes such as boat, bottle, and table accurately, whereas Supervised and ClassMix often misclassify them completely.
D Visualisation on CityScapes (Trained with 1% Labelled Pixel)
With 1% labelled pixels in CityScapes, ReCo's gains are less pronounced than in the full-label setting and mainly affect small object boundaries.
- ReCo's performance improvements are less pronounced in CityScapes with partial labels than with full labels.
- The improvements typically come from more accurate predictions around small object boundaries, including traffic lights and traffic signs.
- Learning semantics from partial labels with minimal boundary information remains an open research question with substantial room for improvement.
E Visualisation on Semantic Class Relationship from Pascal VOC (top) and SUN RGB-D (bottom)
Across the evaluated datasets, ReCo produces more disentangled features than the Supervised baseline, supporting better decision boundaries.
- ReCo features are more disentangled than those of the Supervised baseline across all datasets.
- The learned representations help the segmentation model learn a better decision boundary.