PaperlayerView exact revision on arXiv

Source-linked AI summary

Contextrast++: Robust Multi-Scale Contextual Contrastive Learning for Semantic Segmentation

Changki Sung, Hyungtae Lim, Wanhee Kim, Youngwoo Seo, Hyun Myung

arXiv:2608.22679v1cs.CVcs.RO

TL;DR

Semantic segmentation needs better local-global context modeling and handling of long-tailed classes without increasing inference cost. Contextrast++ combines contextual contrastive learning with adaptive fusion, PA and AA losses, a class-balanced memory bank, and BANE sampling. Experiments on public datasets show improved performance over prior contrastive learning methods while adding no inference overhead.

  • Problem

    Semantic segmentation remains challenged by local-global context integration, inaccurate boundaries and small objects, and long-tailed class distributions under inference-cost constraints.

  • Method

    Contextrast++ uses adaptive multi-scale anchor fusion, pixel-to-anchor and anchor-to-anchor losses, a class-balanced memory bank, and boundary-aware negative sampling.

  • Results

    Contextrast++ outperforms prior contrastive learning methods on the majority of configurations across five public datasets and CNN and transformer backbones.

  • Takeaways & Limitations

    The framework improves semantic and fine-grained feature learning while retaining unchanged inference-time computational requirements.

Abstract

from arXiv · show

Semantic segmentation has rapidly advanced with deep learning; however, challenges remain in effectively capturing local and global contexts as well as addressing the long-tailed distribution problem. To tackle these issues, we present Contextrast++, a robust contrastive learning method for semantic segmentation that improves multi-scale feature integration and mitigates class imbalance issues. Our method consists of two key components: 1) contextual contrastive learning (CCL) and 2) boundary-aware negative (BANE) sampling. CCL includes three subcomponents: adaptive fusion module, pixel-to-anchor (PA) loss, and anchor-to-anchor (AA) loss. The adaptive fusion module dynamically balances local and global feature integration, resulting in a more context-aware representation. While the PA loss leverages the fused multi-scale features to improve feature representation learning, the AA loss focuses on addressing the long-tailed distribution problem by utilizing a memory bank that stores a fixed number of class-balanced representative anchors. Meanwhile, BANE sampling enhances segmentation precision by selecting hard negatives from misclassified boundary regions, which refines fine-grained details during contrastive learning. As verified in extensive experiments using public datasets, we demonstrate that Contextrast++ substantially improves semantic segmentation performance over existing contrastive learning-based state-of-the-art approaches, while introducing no additional computational overhead during inference.

I. INTRODUCTION

Semantic segmentation still struggles with small objects, boundaries, and underrepresented classes, while larger models increase inference cost. Contextrast++ addresses these issues with adaptive multi-scale contrastive learning, class-balanced anchors, and training-only overhead.

  • Semantic segmentation models remain inaccurate on small objects, object boundaries, and underrepresented classes.
  • Increasing model complexity can mitigate these challenges but raises memory consumption and slows inference.
  • Contrastive learning improves these targets during training while preserving identical inference memory consumption and processing time.
  • Contextrast++ adaptively integrates multi-scale anchors using self-attention, concatenation, and an MLP to produce more expressive class-specific representations.
  • Its AA loss uses a memory bank of class-balanced representative anchors to mitigate long-tailed distribution problems.
  • Contextrast++ outperforms prior contrastive learning approaches across CNN and transformer architectures on diverse datasets.

B. Contrastive learning approaches in semantic segmentation

Contrastive segmentation methods have progressed from single-scale features toward multi-scale modeling, but separate cross-scale processes can create inconsistent feature relationships. Contextrast++ extends contextual contrastive learning with adaptive fusion and class-balanced memory-bank supervision.

  • Single-scale contrastive methods miss valuable multi-scale contextual information.
  • Separate multi-scale and cross-scale processes can cause feature shifts and misaligned relationships.
  • Contextrast used static multi-scale anchor fusion and BANE sampling, but its fusion lacked adaptability and mini-batch balancing only partially addressed long-tailed classes.
  • Contextrast++ replaces static fusion with learnable integration and maintains class-balanced anchors across all classes.
  • The framework strengthens encoder representations through supervised contrastive learning without inference-time overhead.
  • Its design combines adaptive multi-scale anchors, enhanced PA and AA losses, and BANE sampling for semantic segmentation.

B. Contextual contrastive learning (CCL)

CCL builds class-level representations from multi-scale encoder features, then adaptively fuses them to improve contextual contrastive learning. Its fusion module replaces fixed summation with self-attention, concatenation, and MLP-based nonlinear integration.

  • B. Contextual contrastive learning (CCL): CCL optimizes class representations in the embedding space using adaptive fusion, PA loss, and AA loss.These components jointly target multi-scale representation learning and class-distribution issues.
  • Embedding and representative anchors: Each encoder layer produces embeddings through a projection head, with deeper layers having progressively smaller spatial resolutions.The projection head maps F_i to Z_i, whose vectors lie in a d-dimensional embedding space.
  • Embedding and representative anchors: Representative anchors are class-wise means of embedded feature vectors aligned with downsampled ground-truth labels.Only classes present at a given layer contribute anchors at that layer.
  • Adaptive fusion module: The prior fusion method used fixed weights and element-wise summation, limiting adaptation and richer cross-scale interactions.The weights satisfy w_l+w_h = 1, while summation imposes a linear superposition of features.
  • Adaptive fusion module: Contextrast++ uses a two-stage refine-then-fuse module with self-attention, concatenation, and MLP to learn more expressive cross-scale anchor representations.Concatenation preserves scale-specific representations, and the lightweight design adds negligible computational overhead.

4) PA loss:

The PA loss reformulates contrastive learning around fused class anchors rather than switching individual samples between anchor roles. It pulls same-class samples toward their fused anchor and pushes different-class samples away, improving stability and class-level consistency.

  • PA loss: Conventional InfoNCE contrasts each sample against positive and negative samples using temperature-scaled dot-product similarities.Positive samples share the anchor’s semantic class, whereas negative samples belong to different classes.
  • PA loss: PA loss reformulates contrastive learning to avoid representation switching during training.The reformulation uses fused representative anchors as stable reference points.
  • PA loss: PA loss pulls positive samples toward the fused anchor and pushes negative samples away from it.The loss is weighted across encoder layers by λ_i and uses temperature-scaled dot-product similarities.
  • PA loss: This anchor-centered optimization improves training stability and class-level consistency.The stated benefit follows from eliminating representation switching.

5) AA loss with a memory bank:

The AA loss uses class-prioritized anchors and a memory bank to provide stable, class-balanced supervision for long-tailed semantic segmentation.

  • Downsampling can remove small-object classes from deeper layers, while storing all layers can overrepresent fine-grained features.These effects make naïve memory-bank designs structurally incompatible with multi-scale segmentation.
  • Class-prioritized anchors select each class’s representative from the deepest available layer where that class appears.The first layer defines the complete set of existing class IDs, while deeper layers provide anchors when available.
  • A memory bank maintains N class-wise queues, each storing up to KM anchors, and replaces the oldest anchor when capacity is exceeded.This design balances storage capacity and memory efficiency.
  • Equal sampling across classes supplies balanced anchors to the AA loss, whose similarities use dot products scaled by temperature τ.Representative anchors also avoid storing large numbers of individual class features.
  • The AA loss is jointly optimized with cross-entropy and the adaptive-fusion PA loss, complementing class prediction with global-local and long-tailed supervision.The PA loss learns relationships between global and local contexts, while AA addresses class imbalance.

6) Loss function:

Contextrast++ jointly optimizes standard cross-entropy with contextual contrastive losses and adds BANE sampling to emphasize difficult boundary negatives.

  • The framework jointly optimizes LCE, LPA, and LAA, with α controlling the weight of the contrastive learning loss.LPA uses adaptively fused representative anchors, while LAA addresses long-tailed distributions.
  • BANE sampling decomposes predictions into class-wise binary error maps, transforms them into distance maps, and selects negatives using those distances.The strategy follows three sequential steps during training.
  • In each error map, value one marks an incorrectly predicted pixel that serves as a negative sample, while zero marks other pixels.The error maps are defined per class and encoder layer.
  • Among misclassified pixels, smaller distance-map values identify locations nearer object boundaries.The distance is measured relative to class edge pixels using a Euclidean distance transform.
  • BANE selects embeddings from the lowest Ks percentile of distances as hard negatives, encouraging separation from incorrect-class anchors.Ks uniformly controls the selection ratio across encoder layers.

A. Experimental setup

The experiments evaluate Contextrast++ on five public, long-tailed segmentation datasets using common training infrastructure, mIoU, and boundary mIoU.

  • ADE20K, PASCAL-C, and COCO-Stuff exhibit more severe class imbalance because they contain more semantic categories and greater class diversity.
  • Training uses MMSegmentation, ImageNet-pretrained weights, and augmentation with color jittering, horizontal flipping, and random scaling.The reported settings include KM = 100 for the memory bank and Ks = 50% for BANE sampling.
  • Mean Intersection over Union (mIoU) is the primary metric, while boundary mIoU (B-mIoU) evaluates accuracy near object boundaries.B-mIoU uses a pixel threshold τB to adjust the distance from the boundary.
  • The evaluation compares Contextrast++ with single-scale methods Pico, Pico+, and Region, and multi-scale methods Multi and Contextrast.

B. Evaluation of semantic segmentation performance

Across datasets, backbones, class-frequency groups, boundaries, and object scales, Contextrast++ improves segmentation performance and qualitative detail over contrastive-learning baselines.

  • Long-tailed robustness: Contextrast++ achieves the highest accuracy across head, mid, and tail class groups, with larger gains for tail classes.Compared with Contextrast, tail-class IoU improves by +1.51%p.
  • Overall performance: Multi degraded OCRNet performance by −0.33%p, whereas Contextrast++ is evaluated as the proposed alternative within the comparative framework.
  • Qualitative analysis: Qualitative visualizations show improved recognition of small, occluded, cropped, and boundary-adjacent objects across Cityscapes, CamVid, ADE20K, and COCO-Stuff.The results also address under-segmentation and over-segmentation relative to other contrastive-learning approaches.
  • Boundaries and small objects: Contextrast++ improves boundary segmentation on ADE20K and COCO-Stuff, where complex class distributions make boundary precision especially relevant.Contextrast and Contextrast++ improve B-mIoU over Multi through BANE sampling.
  • Boundaries and small objects: Contextrast++ consistently outperforms baselines across tiny, small, medium, and large object groups, with notable gains at finer scales.The reported object groups are defined by pixel size, from up to 16 × 16 pixels to larger than 96×96 pixels.
  • Overall performance: Contextrast++ outperforms prior contrastive-learning approaches across CNN and transformer architectures on diverse datasets.The framework is reported as architecture-agnostic and generalizable.

C. Ablation studies

Ablations show that adaptive multi-scale fusion, class-balanced anchor memory, and boundary-aware negative sampling each improve segmentation, with their combination reaching the strongest reported mIoU. Additional analyses identify complementary scale cues, specialized nonlinear fusion, deepest-layer anchors, and moderated memory and negative-sample sizes as important design choices.

  • Component effectiveness: 82.87% mIoU is achieved when all proposed components are combined, including adaptive fusion, LAA, and BANE sampling.The ablation attributes gains to improved anchor representation, class-wise balancing, and more informative negatives.
  • Multi-scale integration: Performance consistently improves as more scales are combined, with all four scales providing complementary fine-detail and global-semantic cues.Removing global features most strongly degrades performance, while fine-grained features remain important for boundaries and small objects.
  • Hyperparameters: 82.87% mIoU is obtained with λ1 = 0.1, λ2 = 0.3, λ3 = 0.7, and λ4 = 1.0, while smaller deep-layer weights reduce performance.The results indicate that relative layer contributions require empirical balancing rather than monotonic weighting.
  • Adaptive fusion: The adaptive fusion design achieves the highest specialization and mIoU because channel specialization and nonlinear refinement work together.Element-wise summation, Concat + MLP, and self-attention-only variants each lack one of these properties.
  • Anchor and memory design: Deepest-layer anchors outperform earliest-layer and averaged anchors, while class-balanced anchor memory outperforms pixel-level memory under the same budget.The reported best memory-bank configuration uses KM = 100; larger banks can let LAA dominate and reduce feature diversity.
  • BANE sampling: Boundary-adjacent misclassified pixels are harder negatives, but excessive negative sampling harms performance; the best reported sampling ratio is 50%.The decline is attributed to optimization toward the hardest negatives causing local minima early in training.
  • Efficiency: Relative to Contextrast, Contextrast++ adds 136 MiB of memory and 21 minutes of training time while leaving inference unchanged.The measured overhead is reported for Cityscapes-val across the entire training schedule.

D. Qualitative analyses of feature representations

Qualitative analyses compare feature activations, Grad-CAM attention, and t-SNE embeddings across baselines and Contextrast++. The visual evidence emphasizes more precise object-focused attention, sharper boundaries, and improved class separability across layers.

  • Feature map visualization: Contextrast++ attends more accurately to small objects and fine-grained boundaries than HRNet, Multi, and Contextrast feature maps.The comparison uses brighter activation regions and zoomed-in boxes on Cityscapes-val.
  • t-SNE analysis: Contextrast++ produces more compact intra-class clusters and clearer inter-class separation across the first four layers.The t-SNE comparison is against HRNet on Cityscapes-val and is presented as evidence of improved semantic discriminability across scales.
  • Grad-CAM: Contextrast++ focuses more precisely on relevant class regions while disregarding irrelevant areas in Grad-CAM visualizations.It reduces attention to an occluding person near a car and ignores a stroller in a person example.

V. CONCLUSION

Contextrast++ extends contextual contrastive learning with adaptive multi-scale integration, class-balanced anchor memory, and boundary-aware hard-negative selection. Across five public datasets and CNN and transformer backbones, it outperforms prior contrastive methods on most configurations, with largest gains on long-tailed datasets and object boundaries.

  • Conclusion: Contextrast++ adaptively integrates multi-scale information and addresses semantic segmentation challenges through enhanced PA and AA losses.The framework builds on CCL and BANE sampling from Contextrast.
  • Conclusion: The adaptive fusion module balances local and global contexts, AA loss mitigates long-tailed imbalance, and BANE improves fine-grained learning near boundaries.These mechanisms use self-attention-based anchor fusion, representative-anchor memory, and hard boundary negatives, respectively.
  • Conclusion: Across five public datasets and CNN and transformer backbones, Contextrast++ outperforms prior contrastive learning methods on most configurations, especially for long-tailed datasets and object boundaries.The authors identify semantic mapping and navigation in urban robotic environments as future applications.
Loading 2608.22679v1…