Source-linked AI summary
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency
Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, Jiaya Jia
TL;DR
Semi-supervised segmentation seeks strong representations from few pixel-labeled images and many unlabeled images, despite models’ excessive reliance on available contexts. The paper enforces same-identity feature consistency across different contexts with directional pixel-wise contrastive learning and sampling strategies. Experiments report new state-of-the-art results and generalization with extra image-level annotations.
Problem
Semi-supervised segmentation must learn robust representations from limited pixel-wise annotations and abundant unlabeled images, while avoiding excessive reliance on training contexts.
Method
The method enforces context-aware consistency between same-identity features under different contexts using Directional Contrastive Loss and two sampling strategies.
Results
The method achieves new state-of-the-art results and generalizes well when extra image-level annotations are available.
Takeaways & Limitations
Context-aware consistency makes representations more robust to contextual variance while retaining useful contextual information through supervised learning and an information-bottleneck projector.
Takeaways & Limitations
The method does not make features completely context-invariant: supervised loss and the projector are used to preserve useful contextual information for segmentation.
Abstract
from arXiv · showhide
Semantic segmentation has made tremendous progress in recent years. However, satisfying performance highly depends on a large number of pixel-level annotations. Therefore, in this paper, we focus on the semi-supervised segmentation problem where only a small set of labeled data is provided with a much larger collection of totally unlabeled images. Nevertheless, due to the limited annotations, models may overly rely on the contexts available in the training data, which causes poor generalization to the scenes unseen before. A preferred high-level representation should capture the contextual information while not losing self-awareness. Therefore, we propose to maintain the context-aware consistency between features of the same identity but with different contexts, making the representations robust to the varying environments. Moreover, we present the Directional Contrastive Loss (DC Loss) to accomplish the consistency in a pixel-to-pixel manner, only requiring the feature with lower quality to be aligned towards its counterpart. In addition, to avoid the false-negative samples and filter the uncertain positive samples, we put forward two sampling strategies. Extensive experiments show that our simple yet effective method surpasses current state-of-the-art methods by a large margin and also generalizes well with extra image-level annotations.
1. Introduction
The paper addresses semi-supervised segmentation with limited pixel-wise labels by reducing excessive reliance on training contexts. It enforces consistency between same-identity features under different contexts using pixel-wise directional contrastive alignment.
- Motivation: Dense pixel-wise annotation is costly, motivating semi-supervised segmentation with a small labeled set and many unlabeled images.Cityscapes annotation takes more than 1.5h per image on average.
- Motivation: Limited training data can make segmentation models over-rely on context and generalize poorly to unseen scenes.The paper specifically notes that models may use surrounding contexts while overlooking the objects themselves.
- Context-aware consistency: The method maintains context-aware consistency between same-identity pixels placed in different environments through overlapping random crops.The overlapping regions have a pixel-wise one-to-one correspondence despite unknown labels.
- Directional alignment: Directional Contrastive Loss performs pixel-wise contextual alignment while directing lower-confidence features toward higher-confidence counterparts.This directional design is intended to prevent more confident features from being corrupted by less confident ones.
- Evaluation and applicability: Two sampling strategies further improve performance, while the method adds few training parameters and leaves the inference model structurally unchanged.Experiments on PASCAL VOC and Cityscapes support the method’s effectiveness, including extension to extra image-level annotations.
2. Related Work
Prior work improves segmentation representations by aggregating contextual information and exploits unlabeled data through adversarial or consistency-based learning. The paper distinguishes its approach from low-level augmentation by targeting contextual variation through overlapping-region feature consistency.
- Semantic Segmentation: Semantic segmentation methods use encoder-decoder structures and context aggregation mechanisms, but generally require substantial pixel-wise annotation.Representative mechanisms include dilated convolution, global pooling, pyramid pooling, and attention.
- Semi-Supervised Learning: Semi-supervised learning uses unlabeled data with limited labels through adversarial distribution alignment or consistency between perturbed samples.Prior consistency methods perturb outputs, inputs, features, or model predictions using approaches such as VAT and Π-Model.
- Semi-Supervised Semantic Segmentation: Semi-supervised segmentation combines adversarial learning, teacher-student frameworks, synthesized samples, self-correction, and output consistency to exploit unlabeled images.Weak labels such as bounding boxes, image-level labels, and scribbles reduce annotation demands but still require human effort.
- Consistency-based Methods: Low-level augmentations alter pixel appearance without changing contextual cues, motivating contextual augmentation and overlapping-region consistency instead.The paper compares contextual and low-level augmentation using t-SNE visualizations of overlapping-region features.
3. Method
The method addresses contextual overfitting in semi-supervised segmentation by enforcing consistency between corresponding pixels observed under different contexts. It uses Directional Contrastive Loss with confidence-aware alignment and sampling strategies to improve feature robustness and negative-sample quality.
- Context-Aware Consistency: The method targets excessive reliance on contextual cues caused by overfitting limited training data, while retaining useful context through supervised learning.The supervised cross-entropy loss prevents the model from completely ignoring context, while the projector is used for alignment.
- Context-Aware Consistency: The framework maintains context-aware consistency between corresponding pixels in overlapping regions of two differently contextualized crops from an unlabeled image.The crops share an overlapping region but differ in non-overlapping contextual regions, creating contextual augmentation while preserving pixel-wise correspondence.
- Directional Contrastive Loss: Directional Contrastive Loss performs pixel-wise contrastive learning, bringing same-pixel features closer while separating features from different original-image locations.Corresponding features in the two overlap maps form positive pairs; features at different locations form negative pairs.
- Directional Contrastive Loss: DCL aligns the less confident feature toward the more confident counterpart, using max C(fi) as feature confidence to avoid corrupting the better representation.This directional design differs from bilateral alignment in vanilla and supervised contrastive losses.
- Sampling Strategies: Pseudo-label negative sampling raises selected-negative precision from around 60% to 89% by filtering likely false negatives, making training more stable and robust.Random sampling leaves fewer than 60% of selected negatives as true negatives, whereas pseudo labels eliminate high-probability false negatives; experiments report substantial improvement.
- Extension with Extra Image-level Annotations: The method can incorporate extra image-level annotations and preserves the original model for inference, enabling adaptation without structural constraints.The extension uses an auxiliary classifier for coarse pseudo labels and discards it during inference.
4. Experiments
Experiments on PASCAL VOC and Cityscapes evaluate the method across labeling regimes, baselines, architectures, and ablations. The method consistently improves segmentation performance, while ablations attribute gains to context-aware consistency, directional contrastive alignment, negative sampling, positive filtering, and the projector.
- Network Architecture: The study uses DeepLabv3+ as the main segmentation network and also implements the method on PSPNet to assess generalization across architectures.DeepLabv3+ uses ResNet101 or ResNet50 backbones in the reported PASCAL VOC comparisons, while PSPNet is included as another base model.
- Experimental Setting: Experiments use PASCAL VOC and Cityscapes, with evaluations across multiple labeled-data proportions and repeated runs using mIoU.The compared settings include full, 1/4, 1/8, and 1/16 labeled data, with results averaged over 3 runs.
- Comparison with State-of-the-Art Methods: The method surpasses competing approaches by a large margin on PASCAL VOC and demonstrates generalization ability on Cityscapes.Comparisons are reproduced within a unified framework where possible, using the same base segmentation model and data lists.
- Ablation Study: Context-aware consistency improves performance by 1.9 points over training with only low-level transformations.The comparison uses experiments VIII and VI, where the latter removes contextual augmentation by setting the crop IoU to [1].
- Ablation Study: DC Loss outperforms simple ℓ2 alignment and the supervised baseline after negative sampling, while directional alignment improves vanilla contrastive learning by 1.2 points.Using ℓ2 loss alone lowers performance from 65.0 to 64.2, whereas the directional mask yields the reported gain over vanilla contrastive loss.
- Ablation Study: Negative sampling and positive filtering further improve the method, with positive filtering contributing 0.8 points and larger negative sets improving performance until an upper bound.Increasing negative samples from 500 to 19.2k raises per-GPU memory by about 800M and iteration time from 0.96s to 1.18s.
- Extra Image-Level Annotations: With extra image-level annotations, the method reaches 76.1% mIoU on PASCAL VOC, 1.0 points above the full pixel-level annotation model.The setting uses pixel-wise annotations for 1464 training images and image-level annotations for 9118 augmented images.
5. Conclusion
The paper addresses semi-supervised semantic segmentation by enforcing context-aware consistency and introducing Directional Contrastive Loss with two sampling strategies. Experiments report new state-of-the-art results and strong generalization with extra image-level annotations.
- Context-aware consistency requires features of the same identity under different contexts to remain consistent.
- Directional Contrastive Loss performs the feature alignment for this consistency.
- Two sampling strategies provide further improvements to the proposed method.
- The method achieves new state-of-the-art results and generalizes well with extra image-level annotations.