Source-linked AI summary
Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning
Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, Liwei Wang
TL;DR
Semi-supervised semantic segmentation must learn from limited, imbalanced data while existing approaches largely treat categories equally. AEL uses category-wise confidence tracking to tilt training toward under-performing categories through adaptive augmentation, sampling, and re-weighting, and reports large gains across benchmarks and protocols.
Problem
Imbalanced pixel categories and limited labeled data leave some categories, especially tailed classes, under-performing, while common approaches largely neglect this biased training problem.
Method
AEL maintains a confidence bank of category-wise performance and uses it to adaptively emphasize under-performing categories through augmentation, sampling, and dynamic re-weighting.
Results
AEL outperforms state-of-the-art methods by a large margin on Cityscapes and PASCAL VOC 2012 under various data partition protocols, including 74.28% and 76.97% reported results.
Takeaways & Limitations
AEL’s adaptive design improves overall segmentation and specifically addresses performance on under-performing and tail categories.
Abstract
from arXiv · showhide
Due to the limited and even imbalanced data, semi-supervised semantic segmentation tends to have poor performance on some certain categories, e.g., tailed categories in Cityscapes dataset which exhibits a long-tailed label distribution. Existing approaches almost all neglect this problem, and treat categories equally. Some popular approaches such as consistency regularization or pseudo-labeling may even harm the learning of under-performing categories, that the predictions or pseudo labels of these categories could be too inaccurate to guide the learning on the unlabeled data. In this paper, we look into this problem, and propose a novel framework for semi-supervised semantic segmentation, named adaptive equalization learning (AEL). AEL adaptively balances the training of well and badly performed categories, with a confidence bank to dynamically track category-wise performance during training. The confidence bank is leveraged as an indicator to tilt training towards under-performing categories, instantiated in three strategies: 1) adaptive Copy-Paste and CutMix data augmentation approaches which give more chance for under-performing categories to be copied or cut; 2) an adaptive data sampling approach to encourage pixels from under-performing category to be sampled; 3) a simple yet effective re-weighting method to alleviate the training noise raised by pseudo-labeling. Experimentally, AEL outperforms the state-of-the-art methods by a large margin on the Cityscapes and Pascal VOC benchmarks under various data partition protocols. Code is available at https://github.com/hzhupku/SemiSeg-AEL
1 Introduction
Semi-supervised semantic segmentation reduces annotation effort but struggles with imbalanced categories, especially tailed classes. AEL addresses this biased training by adaptively emphasizing under-performing categories and achieves strong benchmark results.
- Pixel-level annotation is expensive, while Cityscapes has head classes with hundreds of times more pixels than tailed classes.
- AEL uses a confidence bank to track category-wise performance and adapt training toward under-performing categories.Its strategies include adaptive Copy-Paste, adaptive CutMix, adaptive equalization sampling, and prediction-aware re-weighting.
- 74.28%, 75.83% and 77.90% mIoU were achieved on Cityscapes under 1/32, 1/16 and 1/8 protocols, respectively.These results were +16.39%, +12.87% and +8.09% better than the supervised baseline.
- 76.97%, 77.20% and 77.57% were achieved on PASCAL VOC 2012 under 1/32, 1/16 and 1/8 protocols, respectively.These results were +6.83%, +6.60% and +4.45% better than the supervised baseline.
- 81.95% was achieved on the full Cityscapes train set by leveraging 5,000 coarse images as unlabeled data.The reported improvement over the model trained on the full Cityscapes train set was +1.03%.
2 Related Work
Related work centers on consistency regularization and pseudo-labeling for semi-supervised learning and segmentation. Class imbalance remains relatively under-explored in semantic segmentation, motivating AEL’s focus on under-performing categories.
- Semi-Supervised Learning: Consistency regularization enforces prediction agreement across views, while pseudo-labeling trains on labels generated from model predictions.
- Semi-Supervised Semantic Segmentation: Existing segmentation methods apply techniques including CutMix, feature-level perturbation, network perturbation, and pseudo-label refinement.
- Semi-Supervised Semantic Segmentation: Existing semi-supervised segmentation methods do not explore the biased learning issue despite improving over supervised baselines.
- Class Imbalance in Semi-Supervised Learning: Class imbalance in semi-supervised learning is relatively under-explored, especially for semantic segmentation.
- Class Imbalance in Semi-Supervised Learning: Prior imbalance methods include unlabeled-data utilization, suppressed consistency loss, self-training, resampling, and re-weighting.AEL focuses on improving under-performing categories, which are mostly tailed classes.
3 Method
AEL is a semi-supervised segmentation framework that uses a confidence bank to identify under-performing categories and adapt training toward them through augmentation, sampling, and re-weighting.
- Overview: AEL combines a teacher model for pseudo-labeling, a student model for online learning, and dedicated modules targeting under-performing categories.The teacher is updated by exponential moving average of the student; dedicated modules use labeled and unlabeled data.
- Confidence Bank: The confidence bank dynamically records category-wise performance, enabling identification of categories that are insufficiently trained.It uses category-wise predictions and is updated with exponential moving average during training.
- Adaptive CutMix: Adaptive CutMix increases the frequency of under-performing categories in unlabeled training data by sampling images and crop regions containing selected categories.Sampling probabilities are derived from category-wise confidence, and predictions provide approximate ground truth for the augmented unlabeled data.
- Adaptive Copy-Paste: Adaptive Copy-Paste increases the occurrence of under-performing categories by preferentially copying relevant objects into labeled training images.It adapts the Copy-Paste augmentation strategy to focus training on categories needing more supervision.
- Adaptive Equalization Sampling: Adaptive equalization sampling samples fewer pixels from well-trained categories and more training pixels from under-performing categories for the unsupervised loss.The sampling rate is category-dependent, and sampled-pixel indicators determine which pixels contribute to the reformulated unsupervised loss.
- Dynamic Re-Weighting: Dynamic re-weighting addresses pseudo-label noise while retaining more contributions from convincing samples and complementing adaptive equalization sampling.Strict confidence thresholds reduce recall for under-performing categories, whereas using all pixels introduces more noise.
4 Experiments
AEL is evaluated across multiple datasets, data-partition protocols, and ablations, with results reported using mIoU on validation sets. It consistently improves supervised and state-of-the-art baselines, including for tail classes.
- Experimental Setup: Performance is measured by single-scale mIoU on the Cityscapes and PASCAL VOC validation sets, with ablations conducted on Cityscapes.The reported implementation uses DeepLabv3+ with a ResNet-101 backbone for the benchmark comparisons.
- Comparison with State-of-the-Art Methods: AEL improves Cityscapes over the supervised baseline by +16.4%, +12.9%, +8.1%, +4.8% and +2.8% under 1/32 through 1/2 protocols, respectively.It also exceeds Cutmix-Seg by +5.2% under 1/32 and +1.3% under 1/2.
- Comparison with State-of-the-Art Methods: AEL improves PASCAL VOC 2012 over the supervised baseline by +6.8%, +7.0%, +4.1%, +1.7% and +3.1% under 1/32 through 1/2 protocols, respectively.The method exceeds the previous best method by +3.6% under 1/32 and +2.2% under 1/2.
- Ablation Study: Component ablations show gains from Dynamic Re-weighting, Adaptive Equalization Sampling, Adaptive CutMix, and Adaptive Copy-Paste.Under 1/32 and 1/16 protocols, the reported improvements are +1.1%/+1.7% for DR, +2.5%/+2.0% for AES, +1.3%/+1.7% for ACM, and +0.5%/+0.5% for ACP.
- Results on ADE20K Dataset: AEL consistently improves the supervised baseline on ADE20K by 6.36%, 5.70%, 5.67%, 3.18% and 1.45% across 1/32 through 1/2 protocols.It also exceeds Cutmix-Seg by 2.25%, 3.38%, 2.48%, 1.31% and 1.26%, respectively.
5 Conclusion
AEL addresses biased category learning in semi-supervised semantic segmentation by tracking category-wise performance and adaptively emphasizing under-performing categories. It uses several adaptive training technologies and reports large-margin benchmark improvements.
- AEL adaptively balances training across categories using a confidence bank that identifies under-performing categories during training.The bank records category-wise performance at each training step.
- Adaptive Copy-Paste, CutMix, equalization sampling, and dynamic re-weighting tilt training toward under-performing categories.These components leverage labeled and unlabeled data to make training less biased.
- AEL outperforms state-of-the-art methods by a large margin on Cityscapes and Pascal VOC under various data partition protocols.
A Implementation Details of Adaptive CutMix
Adaptive CutMix selects an unlabeled image containing a sampled category, using dynamically maintained category information for each unlabeled image.
- Adaptive CutMix requires an unlabeled image containing the sampled category.
- A dynamically updated dictionary records category-wise information for each unlabeled image.The dictionary supports determining whether an image contains category c during training.
B Hyper-parameters for PASCAL VOC 2012 Dataset
For PASCAL VOC 2012, AEL uses γ=2, confidence as the category-performance indicator, r*=0.03 for adaptive CutMix, and K=1 sampled category.
- γ is set to 2, and confidence indicates category-wise performance during training.
- The adaptive CutMix ratio r* is set to 0.03.
- The number of sampled categories K is set to 1 for PASCAL VOC 2012.The setting reflects that images contain fewer categories and larger instances.
C Qualitative Results on PASCAL VOC 2012 Dataset
Qualitative comparisons on the PASCAL VOC 2012 validation set show AEL alongside the ground truth, supervised baseline, and basic framework. AEL produces promising visual quality and improves fine details.
- AEL achieves promising visual quality and further improves fine details over the compared methods.
- Figure 4 compares the input image, ground truth, supervised baseline, basic framework, and AEL predictions.