Source-linked AI summary
ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning
Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, Lennart Svensson
TL;DR
Costly pixel-level annotation and ineffective classification augmentations limit semi-supervised semantic segmentation. ClassMix mixes predicted classes from unlabelled images, uses consistency training and pseudo-labelling, and improves reported performance across two benchmarks, with strongest results on Cityscapes.
Problem
Ground-truth labels for semantic segmentation are costly, while common semi-supervised classification augmentations are less effective for segmentation.
Method
ClassMix mixes predicted classes from two unlabelled images into augmented samples and artificial labels, combined with consistency regularization and pseudo-labelling.
Results
ClassMix achieves state-of-the-art semi-supervised results on Cityscapes and competitive results on Pascal VOC, outperforming other mixing methods in an ablation study.
Takeaways & Limitations
Semantic predictions can support augmentation that better respects object boundaries and improves semi-supervised semantic segmentation performance.
Abstract
from arXiv · showhide
The state of the art in semantic segmentation is steadily increasing in performance, resulting in more precise and reliable segmentations in many different applications. However, progress is limited by the cost of generating labels for training, which sometimes requires hours of manual labor for a single image. Because of this, semi-supervised methods have been applied to this task, with varying degrees of success. A key challenge is that common augmentations used in semi-supervised classification are less effective for semantic segmentation. We propose a novel data augmentation mechanism called ClassMix, which generates augmentations by mixing unlabelled samples, by leveraging on the network's predictions for respecting object boundaries. We evaluate this augmentation technique on two common semi-supervised semantic segmentation benchmarks, showing that it attains state-of-the-art results. Lastly, we also provide extensive ablation studies comparing different design decisions and training regimes.
1. Introduction
Semantic segmentation needs costly pixel-level labels, while unlabelled data is abundant. ClassMix addresses weak segmentation augmentations by mixing predicted classes across unlabelled images while respecting semantic boundaries.
- Motivation: Pixel-level ground-truth labels are a major bottleneck, whereas unlabelled data is usually abundant.Using unlabelled data could increase performance at low cost.
- Motivation: Strong augmentations that improve semi-supervised image classification have proved ineffective for semantic segmentation.Prior segmentation approaches instead perturb encoded states or use CutMix.
- ClassMix: ClassMix cuts predicted classes from one unlabelled image and pastes them onto another to form boundary-aware training samples.The method uses pixel-level semantic predictions rather than ground-truth annotations.
- ClassMix: The framework combines ClassMix with consistency regularization, pseudo-labelling, and entropy minimization for semi-supervised segmentation.Predictions on mixed images are trained to remain consistent with predictions before mixing.
- Evaluation: Experiments report state-of-the-art Cityscapes results and competitive Pascal VOC results.The contribution summary also identifies a unified framework and an ablation study of design and experimental choices.
2. Related Work
Related semi-supervised segmentation work uses adversarial learning, consistency regularization, pseudo-labelling, and mask-based image compositing. ClassMix differs by generating masks from predicted semantic segmentations rather than fixed rectangles or ground-truth masks.
- Existing approaches: Semi-supervised semantic segmentation has used adversarial learning, consistency regularization, and pseudo-labelling.The proposed method primarily combines consistency regularization and pseudo-labelling.
- Consistency regularization: Consistency regularization enforces prediction invariance to perturbations, with augmentation anchoring transferring predictions from weakly altered images to strongly augmented ones.ClassMix applies this from unperturbed images to mixed images.
- Consistency regularization: Segmentation consistency methods have used encoder-output perturbations or CutMix, which composites rectangular regions from two images.These approaches address difficulties including violation of the cluster assumption.
- Pseudo-labelling: Pseudo-labelling trains against targets derived from network class predictions and is motivated by entropy minimization on unlabelled images.In ClassMix, it also helps prevent unreasonable labels near mixing borders.
- Mask-based augmentation: ClassMix uses predicted segmentations to generate binary masks instead of CutMix rectangles.This preserves the shared mask-based compositing principle while conditioning the mask on semantic predictions.
- Mask-based augmentation: Unlike several existing object-pasting strategies, ClassMix does not require ground-truth segmentation masks.It combines two images conditioned on predicted semantic maps and can therefore use unlabelled data.
3. Method
ClassMix synthesizes unlabelled training examples by selecting predicted semantic classes from one image and combining them with another, producing artificial labels aligned to the same mask. Mean Teacher predictions, pseudo-labelling, and a supervised-plus-unsupervised loss stabilize and train the resulting samples.
- 3.1. ClassMix: Main Idea: ClassMix uses two unlabelled images to create a new augmented image and corresponding artificial label without ground-truth annotations.The augmented sample mixes inputs and predictions from the two images.
- 3.1. ClassMix: Main Idea: A binary mask randomly selects half of the classes in one image’s pixel-wise argmax prediction, then copies those pixels from that image and the remainder from the other.The same mask is applied to the images and their predictions.
- 3.2. ClassMix: Details: Mean Teacher uses an exponential moving average of network parameters to produce more stable predictions and artificial labels, while the student network trains on mixed images.The student parameters are updated using gradient descent.
- 3.2. ClassMix: Details: Pseudo-labelling converts each artificial-label pixel distribution into a one-hot target for the most probable class, sharpening labels near mixing borders.This is intended to eliminate uncertainty and mitigate label contamination caused by changed context around pasted classes.
- 3.3. Loss and Training: Training minimizes a supervised loss on labelled images plus a weighted unsupervised loss on ClassMix images and artificial labels.The unsupervised weight λ balances the two terms and is increased as prediction quality improves.
- 3.3. Loss and Training: The cross-entropy loss is averaged over pixel positions, with training batches containing 50% labelled data and 50% augmented data.The loss uses predicted class probabilities and target values at each pixel.
4. Experiments
Experiments evaluate ClassMix on Cityscapes and Pascal VOC 2012, then examine dataset properties, mixing choices, loss weighting, training duration, and augmentation regimes. ClassMix performs especially strongly on Cityscapes, while Pascal results remain competitive despite less favorable class distributions.
- Datasets and setup: Experiments evaluate ClassMix on Cityscapes and Pascal VOC 2012, with an extensive ablation study investigating its properties and components.
- Cityscapes results: Cityscapes results exceed the supervised baseline for all four labelled-data proportions, with higher results for three of four proportions.The improvement from baseline to SSL is higher for every training-data amount.
- Cityscapes results: ClassMix achieves the best reported semi-supervised results on Cityscapes among the compared methods using the same DeepLab-v2 framework.The authors attribute performance partly to diverse masks and semantic alignment with object boundaries.
- Pascal VOC 2012 results: Pascal VOC 2012 results are competitive, with strongest performance for two data amounts, but comparisons are complicated by differing baselines and pretraining.The method also exceeds Ouali et al.’s 69.4% mIoU using 1.5k samples with DeepLabv3, according to the authors’ comparison.
- Dataset effects: Pascal’s few classes per image reduce mask diversity, while inconsistent spatial contexts can place pasted objects unrealistically and limit performance relative to Cityscapes.Cityscapes’ clustered class locations make transferred objects more likely to appear in reasonable contexts.
- Ablation study: ClassMix outperforms CutMix and FMix in the ablation, which the authors associate with more varied masks and borders that better follow semantic boundaries.
- Ablation study: Pixel-wise confidence thresholding lowers results by masking boundary and small-object pixels, whereas the default weighting uses the proportion of pixels above threshold 0.968.Sigmoid ramp-up performs somewhat below the default and matches constant unsupervised weighting at one.
- Ablation study: Pseudo-labels improve generalization and help avoid label contamination near object boundaries, where unreasonable targets otherwise affect training.
5. Conclusion
The paper proposes ClassMix for semi-supervised semantic segmentation and reports improved state-of-the-art performance through experiments and ablations.
- ClassMix mixes unlabelled samples using semantic predictions to generate augmented images and artificial labels that better respect object boundaries.
- The method was evaluated on two commonly used semantic segmentation datasets, with reported improvements to the state of the art.
- An extensive ablation study compared different configurations and training regimes to motivate the design choices.