Source-linked AI summary
Guided Collaborative Training for Pixel-wise Semi-Supervised Learning
Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, Rynson W. H. Lau
TL;DR
Pixel-wise SSL is difficult to generalize because dense outputs complicate confidence estimation and existing methods rely on task-specific properties. GCT combines a flaw detector with collaborative constraints that are independent of task-specific properties. Across four pixel-wise tasks, it outperforms applicable state-of-the-art SSL methods by a large margin.
Problem
Pixel-wise SSL performs unsatisfactorily because dense outputs complicate confidence estimation and perturbation design, while existing approaches often require task-specific properties.
Method
GCT uses two task models, a flaw detector producing a flaw probability map, and dynamic consistency and flaw correction constraints for collaborative unlabeled-data learning.
Results
GCT surpasses applicable state-of-the-art SSL methods across semantic segmentation, real image denoising, portrait image matting, and night image enhancement.
Takeaways & Limitations
GCT provides a general pixel-wise SSL framework that applies across diverse tasks without structural adaptation.
Takeaways & Limitations
SSL still has limited performance on tasks requiring highly precise pseudo labels, such as image denoising.
Abstract
from arXiv · showhide
We investigate the generalization of semi-supervised learning (SSL) to diverse pixel-wise tasks. Although SSL methods have achieved impressive results in image classification, the performances of applying them to pixel-wise tasks are unsatisfactory due to their need for dense outputs. In addition, existing pixel-wise SSL approaches are only suitable for certain tasks as they usually require to use task-specific properties. In this paper, we present a new SSL framework, named Guided Collaborative Training (GCT), for pixel-wise tasks, with two main technical contributions. First, GCT addresses the issues caused by the dense outputs through a novel flaw detector. Second, the modules in GCT learn from unlabeled data collaboratively through two newly proposed constraints that are independent of task-specific properties. As a result, GCT can be applied to a wide range of pixel-wise tasks without structural adaptation. Our extensive experiments on four challenging vision tasks, including semantic segmentation, real image denoising, portrait image matting, and night image enhancement, show that GCT outperforms state-of-the-art SSL methods by a large margin. Our code available at: https://github.com/ZHKKKe/PixelSSL.
1 Introduction
The paper targets the poor generalization of SSL to diverse pixel-wise tasks, where dense outputs make confidence estimation and perturbation design difficult. GCT addresses these issues with a flaw detector and collaborative constraints, and is evaluated across four pixel-wise tasks.
- Pixel-wise labels are costly because they require precise annotations for every pixel, motivating SSL for reducing labeled-data requirements.
- Existing pixel-wise SSL methods mainly target specific tasks and rely on task-specific properties, limiting generalization to new vision tasks.
- Dense outputs complicate SSL because pixel-wise confidence is difficult to estimate, especially for regression, and standard perturbations are unsuitable.
- GCT introduces two task models and a flaw detector that estimates pixel-wise confidence through a flaw probability map.
- GCT uses dynamic consistency and flaw correction constraints to exchange reliable pixel predictions and correct unreliable ones without task-specific properties.
- GCT surpasses state-of-the-art SSL methods applicable to semantic segmentation, real image denoising, portrait matting, and night image enhancement.
2 Related Work
Prior pixel-wise SSL research largely focuses on semantic segmentation and often depends on classification-specific or task-specific assumptions. GCT instead uses flaw-detector-based confidence estimation and perturbations suited to dense outputs for broader applicability.
- Image-classification SSL includes adversarial, consistency-based, and self-supervised combinations, but these approaches do not directly resolve dense-output challenges.
- Existing pixel-wise SSL research mainly focuses on semantic segmentation and commonly uses category-dependent properties or pseudo-label filtering.
- Perceptual constraints can support some pixel-wise regression tasks but fail when input and output have different semantic contents, such as segmentation.
- Prediction confidence guides pseudo-label selection, with prior methods obtaining confidence from averaged targets or image-level discriminator outputs.
- GCT estimates location-specific confidence using a dense flaw probability map trained with labeled-data ground truth rather than an image-level average.
- GCT follows Dual Student for perturbation creation but adds flaw-detector-based constraints to support diverse pixel-wise tasks.
3 Guided Collaborative Training
GCT combines two task models with a flaw detector to guide task-independent learning from unlabeled pixel-wise data. Its two constraints exchange reliable pixels and correct unreliable predictions using flaw probability maps.
- Framework overview: GCT contains two task models and a flaw detector whose outputs guide learning from unlabeled data.The flaw detector is trained on labeled data using targets generated from task predictions and ground truth.
- Collaborative training: Together, the constraints make the task models collaborators while the flaw detector and task model objectives become aligned at convergence.The final task-model objective combines supervised learning with dynamic consistency and flaw correction terms.
- Flaw detector: The flaw detector estimates dense, per-pixel prediction confidence from flaw regions converted into probability maps.The labeled-data target is derived from |T_k(x_l)−y| and processed by dilation, blurring, and normalization.
- Comparison with adversarial training: Unlike an adversarial discriminator with constant targets, the flaw detector provides location-specific guidance derived from labeled prediction errors.The discriminator and task model remain adversarial throughout training, whereas GCT uses flaw probabilities to guide correction and knowledge exchange.
- Dynamic consistency constraint: Dynamic consistency exchanges reliable pixel predictions bidirectionally between task models and disables exchange when both flaw probabilities exceed ξ.A pixel with lower flaw probability becomes the pseudo label for the corresponding pixel in the other task model.
- Flaw correction constraint: Flaw correction minimizes each task model’s flaw probability map, applying the constraint to pixels excluded from dynamic consistency.The mask enables flaw correction where both task models have unreliable predictions.
4 Experiments
Experiments evaluate GCT across semantic segmentation, real image denoising, portrait image matting, and night image enhancement under controlled labeled and unlabeled-data settings. GCT consistently improves over supervised and prior SSL baselines, while ablations support the contribution of its constraints and flaw detector.
- Experimental Setup: Experiments use matched computational budgets and standardized training rules to compare SupOnly with AdvSSL, Mean Teacher, S4L, and GCT.SSL models use the same total number of trained samples as the fully supervised baseline.
- Semantic Segmentation: GCT improves Pascal VOC mIOU over SupOnly by 1.26% with 1/2 labels to 3.76% with 1/8 labels.The fully supervised baseline reaches 75.32% mIOU, comparable to DeepLab-v2’s original 75.14%.
- Real Image Denoising: On SIDD, GCT improves PSNR by 0.61dB with 1/16 labels, exceeding previous SSL improvements of at most 0.33dB.The 1/16 setting uses only 10 labeled image pairs; GCT surpasses other SSL methods at every labeled ratio.
- Portrait Image Matting: With 100 labeled portrait images, GCT improves over SupOnly by 1.96dB using 3,850 unlabeled images and 3.99dB using 7,700.Doubling the unlabeled images produces a 2.03dB larger improvement with GCT than with existing SSL methods.
- Ablation Experiments: Ablations show that both the dynamic consistency and flaw correction constraints contribute, while their combination achieves the optimal performance.The flaw detector supports knowledge exchange between task models and correction of unreliable predictions.
- Ablation Experiments: Using the flaw detector to disable harmful Mean Teacher transfers raises mIOU from 69.81% to 70.47% and PSNR from 38.22dB to 38.42dB.These gains are reported under 1/8 labels on Pascal VOC and SIDD, respectively.
5 Conclusions
The paper presents GCT as a general framework for pixel-wise semi-supervised learning and reports effectiveness across diverse vision tasks. It also identifies limited performance when tasks require highly precise pseudo labels.
- GCT is presented as a general framework for pixel-wise semi-supervised learning.
- The experiments support GCT’s effectiveness across a variety of vision tasks.
- SSL still has limited performance on tasks requiring highly precise pseudo labels, such as image denoising.The paper proposes more accurate pseudo labels as a possible direction for future work.
Supplementary Material
The supplementary material identifies the paper’s authors and their affiliations with SenseTime Research.
- The paper lists Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson W.H. Lau as authors.
- Zhanghan Ke and Rynson W.H. Lau are associated with affiliation 1.
- Affiliation 2 is SenseTime Research, with contact addresses for Ke, Qiu, Li, and Yan.
Appendix A: Algorithm of C
Algorithm C computes the flaw detector’s ground truth from task-model predictions and labels using blur, dilation, and normalization. The experiments vary its channel-average coefficient and repetition count across tasks.
- Pipeline C calculates the flaw detector’s ground truth from task-model predictions and corresponding labels on labeled data.
- C applies Gaussian blurring, local-region dilation, and pixel normalization to produce its processed output.Normalization maps all pixels to [0, 1].
- The pipeline repeats dilation and blurring, then normalizes and returns the resulting flaw map.
- The repetition count is ν = 10 for real image denoising, ν = 5 for night image enhancement, and ν = 1 for the other two tasks.
Appendix B: Architecture of Flaw Detector
The flaw detector is a fully convolutional network that processes task-model outputs and inputs, progressively changes feature channels, and rescales its output to input resolution.
- The flaw detector F is a fully convolutional network with eight 4 × 4 convolutional layers.
- Its channel count increases from 64 to 512 across the first seven layers and decreases to 1 in the final layer.
- F concatenates the task-model output and input, uses normalization and leaky ReLU in early layers, and bilinearly interpolates its output to input size.
Appendix C: Training Details
The experiments compare Mean Teacher, S4L, AdvSSL, and GCT using task-specific SSL hyper-parameters selected by grid search. GCT combines flaw correction and dynamic consistency constraints.
- Methods: The evaluation compares Mean Teacher, S4L, AdvSSL, and GCT across four pixel-wise tasks.The tasks are semantic segmentation, real image denoising, portrait image matting, and night image enhancement.
- Hyper-parameters: Mean Teacher scales a consistency constraint, ramps it up over training, and ensembles a teacher model with moving averages.Its listed parameters are λMT, ηMT, and αMT.
- Hyper-parameters: S4L uses λS4L to scale its unsupervised rotation constraint, while AdvSSL scales labeled and unlabeled adversarial constraints.The adversarial coefficients are listed separately for labeled and unlabeled constraints.
- Hyper-parameters: GCT uses λfc for flaw correction, λdc for dynamic consistency, and ηdc to ramp up dynamic consistency during training.It also includes a coefficient for combining the two SSL constraints.
- Hyper-parameter selection: Grid search is used to select suitable SSL hyper-parameters for each validated task.The final experimental settings follow this search procedure.
Appendix D: Visual Comparisons
Visual comparisons cover four pixel-wise tasks under limited labeled-data settings. Red bounding boxes highlight output differences, and GCT surpasses existing SSL methods in visual effects.
- Overall comparison: Red bounding boxes highlight main output differences, and GCT surpasses existing SSL methods in visual effects.The comparison is presented across the four validated tasks.
- Semantic Segmentation: The visual comparisons evaluate semantic segmentation on PASCAL VOC using 1/8 labeled data.The figure compares SSL methods on this benchmark and labeling regime.
- Real Image Denoising: Real image denoising comparisons use the SIDD dataset with 1/8 labeled data.The figure presents visual SSL comparisons under this limited-label setting.
- Portrait Image Matting: Portrait image matting comparisons use 100 labeled data and 3850 unlabeled data from the authors’ dataset.This setting emphasizes the large unlabeled-data component of the experiment.
- Night Image Enhancement: Night image enhancement comparisons use 200 labeled data and 1500 unlabeled data from the authors’ dataset.The figure compares visual outputs under this labeled and unlabeled split.