Source-linked AI summary
Data Distillation: Towards Omni-Supervised Learning
Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, Kaiming He
TL;DR
Omni-supervised learning asks whether labeled datasets can be augmented with internet-scale unlabeled data beyond the fully supervised baseline. The paper proposes data distillation, which ensembles one model’s predictions across transformed unlabeled images to create annotations, and reports gains for COCO keypoint and object detection. The authors note that iterative distillation may improve results further.
Problem
Omni-supervised learning seeks to use all available labeled data together with potentially unlimited unlabeled data, beyond simulated semi-supervised settings bounded by dataset splits.
Method
Data distillation applies one trained model to multiple transformations of unlabeled data, ensembles the predictions into annotations, and retrains on supervised and generated labels.
Results
Data distillation surpasses fully supervised baselines on COCO object and keypoint detection, with 1.2 to 1.5 points of AP increase reported across keypoint backbones.
Takeaways & Limitations
The results support omni-supervised learning as a practical large-scale setting for improving challenging visual recognition tasks with unlabeled data.
Takeaways & Limitations
The authors leave iterative data distillation as future work because it may improve results further.
Abstract
from arXiv · showhide
We investigate omni-supervised learning, a special regime of semi-supervised learning in which the learner exploits all available labeled data plus internet-scale sources of unlabeled data. Omni-supervised learning is lower-bounded by performance on existing labeled datasets, offering the potential to surpass state-of-the-art fully supervised methods. To exploit the omni-supervised setting, we propose data distillation, a method that ensembles predictions from multiple transformations of unlabeled data, using a single model, to automatically generate new training annotations. We argue that visual recognition models have recently become accurate enough that it is now possible to apply classic ideas about self-training to challenging real-world data. Our experimental results show that in the cases of human keypoint detection and general object detection, state-of-the-art models trained with data distillation surpass the performance of using labeled data from the COCO dataset alone.
1. Introduction
The paper frames omni-supervised learning as using all labeled data alongside potentially unlimited unlabeled data, then proposes data distillation to generate annotations from transformed views. Experiments show gains for COCO keypoint and object detection.
- Omni-supervised learning combines well-annotated data with potentially unlimited unlabeled data from internet-scale sources.
- Unlike simulated semi-supervised splits, omni-supervised learning is lower-bounded by performance using all available annotations.
- Data distillation ensembles predictions from one model applied to multiple transformations of unlabeled images to generate training annotations.
- Recent fully supervised models make self-training more practical because their predictions may contain fewer errors than correct predictions.
- Data distillation avoids changing the underlying recognition model or loss definitions and scales to large unlabeled data sources.
- Up to 2 points of AP improvement over a strong Mask R-CNN baseline was observed for COCO keypoint detection, with gains also shown for object detection.
2. Related Work
Related work spans model and knowledge distillation, self-training, perturbation consistency, and large-scale semi-supervised learning. Data distillation combines these themes by using transformed views with a single strong model for structured prediction.
- Model compression and knowledge distillation transfer ensemble knowledge or soft predictions into a student model.
- Prior distillation methods address architecture compression, cross-modal labels, privileged information, and checkpoint-based ensembles.
- Data distillation is closely related to self-training, which uses a model’s predictions on unlabeled data to train itself.
- Other semi-supervised methods exploit multiple views, perturbations, reconstruction consistency, or output consistency across transformations.
- Large-scale predecessors used millions of images or web-scale data, whereas this method uses strong deep networks and extends to structured prediction tasks.
3. Data Distillation
Data distillation applies one trained model to multiple transformations of unlabeled data, aggregates the predictions into labels, and retrains on labeled and automatically labeled examples. The procedure preserves the recognition model and loss while exploiting multi-transform information.
- Data distillation trains a model on labeled data, applies it to transformed unlabeled data, ensembles predictions into labels, and retrains.
- Multi-transform inference applies one model to multiple transformations of each data point and aggregates the resulting predictions.
- Aggregating transformed predictions can produce a single prediction superior to any prediction from an individual transform.
- The method generates hard labels matching the structure and type of manual annotations, using task-specific aggregation logic when needed.
- Inference over multiple transformations is generally less expensive than training multiple models from scratch.
- A student model is trained on the union of supervised data and automatically labeled unlabeled data without changing the loss function.
4. Data Distillation for Keypoint Detection
The keypoint-detection implementation uses geometric multi-transform inference with Mask R-CNN, aggregates transformed keypoint predictions, filters annotations by score, and retrains on mixed manual and generated data. Generated annotations are generally high quality but can contain errors and missing detections.
- 4. Data Distillation for Keypoint Detection: The keypoint-detection instantiation uses Mask R-CNN, a two-stage model with bounding-box and keypoint heads.
- Data transformations: Geometric predictions are inverse-transformed before merging, and the implementation uses scaling and horizontal flipping.
- Data transformations: ResNet-50 keypoint AP increased from 65.1 to 67.8 when the selected transformations were applied for teacher inference.
- Generated annotations: Generated Sports-1M annotations have reasonably high quality, though examples include inverted keypoints and other mistakes.
- Ensembling: The keypoint head averages heatmaps from multiple transformations for the same region of interest and takes the argmax position.
- Selecting predictions: Predictions above a detection-score threshold are selected to generate annotations, using the score as a proxy for prediction quality.
- Selecting predictions: The generated annotations may contain false negatives, so missing annotations should not necessarily be interpreted as background.
- Retraining: Retraining mixes original and generated images at a fixed 6:4 expected ratio and lengthens the schedule to account for extra images.
5. Experiments on Keypoint Detection
Experiments evaluate data distillation for COCO keypoint detection across simulated and real-world omni-supervised settings. The method improves over supervised baselines, including with dissimilar unlabeled data, while benefiting from stronger teachers, more unlabeled images, and longer training.
- Small-scale Data: 60.2 AP surpasses the co-35-only baseline’s 54.9 AP but remains below the co-115 fully-supervised result of 65.1 AP.This sanity-check uses co-35 as labeled data and co-80 as unlabeled data.
- Large-scale Dissimilar-distribution Data: 1.2 to 1.5 points of AP increase is achieved with s1m-180 despite its different distribution from COCO.The improvement is reported consistently across different backbones and is reasonably close to the similar-distribution case.
- Training Iterations: Data distillation improves with training from 90k to 360k iterations, whereas its 90k result is below the fully-supervised counterpart at 63.6 versus 64.2.The authors use 360k iterations for other data-distillation results and suggest performance had not saturated.
- Teacher Quality: Higher-quality teachers generally produce better students, while a ResNet-50 student reaches 68.9 AP versus 67.8 AP for its test-time-augmented teacher.The authors interpret the latter result as evidence that distilled labels may convey knowledge from extra unlabeled data rather than only transform robustness.
6. Experiments on Object Detection
The object-detection experiments test data distillation across small- and large-scale COCO settings, using unlabeled data alongside labeled data. Gains depend on scale: small-scale distillation remains below the fully supervised upper bound, while large-scale distillation consistently improves over supervised baselines.
- Experimental Setup: Data distillation is evaluated for COCO object detection using co-35/80/115 splits and minival testing.The experiments compare labeled and unlabeled data configurations across small- and large-scale settings.
- Small-scale Data: 32.3 vs. 30.5: small-scale data distillation exceeds fully supervised training on co-35 but remains below fully supervised training on co-115.The authors describe this result as closer to the lower bound and farther from the upper bound, requiring further exploration.
- Large-scale Data: 0.8-0.9 points in AP and 0.9-1.1 points in AP50: large-scale data distillation improves over fully supervised co-115 baselines across network backbones and metrics.The gains are consistently observed despite appearing small.
- Large-scale Data: Up to 1.8 points in APM, from 43.7 to 45.5 with ResNeXt-101-32×4, is the largest reported large-scale improvement.This is the biggest improvement among the reported metrics in the large-scale object-detection comparison.
- Findings: The results suggest that object detection with unlabeled data is more challenging, but data distillation can still provide gains in the large-scale setting.The conclusion is drawn from the small- and large-scale comparisons.
7. Conclusion
The paper shows that omni-supervised learning can surpass large-scale supervised learning by combining available labeled data with large amounts of unlabeled data. Data distillation demonstrates this possibility on COCO object and keypoint detection.
- Conclusion: Omni-supervised learning can surpass large-scale supervised learning by combining all available supervised data with large amounts of unlabeled data.The paper applies data distillation to COCO object and keypoint detection.