Source-linked AI summary
End-to-End Semi-Supervised Object Detection with Soft Teacher
Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, Zicheng Liu
TL;DR
Semi-supervised object detection needs to exploit unlabeled images because annotation is costly, while prior pseudo-label methods rely on multi-stage training and initially imperfect labels. This paper trains a teacher-student detector end to end, adding soft teacher and box jittering techniques, and reports large COCO gains across labeling ratios and detector settings.
Problem
Annotation is time-consuming and expensive, while prior pseudo-label methods use multi-stage training whose performance is limited by initially inaccurate pseudo labels.
Method
An end-to-end teacher-student framework simultaneously pseudo-labels unlabeled images and trains the detector, using EMA teacher updates, soft teacher weighting, and box jittering.
Results
The method outperforms previous methods on COCO across 1%, 5% and 10% labeling ratios and improves detection and instance segmentation in fully labeled settings.
Takeaways & Limitations
The framework performs strongly in both partially labeled and fully labeled COCO settings, including with state-of-the-art Swin-Transformer detectors.
Takeaways & Limitations
The framework is theoretically applicable to mainstream detectors, but the paper uses Faster R-CNN as its default framework for fair comparison.
Abstract
from arXiv · showhide
This paper presents an end-to-end semi-supervised object detection approach, in contrast to previous more complex multi-stage methods. The end-to-end training gradually improves pseudo label qualities during the curriculum, and the more and more accurate pseudo labels in turn benefit object detection training. We also propose two simple yet effective techniques within this framework: a soft teacher mechanism where the classification loss of each unlabeled bounding box is weighed by the classification score produced by the teacher network; a box jittering approach to select reliable pseudo boxes for the learning of box regression. On the COCO benchmark, the proposed approach outperforms previous methods by a large margin under various labeling ratios, i.e. 1\%, 5\% and 10\%. Moreover, our approach proves to perform also well when the amount of labeled data is relatively large. For example, it can improve a 40.9 mAP baseline detector trained using the full COCO training set by +3.6 mAP, reaching 44.5 mAP, by leveraging the 123K unlabeled images of COCO. On the state-of-the-art Swin Transformer based object detector (58.9 mAP on test-dev), it can still significantly improve the detection accuracy by +1.5 mAP, reaching 60.4 mAP, and improve the instance segmentation accuracy by +1.2 mAP, reaching 52.4 mAP. Further incorporating with the Object365 pre-trained model, the detection accuracy reaches 61.3 mAP and the instance segmentation accuracy reaches 53.0 mAP, pushing the new state-of-the-art.
1. Introduction
The paper addresses limited-label object detection with an end-to-end pseudo-labeling framework and two techniques for using teacher information. It reports large COCO gains across sparse- and fully labeled settings.
- Annotation is time-consuming and expensive, motivating semi-supervised learning for object detection.
- Previous pseudo-label methods use multi-stage training, but their final performance is limited by pseudo labels from an initially inaccurate detector.
- The end-to-end framework jointly pseudo-labels unlabeled images and trains a detector with labeled and pseudo-labeled data at each iteration.
- Soft teacher weights background classification loss using reliability information from the teacher, while box jittering selects reliable boxes for localization training.
- 20.5, 30.7 and 34.0 mAP are achieved with 1%, 5% and 10% labeled COCO data, surpassing STAC by +6.5, +6.4 and +5.4 mAP, respectively.
- +1.5 mAP detection and +1.2 mAP instance segmentation gains reach 60.4 and 52.4 mAP on a Swin-Transformer detector; Object365 pre-training reaches 61.3 and 53.0 mAP.
2. Related works
Related work divides semi-supervised learning into consistency-based and pseudo-label-based methods. For object detection, the paper distinguishes its end-to-end pseudo-labeling framework from prior multi-stage approaches.
- Semi-supervised image classification methods are broadly consistency-based or pseudo-label-based.
- Semi-supervised object detection likewise includes consistency and pseudo-label methods, with this paper belonging to the pseudo-label category.
- Prior methods use augmentation ensembling, label pasting, or consistency estimation to exploit unlabeled images.
- STAC uses weak augmentation for training and strong augmentation for pseudo-labeling but retains a multi-stage training scheme.
- The proposed method is theoretically compatible with single-stage and two-stage detectors, while Faster R-CNN is used for fair comparison.
3. Methodology
The method uses end-to-end teacher–student pseudo-labeling, jointly training on labeled ground truth and unlabeled pseudo boxes. Soft Teacher weights classification supervision by teacher-derived reliability, while box jittering estimates localization reliability for regression.
- End-to-End Pseudo-Labeling Framework: The student minimizes supervised and unsupervised losses, while an EMA teacher generates pseudo boxes for unlabeled images.Labeled and unlabeled images are sampled into each batch; the overall objective is a weighted sum of supervised and unsupervised loss.
- Pseudo-Label Generation: The framework applies strong augmentation for student detection training and weak augmentation for teacher pseudo-labeling.This augmentation arrangement is used to generate higher-quality pseudo boxes while training the student.
- Pseudo-Label Generation: Foreground-score thresholding retains teacher-generated candidates as pseudo boxes, but stricter thresholds trade recall for precision.A threshold of 0.9 achieves the best reported performance, although retained-box recall falls quickly as the threshold increases.
- Soft Teacher: Soft Teacher weights each student-generated background candidate’s classification loss by a reliability score derived from the teacher model.The teacher processes student-generated candidates through its detection head to obtain background scores used as a reliability proxy.
- Box Jittering: Foreground score is not strongly correlated with localization accuracy, so it is unsuitable as the sole criterion for selecting regression pseudo boxes.Figure 3 compares foreground score and box regression variance against IoU with ground truth for box candidates.
- Box Jittering: Box jittering estimates localization reliability by repeatedly refining perturbed candidates and measuring the variance of their regression predictions.Repeated jittered refinements produce coordinate-wise standard deviations; smaller normalized regression variance indicates higher localization reliability.
4. Experiments
Experiments evaluate Soft Teacher on COCO with partially and fully labeled settings, compare it against prior methods and stronger detectors, and ablate its framework and key design choices.
- System Comparison: +3.6 points are gained over the supervised baseline in the fully labeled setting, exceeding Proposal Learning’s +1.0 points and STAC’s -0.3 points.The comparison uses all train2017 labeled data and unlabeled2017 as additional unlabeled images.
- System Comparison: +1.8 detection AP and +1.4 mask AP are obtained on HTC++ with a Swin-L backbone, while test-dev2017 detection improves by 1.5 mAP.The test-dev result surpasses 60 mAP on COCO.
- Ablation Studies: 1.3 points are gained by switching from multi-stage to end-to-end training, and EMA teacher updates further reach 31.2 mAP.The ablation isolates the framework change and then adds EMA updating.
- Ablation Studies: Soft teacher improves E2E+EMA by 2.4 points, while box jittering raises performance to 34.2 mAP, 3 points above E2E+EMA.The two techniques are evaluated sequentially within the end-to-end model with EMA.
- Ablation Studies: The teacher’s background score is the best soft-teacher reliability indicator, whereas using the student performs worse and IoU-based improvements are negligible.The ablation compares alternative indicators for reliability estimation.
- Ablation Studies: The best foreground threshold is 0.9, the best box-regression variance threshold is 0.02, and performance saturates at 10 jittered boxes.Lower or higher foreground thresholds cause significant degradation.
5. Conclusion
The paper proposes an end-to-end semi-supervised detection framework that jointly improves the detector and pseudo labels through student training and EMA-updated teacher labeling. Soft teacher and box jittering support use of teacher guidance, and the method outperforms state-of-the-art methods on COCO in partially and fully labeled settings.
- Conclusion: The framework simultaneously trains a student detector and updates a teacher through exponential moving average for online pseudo-labeling.This replaces the complicated multi-stage schema used by previous approaches.
- Conclusion: Soft teacher and box jittering facilitate efficient use of teacher guidance within the end-to-end framework.They address classification weighting and reliable localization-box selection, respectively.
- Conclusion: The proposed framework outperforms state-of-the-art methods by a large margin on MS-COCO with both partially labeled and fully labeled data.The conclusion reports this outcome across both evaluation settings.