Source-linked AI summary
MegDet: A Large Mini-Batch Object Detector
Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, Jian Sun
TL;DR
Object detection had not well studied mini-batch size despite its importance, while typical detector batches remained small. MegDet combines a learning-rate policy with Cross-GPU Batch Normalization to train detectors with much larger batches, reducing COCO training from 33 hours to 4 hours with higher accuracy and supporting a first-place COCO 2017 Detection Challenge result.
Problem
Object detection commonly uses small mini-batches, leaving the effects and practical training of much larger mini-batches insufficiently studied.
Method
MegDet combines a warmup-based learning-rate policy with Cross-GPU Batch Normalization for large-mini-batch object detector training.
Results
4 hours versus 33 hours: MegDet completes COCO training on 128 GPUs with higher accuracy than the small-mini-batch counterpart.
Takeaways & Limitations
MegDet substantially accelerates the research cycle while maintaining higher accuracy and underpins a 1st-place COCO 2017 Detection Challenge result.
Takeaways & Limitations
The authors leave unexplained why the large-mini-batch detector is worse at early epochs but catches up after the second learning-rate decay.
Abstract
from arXiv · showhide
The improvements in recent CNN-based object detection works, from R-CNN [11], Fast/Faster R-CNN [10, 31] to recent Mask R-CNN [14] and RetinaNet [24], mainly come from new network, new framework, or novel loss design. But mini-batch size, a key factor in the training, has not been well studied. In this paper, we propose a Large MiniBatch Object Detector (MegDet) to enable the training with much larger mini-batch size than before (e.g. from 16 to 256), so that we can effectively utilize multiple GPUs (up to 128 in our experiments) to significantly shorten the training time. Technically, we suggest a learning rate policy and Cross-GPU Batch Normalization, which together allow us to successfully train a large mini-batch detector in much less time (e.g., from 33 hours to 4 hours), and achieve even better accuracy. The MegDet is the backbone of our submission (mmAP 52.5%) to COCO 2017 Challenge, where we won the 1st place of Detection task.
1. Introduction
MegDet addresses the underexplored role of mini-batch size in object detection by combining large-batch training with a learning-rate policy and Cross-GPU Batch Normalization. It achieves faster COCO training with higher accuracy and won the COCO 2017 Detection Challenge.
- Motivation: Object detection commonly uses mini-batches of 2–16, unlike image classification settings using much larger batches.The paper identifies mini-batch size as an underexplored training factor in object detection.
- Problems with Small Mini-Batch Size: Small mini-batches lengthen training, provide poor Batch Normalization statistics, and can produce imbalanced positive and negative proposals.The paper presents these as potential drawbacks of small mini-batch training.
- Challenge and Solution: Large mini-batches create a learning-rate dilemma because increasing the learning rate may harm convergence, while reducing it often lowers accuracy.MegDet uses warmup to gradually increase the learning rate during the early training stage.
- Technical Contributions: Cross-GPU Batch Normalization improves Batch Normalization statistics, accuracy, and training stability for large mini-batches.The method aggregates information across GPUs to make large-batch detector training easier to converge.
- Results: 4 hours versus 33 hours: MegDet trains COCO on 128 GPUs with higher accuracy than its small-mini-batch counterpart.The result corresponds to ResNet-50-based detectors and represents nearly an order-of-magnitude acceleration.
- Results: MegDet contributed to the 1st-place result in the COCO 2017 Detection Challenge.The paper reports this outcome for the MegDet-based submission.
2. Related Work
Related work spans two-stage and one-stage CNN detectors, while large mini-batch research has mainly focused on image classification rather than object detection.
- CNN-Based Detectors: CNN-based detectors are broadly divided into one-stage methods such as SSD, YOLO, and RetinaNet, and two-stage methods such as Faster R-CNN, R-FCN, and Mask R-CNN.The distinction is based on whether detection involves proposal generation and refinement.
- Two-Stage Detectors: R-CNN methods evolved from selective-search proposals toward Faster R-CNN’s Region Proposal Network.Fast R-CNN introduced ROI pooling, while Faster R-CNN replaced traditional proposal generation with an RPN.
- One-Stage Detectors: One-stage detectors directly predict classification and regression outputs using convolutional architectures and are typically faster than two-stage detectors.YOLO and SSD use grid- or feature-layer-based prediction, while RetinaNet uses focal loss.
- Large Mini-Batch Training: Large mini-batch training has been actively studied in image classification, including rapid ResNet-50 training and methods addressing its generalization gap.The paper contrasts this literature with the smaller mini-batches commonly used in object detection.
3. Approach
MegDet addresses the training-time, normalization, and sample-imbalance problems of small detection mini-batches with a large-mini-batch approach combining scaled learning rates, warmup, and Cross-GPU Batch Normalization.
- Approach: MegDet combines large-mini-batch training with learning-rate policy and Cross-GPU Batch Normalization to reduce training time while maintaining or improving accuracy.The approach is presented as a way to finish training in less time while achieving higher accuracy.
- Problems with Small Mini-Batch Size: Small mini-batches lengthen detector training, prevent retraining Batch Normalization with task-specific statistics, and increase positive–negative sample imbalance.A ResNet-50 detector with mini-batch size 16 takes more than 30 hours, while batch size 2 can take more than one week.
- Learning Rate for Large Mini-Batch: Large mini-batches create a convergence–accuracy tradeoff: scaling the learning rate can cause failure to converge, while reducing it often lowers accuracy.The challenge motivates a learning-rate policy specifically for large-mini-batch object detection.
- Learning Rate for Large Mini-Batch: MegDet applies linear learning-rate scaling under a variance-equivalence interpretation, matching the variance of one large-batch update to k accumulated small-batch steps.For a large mini-batch N̂ = k · N, the derivation yields r̂ = k · r, the same final scaling rule as prior work.
- Learning Rate for Large Mini-Batch: Linear Gradual Warmup starts with a small learning rate and increases it until the target rate, helping convergence during the early training stage.Warmup is reported as insufficient by itself for mini-batch sizes such as 128 or 256.
- Cross-GPU Batch Normalization: Cross-GPU Batch Normalization aggregates statistics across devices so high-resolution detection inputs can use sufficient samples for normalization.The implementation uses AllReduce to aggregate means and variances; it operates across GPUs on the same machine and can normalize 16 images when each GPU holds two.
4. Experiments
Experiments evaluate MegDet across training configurations, showing that warmup plus Cross-GPU Batch Normalization enables large mini-batch training with shorter cycles and maintained or improved accuracy.
- COCO experiments use ResNet-50 with FPN, SGD momentum 0.9, weight decay 0.0001, 118,000 training images, and 5,000 validation images.
- Large mini-batch size, no BN: Without batch normalization, mini-batch 32 nearly linearly accelerates training without accuracy loss, while larger batches become harder to converge or lose accuracy under reduced learning rates.
- Large mini-batch size, with CGBN: Warmup plus CGBN makes all tested mini-batch trainings converge easily, avoiding the accuracy loss associated with using a smaller learning rate.
- Large mini-batch size, with CGBN: 4.1 hours is the COCO training time for the 256-mini-batch experiment on 128 GPUs, an 8× acceleration over the 33.2-hour baseline.
- Large mini-batch size, with CGBN: A BN size of 32 gives the best accuracy; smaller sizes produce inaccurate statistics, while increasing the size to 64 reduces accuracy.
- Large mini-batch size, with CGBN: 256 (long) trails 16 (long) early but catches up after the second learning-rate decay, while longer training slightly improves accuracy.
5. Concluding Remarks
The paper concludes that large mini-batch training achieves better accuracy in much shorter time and supported the team’s first-place COCO 2017 detection result.
- MegDet achieves better accuracy in much shorter time and accelerates the research cycle.
- MegDet is associated with first place in the COCO 2017 detection challenge.
- Figure 5 presents illustrative MegDet examples on the COCO dataset.
Appendix
The enhanced MegDet combines multiple architectural, supervision, training, and testing techniques for COCO 2017 object detection.
- 50.5 mmAP is achieved on the COCO validation set and 50.6 mmAP on test-dev.
- An ensemble of four detectors achieves 52.5 mmAP on the COCO 2017 challenge.
- The enhanced system integrates OHEM, atrous convolution, stronger base models, segmentation supervision, contextual modules, ROIAlign, and multi-scale training and testing.