Source-linked AI summary
Rethinking ImageNet Pre-training
Kaiming He, Ross Girshick, Piotr Dollár
TL;DR
The paper asks whether ImageNet pre-training is necessary for target vision tasks. It tests standard models trained from random initialization on COCO with suitable normalization and longer schedules, finding scratch training can match pre-trained counterparts across varied settings while pre-training mainly speeds early convergence.
Problem
Object detection and related tasks commonly rely on ImageNet pre-training, but the benefits of increasingly large classification-level pre-training datasets are small and scale poorly for detection.
Method
The paper compares standard, unspecialized detection systems trained from random initialization or ImageNet pre-training, changing normalization and training length for controlled scratch-training experiments.
Results
Training from scratch is on par with ImageNet-pre-trained counterparts across baselines spanning Average Precision from 40 to over 50, including experiments with 10% COCO data and larger models.
Takeaways & Limitations
ImageNet pre-training speeds convergence, especially early in training, but does not necessarily improve final accuracy or regularization when sufficient target data and computation are available.
Takeaways & Limitations
With 3.5k COCO training images, pre-training reaches 16.0 bbox AP versus 9.3 AP from scratch, placing the breakdown point between 3.5k and 10k images.
Abstract
from arXiv · showhide
We report competitive results on object detection and instance segmentation on the COCO dataset using standard models trained from random initialization. The results are no worse than their ImageNet pre-training counterparts even when using the hyper-parameters of the baseline system (Mask R-CNN) that were optimized for fine-tuning pre-trained models, with the sole exception of increasing the number of training iterations so the randomly initialized models may converge. Training from random initialization is surprisingly robust; our results hold even when: (i) using only 10% of the training data, (ii) for deeper and wider models, and (iii) for multiple tasks and metrics. Experiments show that ImageNet pre-training speeds up convergence early in training, but does not necessarily provide regularization or improve final target task accuracy. To push the envelope we demonstrate 50.9 AP on COCO object detection without using any external data---a result on par with the top COCO 2017 competition results that used ImageNet pre-training. These observations challenge the conventional wisdom of ImageNet pre-training for dependent tasks and we expect these discoveries will encourage people to rethink the current de facto paradigm of `pre-training and fine-tuning' in computer vision.
1. Introduction
The paper questions ImageNet pre-training as a default for vision tasks by testing standard COCO models trained from random initialization. Across data scales and evaluation settings, scratch training can match pre-trained models after longer optimization, while pre-training mainly accelerates early convergence.
- ImageNet pre-training became a standard route to transferable representations and state-of-the-art results across detection, segmentation, and action recognition.
- Object-detection gains from increasingly large pre-training datasets are small and scale poorly, leaving the prospect of solving computer vision through universal pre-training uncertain.
- Training from scratch achieves competitive detection and instance-segmentation accuracy with standard baseline systems and fine-tuning hyper-parameters, provided normalization is appropriate and training is sufficiently long.
- Average Precision from 40 to over 50 shows scratch training can be on par with ImageNet-pre-trained counterparts across multiple baselines, even with 10% of COCO training data and models up to 4× larger than ResNet-101.
- ImageNet pre-training speeds early convergence, but scratch models can catch up after training for roughly the combined pre-training and fine-tuning computation because they must learn low- and mid-level features.
- ImageNet pre-training does not automatically improve regularization, and scratch models can match pre-trained accuracy with 10% of COCO data using the same hyper-parameters.
- Pre-training shows no benefit for more spatially localized targets in these experiments, including improved high-overlap box AP and relatively faster scratch convergence for keypoint AP.
- The findings suggest target-domain data and computation may be worth prioritizing, especially when the source pre-training task differs substantially from the target task.
2. Related Work
Related work established pre-training and fine-tuning as the dominant paradigm for object detection, while earlier and specialized studies showed that scratch training could remain viable. This paper instead isolates pre-training’s role on standard, unspecialized architectures.
- Modern vision algorithms commonly fine-tune ImageNet-pre-trained networks, following early object-detection breakthroughs based on this paradigm.
- Before pre-training became prevalent, object detectors were trained without pre-training, a history that is now somewhat overlooked.
- Earlier analysis found pre-training important with limited detection data but showed scratch training on more detection data could reach 90% of fine-tuning accuracy.
- Because modern detectors evolved under pre-training, training from scratch became viewed as non-trivial, motivating specialized architectures and design principles.
- Unlike specialized scratch-trained detectors, this study compares pre-training with no pre-training on unspecialized architectures and often matches fine-tuning accuracy without architectural specialization.
3. Methodology
The methodology isolates ImageNet pre-training by making only minimal changes to standard detector systems: normalization suited to small batches and longer training schedules for scratch models. The experiments compare training cost through images, instances, and pixels seen.
- 3. Methodology: The study ablates ImageNet pre-training while retaining typical architectures, changing only normalization and training length as necessary for scratch training.
- 3.1. Normalization: High-resolution detection inputs create small batches that degrade Batch Normalization, while pre-training can mask this issue through fixed batch statistics.
- 3.1. Normalization: Group Normalization is batch-size insensitive, and Synchronized Batch Normalization computes statistics across devices to increase effective batch size.
- 3.1. Normalization: Both Group Normalization and Synchronized Batch Normalization enable detection models to train from scratch, while normalized initialization also enables VGG detectors without BN or GN.
- 3.2. Convergence: Figure 2 compares cumulative images, instances, and pixels across pre-training plus fine-tuning versus scratch schedules, with scratch using three times more COCO iterations.
- 3.2. Convergence: Scratch models are trained longer because they must learn low- and high-level features that pre-training supplies, making sample-count comparisons more informative.
- 3.2. Convergence: The scratch schedule can catch up despite seeing fewer image-level samples because the 1.28 million ImageNet images dominate that count, while pixel-level counts become closer.
4. Experimental Settings
Experiments use minimally modified Mask R-CNN-family baselines with ResNet or ResNeXt FPN backbones, consistent optimization settings, and varied training schedules. Normalization replaces frozen batch-normalization layers for both scratch and pre-trained comparisons.
- The baselines follow publicly available Detectron Mask R-CNN settings, except for normalization choices and the number of training iterations.
- The study evaluates Mask R-CNN with ResNet or ResNeXt plus FPN backbones and trains region proposals jointly with the detector.
- GroupNorm or Synchronized BatchNorm replaces frozen BN layers, and pre-trained comparison models use the same normalization for fairness.
- Training schedules extend Detectron’s 1× and 2× conventions, with a 6× schedule representing 540k iterations and late learning-rate reductions.
- All other settings use Detectron hyper-parameters, including a 0.02 initial learning rate, 0.0001 weight decay, 0.9 momentum, and synchronized SGD on 8 GPUs.
- Testing uses no augmentation, training uses horizontal flipping, and images are resized to 800 pixels on the shorter side unless noted.
5. Results and Analysis
Across COCO detection, segmentation, and keypoint tasks, models trained from random initialization can match or exceed ImageNet-pretrained models after longer training, across architectures, metrics, and enhanced baselines. The advantage of pre-training is mainly faster early convergence, while severe data scarcity and some datasets remain breakdown regimes.
- 5.1. Training from scratch to match accuracy: Models trained from scratch can catch up with fine-tuned detectors when using 5× or 6× schedules, reaching detection AP no worse than pre-trained counterparts.Typical 2× fine-tuning schedules are insufficient for randomly initialized models.
- 5.1. Training from scratch to match accuracy: ImageNet pre-training mainly speeds early convergence on COCO, with little or no evidence of improving final detection accuracy.Randomly initialized models must learn low- and mid-level features that pre-training supplies initially.
- 5.1. Training from scratch to match accuracy: 41.6 mask AP versus 40.9 for R50 and 42.5 versus 41.9 for R101 shows scratch training is slightly better in an enhanced baseline.These comparisons use Mask R-CNN models with stronger baseline configurations.
- 5.1. Training from scratch to match accuracy: 50.9 bbox AP and 43.2 mask AP were achieved by a large ResNeXt-152 model trained from random initialization on COCO.The same model with ImageNet pre-training reached 50.3 bbox AP and 42.5 mask AP.
- 5.1. Training from scratch to match accuracy: 65.6 versus 65.5 keypoint AP under 2× schedules shows random initialization can quickly match pre-training without increasing iterations for keypoint detection.The task is sensitive to fine spatial localization, which is not explicit in classification-based ImageNet pre-training.
6. Discussions
The discussion concludes that training from scratch is often sufficient when target data and computation are adequate, while ImageNet pre-training mainly accelerates convergence and can matter in very small-data regimes. These findings motivate more careful evaluation of universal representations and greater attention to the costs of generic pre-training data.
- Training from scratch is possible without architectural changes, but it requires more iterations to sufficiently converge.
- Training from scratch can be no worse than ImageNet pre-training across many circumstances, including as few as 10k COCO images.
- ImageNet pre-training speeds up convergence on the target task but does not necessarily improve final accuracy.
- ImageNet pre-training does not necessarily reduce overfitting unless the target dataset enters a very small-data regime.
- ImageNet pre-training helps less when the target task is more sensitive to localization than classification.
- The authors argue that target-data annotation may be more useful than generic pre-training data when classification-level pre-training gains diminish and its collection costs are counted.
- The results do not reject universal representations, but suggest evaluating pre-trained features more carefully because random initialization can also produce excellent results.