Source-linked AI summary
ImageNet pre-trained models with batch normalization
Marcel Simon, Erik Rodner, Joachim Denzler
TL;DR
ImageNet-pre-trained CNNs are widely used, but training large architectures and reproducing strong models remains costly. This paper releases Caffe models with batch-normalized AlexNet and VGG19 variants plus residual-network training code; all models outperform prior same-architecture models, including an AlexNet error decrease of over 2.6%.
Problem
ImageNet-pre-trained CNNs support accurate models across computer-vision tasks, while training large models such as VGG19 can take several months on a high-end GPU.
Method
The paper releases Caffe pre-trained batch-normalized AlexNet and VGG19 models, residual networks, and code and files for reproducing training.
Results
All released models outperform previous same-architecture pre-trained models, with AlexNet showing an error decrease of over 2.6%.
Takeaways & Limitations
The release provides directly fine-tunable batch-normalization models and reproducible residual-network training resources for ImageNet-based research.
Takeaways & Limitations
Residual-network training sometimes diverged at random time points and required restarting from the last snapshot with a different random seed.
Abstract
from arXiv · showhide
Convolutional neural networks (CNN) pre-trained on ImageNet are the backbone of most state-of-the-art approaches. In this paper, we present a new set of pre-trained models with popular state-of-the-art architectures for the Caffe framework. The first release includes Residual Networks (ResNets) with generation script as well as the batch-normalization-variants of AlexNet and VGG19. All models outperform previous models with the same architecture. The models and training code are available at http://www.inf-cv.uni-jena.de/Research/CNN+Models.html and https://github.com/cvjena/cnn-models
1. Introduction
CNNs advanced as GPUs increased computational speed and large-scale datasets supplied enough data to train models with over one hundred million parameters. ImageNet pre-training became central to accurate models across many computer-vision tasks, motivating this release of Caffe models.
- Drivers of CNN progress: GPUs and large-scale datasets enabled the recent resurgence of convolutional neural networks.The datasets provide training data for models with more than one hundred million parameters.
- Role of ImageNet: ImageNet pre-training underlies most cited advances in classification, detection, segmentation, captioning, visual question answering, and 3D information prediction.The ILSVRC classification task contains roughly 1.2 million training images across 1,000 categories.
- Paper contribution: The release provides Caffe pre-trained models, including batch-normalized AlexNet and VGG19 variants and reproducible residual-network training code.It also includes files and logs required to reproduce model training.
2. Batch normalization for CNNs
Batch normalization supports training and adaptation of CNNs, especially larger models. The authors report slower initial convergence at higher learning rates but lower final error and direct fine-tuning compatibility.
- Training: Batch normalization is crucial for successful training and convergence of larger models such as VGG19.The authors report this observation specifically for larger architectures.
- Training: Higher learning rates with batch normalization can produce better generalization despite slower initial convergence.The authors observed lower final error for both AlexNet and VGG19.
- Fine-tuning: Adding batch normalization after training can severely increase error because output statistics mismatch.The passage contrasts post hoc addition with models trained using batch normalization.
- Fine-tuning: Fine-tuning the released batch-normalized models is directly possible, supporting adaptation to new tasks.The passage presents this as an advantage over adding batch normalization afterward.
3. Implementation details
The models modify AlexNet and VGG19 with batch normalization, use specified ImageNet training procedures and preprocessing, and address residual-network instability by restarting from snapshots.
- Architecture: Batch normalization was inserted between convolutional or inner-product layers and activation units, replacing local response normalization and dropout.Input mean subtraction was also replaced by batch normalization.
- Architecture: The input batch-normalization layer adaptively calculates the mean during training and fine-tuning, removing manual mean computation and storage.This design also relieves users from manually subtracting the mean during feature computation.
- Training setup: Training used 64 epochs on ILSVRC 2012–2016, batch size 256, architecture-specific initial learning rates, and linear learning-rate decay.The initial learning rates were 0.05 for AlexNet, 0.01 for VGG19, and 0.1 for ResNet.
- Preprocessing: Images were resized with the smaller side at 256 pixels, then randomly cropped during training and center-cropped during validation.Training crops were 224 × 224 for ResNet and VGG19 and 227 × 227 for AlexNet, without color, scale, or aspect-ratio augmentation.
- Training stability: Residual-network training sometimes diverged suddenly, so training was restarted from the last snapshot with a different random seed.The changed image order prevented divergence at the same time point in the reported case.
- Model selection: Final models were provided after full training rather than being selected by validation error or manually adjusted learning rates.The authors specifically avoided intervention associated with step-based learning-rate policies.
4. Results
The released models improve prior same-architecture error rates, with batch normalization benefiting the trained models. AlexNet’s validation error decreases consistently during training, including a late drop caused by linear learning-rate decay.
- Final results: All provided models slightly improve the error rates of previously trained models, while added batch-normalization layers benefit error rates.The reported metrics are single-crop top-1 and top-5 error on the ILSVRC 2012 validation set.
- Final results: AlexNet achieves an error decrease of over 2.6% relative to the referenced previous model.The passage reports this as the AlexNet example accompanying Table 1.
- Training curve: AlexNet’s single-crop top-1 validation error decreases consistently and fairly quickly during training.The figure tracks error against training time.
- Training curve: The steep error decrease near training’s end follows the linear learning-rate decay, not evidence that further improvement remains available.The learning rate approaches zero, and constant-learning-rate experiments showed no additional improvement.
5. Conclusions
The paper introduces ImageNet pre-trained Caffe models centered on batch-normalized AlexNet and VGG19 and residual networks. These models outperform previous pre-trained models, while residual-network ImageNet results are reproduced.
- The release provides batch-normalization variants of AlexNet and VGG19 for ImageNet in Caffe.
- It includes residual-network models and reproduces their ImageNet results.
- All released models outperform previous pre-trained models.