Source-linked AI summary
Rethinking the Inception Architecture for Computer Vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna
TL;DR
Computer-vision networks benefit from increased depth and computation, but efficiency and low parameter count remain important for mobile and big-data settings. The paper scales Inception using factorized convolutions and regularization, achieving state-of-the-art ILSVRC 2012 classification results with modest computational and parameter costs.
Problem
High-performing convolutional networks often require substantial computation, while efficiently scaling Inception is difficult because its architecture is complex and computational budgets are constrained.
Method
The paper develops design principles for scaling Inception through factorized convolutions, dimension reduction, and label smoothing regularization.
Results
21.2% top-1 and 5.6% top-5 error were achieved for single-crop ILSVRC 2012 classification with Inception-v3 at a 2.5× computational increase over the cited prior network.
Takeaways & Limitations
The design guidance produces high-performance vision networks with relatively modest computation compared with simpler, more monolithic architectures.
Takeaways & Limitations
The lower-resolution evaluation warns that naively shrinking the network creates an unfair comparison because it yields a 16 times cheaper model for a harder task.
Abstract
from arXiv · showhide
Convolutional networks are at the core of most state-of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we explore ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21.2% top-1 and 5.6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3.5% top-5 error on the validation set (3.6% error on the test set) and 17.3% top-1 error on the validation set.
1. Introduction
Deep and wide convolutional networks improved vision performance, but their computational and architectural costs remain important constraints. Inception offers lower computation and parameter count, yet its complexity makes principled scaling difficult.
- Since 2014, deeper and wider convolutional architectures substantially improved classification performance and transferred gains across computer vision tasks.
- VGGNet achieves architectural simplicity at high computational cost, whereas Inception was designed for constrained memory and computation.
- 5 million parameters made GoogLeNet 12× smaller than AlexNet, which used 60 million parameters.
- Inception’s lower computational cost enabled big-data and mobile-vision use cases where processing or hardware capacity is limited.
- Inception’s complexity makes modifications difficult, and naively doubling all filter-bank sizes causes a 4x increase in computation.
2. General Design Principles
The paper proposes design principles for scaling convolutional networks efficiently: preserve useful representations, process high-dimensional features locally, reduce dimensions before spatial aggregation, and balance width with depth. These principles remain guidance rather than an automatic recipe.
- The principles are speculative and should be applied judiciously because they are not straightforward to use as an out-of-the-box improvement method.
- The authors advise avoiding extreme representational bottlenecks, especially early in the network, while gently reducing representation size toward the output.
- Higher-dimensional representations are easier to process locally because more activations per tile allow more disentangled features and faster training.
- Spatial aggregation can use lower-dimensional embeddings with little representational loss because adjacent activations are strongly correlated.
- For a fixed computation budget, performance improves most when network width and depth increase in parallel rather than independently.
3. Factorizing Convolutions with Large Filter Size
The paper factorizes expensive spatial convolutions into smaller or asymmetric convolutional sequences to reduce computation while retaining useful receptive fields. Experiments support ReLU-based factorization and show that asymmetric factorization is especially effective on medium-sized grids.
- Dimension reduction before spatial aggregation exploits correlations among nearby activations and can preserve expressive local representations.
- Factorizing convolutions reduces computation and parameters, enabling larger filter banks while retaining trainability on a single computer.
- Factorization into smaller convolutions: A 5 × 5 convolution costs 2.78 times as much computation as a 3 × 3 convolution with the same filter counts.
- Factorization into smaller convolutions: In control experiments, factorization with linear plus ReLU layers reached 76.2% top-1 accuracy, versus 77.2% with two ReLU layers after 3.86 million operations.
- Factorization into smaller convolutions: Replacing a 5 × 5 convolution with two 3 × 3 convolutions yields a 28% relative computational gain and the same parameter-count saving under the stated assumptions.
- Spatial Factorization into Asymmetric Convolutions: A 3 × 1 convolution followed by 1 × 3 convolution matches a 3 × 3 receptive field while costing 33% less when input and output filter counts are equal.
- Spatial Factorization into Asymmetric Convolutions: Asymmetric factorization performs poorly in early layers but gives very good results on medium grids, including 1 × 7 followed by 7 × 1 convolutions.
4. Utility of Auxiliary Classifiers
Auxiliary classifiers did not improve early training convergence, but later produced a slightly higher accuracy plateau and appear to function primarily as regularizers.
- Auxiliary classifiers did not improve convergence early in training, with or without side heads showing virtually identical progression before high accuracy.
- Near the end of training, networks with auxiliary branches overtook networks without them and reached a slightly higher accuracy plateau.
- Removing the lower auxiliary branch had no adverse effect on the network’s final quality.
- The authors argue that auxiliary classifiers act as regularizers rather than primarily improving low-level feature learning.Batch normalization or dropout in the side branch further improved the main classifier, providing supporting evidence for this interpretation.
5. Efficient Grid Size Reduction
The proposed grid-size reduction avoids representational bottlenecks while reducing computation by combining parallel pooling and convolutional stride-2 paths.
- Pooling after expanding filter dimensions is computationally expensive because the convolution operates on the larger grid.The described cost is dominated by the stride-1 convolution before pooling.
- Expanded filter-bank outputs are used only on the coarsest 8 × 8 grids, where high-dimensional sparse representations are most critical.At that stage, local processing by 1 × 1 convolutions occupies a larger ratio relative to spatial aggregation.
- Reducing grid size through pooling with convolution lowers computation but creates a representational bottleneck and less expressive networks.
- The proposed alternative uses two parallel stride-2 blocks, P and C, whose filter-bank outputs are concatenated.P is a pooling layer, while both branches use stride 2.
- The grid-reduction module is designed to reduce computational cost while avoiding the representational bottleneck.
6. Inception-v2
Inception-v2 combines factorized convolutions, grid reduction, and balanced filter-bank design into a 42-layer architecture that improves performance while remaining computationally efficient.
- The proposed network uses factorized convolutions and grid-reduction modules across 35 × 35, 17 × 17, and 8 × 8 stages.It includes three traditional Inception modules, five factorized modules, and two coarsest-grid Inception modules.
- The 7 × 7 convolution is factorized into three 3 × 3 convolutions, while filter-bank sizes follow the balanced-width-and-depth principle.
- 42 layers require only about 2.5 times GoogLeNet’s computation and remain substantially more efficient than VGGNet.
- Network quality remains relatively stable across architectural variations when the stated design principles are observed.
7. Model Regularization via Label Smoothing
The paper introduces label-smoothing regularization, which replaces one-hot ground-truth labels with a mixture involving a fixed prior to discourage excessive confidence. On ImageNet, this produced a consistent absolute improvement in both top-1 and top-5 error.
- Motivation: The method is motivated by over-confident predictions, which can cause over-fitting when the model assigns nearly all probability to the ground-truth label.The proposed mechanism encourages the model to be less confident and thereby regularizes it.
- Mechanism: Label-smoothing regularization replaces the one-hot ground-truth distribution with a mixture of that distribution and a fixed label distribution.The mixture weights are 1 − ϵ and ϵ, respectively, and the paper uses the label prior as the fixed distribution.
- Effect: Label smoothing prevents the largest logit from becoming much larger than the others because every smoothed target probability has a positive lower bound.This makes an extremely concentrated prediction incur large cross-entropy with the smoothed target distribution.
- Objective: With a uniform prior u(k) = 1/K, label smoothing adds a loss term that penalizes deviation of the predicted distribution from that prior.The added term has relative weight ϵ/(1−ϵ) and, for a uniform prior, measures dissimilarity from uniform.
- Results: 0.2% absolute improvement was observed consistently for both top-1 error and top-5 error on ILSVRC 2012 with K = 1000 and ϵ = 0.1.The experiments used u(k) = 1/1000.
8. Training Methodology
The networks were trained with distributed stochastic-gradient optimization across 50 GPU replicas. The best models used RMSProp with specified decay, learning-rate scheduling, and gradient clipping.
- Training setup: Training used stochastic gradient optimization in TensorFlow with 50 replicas, each running on an Nvidia Kepler GPU with batch size 32 for 100 epochs.The setup distributed training across multiple GPU replicas.
- Optimizer: The best models used RMSProp with decay 0.9 and ϵ = 1.0, rather than the earlier momentum-based experiments.Earlier experiments used momentum with decay 0.9.
- Learning-rate schedule: The learning rate was 0.045 and was exponentially decayed every two epochs at rate 0.94.This schedule was part of the reported training configuration.
- Optimization controls: Gradient clipping was included in the training procedure.The supplied passage reports clipping but truncates its threshold.
9. Performance on Lower Resolution Input
The study compares receptive-field resolution while holding computational cost nearly constant. Lower-resolution networks achieved final quality close to higher-resolution counterparts, whereas naively shrinking network size performed much worse.
- Experimental design: Three networks with 299 × 299, 151 × 151, and 79 × 79 receptive fields were evaluated under nearly identical computational cost.The configurations varied stride and first-layer pooling: stride 2 with pooling, stride 1 with pooling, and stride 1 without pooling, respectively.
- Evaluation: The comparison measures recognition performance on the ILSVRC 2012 validation set after training each network until convergence.The table is described as comparing receptive-field size while computational cost remains constant.
- Result: Lower-resolution networks took longer to train, but their final quality was quite close to that of higher-resolution counterparts.The reported quality comparison concerns the converged validation results.
- Caveat: Naively reducing network size according to input resolution caused much poorer performance.The paper characterizes that comparison as unfair because it contrasts a 16-times-cheaper model with a more difficult task.
- Application: The results suggest that dedicated high-cost, low-resolution networks may be considered for smaller objects in the R-CNN context.This proposed use follows the reported lower-resolution recognition results.
10. Experimental Results and Comparisons
The experiments evaluate cumulative architectural changes to Inception-v2 and the resulting Inception-v3 model across single-crop, multi-crop, and ensemble settings on ILSVRC-2012 validation data.
- Cumulative architectural changes: Inception-v2 results report cumulative effects, with each successive line adding a new modification to earlier changes.The tested changes include label smoothing, factorizing the first 7 × 7 convolution into sequential 3 × 3 convolutions, and batch-normalizing the auxiliary classifier’s fully connected layer.
- Model evaluation: Inception-v3 denotes the final model containing all listed cumulative changes and is evaluated with multi-crop and ensemble methods.The last row of Table 3 is identified as Inception-v3 before those broader evaluations.
- Evaluation setting: 48238 nonblacklisted ILSVRC-2012 validation examples were used for evaluation.Using all 50000 examples produced roughly 0.1% worse top-5 error and around 0.2% worse top-1 error.
- Ensemble comparison: 3.46% top-5 error was achieved by the ensemble on the validation set under multi-model, multi-crop evaluation.Table 5 compares ensemble results with published ILSVRC-2012 ensemble inference results; other reported results are on the validation set.
11. Conclusions
The paper proposes design principles for scaling convolutional networks and studies them in Inception, achieving high classification accuracy with relatively modest computational cost.
- Design principles: The proposed design principles scale convolutional networks while maintaining relatively modest computation compared with simpler, more monolithic architectures.They are studied specifically in the context of the Inception architecture.
- Single-crop results: 21.2% top-1 and 5.6% top-5 error were obtained by Inception-v3 with single-crop evaluation on ILSVRC 2012 classification.This required a 2.5× computational-cost increase compared with the network described by Ioffe et al.
- Efficiency comparison: 25% and 14% relative reductions in top-5 and top-1 error, respectively, were reported against He et al., while using six times less computation and at least five times fewer parameters.The comparison concerns the best published results based on denser networks.
- Ensemble results: 3.5% top-5 error was reached by an ensemble of four Inception-v3 models with multi-crop evaluation.This represented an over 25% reduction relative to the best published results and was almost half the error of the ILSVRC 2014 GoogLeNet ensemble.