Source-linked AI summary
Going Deeper with Convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich
TL;DR
Efficiently scaling convolutional networks is difficult because uniformly increasing size can waste finite computational resources. The paper introduces Inception, whose dense modules approximate sparse structure, and reports state-of-the-art classification and detection performance, including 6.67% top-5 error in ILSVRC 2014.
Problem
Uniformly enlarging convolutional networks can waste finite computational resources, motivating more efficient allocation of model capacity.
Method
Inception uses dimension-reducing 1×1 convolutions and a new multi-scale module to increase network depth and width within a fixed computational budget.
Results
GoogLeNet achieved state-of-the-art classification and detection performance, including a 6.67% top-5 error and first-place ranking in ILSVRC 2014 classification.
Takeaways & Limitations
The results support approximating optimal sparse structures with dense building blocks as a viable way to improve computer-vision networks efficiently.
Takeaways & Limitations
It remains uncertain whether Inception’s gains arise from its guiding design principles, requiring broader analysis and verification.
Abstract
from arXiv · showhide
We propose a deep convolutional neural network architecture codenamed "Inception", which was responsible for setting the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC 2014). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. This was achieved by a carefully crafted design that allows for increasing the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC 2014 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.
1 Introduction
The paper introduces Inception, an efficient deep neural network architecture for computer vision, motivated by advances in convolutional networks and the growing importance of computational efficiency. Its design considers both accuracy and resource use, with experiments targeting a 1.5-billion multiply-add inference budget.
- Recent progress in image recognition and object detection has been driven mainly by new ideas, algorithms, and network architectures rather than only hardware, data, or model scale.
- 1.5 billion multiply-adds at inference was the computational budget used for most experiments, reflecting concern for algorithmic power and memory efficiency.
- The paper focuses on Inception, an efficient computer-vision architecture that introduces the Inception module as a new organizational level while increasing network depth.
2 Related Work
Prior CNNs generally used stacked convolutional and pooling layers followed by fully connected layers, achieving strong results in image classification and related vision tasks. Inception builds on multi-scale processing and Network-in-Network ideas while relating to established detection approaches such as R-CNN.
- Convolutional neural networks: CNNs typically stack convolutional layers, optional contrast normalization and max-pooling, then one or more fully connected layers.This design has produced leading results on MNIST, CIFAR, and ImageNet classification.
- Applications and multi-scale processing: Despite concerns about spatial-information loss from max-pooling, similar CNN architectures have supported localization, object detection, and human pose estimation.These applications include the approaches cited as [14] for localization, [6] [14] [5] for detection, and [19] for pose estimation.
- Applications and multi-scale processing: Serre et al. used fixed Gabor filters of different sizes to handle multiple scales, similarly to the Inception model.The approach was inspired by a neuroscience model of the primate visual cortex.
- Network-in-Network: Network-in-Network increases neural-network representational power through additional 1×1 convolutional layers, typically followed by rectified linear activation.The method integrates readily into existing CNN pipelines and is used heavily in the Inception architecture.
- Object detection: R-CNN frames object detection as two stages: category-agnostic object-proposal generation from low-level cues, followed by CNN-based category classification at proposed locations.This decomposition leverages CNN accuracy for identifying object categories at candidate locations.
3 Motivation and High Level Considerations
Increasing network depth and width improves model quality but causes overfitting and rapidly rising computational costs. Inception explores an intermediate design that approximates sparse connectivity using dense components, while acknowledging limitations in current hardware and uncertainty about the guiding principles.
- Motivation: Increasing network depth and width is a straightforward way to improve deep neural network performance, especially with abundant labeled data.The enlarged network increases both the number of levels and the units at each level.
- Motivation: Larger networks contain more parameters, increasing overfitting risk when labeled training examples are limited.High-quality training sets can also be difficult and expensive to create, particularly for fine-grained visual categories.
- Motivation: Uniformly increasing network size sharply raises computation: expanding filters in two chained convolutional layers produces a quadratic increase.Inefficiently used capacity can waste computation, while the available computational budget remains finite.
- Motivation: Sparse connectivity could address both overfitting and computational waste, but current hardware and numerical libraries favor dense matrix multiplication.Even a 100× reduction in arithmetic operations may not compensate for lookup and cache-miss overhead on sparse data structures.
- Motivation: Inception investigates a practical intermediate approach that approximates sparse topologies with dense components, although its gains do not yet establish that the guiding principles caused its success.The architecture began as a speculative case study and showed modest gains after two topology iterations, while stronger automated verification was still needed.
4 Architectural Details
Inception approximates optimal local sparse structures with repeated dense convolutional modules, using dimension reductions to control computational cost. The design supports wider and deeper networks while enabling controlled tradeoffs between computational resources and performance.
- Inception modules: Inception modules approximate optimal local sparse structures using repeated convolutional building blocks under translation invariance.The architecture searches for an effective local construction and repeats it spatially.
- Inception modules: 3×3 and 5×5 convolutions should become more prevalent in higher layers as feature abstraction increases and spatial concentration decreases.The expected change in output correlation statistics motivates increasing these convolution ratios across layers.
- Computational control: Dimension reductions and projections prevent the computational blow-up caused by applying large convolutions and pooling to layers with many filters.The reductions use compressed representations while retaining information from relatively large image patches.
- Network organization: Inception networks stack these modules with occasional stride-2 max-pooling, while traditionally convolutional lower layers can improve training memory efficiency.Using Inception modules only at higher layers is an infrastructure-driven choice rather than a strict architectural requirement.
- Computational control: Dimension reduction allows substantially more units at each stage without uncontrolled computational growth by shielding later layers from large input-filter counts.Input dimensions are reduced before applying convolutions with large patch sizes.
- Computational tradeoffs: 2−3× faster networks can be built by balancing computational resources through Inception’s adjustable design choices, albeit with slightly inferior performance.The architecture also increases stage width and depth without computational difficulties.
5 GoogLeNet
GoogLeNet is the successful Inception incarnation used in the ILSVRC14 submission, designed for computational efficiency and practical inference. Its 22 parameterized layers include auxiliary classifiers to support gradient propagation and intermediate-feature discrimination.
- Architecture and ensemble: GoogLeNet denotes the most successful particular Inception instance used in the ILSVRC14 submission, with the same topology used for 6 of 7 ensemble models.A deeper and wider Inception network was also included in the ensemble, although its individual quality was slightly inferior.
- Architecture and efficiency: 22 layers with parameters, or 27 including pooling, make GoogLeNet substantially deep while targeting computational efficiency, practicality, and low-memory inference on individual devices.The construction uses about 100 independent building blocks, although that count depends on the machine learning system.
- Auxiliary classifiers: Auxiliary classifiers attached to intermediate layers were added to encourage discrimination in lower stages and improve effective gradient propagation through the deep network.The motivation was that middle-layer features should be highly discriminative, based on the strong performance of relatively shallower networks.
- Auxiliary classifiers: Each auxiliary classifier begins with 5×5 average pooling at stride 3, producing 4×4×512 or 4×4×528 outputs for stages (4a) and (4d), respectively.It then applies a 1×1 convolution with 128 filters, a 1024-unit fully connected layer, dropout, and a softmax classifier.
6 Training Methodology
GoogLeNet was trained with DistBelief using modest model and data parallelism, asynchronous stochastic gradient descent, and a 0.9 momentum setting. Training practices varied across models, crops, and hyperparameters, making it difficult to identify one definitive best procedure.
- Distributed training: Training used DistBelief with modest model and data parallelism, despite relying on a CPU-based implementation.The authors estimated that convergence could be reached with a few high-end GPUs within a week, with memory usage as the main limitation.
- Optimization: 0.9 momentum was used with asynchronous stochastic gradient descent and a fixed schedule that decreased the learning rate.The supplied passage specifies the momentum value and learning-rate schedule but truncates the schedule’s reduction factor.
- Training variation: Sampling methods, crop sizes, dropout, and learning rates varied across models, so no single training method could be identified as definitively most effective.Some models used smaller relative crops while others used larger crops inspired by prior work.
7 ILSVRC 2014 Classification Challenge Setup and Results
GoogLeNet’s ILSVRC 2014 submission used ensemble prediction, aggressive multi-scale cropping, and no external training data. It achieved a 6.67% top-5 error on both validation and testing data, ranking first among participants.
- Challenge setup: The challenge classified images into 1000 ImageNet leaf-node categories using about 1.2 million training, 50,000 validation, and 100,000 test images.Each image had one ground-truth category, and performance was measured from the highest-scoring classifier predictions.
- Submission strategy: The submission used no external training data and applied additional testing techniques to improve performance.The paper analyzes multiple factors contributing to the final submission’s performance.
- Submission strategy: Seven independently trained GoogLeNet versions, including one wider model, were combined through ensemble prediction.The models shared initialization and learning-rate policies but differed in sampling methodologies and input-image order.
- Submission strategy: The testing procedure generated 144 crops per image from four resize scales, three spatial squares, six crop positions, and mirrored versions.The authors note that aggressive cropping may be unnecessary in real applications because additional crops provide marginal benefit after a reasonable number.
- Results: 6.67% top-5 error was achieved on both validation and testing data, ranking first among participants.This represented a 56.5% relative reduction versus SuperVision in 2012 and about 40% versus the previous year’s best approach, Clarifai.
- Results analysis: Testing choices were evaluated by varying the number of models and crops, with results reported on the validation dataset to avoid overfitting testing-data statistics.For one-model evaluations, the model with the lowest validation top-1 error was selected.
8 ILSVRC 2014 Detection Challenge Setup and Results
The ILSVRC 2014 detection task evaluated object bounding boxes across 200 classes using overlap-based correctness and mean average precision. GoogLeNet used an Inception-based R-CNN-style detector with improved region proposals, while results showed rapid progress and strong convolutional-network performance.
- Task setup: Detection required correctly classified bounding boxes for 200 possible object classes, with at least 50% Jaccard overlap; false positives were penalized.Unlike classification, images could contain multiple objects or none, with object scales ranging from large to tiny.
- Detection approach: GoogLeNet’s detector followed R-CNN and used Inception as the region classifier.Its proposal stage combined Selective Search with multi-box predictions to improve bounding-box recall.
- Detection approach: 2× larger superpixels halved Selective Search proposals to reduce false positives.The approach increased superpixel size as part of its proposal-processing strategy.
- Challenge results: Accuracy had almost doubled compared with the 2013 result, and the top-performing teams all used Convolutional Networks.Official scores and team strategies were reported in Table 4, including external data, ensemble models, and contextual models.
- Challenge results: The top single-model result improved by only 0.3 points with an ensemble of 3 models, while GoogLeNet achieved significantly stronger ensemble results.Table 5 compared detection performance using a single model only before discussing ensemble effects.
9 Conclusions
The results support approximating optimal sparse structures with available dense building blocks as a viable way to improve computer-vision neural networks. Inception achieved quality gains with modestly higher computational requirements and suggests that sparser architectures are feasible and useful.
- Conclusions: The results support approximating expected optimal sparse structures with readily available dense building blocks to improve computer-vision neural networks.The authors identify this approach as a viable method for network improvement.
- Conclusions: Inception provided significant quality gains with a modest increase in computational requirements over shallower and less wide networks.The stated advantage is improved quality at modest additional computational cost.
- Conclusions: The detection results were competitive despite using neither context nor bounding-box regression, further supporting Inception’s strength.The passage presents these omitted components as evidence for the architecture’s effectiveness.
- Conclusions: The approach suggests that moving toward sparser architectures is feasible and useful, motivating automated creation of sparser, more refined structures.The authors describe this as a direction for future work based on [2].