Source-linked AI summary
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alex Alemi
TL;DR
The paper asks whether residual connections can improve deep Inception networks while preserving their efficiency. It introduces residual and streamlined Inception variants, finding faster training and competitive recognition, including 3.1% top-5 validation error from a four-model ensemble.
Problem
The paper examines whether residual connections benefit very deep Inception networks while retaining their computational efficiency.
Method
The authors combine Inception with residual connections, design streamlined Inception-v4 architectures, and scale residuals to stabilize very wide networks.
Results
Residual connections dramatically improve Inception training speed; newer models outperform previous networks, while a four-model ensemble achieves 3.1% top-5 validation error.
Takeaways & Limitations
Residual Inception networks provide faster training and strong recognition performance, with Inception-ResNet-v2 significantly improving recognition and Inception-v4 performing similarly.
Takeaways & Limitations
Residual variants became unstable and effectively stopped learning when the number of filters exceeded 1000.
Abstract
from arXiv · showhide
Very deep convolutional networks have been central to the largest advances in image recognition performance in recent years. One example is the Inception architecture that has been shown to achieve very good performance at relatively low computational cost. Recently, the introduction of residual connections in conjunction with a more traditional architecture has yielded state-of-the-art performance in the 2015 ILSVRC challenge; its performance was similar to the latest generation Inception-v3 network. This raises the question of whether there are any benefit in combining the Inception architecture with residual connections. Here we give clear empirical evidence that training with residual connections accelerates the training of Inception networks significantly. There is also some evidence of residual Inception networks outperforming similarly expensive Inception networks without residual connections by a thin margin. We also present several new streamlined architectures for both residual and non-residual Inception networks. These variations improve the single-frame recognition performance on the ILSVRC 2012 classification task significantly. We further demonstrate how proper activation scaling stabilizes the training of very wide residual Inception networks. With an ensemble of three residual and one Inception-v4, we achieve 3.08 percent top-5 error on the test set of the ImageNet classification (CLS) challenge
1. Introduction
This work studies combining residual connections with Inception networks and introduces Inception-v4, a deeper and wider streamlined architecture. It compares pure and hybrid variants, evaluates an ensemble, and finds remaining room for improvement because predictions have not reached the dataset’s annotation label noise.
- Motivation: The study combines residual connections with the revised Inception architecture to address training very deep networks.Because Inception networks tend to be very deep, the authors propose replacing filter concatenation with residual connections.
- Architecture: Inception-v4 is introduced as a more uniform, simplified architecture with greater depth and width than Inception-v3.The design removes historical baggage inherited from earlier Inception versions.
- Experiments: The experiments compare Inception-v3 and Inception-v4 with similarly expensive Inception-ResNet variants.Models were selected with approximately similar parameter counts and computational complexity to their non-residual counterparts.
- Ensemble evaluation: An ensemble of the best-performing models is evaluated because Inception-v4 and Inception-ResNet-v2 achieved similarly strong single-frame ImageNet validation performance.The experiment tests whether combining these models can further advance performance on the dataset.
- Limitations: The ensemble remains above the annotation label-noise level, indicating room for improving predictions.The authors report that gains in single-frame performance do not translate into similarly large gains for the ensemble.
2. Related Work
Large-scale image recognition was shaped by milestones including Network-in-Network, VGGNet, and GoogLeNet. Inception was subsequently refined through batch normalization and factorization, while residual connections were proposed for training very deep convolutional models.
- 2. Related Work: Large-scale image recognition advanced through Network-in-Network, VGGNet, and GoogLeNet (Inception-v1).These followed the growing popularity of convolutional networks after Krizhevsky et al.
- 2. Related Work: Residual connections introduced additive signal merging and were argued to be inherently necessary for training very deep convolutional models.He et al. provided theoretical and practical evidence for benefits in image recognition and especially object detection.
- 2. Related Work: Inception-v1 was refined with batch normalization into Inception-v2 and with additional factorization ideas into Inception-v3.The cited work attributes these refinements to Ioffe et al. and later architectural improvements.
3. Architectural Choices
The paper simplifies Inception-v4’s architectural choices and defines residual variants with cheaper blocks, dimensionality-expansion layers, and cost targets aligned with Inception-v3 and Inception-v4. Very wide residual networks require residual scaling to avoid training instability, with scaling factors of 0.1–0.3 stabilizing training without harming final accuracy.
- Inception-v4 architectural simplification: Inception-v4 uses more uniform Inception-block choices after earlier designs retained unnecessary architectural complexity.The authors attribute this change to shedding unnecessary baggage from prior conservative experiments.
- Residual Inception blocks: Residual Inception networks use cheaper blocks followed by 1 × 1 filter-expansion layers without activation to match input depth before addition.The expansion compensates for dimensionality reduction induced by the Inception block.
- Residual variants: Inception-ResNet-v1 roughly matches Inception-v3’s computational cost, while Inception-ResNet-v2 matches the raw cost of Inception-v4.In practice, Inception-v4 had significantly slower step time, probably because it used more layers.
- Residual scaling: When residual networks exceeded 1000 filters, training became unstable and the network could die early by producing only zeros before average pooling.Lowering the learning rate or adding batch normalization to that layer did not prevent the instability.
- Residual scaling: Scaling residuals by factors between 0.1 and 0.3 before addition stabilized training in very wide residual networks.The authors found scaling preferable to relying on very low learning rates, which could remain insufficient at very high filter counts.
- Residual scaling: Residual scaling helped stabilize training and did not appear to harm final accuracy even when it was not strictly necessary.This supports using scaling beyond only the cases where instability is observed.
4. Training Methodology
The networks were trained with distributed stochastic gradient methods across 20 GPUs, using RMSProp with scheduled learning-rate decay and parameter-averaged evaluation. Compared with similarly costly Inception-v3, the residual model trained faster but achieved slightly worse final accuracy.
- Distributed training: Training used stochastic gradient optimization in TensorFlow with 20 replicas, each running on an NVIDIA Kepler GPU.Earlier experiments used momentum with decay 0.9.
- Training comparison: The residual network trained much faster than similarly costly Inception-v3 but reached slightly worse final accuracy.Evaluation used a single crop on non-blacklist images from the ILSVRC-2012 validation set.
- Optimization schedule: RMSProp used decay 0.9, ϵ = 1.0, learning rate 0.045, and exponential 0.94 decay every two epochs.Model evaluations used a running average of parameters computed over time.
5. Experimental Results
Residual Inception models trained substantially faster than similarly costly pure Inception models and achieved slightly better final validation performance. Across larger models, final accuracy appeared primarily dependent on model size, while the ensemble achieved 3.08% top-5 test error.
- Evaluation caveat: Validation comparisons used a non-blacklisted subset that omitted about 1700 entities, producing somewhat incomparable and more optimistic numbers than other reports.The omission was intended only for the CLSLOC benchmark, not these broader comparisons.
- Training and validation comparisons: Residual Inception trained much faster and reached slightly better final recall than similarly costly pure Inception-v3.The comparison used single-crop evaluation on non-blacklisted ILSVRC 2012 validation images.
- Training and validation comparisons: Residual Inception trained much faster and reached slightly better final accuracy than traditional Inception-v4.This comparison also used similarly costly models and single-crop validation evaluation.
- Training and validation comparisons: Final accuracy seemed to depend mainly on model size, although residual models converged faster across the four-model comparison.Top-1 and top-5 error evolutions showed similar behavior.
- Ensemble results: 3.08% top-5 error was achieved by an ensemble of one pure Inception-v4 and three Inception-ResNet-v2 models on the ILSVRC test set.The ensemble used 144 crops/dense evaluation, and the test result was used to verify that the models did not overfit the validation set.
6. Conclusions
The paper presents three new architectures: two hybrid Inception-ResNet variants and the pure Inception-v4. Residual connections dramatically improve Inception training speed, while the latest larger models outperform previous networks.
- The authors present three new network architectures in detail.
- Inception-ResNet-v1 is a hybrid Inception model with computational cost similar to Inception-v3.
- Inception-ResNet-v2 is a costlier hybrid Inception model with significantly improved recognition performance.
- Inception-v4 is a pure Inception model without residual connections, with roughly the same recognition performance as Inception-ResNet-v2.
- Residual connections dramatically improve Inception training speed, and the latest larger models outperform all previous networks.The performance improvement is attributed to increased model size.