Source-linked AI summary
Convolutional Neural Networks at Constrained Time Cost
Kaiming He, Jian Sun
TL;DR
CNNs are increasingly accurate but costly, creating a need to understand accuracy under constrained training and testing time. The paper uses controlled layer replacements to compare architectures at fixed time complexity and finds a model with 11.8% ImageNet top-5 error that is 20% faster than AlexNet.
Problem
Increasing CNN complexity improves recognition accuracy but makes models costly for practical applications with constrained training or testing time.
Method
The paper progressively modifies a baseline CNN using controlled layer replacements that preserve time complexity while varying depth, width, filter sizes, and strides.
Results
11.8% top-5 error on ImageNet with a 10-view test was achieved by a model that was 20% faster than AlexNet.
Takeaways & Limitations
Fixing time complexity exposes architecture trade-offs and yields models that are fast for practical applications yet more accurate than existing fast models.
Takeaways & Limitations
The deeper models consume more memory than the baseline, leaving constrained-memory CNN design for future investigation.
Abstract
from arXiv · showhide
Though recent advanced convolutional neural networks (CNNs) have been improving the image recognition accuracy, the models are getting more complex and time-consuming. For real-world applications in industrial and commercial scenarios, engineers and developers are often faced with the requirement of constrained time budget. In this paper, we investigate the accuracy of CNNs under constrained time cost. Under this constraint, the designs of the network architectures should exhibit as trade-offs among the factors like depth, numbers of filters, filter sizes, etc. With a series of controlled comparisons, we progressively modify a baseline model while preserving its time complexity. This is also helpful for understanding the importance of the factors in network designs. We present an architecture that achieves very competitive accuracy in the ImageNet dataset (11.8% top-5 error, 10-view test), yet is 20% faster than "AlexNet" (16.0% top-5 error, 10-view test).
1. Introduction
The paper studies CNN accuracy when training and testing time are constrained, treating architecture design as trade-offs among depth, width, filter size, and stride. Controlled layer replacements preserve time cost while revealing depth effects and producing a faster, accurate ImageNet model.
- Motivation: Constrained time budgets make recent CNNs impractical or unnecessary for real-time services, large-scale indexing, and low-power devices.Training can also require high-end GPUs or clusters for one week or several weeks.
- Approach: The study varies depth, width, filter sizes, and strides while preserving time cost through layer replacement.When depth increases, width and/or filter sizes must be reduced to compensate.
- Findings: Deeper networks improve accuracy even when reduced width or filter sizes compensate for time cost.The paper frames this as a controlled comparison rather than an unconstrained complexity increase.
- Findings: Overly increasing depth eventually saturates or degrades accuracy, even when width and filter sizes remain unchanged.The degradation is also visible in training error, so the paper reports that it is not due to over-fitting.
- Result: 11.8% top-5 error on ImageNet with a 10-view test was achieved by a model that was 20% faster than AlexNet.The model also used 40% less complexity and trained for 3 to 4 days on a single GPU.
2. Related Work
Related work accelerates trained CNNs or constrains parameters, while this paper fixes time complexity during architecture investigation. It focuses on untied ImageNet CNNs and additionally studies filter sizes and strides.
- Test-time acceleration: Post-processing methods accelerate CNN testing by approximating or simplifying trained networks, but they can degrade accuracy and do not address training time.Their speedup is also limited when a testing-time budget is given.
- Complexity constraints: Prior work evaluates tied and untied CNNs under varying width, depth, and parameter counts, whereas this paper fixes time complexity.The paper studies untied CNNs trained on ImageNet as generic feature extractors.
- Architecture choices: Inception uses multi-path convolutional branches with varied filter sizes and numbers, while this paper studies single-path designs.The paper notes that the influence of each inception branch remains unclear.
3. Prerequisites
The experiments begin with a narrow eight-layer Overfeat-like ImageNet baseline and construct alternatives under matched convolutional time complexity. The analysis fixes fully connected and pooling dimensions while using theoretical complexity as the design basis.
- Baseline Model: The baseline is an eight-layer Overfeat-like CNN with five convolutional layers, three fully connected layers, ReLUs, and an SPP layer.Its convolutional filters progress from 7×7 and 5×5 to three 3×3 layers, with 64, 128, and 256 filters respectively.
- Baseline Model: The baseline reaches 37.4/15.9 top-1/top-5 error on ImageNet using a 10-view test after 75 epochs.Training takes about 3 days.
- Controlled Designs: New models preserve the baseline’s time complexity while trading off depth, width, filter size, and stride.The baseline is chosen for its common three-stage design and relatively small filter counts.
- Time Complexity: Convolutional time complexity depends on layer depth, filter counts, input channels, filter sizes, and output feature-map sizes.The formulation applies to training and testing, with training time per image roughly three times testing time per image.
- Model Configurations: Table 1 lists constrained-complexity configurations, validation errors, and theoretical complexity relative to baseline A.Its notation also encodes filter sizes, filter counts, strides, repeated layers, and layer sequencing.
- Model Configurations: Table 2 describes variants that delay subsampling by setting pooling strides to 1 and moving the original strides to subsequent convolutional layers.
- Time Complexity: Fully connected and pooling costs are excluded from the comparison and their input/output dimensions are fixed across models.These layers typically account for 5–10% of computational time.
- Time Complexity: Theoretical rather than actual running time is used to design networks because hardware and implementation affect measured runtime.Most models nevertheless scale reasonably with the theoretical complexity.
4. Model Designs by Layer Replacement
The paper designs CNN variants through layer replacement, preserving or approximately preserving time complexity while trading depth, width, and filter sizes. Controlled comparisons show that depth generally has higher priority than filter size or width, but excessive depth can hurt accuracy.
- Design procedure: Layer replacement progressively substitutes a few layers while preserving complexity and leaving the remaining layers unchanged.The study mainly trades factors within stages and fixes each stage’s output-filter count.
- Depth and filter sizes: Replacing larger filters with cascades of smaller filters yields deeper networks that outperform shallower networks at roughly equal time complexity.Models A–E compare depth and filter sizes using 3×3, 2×2, and 5×5 replacements.
- Depth and width: Increasing depth while reducing width produces substantial gains: models F and G achieve 14.8 and 14.7 top-5 error versus 15.9 for model A.Models H and I also have lower error than models C and D.
- Depth and width: The gain from increasing depth saturates, with model G only marginally better than model F.This comparison motivates further investigation of whether excessive depth degrades accuracy.
- Width and filter sizes: Width and filter size show no apparent priority relative to each other under the tested trade-offs.Models B versus F and E versus I have comparable top-1/top-5 errors despite exchanging width and filter sizes.
- Depth limits: Overly increasing depth can worsen accuracy even when width and filter sizes remain unchanged, so reducing them to deepen the network is not always beneficial.The degradation is also observed in Network-in-Network variants, whose added 1×1 layers increase depth and worsen error rates.
5. Implementation Details
The experiments use a common single-GPU implementation, augmentation protocol, optimization schedule, padding scheme, and spatial pyramid pooling design.
- All models are implemented with cuda-convnet2 and run on a single GPU.
- Data Augmentation: Training uses random 224×224 crops from images with shorter side 256, horizontal flips, and color alteration; testing uses 10 cropped views and their flips.
- Settings: Optimization uses mini-batches of 128, staged learning rates, weight decay 0.0005, momentum 0.9, Gaussian weight initialization, zero biases, and 50% dropout in the first two fully connected layers.
- Padding is adjusted across filter sizes and subsampling stages to maintain dominant feature-map sizes of 36×36, 18×18, and 6×6.
- A four-level spatial pyramid pooling layer with 50 total bins is concatenated and fed into the subsequent fully connected layer.
6. Comparisons
The proposed models are compared with fast and accurate ImageNet models under controlled augmentation and single-model 10-view evaluation. They achieve strong accuracy while reducing practical running time relative to several baselines.
- Comparisons use the 1000-category ImageNet 2012 dataset and matching data augmentation to isolate architectural differences.
- Comparisons with Fast Models: 4.2% lower top-5 error and 5.8% lower top-1 error than AlexNet are obtained with 40% less complexity and 20% faster actual running time.
- Comparisons with Fast Models: The actual running-time difference from convolutional complexity is mainly attributed to fully connected and pooling-layer overhead.
- Comparisons with Accurate Models: The proposed model is more accurate than CNN-M, CNN-S, and SPPnet (O5), while SPPnet (O7) is 0.7% better in top-5 error but requires 4.5× the running time.
- VGG-16 is evaluated using its released model, resized-image 10 views with shorter side 384, and Caffe on one K40 GPU because cuda-convnet2 cannot support its scale.
- GoogLeNet has 9.15% top-5 error and 2.1× the proposed model’s convolution complexity, but its measured running time is 7.8× higher on the K40 GPU.
7. Conclusion and Future Work
The paper argues that constrained time cost is both practically important and useful for studying CNN design factors. It identifies memory cost as an additional unresolved practical issue.
- Constrained time cost matters for industrial and commercial requirements and enables controlled analysis of architecture factors.
- The proposed models are fast for practical applications yet more accurate than existing fast models.
- Deeper models consume more memory than the baseline, although smaller test-time mini-batches can partially address the limitation.
- Investigating CNNs under constrained memory cost remains future work.