Source-linked AI summary
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
TL;DR
Rectifier networks had rarely been studied in light of their activation properties, limiting guidance for training powerful deep models. The paper introduces PReLU and a rectifier-aware initialization method, achieving 4.94% top-5 error on ImageNet and surpassing reported human-level performance on the challenge.
Problem
Despite rectifiers’ importance in deep networks, their properties had rarely guided model design or training theory for image classification.
Method
The paper proposes learnable PReLU activations and a rectifier-aware initialization method for training very deep models from scratch.
Results
4.94% top-5 error on ImageNet 2012 is a 26% relative improvement over GoogLeNet’s 6.66% and surpasses reported human-level performance of 5.1%.
Takeaways & Limitations
The result demonstrates that rectifier-based networks can surpass reported human-level performance on the 1000-class ImageNet recognition challenge.
Takeaways & Limitations
The superior result on ImageNet does not establish that machine vision outperforms human vision for object recognition in general.
Abstract
from arXiv · showhide
Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on our PReLU networks (PReLU-nets), we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66%). To our knowledge, our result is the first to surpass human-level performance (5.1%, Russakovsky et al.) on this visual recognition challenge.
1. Introduction
The paper focuses on rectifier-driven neural networks, proposing PReLU and studying the training of very deep rectified models. PReLU-nets achieve 4.94% top-5 ImageNet error, surpassing both the ILSVRC 2014 winner and reported human-level performance.
- Rectifier neurons accelerate convergence and produce better solutions than conventional sigmoid-like units, yet their properties have received limited focused study.
- The paper proposes PReLU, an adaptive generalization of ReLU that learns rectifier parameters while improving accuracy at negligible extra computational cost.For PReLU, the coefficient of the negative part is adaptively learned.
- The authors also investigate why very deep rectified models are difficult to train by explicitly modeling ReLU/PReLU nonlinearities.
- 4.94% top-5 error is achieved by the multi-model PReLU result on ImageNet 2012, a 26% relative improvement over GoogLeNet’s 6.66%.The single-model PReLU-net result is 5.71% top-5 error, surpassing all existing multi-model results; 4.94% also surpasses reported human-level performance of 5.1%.
2. Approach
The approach presents the PReLU activation function, derives an initialization method for deep rectifier networks, and discusses the architecture designs.
- Approach: The section covers PReLU activation, initialization for deep rectifier networks, and architecture designs.These topics correspond to Sections 2.1, 2.2, and 2.3, respectively.
2.1. Parametric Rectifiers
PReLU replaces fixed ReLU slopes with learned, channel-specific negative-side coefficients, improving classification with negligible computational and parameter overhead. Experiments show accuracy gains over ReLU and reveal depth-dependent activation patterns.
- Parametric Rectifiers: PReLU learns the negative-part slope ai, allowing activation functions to vary across channels; ai = 0 recovers ReLU.The equivalent form is f(yi) = max(0, yi) + ai min(0, yi).
- Parametric Rectifiers: PReLU introduces only one learned slope per channel, yielding negligible extra parameters and no expected extra overfitting risk.Its time complexity is negligible during both forward and backward propagation.
- Parametric Rectifiers: 32.64% top-1 error represents a 1.2% gain over the 33.82% ReLU baseline on ImageNet 2012.Channel-wise and channel-shared PReLUs perform comparably, while the shared variant adds only 13 free parameters.
- Parametric Rectifiers: 0.681 and 0.596 are the conv1 learned coefficients, indicating that both positive and negative filter responses are retained.The first-layer filters are mostly Gabor-like edge or texture detectors.
- Parametric Rectifiers: Deeper convolutional layers generally learn smaller channel-wise coefficients, making activations more nonlinear and more discriminative with depth.The model retains more information in earlier stages and becomes more discriminative in deeper stages.
2.2. Initialization of Filter Weights for Rectifiers
The paper derives a ReLU/PReLU-aware initialization by controlling response and gradient variances across layers, avoiding exponentially amplified or diminished signals. This initialization enables extremely deep rectifier models to converge where fixed-variance or Xavier initialization can stall.
- Derivation: The method derives initialization from rectifier-specific response variances because Xavier initialization assumes linear activations, an assumption invalid for ReLU and PReLU.The derivation mainly follows prior variance analysis but accounts for the nonzero mean introduced by ReLU outputs.
- Experimental validation: 30 conv/fc layers converge with the proposed initialization, whereas Xavier cannot make the extremely deep model converge.The 30-layer comparison uses ReLU for both initialization methods, with Xavier completely stalling and its gradients diminishing.
- Forward propagation: The initialization chooses weight variance so the product of layerwise signal-scaling factors remains controlled, and initializes biases to zero.The target is to avoid reducing or magnifying input signals exponentially; the resulting distribution is zero-mean Gaussian.
- Backward propagation: The same variance analysis applies to back-propagated gradients, and either the forward- or backward-derived condition alone is sufficient for convergence in all models tested.The forward and backward conditions differ in connection counts, but properly scaling one signal also preserves the other in common network designs.
- Experimental validation: 1/(1.7 × 10^4) is the gradient magnitude obtained with std 0.01 relative to the derived initialization in the VGG model’s conv10-to-conv2 propagation.For the 10-convolution-layer model, the derived std values are 0.059, 0.042, 0.029, and 0.021 for filter counts 64, 128, 256, and 512, respectively.
2.3. Architectures
The architectures use a modified 19-layer baseline, followed by deeper and wider variants. The authors favor widening convolutional layers because greater depth shows diminishing or degraded accuracy, while augmentation limits overfitting.
- Baseline architecture: Model A is the 19-layer baseline, modifying VGG-19 with a 7×7 first-layer filter, stride 2, and later convolutional layers moved to smaller feature maps.The time complexity is described as roughly unchanged because deeper layers use more filters.
- Model variants: Model B adds three convolutional layers to A, while model C widens B with more filters.C’s increased width substantially raises complexity.
- Model variants: 2.3× of B is the approximate time complexity of wider model C.Training A/B on four K20 GPUs or C on eight K40 GPUs takes about 3-4 weeks.
- Width versus depth: The authors increase width instead of depth because deeper models yield diminishing improvement, saturated accuracy, or degradation in several cited experiments.They argue that increasing convolutional-layer parameters can improve accuracy, motivating wider convolutional layers as a higher-capacity design.
- Regularization: Despite their large size, the models show no severe overfitting, which the authors attribute to aggressive data augmentation throughout training.The passage attributes this outcome to the training procedure rather than reporting a separate quantitative overfitting measure.
3. Implementation Details
The implementation combines scale-jittered, augmented crop training with direct end-to-end optimization of very deep models using the proposed initialization. Evaluation uses dense multi-view, multi-scale feature-map testing, while training employs specified optimization settings and simplified multi-GPU data parallelism.
- Training: Training randomly samples 224×224 crops, jitters the shorter-side scale from 256 to 512, subtracts per-pixel means, and applies horizontal flips and color alteration.Half of random samples are flipped horizontally.
- Training: Scale jittering begins at the start of training, and very deep models are trained directly with Eqn.(14) rather than finetuned from shallower models.The authors suggest end-to-end training may avoid poorer local optima.
- Optimization: Optimization uses weight decay 0.0005, momentum 0.9, 50% dropout in the first two fc layers, mini-batches of 128, and learning rates from 1e-2 to 1e-4 switched at plateaus.Each model is trained for about 80 epochs.
- Testing: Testing applies convolution to resized full images, pools 14×14 feature-map windows with SPP, averages dense-window scores across flips, and combines multiple scales.This extends SPP-net multi-view testing with dense sliding windows.
- Parallel training: 3.8x speedup is observed with 4 GPUs and 6.0x with 8 GPUs using convolutional-layer data parallelism while fc-layer propagation remains on one GPU.The implementation is based on a modification of Caffe and keeps the mini-batch size at 128.
4. Experiments on ImageNet
On ImageNet 2012, PReLU improves top-1 and top-5 error with almost no computational cost, while wider and deeper models achieve strong single- and multi-model results. The final 4.94% top-5 test error surpasses reported human performance on this dataset, though the authors limit the conclusion to this challenge rather than object recognition generally.
- PReLU versus ReLU: PReLU reduces multi-scale top-1 error by 1.05% and top-5 error by 0.23% versus ReLU, with almost no computational cost.The comparison uses channel-wise PReLU on large model A under matched training epochs and learning-rate schedules.
- Single-model results: 7.38% is the best 10-view single-model result, and the authors report that their other models also outperform existing results.Each test view is a 224-crop; the VGG-16 10-view result was obtained by testing its publicly released model.
- Single-model results: 5.71% top-5 error from the best single model, C with PReLU, beats all previous multi-model results; increasing width improves accuracy once models are deep enough.The 19-layer and 22-layer PReLU models perform comparably, while the wider C model improves over B.
- Multi-model results: 4.94% top-5 error on the ImageNet 2012 test set is 1.7% better than GoogLeNet’s 6.66%, a ∼26% relative improvement, and a ∼17% relative improvement over Baidu’s 5.98%.The ILSVRC server evaluated this result because test-set labels are unpublished.