Source-linked AI summary
Training Deeper Convolutional Networks with Deep Supervision
Liwei Wang, Chen-Yu Lee, Zhuowen Tu, Svetlana Lazebnik
TL;DR
Deeper CNNs improve accuracy but are harder and more expensive to train. The paper adds auxiliary supervision branches at selected intermediate layers using a gradient-based placement heuristic. Deep supervision improves classification results on ImageNet and MIT Places, while the approach has reported scope and setting limitations relative to VGG models.
Problem
Increasing CNN depth can improve accuracy, but adding layers makes training more difficult and computationally expensive.
Method
The method adds auxiliary neural-network supervision branches at selected intermediate convolutional layers, placing them where mean gradients vanish during early training.
Results
Deep supervision improves classification across ImageNet and MIT Places: CNDS-8 is 1% more accurate than CNN-8 on ImageNet and about 1% better on Places.
Takeaways & Limitations
Deep supervision provides a less complex, faster-training alternative to iterative pre-training of progressively deeper networks and can support strong Places performance with faster feature extraction than GoogleNet.
Takeaways & Limitations
The models remain less accurate than same-depth VGG models, with differences attributed mainly to network settings such as stride, filter size, and training scale.
Abstract
from arXiv · showhide
One of the most promising ways of improving the performance of deep convolutional neural networks is by increasing the number of convolutional layers. However, adding layers makes training more difficult and computationally expensive. In order to train deeper networks, we propose to add auxiliary supervision branches after certain intermediate layers during training. We formulate a simple rule of thumb to determine where these branches should be added. The resulting deeply supervised structure makes the training much easier and also produces better classification results on ImageNet and the recently released, larger MIT Places dataset
1. Introduction
Increasing CNN depth and width can improve accuracy, but larger networks are slower to train and more prone to overfitting. This work addresses deeper-network training with intermediate auxiliary supervision and evaluates the approach on ImageNet and MIT Places.
- Increasing network depth and width can improve CNN accuracy, but larger models introduce more parameters, slower back-propagation convergence, and greater overfitting risk.
- The proposed method adds auxiliary classifiers after selected intermediate convolutional layers to train deeper networks more efficiently.
- A simple rule of thumb determines branch placement using vanishing-gradient analysis in deep networks.
- Experiments use 8- and 13-convolutional-layer models, deeper than AlexNet’s 5 convolutional layers but shallower than the 16- and 21-layer networks of prior work.
- Results on ImageNet and MIT Places show deeper models are more accurate, while deep supervision offers similar MIT Places accuracy to GoogleNet with faster feature extraction.
2. Our Method
The method trains deeper convolutional networks by adding auxiliary classifiers at selected intermediate layers, using gradient behavior to choose their locations. These branches augment the final-layer loss during training and are applied to 8- and 13-layer models.
- Method design: CNDS adds auxiliary classifiers after selected intermediate convolutional layers instead of supervising only the final layer.Unlike deeply-supervised nets that add supervision after every hidden layer, CNDS uses a gradient-based placement heuristic.
- Auxiliary branch: The auxiliary branch uses dimensionality reduction and discriminative nonlinear mapping before classification because lower-layer feature maps are noisy.Its classifier is a small neural network with a convolutional layer, fully connected layers, and a softmax classifier.
- Branch placement: The placement rule adds supervision after the layer whose mean gradient vanishes, operationalized as a value below 10^-7.The authors estimate intermediate-layer gradients after 10–50 back-propagation iterations using standard AlexNet initialization.
- Branch placement: For the eight-layer model, auxiliary supervision follows the fourth convolutional layer, where gradients tend to vanish.The resulting architecture contains a main branch and an auxiliary branch with weights Ws5 through Ws8.
- Optimization: The combined objective is the main loss plus a weighted auxiliary loss, with the auxiliary weight α_t decaying over training to act mainly as regularization.Early convolutional weights receive gradients from both loss terms, while later main- and auxiliary-branch weights are affected by their respective losses.
- Branch placement: For the 13-layer model, the authors place supervision after layers 10, 7, and 4 because gradients tend to decay every three to four layers.All auxiliary losses use the same α_t, starting at 0.3 and then decaying according to the training schedule.
3. Experiments
The experiments evaluate deeper CNNs with and without deep supervision on ImageNet and Places, using comparable architectures and reporting accuracy, training time, and dataset-specific evaluation procedures. Deep supervision improves results over the corresponding baselines while reducing training time for the ImageNet 8-layer comparison.
- ImageNet experiments: ImageNet evaluation uses 1000 categories with 1.2M training, 50K validation, and 100K test images, measured by top-1 and top-5 classification error.Top-5 error counts an image as incorrect when its ground-truth class is absent from the five highest-scoring predictions.
- ImageNet experiments: The ImageNet-CNN-8 baseline requires about 6 days, including shallower-network pretraining, using 80 epochs on two NVIDIA Tesla K40 GPUs.The baseline initializes some deeper-network layers from a trained five-layer network and the remaining intermediate layers randomly.
- ImageNet experiments: ImageNet-CNDS-8 adds auxiliary supervision after the fourth convolutional layer and takes about 5 days with 65 epochs on two K40 GPUs.The same network and parameter settings are used apart from initialization and training procedure.
- ImageNet experiments: ImageNet-CNDS-13 is trained with deep supervision for about 5 days on four GPUs over 67 epochs because initialization for this depth remains an open problem.The 13-layer model is trained only with CNDS rather than compared against a pretrained-initialization baseline.
- ImageNet results: Both 8-layer ImageNet models outperform reported 5-layer models, while the 13-layer model outperforms the 8-layer models; CNDS-8 is 1% more accurate than CNN-8 and trains faster.The two 8-layer models have identical test-time structures, and the auxiliary-loss weight eventually decays to zero.
- Comparison and caveat: The authors report that their models remain less accurate than same-depth VGG models, while attributing much of the difference to network settings and training scale.They identify first-layer stride and filter size, plus single-scale rather than multi-scale training, as key differences.
- Places experiments: The Places evaluation reports top-1 and top-5 accuracy on validation and test sets, with test predictions submitted to the MIT evaluation website because labels are withheld.Places-CNN-8 and Places-CNDS-8 use the same structures and parameters as their ImageNet counterparts but are trained with Caffe.
4. Discussion
The discussion frames deep supervision as a training strategy for very deep networks and reports advantages over shallower-network pretraining, including new state-of-the-art Places results.
- Contribution: The work inserts auxiliary supervision at intermediate layers and seeks design principles for choosing where and how to add it.The authors describe this as training very deep networks with deep supervision.
- Comparison: The experiments show an advantage over alternative training methods that require pre-training shallower networks.This comparison is presented as part of the paper’s overall assessment of deep supervision.
- Results: The paper reports new state-of-the-art results on the recently released, very large Places dataset.Table 1 is separately described as covering ILSVRC 2012–2014 validation-set top-1 and top-5 error for literature and proposed models.
- Results: Table 2 compares top-1 and top-5 accuracies on Places validation and test sets for the five-layer model from [13] and the proposed eight-convolution-layer models.The table organizes results by dataset split and model depth.