PaperlayerView exact revision on arXiv

Source-linked AI summary

BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks

Surat Teerapittayanon, Bradley McDanel, H. T. Kung

arXiv:1709.01686v1cs.NEcs.CVcs.LG

TL;DR

Deep neural networks improve classification but make feedforward inference increasingly costly in latency and energy. BranchyNet adds side classifiers and exits confident samples early, while jointly optimizing all exit losses. Across LeNet, AlexNet, and ResNet evaluations, the paper reports reduced inference cost and 2x-6x speedup on CPU and GPU.

  • Problem

    Increasing network depth improves learned representations and classification but raises feedforward inference latency and energy usage, creating a challenge for real-time and energy-sensitive applications.

  • Method

    BranchyNet augments a baseline network with side branches, uses entropy thresholds to exit confident samples early, and jointly optimizes the losses at all exit points.

  • Results

    2x-6x speedup was reported on both CPU and GPU, with BranchyNet reducing inference cost across evaluated LeNet, AlexNet, and ResNet networks.

  • Takeaways & Limitations

    Many samples can be classified at earlier stages, allowing BranchyNet to reduce inference computation while retaining the baseline network’s later exits for harder samples.

Abstract

from arXiv · show

Deep neural networks are state of the art methods for many learning tasks due to their ability to extract increasingly better features at each network layer. However, the improved performance of additional layers in a deep network comes at the cost of added latency and energy usage in feedforward inference. As networks continue to get deeper and larger, these costs become more prohibitive for real-time and energy-sensitive applications. To address this issue, we present BranchyNet, a novel deep network architecture that is augmented with additional side branch classifiers. The architecture allows prediction results for a large portion of test samples to exit the network early via these branches when samples can already be inferred with high confidence. BranchyNet exploits the observation that features learned at an early layer of a network may often be sufficient for the classification of many data points. For more difficult samples, which are expected less frequently, BranchyNet will use further or all network layers to provide the best likelihood of correct prediction. We study the BranchyNet architecture using several well-known networks (LeNet, AlexNet, ResNet) and datasets (MNIST, CIFAR10) and show that it can both improve accuracy and significantly reduce the inference time of the network.

I. INTRODUCTION

BranchyNet addresses the latency and energy costs of increasingly deep networks by adding side branches that let confident samples exit early. Joint optimization also uses the exit classifiers for regularization and additional gradient signals.

  • Motivation: Deepening networks improves classification but substantially increases feedforward inference latency and energy costs.The cited context contrasts AlexNet, VGGNet, and ResNet depths and notes a 20x runtime increase for VGGNet over AlexNet on a Titan X GPU.
  • Approach: BranchyNet adds side branches to a baseline network so confidently classified samples can exit before reaching later layers.Samples with low-entropy predictions leave at an early exit; uncertain samples continue to subsequent exits.
  • Training: Jointly optimizing weighted losses across exit points provides mutual regularization that can prevent overfitting and improve test accuracy.The paper describes each exit point as regularizing the others.
  • Approach: Entropy thresholds determine whether each sample exits early or continues, while the final baseline exit always performs classification.At each exit, entropy below a learned threshold indicates sufficient confidence.
  • Contributions: The paper identifies fast inference, joint-optimization regularization, and vanishing-gradient mitigation as its three main contributions.These contributions target runtime and energy savings, test accuracy, and lower-layer feature learning.
  • Training: Early exit branches provide more immediate backpropagation signals, producing more discriminative lower-layer features and improving accuracy.This is presented as a mitigation of vanishing gradients.

II. BACKGROUND AND RELATED PRIOR WORK

Prior work improves deep-network efficiency through compression, implementation optimization, regularization, or gradient-propagation mechanisms. BranchyNet is distinguished by combining general side branches with early exits and joint training for confident samples.

  • Deep CNNs: Deep CNNs use stacked convolutional and fully connected layers, with AlexNet, VGG, and ResNet extending depth and size to improve classification accuracy.LeNet-5 established the standard CNN structure described here.
  • Efficiency approaches: Network compression reduces parameters but may be difficult to convert into GPU speedups because sparse connections lack exploitable regularity.Tensor decomposition can compact models at the cost of a small amount of accuracy loss.
  • Efficiency approaches: Implementation optimization accelerates convolution algorithms, whereas BranchyNet changes network structure to reduce inference computation for selected samples.Examples include CPU code optimization, FFT convolution, and specialized 3x3 convolution algorithms.
  • Regularization: Regularization methods include dropout, L1 and L2 penalties, and intermediate softmax branches, but BranchyNet additionally targets early exits for confidently classified test samples.The related softmax-branch approach is described as regularizing the main network.
  • Gradient propagation: Skip connections in Highway Networks, ResNet, and Stochastic Depth help propagate gradients during backpropagation and address vanishing gradients.The cited mechanism is an identity shortcut between layers.
  • Early-exit methods: Conditional Deep Learning uses a cascade of linear classifiers, while BranchyNet permits more general branch structures and jointly trains classifiers with the original network.The comparison distinguishes branch architecture and training procedure.

III. BRANCHYNET

BranchyNet modifies a standard network with side branches and exit classifiers that support early prediction while also addressing regularization and vanishing gradients. Its design involves choosing branch locations, structures, classifiers, exit criteria, and training procedures, and it can extend beyond classification.

  • Architecture: BranchyNet adds exit branches at selected network locations so samples accurately classified at early stages can leave before later computation.Branches are integrated into the standard deep-network structure.
  • Training: Branch training regularizes the main branch and provides more immediate gradient signals from shallower branches, producing more discriminative lower-layer features.The architecture addresses both overfitting and vanishing gradients during training.
  • Design considerations: Design choices include branch locations, branch size and depth, exit classifiers, exit criteria and test costs, and joint training of all exit classifiers.The branch concept can also be applied recursively.
  • Scope: Although described for classification, BranchyNet is presented as general enough for image segmentation and object detection.The cited scope extends the architecture beyond the paper’s primary classification setting.

A. Architecture

BranchyNet augments a baseline network with contiguous-layer branches and exit points, training all exits through a weighted joint loss. Its training procedure records outputs from main and side branches, computes error, and updates weights by backpropagation.

  • A. Architecture: A BranchyNet network partitions contiguous layers into non-overlapping branches, each followed by an exit point, with the original network serving as the main branch.
  • A. Architecture: Each exit branch minimizes its classification loss using parameters from the entry point through that exit.
  • A. Architecture: The training objective is a weighted sum of the loss functions associated with all exit branches.
  • A. Architecture: Training performs a feedforward pass through main and side branches, records every exit output, computes error, and updates weights during backward propagation.The described updates use gradient descent with Adam.

C. Fast Inference with BranchyNet

BranchyNet performs inference by evaluating exits from the lowest to the highest, using entropy thresholds to decide whether a sample can leave early. Samples below the relevant threshold return their highest-probability label without higher-branch computation.

  • C. Fast Inference with BranchyNet: BranchyNet uses entropy of each exit classifier’s output as a confidence measure during inference.The output probabilities are used to compute entropy at each exit point.
  • C. Fast Inference with BranchyNet: When a sample exits early, BranchyNet returns the class label with the maximum score and performs no further computation in higher branches.
  • C. Fast Inference with BranchyNet: Inference iterates from the lowest exit point to the highest, feeding the sample through each corresponding branch.
  • C. Fast Inference with BranchyNet: At exit n, the sample exits when its entropy is below threshold Tn; otherwise, processing continues to a higher exit.The threshold vector T contains one threshold for each exit point, and N denotes the number of exit points.

IV. RESULTS

BranchyNet was evaluated by adapting LeNet, AlexNet, and ResNet for MNIST and CIFAR10 on CPU and GPU. Across networks, it generally improved the accuracy–runtime trade-off by exiting many samples before the final layer.

  • Evaluation Setup: BranchyNet was evaluated as B-LeNet on MNIST and as B-AlexNet and B-ResNet on CIFAR10 using both CPU and GPU.The experiments used a 3.0GHz CPU and an NVIDIA GeForce GTX TITAN X GPU.
  • Overall Results: BranchyNet outperformed the original baseline network on GPU performance across LeNet, AlexNet, and ResNet.The reported runtime is averaged over all test samples.
  • Accuracy–Runtime Trade-off: The knee point represents a threshold where BranchyNet accuracy is comparable to the main network while inference is significantly faster.For B-ResNet, the reported knee-point accuracy is slightly below baseline, although another threshold can yield higher accuracy with less inference-time savings.
  • Speedup and Early Exit: 94% of B-LeNet, 65% of B-AlexNet, and 41% of B-ResNet samples exited before the last layer, producing CPU/GPU speedups of 5.4/4.7x, 1.5/2.4x, and 1.9/1.9x, respectively.These values are selected knee-point results from Table I.

V. ANALYSIS AND DISCUSSION

This section provides additional analysis of key aspects of BranchyNet.

  • Additional Analysis: The section examines key aspects of BranchyNet beyond the primary evaluation.The supplied passage introduces additional analysis without specifying the individual aspects.
  • Additional Analysis: BranchyNet is analyzed through supplementary discussion after the main results.The passage identifies this material as a separate section of analysis.
  • Additional Analysis: The section focuses on aspects considered important for understanding BranchyNet.No further aspect-specific findings are stated in the supplied passage.

A. Hyperparameter Sensitivity

BranchyNet performance depends on joint-loss weights, branch locations, and entropy thresholds. The analysis links these choices to early-layer accuracy, dataset difficulty, and latency–accuracy requirements.

  • Joint-Optimization Weights: Giving early branches more weight improves the accuracy of later branches through added regularization.The weighting choice affects how discriminative features are learned across the network.
  • Exit Thresholds: Figure 3 evaluates accuracy–runtime trade-offs by sweeping entropy thresholds for branch exit points.Each plotted point represents a threshold combination, and the star marks a knee point.
  • Joint-Optimization Weights: A 1.0 weight on the first branch and 0.3 on the last branch increased B-AlexNet classification accuracy by 1% over equal weighting.This result used a simplified BranchyAlexNet with only the first and last branch.
  • Exit Thresholds: Exit thresholds should satisfy an application’s inference-latency requirement while maintaining required accuracy.The analysis notes that threshold selection depends on the application and dataset.
  • Branch Placement: Branch locations should reflect dataset difficulty: MNIST permits an early first branch, whereas harder datasets require branches placed higher in the network.Additional branches were placed at equidistant points, while automatic placement was left for future work.

B. Tuning Entropy Thresholds

Entropy thresholds control BranchyNet’s accuracy–runtime trade-off and can be selected by screening candidate values against a target constraint. The paper also discusses adapting thresholds automatically for unseen samples.

  • Threshold Trade-off: Increasing entropy thresholds trades classification accuracy for faster runtime in BranchyNet.The results show accuracy and runtime across a range of threshold values.
  • Threshold Selection: Thresholds can be selected by screening values until a specified runtime or accuracy constraint is satisfied.The authors provide code implementing this screening procedure.
  • Threshold Trade-off: Figure 4 measures B-AlexNet’s overall classification accuracy as the first-exit entropy threshold varies.Samples not exiting at the first branch are sent to the final exit.
  • Threshold Selection: A possible automatic strategy uses Meta-Recognition to estimate unseen-test-sample characteristics and adjust thresholds toward a runtime or accuracy goal.The proposed implementation trains an MLP for each exit point using its output softmax probability vector.

C. Effects of Structure of Branches

Branch structure affects both final-branch accuracy and inference efficiency. Earlier branches benefit from careful sizing, while entropy thresholds trade off runtime and cache behavior.

  • An optimal number of convolutional layers in an earlier branch can improve final-branch accuracy, but adding too many can harm overall accuracy.Adding a few fully connected layers can also help combine local and global features into more discriminative features.
  • Earlier branch points should generally have more layers, while later branch points should have fewer layers.Each branch should remain smaller than the computation required to reach a later exit point.
  • Aggressive entropy thresholds produce faster CPU runtime and lower cache miss rates for B-AlexNet.Because most samples exit early, early-branch weights are cached more efficiently.

VI. CONCLUSION

BranchyNet uses side branches, exit criteria, and joint loss optimization to let many samples leave deep networks early. Across several network architectures, it reduces inference cost and achieves 2x-6x speedup on CPU and GPU, while also serving as a flexible research toolbox.

  • BranchyNet combines branching structures, exit criteria, and joint optimization across exit losses to accelerate deep-network inference.
  • The architecture was evaluated on several popular network architectures and reduced deep-neural-network inference cost with 2x-6x speedup on both CPU and GPU.
  • BranchyNet can be combined with network pruning and compression and adapted to tasks such as image segmentation.
Loading 1709.01686v1…