PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Deep ResNet Blocks Sequentially using Boosting Theory

Furong Huang, Jordan Ash, John Langford, Robert Schapire

arXiv:1706.04964v4cs.LG

TL;DR

Deep networks are difficult to train because back-propagation can be unstable, motivating a theory and alternative training method for ResNets. The paper models a ResNet as telescoping-sum boosting over representations and introduces BoostResNet, which trains shallow residual modules sequentially. Under a weak-learning condition, training error decreases exponentially with depth, while the method reduces training complexity and shows accuracy or convergence benefits in experiments.

  • Problem

    Deep neural networks are difficult to train because back-propagation is unstable, motivating a theoretical account and alternative training procedure for ResNets.

    Source: §1 · p. 1

  • Method

    The paper develops telescoping-sum boosting for ResNets and trains residual blocks sequentially as weak module classifiers, including auxiliary classifiers for representation boosting.

    Source: §1.1 · p. 2, §3 · p. 4, §4.2 · p. 5

  • Results

    Under a weak-learning condition, BoostResNet reduces training error exponentially with depth and requires substantially lower training complexity than end-to-end back-propagation.

    Source: §1.1 · p. 2

  • Takeaways & Limitations

    The boosting framework applies to multiclass ResNets and can extend to other nonlinear, including non-differentiable, hypothesis units.

    Source: §1.1 · p. 2, §1.2 · p. 3

Abstract

from arXiv · show

Deep neural networks are known to be difficult to train due to the instability of back-propagation. A deep \emph{residual network} (ResNet) with identity loops remedies this by stabilizing gradient computations. We prove a boosting theory for the ResNet architecture. We construct $T$ weak module classifiers, each contains two of the $T$ layers, such that the combined strong learner is a ResNet. Therefore, we introduce an alternative Deep ResNet training algorithm, \emph{BoostResNet}, which is particularly suitable in non-differentiable architectures. Our proposed algorithm merely requires a sequential training of $T$ "shallow ResNets" which are inexpensive. We prove that the training error decays exponentially with the depth $T$ if the \emph{weak module classifiers} that we train perform slightly better than some weak baseline. In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition. Our results apply to general multi-class ResNets. A generalization error bound based on margin theory is proved and suggests ResNet's resistant to overfitting under network with $l_1$ norm bounded weights.

1. Introduction

The paper asks why ResNets are easier to train and develops a boosting-based explanation and training framework. It introduces telescoping-sum boosting and BoostResNet, with theoretical guarantees and empirical improvements over end-to-end back-propagation.

1.1. Summary of Results

  • ResNets are characterized as telescoping-sum boosting over representations rather than naive ensembles of labels or score functions.The paper provides an error bound for feature boosting and introduces telescoping-sum boosting as a new analysis framework.Source: §1.1 · p. 2
  • BoostResNet trains residual blocks sequentially under a weak-learning condition, guaranteeing exponentially decreasing error as depth increases.Each residual block must perform better than a weak baseline in predicting labels.Source: §1.1 · p. 2
  • BoostResNet requires substantially less training computation and GPU memory than end-to-end back-propagation.It stores only individual layers during training, whereas end-to-end back-propagation keeps the full network in GPU memory.Source: §1.1 · p. 2
  • Across MLP-ResNet and CNN-ResNet experiments, BoostResNet improves computational performance, improves accuracy for MLP-ResNet, and converges faster for CNN-ResNet.The comparison uses multiple datasets and two feed-forward ResNet architectures.Source: §1.1 · p. 2
  • The multi-channel framework distinguishes task classes from learned channels and can extend to non-differentiable units such as decision trees or tensor decompositions.The authors present the framework as broader than explaining ResNet alone.Source: §1.1 · p. 2

1.2. Related Works

  • Unlike traditional boosting and AdaNet, BoostResNet boosts representations over multiple channels, producing a less bushy ResNet-style architecture.The paper introduces a new weak-learning condition for this telescoping-sum framework.Source: §1.2 · pp. 2–3

2. Preliminaries

The preliminaries define ResNets as stacks of residual blocks combining nonlinear modules with identity loops. They then connect the resulting recursive representation to boosting, where each module is trained as a weak learner in sequence.

2. Preliminaries

  • A ResNet stacks residual blocks, each combining a nonlinear module with an identity shortcut, and trains on examples drawn i.i.d. from a distribution.The paper considers MLP and CNN modules and defines the training sample over input-label pairs.Source: §2 · p. 3, A Residual Block of ResNet · p. 3

A Residual Block of ResNet

  • The identity-loop recursion makes the final representation a sum of module outputs, followed by a linear classifier for binary or multiclass prediction.For multiclass tasks, the classifier is a matrix mapping the final representation to class scores.Source: A Residual Block of ResNet · p. 3
  • Boosting combines weak classifiers trained on adapted examples to improve their collective prediction power.This weak-learning perspective motivates treating residual modules as sequential learners.Source: A Residual Block of ResNet · p. 3
  • Identity loops are empirically associated with reduced training-error degradation, motivating a sequential alternative to unstable end-to-end back-propagation.The paper seeks a theoretical justification and an algorithm that avoids back-propagating through the full deep network.Source: A Residual Block of ResNet · pp. 3–4

3. ResNet in Telescoping Sum Boosting Framework

The paper reframes a ResNet as telescoping-sum boosting over feature representations, enabling sequential training of residual modules rather than traditional label-based boosting. Each module combines a residual block with an auxiliary classifier, and their ensemble recovers the ResNet output.

ResNet: Ensemble of Weak Module Classifiers

  • ResNet boosting ensembles estimated feature representations rather than estimated hypotheses or labels, requiring a distinct telescoping-sum framework.Traditional boosting results therefore do not directly apply.Source: §3 · p. 4

Hypothesis Modules

  • An auxiliary linear classifier on each residual block forms the hypothesis module needed to train modules sequentially.Given the previous block output, only the current residual function and auxiliary classifier need training.Source: §3 · p. 4

Weak Module Classifiers

  • The weak module classifier restricts each learner to a telescoping-sum form with a scalar coefficient α_t.This restriction defines the paper’s telescoping sum boosting framework.Source: §3 · p. 4

ResNet: Ensemble of Weak Module Classifiers

  • The ensemble of T weak module classifiers, scaled by α_T +1, is identical to the depth-T ResNet output F(x).The final ResNet output is subsequently mapped to labels by a top function.Source: §3 · p. 4

Sequential Training

  • This equivalence enables sequential ResNet training, with the analysis covering binary and multiclass settings while focusing on binary classification in the main text.The paper states that the proof appears in Appendix B and multiclass analysis is deferred to Appendix F.Source: §3 · p. 4

4. Telescoping Sum Boosting for Binary Classification

The paper develops telescoping-sum boosting for ResNets, with a weak learning condition supporting sequential shallow-block training and exponential training-error decay. It also analyzes implementation efficiency and margin-based generalization under bounded weights.

4.1. Weak Learning Condition

  • The weak learning condition requires each module to improve slightly over its predecessor, with a non-adversarial covariance assumption motivating the telescoping-sum construction.The condition is deliberately weaker than requiring a fixed additive improvement, especially when the current module is already highly accurate.Source: §4.1 · pp. 4–5

4.2. BoostResNet

  • BoostResNet trains residual blocks sequentially as shallow ResNets, using auxiliary classifiers only during training while retaining a standard ResNet model.Each module receives the previous module’s output, and auxiliary classifiers guide training without appearing in the final network.Source: §4.2 · p. 5, §4.3 · p. 6
  • Training error decays exponentially with ResNet depth when every weak module classifier satisfies the γ-weak learning condition.The guarantee applies even when each successive hypothesis module is only slightly better than the previous one, and multiclass extensions are provided.Source: §4.2 · pp. 5–6

4.3. Oracle Implementation for ResNet

  • BoostResNet uses memory M1 + M2 and computation C1 + C2, compared with M1T + M2 and C1T + C2 for end-to-end backpropagation.The savings come from keeping only consecutive residual blocks in memory and reducing communication and shallow-gradient overhead.Source: §4.3 · p. 6

4.4. Generalization Error Analysis

  • The margin-based bound links better generalization to larger margins and smaller l1-bounded weight norms, whose dominant complexity term scales at least linearly with depth.Stronger weak modules can produce larger margins, while larger edge values permit larger margin thresholds with a small or zero error term under the stated condition.Source: §4.4 · p. 6

5. Experiments

Experiments compare BoostResNet with end-to-end backpropagation across MLP and CNN ResNets on MNIST, SVHN, and CIFAR-10. BoostResNet generally converges faster, with especially strong SVHN accuracy and a CIFAR-10 refinement trade-off.

Experimental Setup

  • Experiments use identical architectures and random seeds for BoostResNet and e2eBP, with AdaBoost as an additional convolutional-module baseline.Models are trained with Adam-based SGD; hyperparameters are selected by random search, and stopping controls are especially important for BoostResNet.Source: §5 · pp. 7–8

MLP-ResNet on MNIST

  • BoostResNet continues performing well as MLP-ResNet depth increases, while e2eBP training error degrades on MNIST.The sequential procedure is reported to relieve gradient instability as depth grows.Source: §5 · pp. 7–8, Figure 2 · p. 7

CNN-ResNet on SVHN

  • BoostResNet converges much faster than e2eBP on the 50-layer SVHN CNN-ResNet, with comparable test accuracy.The experiment uses 25 residual blocks, and the figure marks the transition from BoostResNet to e2eBP refinement.Source: Figure 3 · p. 7, §5 · p. 8

CNN-ResNet on CIFAR-10

  • On CIFAR-10, BoostResNet converges to the optimal solution faster than e2eBP, but its e2eBP-refined test accuracy is slightly lower.Source: §5 · p. 8

Weak Learning Condition Check

  • The weak learning condition is nonvacuous on SVHN: the required edge γt remains greater than zero, while learned representations improve with depth.Figure 4 examines a 50-residual-block convolutional network.Source: Figure 4 · p. 7, §5 · p. 8

CNN-ResNet on SVHN

  • 93.8% test accuracy versus 83% for e2eBP on SVHN using 3 × 10^8 gradient updates.BoostResNet also trains faster and retains comparable accuracy after e2eBP refinement.Source: §5 · p. 8, Table 1 · p. 8

6. Conclusions and Future Works

The conclusion presents BoostResNet as an exponentially convergent, computationally efficient alternative to end-to-end backpropagation. It also identifies non-differentiable weak learning oracles as a future extension.

6. Conclusions and Future Works

  • Under the weak learning condition, BoostResNet achieves exponentially decaying training error with depth while requiring much less computation and memory than e2eBP.Only individual layers need to reside in GPU memory during BoostResNet training.Source: §6 · p. 8
  • The framework is intended to accommodate non-differentiable data and future weak learning oracles such as tensor-decomposition methods.The paper cites tensor decomposition as a spectral learning framework with theoretical guarantees and proposes extending the framework using general weak learning oracles.Source: §6 · p. 9

A.1. Loss function and architecture selection

The paper situates BoostResNet against unresolved choices in loss functions, architectures, and learning algorithms. It contrasts limited existing guarantees with an oracle-based approach for shallow modules.

A.1. Loss function and architecture selection

  • Existing work lacks general rigorous principles for choosing loss functions and theoretical training-error guarantees for nonlinear identity-skip architectures.Prior theory includes linear neural networks, but the paper characterizes that setting as unrealistic in practice.Source: §A.1 · p. 11

A.2. Learning algorithm design

  • Unlike SGD modifications such as momentum, Nesterov acceleration, Adagrad, and Adam, BoostResNet can use an arbitrary oracle solving a shallow neural network.The oracle need not be an SGD solver.Source: §A.2 · p. 11

B. Proof for Lemma 3.2: the strong learner is a ResNet

The proof establishes the ResNet structure by showing that sequential weak modules combine through a telescoping sum. Each module receives the preceding module’s output, yielding the strong learner’s residual composition.

B. Proof for Lemma 3.2: the strong learner is a ResNet

C. Proof for Theorem 4.2: binary class telescoping sum boosting theory

The binary boosting analysis bounds training error through exponential loss and module-wise normalization, yielding consistent learning as the number of modules grows.

D. Proof for Corollary 4.3: Generalization Bound

The paper derives generalization bounds for ResNet by combining Rademacher-complexity analysis of module hypothesis spaces with margin-based control under weak-learning conditions.

D. Proof for Corollary 4.3: Generalization Bound

D.1. ResNet Module Hypothesis Space

  • Each ResNet module is analyzed through neuron functions with l1-bounded weights and nonlinear transformations of preceding features.Source: §D.1 · pp. 13–14
  • The module complexity is bounded as a function of the output range, weight-norm bound, and number of channels.Source: §D.1 · p. 14

E. Proof for Theorem E: Margin and Generalization Bound

F. Telescoping Sum Boosting for Multi-calss Classification

F.2. Weak Learning Condition

  • The multiclass algorithm uses an optimal edge-over-random cost function and proves exponentially decaying training error when every module satisfies the condition.Source: §F.2 · p. 16

F.3. Oracle Implementation

  • Algorithms 3 and 4 initialize costs, train modules through an oracle, update scores, and iterate until the weak-learning threshold is reached.Source: §F.2 · p. 17

G. Proof for Theorem F.3 multiclass boosting theory

The multiclass proof defines exponential loss over accumulated module scores, minimizes a per-module normalization factor, and derives a training-error bound for the resulting ResNet.

H.1. Training error degradation of e2eBP on ResNet

The section examines how end-to-end back-propagation behaves on MLP-ResNets of varying depth using MNIST convergence experiments.

  • Figure 5 reports e2eBP convergence on multilayer perceptron residual networks of various depths on MNIST.Source: Figure 5 · p. 19
  • The experiment focuses on the baseline training behavior rather than the sequential BoostResNet procedure.Source: Figure 5 · p. 19, §H.1 · p. 19

Related papers

Loading 1706.04964v4…