PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-level Residual Networks from Dynamical Systems View

Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, David Begert

arXiv:1710.10348v2stat.MLcs.CV

TL;DR

ResNet design and training remain theoretically underexplained, including why lesioning layers can preserve performance and why deep models train slowly. The paper analyzes lesion properties through a dynamical-systems view, proposes multi-level training, and reports more than 40% lower training time with superior or on-par accuracy across three benchmarks.

  • Problem

    Theoretical principles for designing and training ResNets remain incompletely understood, including unexplained layer-lesioning behavior and the long training time of deep models.

  • Method

    The paper combines theoretical and empirical dynamical-systems analysis of ResNet lesion properties with a multi-level training method for ResNets and Wide ResNets.

  • Results

    More than 40% training time reduction is achieved across ResNets and Wide ResNets on three image-classification benchmarks, with superior or on-par accuracy.

  • Takeaways & Limitations

    The dynamical-systems analysis supports an efficient multi-level approach that accelerates training while maintaining similar or better accuracy in the evaluated residual architectures and benchmarks.

  • Takeaways & Limitations

    Time savings increase with more interpolations but have diminishing marginal benefit, and large interpolation counts may leave cycles with too few training steps, creating an efficiency–accuracy trade-off.

Abstract

from arXiv · show

Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet theoretically, such as unraveled view, unrolled iterative estimation and dynamical systems view. In this paper, we adopt the dynamical systems point of view, and analyze the lesioning properties of ResNet both theoretically and experimentally. Based on these analyses, we additionally propose a novel method for accelerating ResNet training. We apply the proposed method to train ResNets and Wide ResNets for three image classification benchmarks, reducing training time by more than 40% with superior or on-par accuracy.

1 INTRODUCTION

The paper extends the dynamical-systems interpretation of ResNets with empirical lesion analysis and introduces multi-level training to address their long training time. Across ResNets and Wide ResNets on three datasets, the method reduces training time by more than 40% with superior or on-par accuracy.

  • Motivation: The dynamical systems view interprets ResNets as ordinary differential equations, but previously offered limited empirical explanation for why removing layers may not reduce performance.The paper addresses this gap through theoretical and empirical analysis of lesion studies.
  • Motivation: Deep ResNets can require several days or weeks to train on large datasets or at extreme depths, motivating a faster training method.The paper identifies long training time as a major challenge, especially for ImageNet and 1000-layer networks.
  • Contributions: The paper analyzes lesion properties from the dynamical-systems view through empirical analysis.This is presented as a central contribution alongside the proposed training method.
  • Contributions: The paper proposes a simple, efficient multi-level training method for ResNets based on the dynamical systems view.The method is intended to accelerate training.
  • Results: More than 40% training time reduction is achieved across ResNets and Wide ResNets on three widely used datasets, with superior or on-par accuracy.The result is reported for both architecture families across the three benchmarks.

2 RELATED WORK

Prior work interprets ResNets as residual-block compositions, ensembles of paths, iterative representation refinement, or discretized dynamical systems. These views explain different behaviors, while efficient-training methods reduce computation or modify optimization.

  • ResNets and variants: A residual block updates its feature map by adding a residual module to the preceding feature map.The update is written as Y_j+1 = Y_j + G(Y_j,θ_j).
  • ResNets and variants: ResNets are composed of stages whose first blocks halve feature-map size and double filter count, while later blocks preserve dimensionality.The residual module G consists of two convolutional layers.
  • Interpretations of ResNets: The unraveled view treats a network with n residual blocks as 2^n paths, explaining resilience to block removal through partially preserved paths.Lesion studies in this view find that paths do not strongly depend on one another and behave like an ensemble.
  • Interpretations of ResNets: The unrolled iterative-estimation view treats blocks within a stage as progressively refining one representation, so removing blocks has only a mild effect.The first block provides a rough estimate and subsequent blocks refine it.
  • Interpretations of ResNets: The dynamical-systems view models ResNets as discretized dynamical systems, but prior work had not interpreted layer deletion from this perspective.It also connects large-system behavior with exploding or vanishing gradients.
  • Efficient training methods: Prior efficiency methods include stochastic depth, which bypasses residual blocks during training and saves approximately 25% of training time.Other approaches target inference-time computation, resource-limited testing, sparse aggregation, or optimization difficulty.

3 RESNETS FROM DYNAMICAL SYSTEMS VIEW

The paper interprets ResNets as discretized dynamical systems and uses this view to explain residual-module behavior, lesion effects, and a multi-level training procedure.

  • Dynamical systems view: ResNets can be viewed as forward Euler discretizations of an ODE, with network depth determining the discretization step h = T/d.The input and output feature maps correspond to ODE states at times 0 and T, respectively.
  • Empirical analysis: The average residual-module L2-norm follows a reciprocal trend as the number of residual blocks increases, consistent with the dynamical-systems formulation.The empirical quantity is γ, the average L2-norm across residual modules.
  • Lesion study: Removing a residual block skips one time step and squeezes adjacent steps together, but its effect is negligible when the removed module output is sufficiently small.The difference before and after removal equals the removed residual module output G(Y(t1)).
  • Residual-module behavior: In deep networks, residual modules usually produce small outputs relative to identity mappings, so later blocks mainly refine features incrementally.The largest feature change occurs in the first block, while subsequent blocks make smaller refinements.
  • Multi-level training: The multi-level method begins training with a shallow network, then repeatedly doubles residual blocks while halving the explicit step size and copying adjacent-block parameters.Interpolation is applied across stages and provides initialization because it preserves the same differential equation under the dynamical-systems view.

4 EFFICIENT TRAINING WITH MULTI-LEVEL METHOD

The multi-level method trains progressively deeper ResNets by interpolating residual blocks and halving the step size, inspired by multi-grid methods for differential equations. It reduces training time, although additional interpolations offer diminishing savings and may trade efficiency against accuracy.

  • 4.1 MULTI-LEVEL TRAINING: The method applies multi-grid ideas to ResNets by training networks through a hierarchy of discretizations with varying step sizes.The dynamical-systems connection motivates treating residual networks as discretizations of differential equations.
  • 4.1 MULTI-LEVEL TRAINING: Training starts with a shallow network and large step size, then doubles residual blocks and halves the step size after several updates.Interpolation inserts a residual block after each existing block and copies adjacent convolutional and batch-normalization parameters.
  • 4.1 MULTI-LEVEL TRAINING: Interpolations divide training into cycles, while preserving the same differential equation and providing parameter initialization for each deeper cycle.Each interpolation is performed across all stages simultaneously.
  • 4.1 MULTI-LEVEL TRAINING: Each cycle resets the learning rate to ηmax and cosine-anneals it to ηmin over that cycle.Tcur tracks progress within the cycle, while T denotes the cycle's total training steps.
  • 4.2 TRAINING TIME: A two-cycle schedule reduces training time by 1/4 relative to the full model trained for the same total steps.The shallow model runs for N steps and the deep model for another N steps, while the deep baseline runs for 2N steps.
  • 4.2 TRAINING TIME: Time saved increases monotonically but with diminishing marginal benefit as interpolations increase, creating an efficiency–accuracy trade-off.With many interpolations, individual cycles may not contain enough training steps.

5 EXPERIMENTS

The experiments evaluate multi-level training on ResNets and Wide ResNets across three image-classification benchmarks. The method achieves over 40% lower training time with competitive or better accuracy than the last-cycle models.

  • 5 EXPERIMENTS: Experiments evaluate ResNets and Wide ResNets across CIFAR-10, CIFAR-100, and STL-10.These are three standard image-classification benchmarks.
  • 5 EXPERIMENTS: The multi-level method uses two interpolations and three cycles, compared with first-cycle and last-cycle models trained for the same number of steps.Baseline models use cyclic learning-rate schedules for fair comparison.
  • 5 EXPERIMENTS: The experiments train for 160 epochs, interpolating at epochs 60 and 110, with ηmin=0.001 and ηmax=0.5.Batch sizes are 100 for CIFAR-10/100 and 32 for STL-10.
  • 5.3 MAIN RESULTS AND ANALYSIS: 42% theoretical time savings from two interpolations is consistent with the experimental results.This supports the efficiency predicted by the multi-level schedule.
  • 5.3 MAIN RESULTS AND ANALYSIS: Across both architectures and all three datasets, test error is lower than the first-cycle model and competitive or slightly lower than the last-cycle model, while training time falls over 40%.The result holds for models spanning multiple network depths.
  • 5.3 MAIN RESULTS AND ANALYSIS: Training and test accuracy temporarily drop after each cycle begins but eventually surpass the preceding cycles for both ResNets and Wide ResNets.Similar curves across the two architectures indicate effectiveness across network widths.

6 CONCLUSION

The paper interprets ResNets through dynamical systems and develops multi-level training to accelerate them. Across residual architectures and benchmarks, the method reduces training time by more than 40% with similar accuracy.

  • 6 CONCLUSION: The paper combines theoretical and empirical dynamical-systems analyses to explain lesion studies in ResNets.It uses these analyses as the basis for the proposed training method.
  • 6 CONCLUSION: The multi-level method accelerates training for ResNets and Wide ResNets across three classification benchmarks.The conclusion describes the method as simple yet effective.
  • 6 CONCLUSION: The method reduces training time by more than 40% while maintaining similar accuracy.Figure 6 shows temporary accuracy drops at cycle boundaries followed by performance surpassing earlier cycles.

A DATASETS AND MODEL DETAILS

The evaluation uses CIFAR-10, CIFAR-100, and STL-10, with dataset-specific image resolutions and sample counts. Model details include three-stage ResNet and Wide ResNet configurations and parameter counts reported in Table 5.

  • A DATASETS AND MODEL DETAILS: CIFAR-10 and CIFAR-100 each contain 50,000 training images and 10,000 testing images, with 32×32 resolution.CIFAR-10 has 10 classes, whereas CIFAR-100 has 100 classes.
  • A DATASETS AND MODEL DETAILS: STL-10 contains 5,000 training images and 8,000 testing images across 10 classes at 96 × 96 resolution.Its classes have fewer training samples but higher image resolution than CIFAR-10/100.
  • A DATASETS AND MODEL DETAILS: The paper lists the number of parameters for each network model in Table 5.The supplied passage identifies the table but gives no parameter values.

B IMPLEMENTATION DETAILS

Interpolation requires special handling for the first block in each stage because it changes feature-map size and channel count, producing parameter shapes unlike later blocks.

  • The first block in each stage changes the feature-map size and number of channels during interpolation.
  • Parameters from block 1’ cannot simply be copied to block 2’ because their convolution and batch-normalization shapes differ.

C FURTHER ANALYSIS OF TIME STEP SIZE

The paper treats explicit residual scaling as a controllable time step and tests whether networks with different depths follow the same underlying dynamical trajectory. The plotted residual norms support this shared-function prediction.

  • Explicit step size: Explicit h multiplies each residual output before addition to the identity mapping, making step size a model hyperparameter.
  • Experimental setting: The empirical analysis evaluates the theory on CIFAR-10 and CIFAR-100 using ResNets with varying depths.
  • Cross-depth comparison: ResNet-32, ResNet-62, and ResNet-122 are modeled as different discretizations of the same differential equation over 15, 30, and 60 time steps.
  • Cross-depth comparison: The three scaled residual-norm curves follow the same trend across corresponding conceptual time points, consistent with a shared underlying function F(t).

D MORE EXPERIMENTAL RESULTS

Experiments examine cyclic learning-rate choices, resetting between cycles, and comparisons with shallow and deep networks. The results identify a preferred learning-rate range and show that resetting can improve validation accuracy, while cycle behavior and training curves provide additional comparisons.

  • Learning-rate effects: ηmax = 0.5 and ηmin = 0.001 achieve the best accuracy in the learning-rate study.
  • Learning-rate effects: As ηmax increases, test accuracy generally rises initially and then declines.
  • Multi-level training curves: On STL10, training and testing accuracy temporarily drop at each cycle's start before eventually surpassing previous cycles for ResNet-50-i and Wide ResNet-50-i.
  • Learning-rate effects: With a small learning rate, learning is slow, whereas a high learning rate leaves too much kinetic energy to reach deeper, narrower loss regions.
  • Cycle resetting: Resetting the learning rate at the beginning of each cycle gives better validation accuracy than continuing the same schedule across cycles.
  • Depth comparison: Figure 11 compares training-accuracy curves for shallow and deep ResNets matching the starting and final model depths.
Loading 1710.10348v2…