PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Yiping Lu, Aoxiao Zhong, Quanzheng Li, Bin Dong

arXiv:1710.10121v3cs.CVcs.LGstat.ML

TL;DR

The paper addresses how numerical differential equations can guide the design of effective deep architectures. It interprets popular networks as ODE discretizations and introduces the LM-architecture for ResNet-like models. LM variants improve accuracy with comparable parameters, while stochastic depth further improves LM-ResNet on CIFAR10.

  • Problem

    Deep-network design needs effective architectures with strong accuracy and parameter efficiency, but existing successful designs lack a unified numerical-differential-equation perspective.

  • Method

    The paper interprets deep architectures as ODE discretizations, introduces a linear multi-step architecture for ResNet-like networks, and connects noise injection with stochastic dynamic systems.

  • Results

    LM-ResNet and LM-ResNeXt improve over their original networks on CIFAR and ImageNet with comparable trainable parameters, while LM-ResNet110 with stochastic depth reaches 4.80% testing error on CIFAR10.

  • Takeaways & Limitations

    Numerical-analysis viewpoints can guide new deep-network architectures and provide a framework for extending stochastic training to LM-based networks.

Abstract

from arXiv · show

In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress ($>50$\%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

1 INTRODUCTION

The paper frames deep-network architecture design through numerical differential equations, interpreting several successful architectures as ODE discretizations. This perspective motivates new designs and connects stochastic training with stochastic dynamic systems.

  • Network design seeks strong generalization using as few parameters as possible, motivating structural improvements beyond simply increasing depth or width.
  • ResNet residual blocks are forward Euler steps for an ODE, suggesting that skip-connected networks can be viewed as discrete dynamic systems.The block update is un+1 = un + ∆tf(un), corresponding to ut = f(u).
  • PolyNet, FractalNet, and RevNet can be interpreted as different numerical discretizations of ODEs, linking their effectiveness to approximating dynamic systems.
  • The differential-equation perspective supports architecture design using numerical-analysis knowledge, including the proposed LM-architecture for ResNet-like networks.The paper presents this as a way to design new and potentially more effective deep networks.
  • Noise injection in ResNet-like networks can be interpreted as discretizing stochastic dynamic systems, providing a stochastic-control perspective on stochastic learning.This framework also enables stochastic training for networks using the LM-architecture.
  • The paper distinguishes its approach from PDE-based and optimization-inspired network design by learning f(u) for supervised tasks without requiring an optimization interpretation or differential structure.

2 NUMERICAL DIFFERENTIAL EQUATION, DEEP NETWORKS AND BEYOND

The paper interprets several deep network architectures as numerical schemes for differential equations and uses this perspective to design LM-architectures. LM-ResNet and LM-ResNeXt improve accuracy or reduce depth and parameters across CIFAR and ImageNet, with modified equations providing a mathematical explanation.

  • Numerical schemes and network architectures: Many deep networks can be viewed as numerical approximations of ODEs, including PolyNet, FractalNet, RevNet, ResNet, and ResNeXt-like architectures.PolyNet resembles backward Euler, FractalNet resembles a second-order Runge–Kutta scheme, and RevNet corresponds to a forward Euler approximation of a dynamic system.
  • Numerical schemes and network architectures: PolyNet’s implicit-scheme interpretation explains how larger step sizes can reduce network depth while increasing the width of residual blocks.The paper connects PolyNet to backward Euler and attributes its reduced depth to the larger step sizes permitted by implicit schemes.
  • LM-architecture: The LM-architecture is a two-step linear multi-step method inspired by numerical ODE solvers and applicable to any ResNet-like network.It introduces a trainable parameter k_n for each layer and produces LM-ResNet and LM-ResNeXt when applied to ResNet and ResNeXt.
  • Empirical results: LM-ResNet and LM-ResNeXt improve over their original counterparts on CIFAR and ImageNet with comparable numbers of trainable parameters.On CIFAR100, LM-ResNeXt29 is reported as having the best result; on ImageNet, LM-ResNet improves over ResNet with comparable parameter counts.
  • Empirical results: LM-architecture can preserve similar performance while substantially compressing networks, including ResNet with half the parameters on CIFAR10.The paper also reports comparable errors between a 56-layer LM-ResNet and a 110-layer ResNet on CIFAR10, a 164-layer LM-ResNet and a 1001-layer ResNet on CIFAR100, and a 50-layer LM-ResNet and a 101-layer ResNet on ImageNet.
  • Modified-equation explanation: Modified equations explain how the LM scheme changes numerical behavior, including acceleration or dispersion effects that can speed the network’s flow.The paper relates these effects to faster convergence for gradient flows and faster flows when the learned operator is elliptic.

3 STOCHASTIC LEARNING STRATEGY: A STOCHASTIC DYNAMIC SYSTEM PERSPECTIVE

The paper interprets stochastic training methods as approximations of stochastic dynamic systems and extends this perspective to LM-architectures. It applies stochastic depth to LM-ResNet, obtaining a 4.80% CIFAR10 testing error.

  • 3.1 NOISE INJECTION AND STOCHASTIC DYNAMIC SYSTEMS: Stochastic learning strategies, including residual-block dropout and branch-wise shake-shake regularization, are interpreted as approximations of stochastic dynamic systems.The paper frames this interpretation through stochastic control and weak approximation arguments.
  • 3.1 NOISE INJECTION AND STOCHASTIC DYNAMIC SYSTEMS: For stochastic depth, residual blocks are randomly retained or dropped, with η_n=1 occurring with probability p_n and η_n=0 otherwise.With Δt=1, the formulation reduces to the original stochastic dropout training strategy.
  • 3.1 NOISE INJECTION AND STOCHASTIC DYNAMIC SYSTEMS: Deeper network blocks should use p_n closer to 1/2, matching an observation reported for stochastic depth.The paper derives this tendency from the stochastic approximation analysis.
  • 3.1 NOISE INJECTION AND STOCHASTIC DYNAMIC SYSTEMS: The stochastic-control perspective represents training procedures using a running loss cost, terminal time, and regularization term.Here, L(·) denotes the loss function, T the terminal time, and R a regularization term.
  • 3.2 STOCHASTIC TRAINING FOR NETWORKS WITH LM-ARCHITECTURE: The stochastic-depth analysis extends to LM-architectures by converting a second-order system into a first-order system and weakly approximating its stochastic process.The resulting LM-architecture training strategy can be implemented by randomly dropping residual blocks.
  • 3.2 STOCHASTIC TRAINING FOR NETWORKS WITH LM-ARCHITECTURE: 4.80% testing error was achieved by LM-ResNet110 with stochastic depth on CIFAR10.This error was lower than the ResNet1202 result reported in the original paper.

4 CONCLUSION AND DISCUSSION

The paper connects deep-network architectures with ODE discretizations, uses this connection to design LM architectures, and extends the perspective to stochastic training. LM architectures improve original networks, while stochastic depth improves LM-ResNet on CIFAR10.

  • The paper develops a comprehensive connection between popular deep-network architectures and ODE discretizations.
  • LM-architecture modifications improve the accuracy of ResNet and ResNeXt and outperform FractalNet and DenseNet on CIFAR100.
  • Networks with stochastic training can be interpreted as weak approximations to stochastic dynamic systems and cast as stochastic control problems.
  • Significant improvement over the original LM-ResNet on CIFAR10 is achieved by introducing stochastic depth.
  • Future work may use mathematical analysis, geometry, physical laws, numerical schemes, and control theory to guide network design and training.

A NUMERICAL ODE

The paper introduces the ordinary differential equation form used as the basis for its numerical-method discussion.

  • The ODE considered in this section has the form u_t = f(u, t).

A.1 FORWARD AND BACKWARD EULER METHOD

Forward and backward Euler methods approximate ODE solutions using different evaluations of the right-hand side, trading simplicity against stability.

  • Forward Euler discretizes the time derivative and evaluates the right-hand side at the current state and time.
  • Backward Euler evaluates the right-hand side at the next state and time.
  • Backward Euler has better stability properties than forward Euler but requires solving a nonlinear equation at each step.

A.2 RUNGE-KUTTA METHOD

Runge-Kutta methods are higher-order one-step ODE methods that use intermediate solution approximations and adjustable coefficients.

  • Runge-Kutta methods form a set of higher-order one-step methods for approximating ODE solutions.
  • The method computes an update from weighted evaluations of f at intermediate approximations and times.
  • Intermediate approximations represent the solution at times t_n + c_j∆t, while coefficients {c_j} can be adjusted for higher-order accuracy.
  • The popular second-order Runge-Kutta method is presented as a concrete example.

A.3 LINEAR MULTI-STEP METHOD

The linear multi-step method extends forward Euler to higher orders and uses an explicit formulation for the linear multi-step structure.

  • The linear multi-step method generalizes the classical forward Euler scheme to higher orders.
  • The general k-step method uses scalar parameters α_j and β_j, with α_0 ≠ 0 and |α_j| + |β_j| ≠ 0.
  • The method is explicit when β_0 = 0, which is the formulation used to design the linear multi-step structure.

B.1 IT ˆO PROCESS

This section introduces Brownian motion and the Itô-process framework, including the stochastic differential and integral representations used in subsequent analysis.

  • Brownian motion is defined as a stochastic process with independent increments across ordered time instances.
  • An Itô process satisfies dX_t = f(X_t, t)dt + g(X_t, t)dB_t, where B_t is standard Brownian motion.
  • The stochastic differential can be represented as an integral equation involving the drift and Itô-integral terms.

B.2 WEAK CONVERGENCE OF NUMERICAL SCHEMES

The section describes forward Euler approximation for Itô processes and explains how replacing Gaussian increments yields a simplified weak Euler scheme with weak convergence conditions.

  • The forward Euler scheme approximates an Itô process driven by drift, diffusion, and Brownian-motion increments.
  • Brownian increments ΔW_n are Gaussian with variance Δt, and the forward Euler scheme converges strongly to the Itô process.
  • Replacing ΔW_n with a non-Gaussian increment produces the simplified weak Euler scheme, which converges weakly when the replacement satisfies condition (5).
  • A zero-mean uniform random variable or a properly scaled Bernoulli variable taking values 1 or −1 satisfies the required condition.
Loading 1710.10121v3…