PaperlayerView exact revision on arXiv

Source-linked AI summary

On Robustness of Neural Ordinary Differential Equations

Hanshu Yan, Jiawei Du, Vincent Y. F. Tan, Jiashi Feng

arXiv:1910.05513v4cs.LGstat.ML

TL;DR

The paper addresses the limited understanding of neural ODE robustness by combining empirical perturbation tests with a theoretical analysis of ODE flows. It finds that ODENets are more robust than CNNs, explains this using non-intersecting integral curves, and proposes TisODE to strengthen robustness and complement other robust architectures.

  • Problem

    The robustness of neural ODEs remains unclear despite prior work on their optimization and approximation properties.

  • Method

    The paper evaluates ODENets under Gaussian and adversarial perturbations, analyzes non-intersecting ODE integral curves, and introduces TisODE with time-invariant dynamics and a steady-state constraint.

  • Results

    ODENets consistently outperform CNNs under tested perturbations, including 64.7% versus 32.9% accuracy on MNIST under PGD-0.2, while TisODE outperforms vanilla neural ODEs.

  • Takeaways & Limitations

    Neural ODEs can serve as robust deep-network building blocks, and TisODE can function as a drop-in module alongside other robust architectures.

  • Takeaways & Limitations

    The formulation assumes fθ is continuous in time and globally Lipschitz continuous in the state, and the non-intersection theorem relies on these conditions.

Abstract

from arXiv · show

Neural ordinary differential equations (ODEs) have been attracting increasing attention in various research domains recently. There have been some works studying optimization issues and approximation capabilities of neural ODEs, but their robustness is still yet unclear. In this work, we fill this important gap by exploring robustness properties of neural ODEs both empirically and theoretically. We first present an empirical study on the robustness of the neural ODE-based networks (ODENets) by exposing them to inputs with various types of perturbations and subsequently investigating the changes of the corresponding outputs. In contrast to conventional convolutional neural networks (CNNs), we find that the ODENets are more robust against both random Gaussian perturbations and adversarial attack examples. We then provide an insightful understanding of this phenomenon by exploiting a certain desirable property of the flow of a continuous-time ODE, namely that integral curves are non-intersecting. Our work suggests that, due to their intrinsic robustness, it is promising to use neural ODEs as a basic block for building robust deep network models. To further enhance the robustness of vanilla neural ODEs, we propose the time-invariant steady neural ODE (TisODE), which regularizes the flow on perturbed data via the time-invariant property and the imposition of a steady-state constraint. We show that the TisODE method outperforms vanilla neural ODEs and also can work in conjunction with other state-of-the-art architectural methods to build more robust deep networks.

1 INTRODUCTION

The paper studies the unclear robustness of neural ODE-based networks and finds that ODENets are more robust than CNNs under diverse perturbations. It explains this behavior through non-intersecting ODE integral curves and proposes TisODE to further improve robustness and integrate with other robust architectures.

  • ODENet architecture: ODENets use a convolutional feature extractor, a neural ODE representation mapping, and a fully connected classifier for image classification.The neural ODE block provides the nonlinear representation mapping between feature extraction and classification.
  • Motivation and empirical study: The paper systematically evaluates ODENet robustness against Gaussian noise and adversarial examples, comparing models trained on original images alone or also on Gaussian-perturbed images.ODENets are reported as more robust than CNNs across all tested perturbations and both training settings.
  • Robustness mechanism: Non-intersecting integral curves provide the paper’s theoretical explanation for ODENet robustness.The flow property concerns solutions starting from distinct initial states and motivates analyzing how perturbed inputs evolve.
  • TisODE: TisODE removes time dependence from ODE dynamics and imposes a steady-state constraint to regulate changes in perturbed trajectories.The method is designed to control output differences after the terminal time and is reported to be more robust than vanilla neural ODEs.
  • TisODE: TisODE can work alongside other architectural robustness methods to further improve deep-network robustness.The paper presents it as a generally applicable component and drop-in module rather than a replacement restricted to standalone neural ODE models.

2 PRELIMINARIES ON NEURAL ODE

Neural ODEs model input-output mappings by evolving a state continuously under trainable dynamics from an initial input state to a terminal output state. The paper relates them to residual networks and uses automatic differentiation to train their parameters.

  • Neural ODE formulation: Neural ODEs arise as the zero-step-size limit of residual-block discretizations, producing a family of continuous-time neural networks.This frames neural ODEs as continuous counterparts of residual architectures.
  • Neural ODE formulation: The dynamics fθ are trainable layers, while z(t) is the d-dimensional state evolving over nonnegative time.The formulation assumes continuity in time and global Lipschitz continuity in the state.
  • Input-output mapping: The input zin is the state at t = 0, and the output zout is the state z(T) at a chosen terminal time.For fixed T, the output depends on the input and the dynamics fθ.
  • Training: The terminal time is set to 1 in practice, and this work trains θ using automatic differentiation in PyTorch.The paper distinguishes this implementation choice from other available training methods such as adjoint sensitivity.

3 AN EMPIRICAL STUDY ON THE ROBUSTNESS OF ODENETS

Experiments compare ODENets with parameter-matched CNNs across datasets, perturbations, and training settings, finding stronger ODENet robustness. The paper connects this behavior to non-intersecting ODE integral curves, which bound output deviations from small feature perturbations.

  • Experimental settings: Models are evaluated under Gaussian noise, FGSM attacks, and PGD attacks, with training either on clean images alone or clean images plus Gaussian-perturbed images.The study reports mean classification accuracy and standard deviation on perturbed images.
  • Experimental settings: Experiments compare ODENets and CNNs on MNIST, SVHN, and ImgNet10 using matched architectural components and comparable parameter counts.Both models use the same feature extractor and final classifier; their representation-mapping components differ.
  • Robustness without perturbed-image training: ODENets outperform CNNs across all tested perturbation types when trained only on non-perturbed images.For MNIST with Gaussian noise of σ = 100, accuracy is 73.2% for ODENet versus 56.4% for CNN; for SVHN FGSM-5/255, it is 43.0% versus 13.7%.
  • Insights on robustness: The proposed robustness explanation uses non-intersecting integral curves: a small feature perturbation produces an output deviation bounded by the separation of neighboring trajectories.The argument assumes nearby initial conditions and a globally Lipschitz ODE vector field; the authors contrast this bound with CNNs.

4 TISODE: BOOSTING THE ROBUSTNESS OF NEURAL ODES

TisODE strengthens neural ODE robustness by removing time dependence, imposing a steady-state constraint, and regularizing integral curves. Experiments show improved robustness over vanilla ODENets and further gains when combined with other architectural methods.

  • 4.1 TIME-INVARIANT STEADY NEURAL ODES: TisODE removes time dependence from ODE dynamics and imposes a steady-state constraint on integral curves.The time-invariant property makes shifted solutions horizontal translations of one another.
  • 4.1 TIME-INVARIANT STEADY NEURAL ODES: Perturbation-output differences can be bounded using information from the original trajectory, motivating stronger flow regularization.The bound applies to nearby initial states and uses the trajectory’s dynamics over the integration interval.
  • 4.1 TIME-INVARIANT STEADY NEURAL ODES: The steady-state loss Lss regularizes trajectories so outputs from nearby points stabilize around the terminal state.The loss is added during training and aggregates the dynamics along each sample’s solution trajectory.
  • 4.2 EVALUATING ROBUSTNESS OF TISODE-BASED CLASSIFIERS: TisODE-based models are more robust than vanilla ODENets across tested perturbations and datasets.On MNIST, gains exceed 4 percentage points for FGSM-0.3 and reach 6 percentage points for FGSM-0.5; ImgNet10 gains exceed 2 percentage points for FGSM and PGD-5/255.
  • 4.3 TISODE - A GENERALLY APPLICABLE DROP-IN TECHNIQUE FOR IMPROVING THE ROBUSTNESS OF DEEP NETWORKS: TisODE can serve as a drop-in module alongside feature denoising and input randomization methods.The combined methods are evaluated on MNIST and SVHN using perturbed training images.
  • 4.3 TISODE - A GENERALLY APPLICABLE DROP-IN TECHNIQUE FOR IMPROVING THE ROBUSTNESS OF DEEP NETWORKS: Adding TisODE raises PGD-0.3 accuracy by at least 10 percentage points for both FDn and IRd on MNIST.FDn increases from 8.2% to 28.2%, while IRd increases from 55.5% to 66.0%; TisODE also ameliorates Gaussian-noise degradation.

5 RELATED WORKS

Related work covers neural ODE formulations and studies of their optimization and approximation properties, alongside established approaches for improving deep-network robustness.

  • Neural ODE: Neural ODEs model inputs and outputs as states of continuous-time dynamical systems using trainable layers.Earlier work also interpreted several network architectures as discretizations of continuous-time ODEs.
  • Neural ODE: Research on neural ODEs has examined optimization, approximation capabilities, and generalization.Adjoint sensitivity methods address the computational and memory inefficiency of autodifferentiation during training.
  • Robust Improvement: Robustness methods include smoothing the loss surface through Jacobian spectral-norm control and training with adversarial examples.These approaches target robustness against adversarial perturbations through regularization or adversarial training.

6 CONCLUSION

The paper finds that neural ODE-based models are more robust than CNNs and introduces TisODE to further improve robustness. TisODE also functions with other robustness techniques as a drop-in module for robust deep models.

  • Neural ODE-based models are more robust than CNN models.
  • TisODE outperforms vanilla neural ODEs in robustness.
  • TisODE can work with other state-of-the-art techniques to improve deep-network robustness.
  • TisODE is presented as an effective drop-in module for building robust deep models.

7 APPENDIX

The appendix documents ODENet architectures, formalizes assumptions used in the theoretical analysis, and reports robustness experiments under adversarial training and image perturbations.

  • 7.1 NETWORKS USED ON THE MNIST, THE SVHN, AND THE IMGNET10 DATASETS: Table 5 specifies ODENet layer arguments, while BasicBlock uses GroupNorm so each datum's dynamics remain independent within a mini-batch.
  • 7.3 GRONWALL’S INEQUALITY: Theorem 2 states Gronwall's inequality for continuous functions and solutions of initial value problems under a bounded difference condition.
  • Adversarially trained neural ODE-based models are consistently more robust than CNN models, and TisODE outperforms vanilla neural ODE.
  • With equal parameter counts and function-evaluation budgets, neural ODE-based models still outperform CNNs across the reported FGSM and PGD settings.
Loading 1910.05513v4…