PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending Against Adversarial Attacks

Qiyu Kang, Yang Song, Qinxu Ding, Wee Peng Tay

arXiv:2110.12976v1cs.LGcs.CRcs.CV

TL;DR

Human-imperceptible adversarial perturbations can fool DNNs, while existing neural ODE constraints do not ensure robustness to such inputs. SODEF trains features toward Lyapunov-stable equilibria and separates class equilibria; under PGD, it reports higher adversarial-example accuracy than TisODE across three datasets.

  • Problem

    Human-imperceptible perturbations can fool DNNs, and nonintersection or steady-state constraints in prior neural ODEs do not ensure robustness to input perturbations.

    Source: §1 · p. 1

  • Method

    SODEF constrains extracted features to lie near Lyapunov-stable equilibria and uses a final fully connected layer designed to maximize their separation.

    Source: §1 · p. 2

  • Results

    Under PGD, adversarial-example classification accuracy improves over TisODE by up to 44.02, 52.54, and 18.91 percentage points on MNIST, CIFAR-10, and CIFAR-100, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SODEF’s stability-based robustness is compatible with existing neural networks and can be appended to increase their robustness to adversarial attacks.

    Source: §5 · p. 10

    The stated formulation assumes a compact metric input space, an injective continuous feature extractor, and pairwise disjoint class-support sets.

    Source: §3.2 · pp. 5–6

Abstract

from arXiv · show

Deep neural networks (DNNs) are well-known to be vulnerable to adversarial attacks, where malicious human-imperceptible perturbations are included in the input to the deep network to fool it into making a wrong classification. Recent studies have demonstrated that neural Ordinary Differential Equations (ODEs) are intrinsically more robust against adversarial attacks compared to vanilla DNNs. In this work, we propose a stable neural ODE with Lyapunov-stable equilibrium points for defending against adversarial attacks (SODEF). By ensuring that the equilibrium points of the ODE solution used as part of SODEF is Lyapunov-stable, the ODE solution for an input with a small perturbation converges to the same solution as the unperturbed input. We provide theoretical results that give insights into the stability of SODEF as well as the choice of regularizers to ensure its stability. Our analysis suggests that our proposed regularizers force the extracted feature points to be within a neighborhood of the Lyapunov-stable equilibrium points of the ODE. SODEF is compatible with many defense methods and can be applied to any neural network's final regressor layer to enhance its stability against adversarial attacks.

1 Introduction

The introduction frames adversarial perturbations as a safety and performance concern and argues that existing neural ODE constraints do not ensure robustness to them. SODEF targets Lyapunov-stable equilibria and reports improved adversarial-example accuracy under PGD.

1 Introduction

  • Human-imperceptible input perturbations can fool vanilla DNNs into misclassification, raising performance and safety concerns for critical applications.The introduction situates this vulnerability amid applications including image classification, speech recognition, and natural language processing.Source: §1 · p. 1
  • Prior neural ODE constraints based on nonintersecting trajectories or steady states do not ensure inputs lie near Lyapunov-stable equilibria, so they cannot guarantee robustness to small perturbations.An identity mapping satisfies the cited TisODE constraints but is not robust to input perturbations.Source: §1 · p. 2
  • SODEF aims to place extracted features near Lyapunov-stable equilibrium points, constraining the ODE Jacobian’s eigenvalues to have negative real parts.Rather than constraining eigenvalues directly, the proposed approach constrains matrix elements to reduce computational complexity for large Jacobians.Source: §1 · p. 2
  • The proposed final fully connected layer uses unit-norm rows with minimized maximum pairwise cosine similarity, and can be constructed offline.The paper presents this diversity-promoting design as a way to improve neural ODE robustness.Source: §1 · p. 2
  • Under PGD, adversarial-example classification accuracy improves over TisODE by up to 44.02, 52.54, and 18.91 percentage points on MNIST, CIFAR-10, and CIFAR-100, respectively.The reported improvements over ODE net are up to 43.69, 52.38, and 18.99 percentage points on the same datasets.Source: §1 · p. 2

2 Preliminaries: Neural ODE and Stability

The preliminaries describe neural ODEs as continuous-time feature mappings and use local linearization and eigenvalue conditions to characterize stability. Near a Lyapunov-stable equilibrium, small perturbations converge toward the same equilibrium as time increases.

2 Preliminaries: Neural ODE and Stability

  • A neural ODE maps input state z(0) to output z(T) through a differential equation with a trainable function fθ.The paper treats neural ODEs as continuous analogues of residual networks and, for simplicity, considers autonomous systems that do not explicitly depend on time.Source: §2 · p. 3
  • Near a hyperbolic equilibrium, the Hartman–Grobman result allows the nonlinear system’s local behavior to be studied through its linearization.A hyperbolic equilibrium has no Jacobian eigenvalue with zero real part.Source: §2 · p. 3
  • For the linear system, asymptotic stability holds if and only if every eigenvalue of its matrix has negative real part.The paper distinguishes this from marginal stability, which permits simple eigenvalues with zero real part.Source: §2 · p. 3
  • Near a Lyapunov-stable equilibrium, a sufficiently small input perturbation converges toward the same equilibrium, limiting its effect on the later network output.The paper describes this perturbation weakening as acting like a noise filter against adversarial attacks.Source: §2 · p. 3
  • Strict diagonal dominance with real negative diagonal entries guarantees that a matrix is nonsingular and all its eigenvalues have negative real parts.This criterion supplies a matrix-element condition relevant to enforcing stability.Source: §2 · p. 3

3 SODEF Architecture

SODEF maps extracted features through a neural ODE layer and a final fully connected layer. Its architecture uses a diversity-promoting final layer and an optimization framework intended to place features near Lyapunov-stable equilibrium points.

3 SODEF Architecture

  • The feature extractor produces an embedding, the neural ODE stabilizes it, and the final fully connected layer maps the ODE output to predictions.The model parameters are φ, θ, and V for these three components, respectively.Source: §3 · p. 4

3.1 Maximizing the Distance between Lyapunov-Stable Equilibrium Points

  • The final layer maps each class to a vector and minimizes the maximum cosine similarity between class vectors to separate their equilibrium points.The layer is added because stable neighborhoods may otherwise be small when different classes’ equilibrium points lie near one another.Source: §3.1 · p. 4
  • On MNIST and CIFAR10, the proposed fixed FC layer improves ODE net and TisODE defense, with class features well separated even under attacks.The paper visualizes MNIST test-set features and FGSM adversarial features before the final FC layer.Source: §3.1 · p. 5, Figure 2 · p. 5, Figure 3 · p. 6

3.2 Objective Formulation and Stability

  • SODEF’s constraints place input features near Lyapunov-stable equilibria with strictly diagonally dominant derivatives, while bounded regularizer terms are added to the loss.The regularizers use monotonically increasing functions bounded below; the implementation sets both functions to exp(·).Source: §3.2 · pp. 6–7
  • Lemma 3 rules out making almost every point stable when the feature measure is continuous on an open set.The paper therefore relaxes the requirement to finitely many stable equilibria and an expected vector-field norm below ε.Source: §3.2 · p. 7
  • Under Theorem 4’s measure conditions, the constraint-satisfying function space is nonempty for every ε > 0, and a stronger condition ensures at least one stable equilibrium in each class support.The first condition excludes a continuous uniform measure on each class support; the stronger one applies to restrictions on open subsets of those supports.Source: §3.2 · p. 7

4 Experiments

Experiments evaluate SODEF across standard datasets, attack types, integration times, and defense settings. They report improved robustness under AutoAttack and PGD, while also examining diversity promotion and transfer attacks.

4 Experiments / 4.1 Setup

  • Experiments compare ODE net and TisODE with SODEF on MNIST, CIFAR-10, and CIFAR100, using pretrained feature extractors, a fixed designed FC layer, and FGSM, PGD, and AutoAttack.Training first fixes the ODE function for 30 epochs, then trains it with the feature extractor released for 120 epochs.Source: §4 · p. 7, §4.1 · pp. 7–8

4.2 Compatibility of SODEF

  • Appending SODEF to TRADES improves its CIFAR-10 robustness under all four AutoAttack components and the ensemble, raising strong L2 AutoAttack accuracy from 59.42% to 67.75%.The experiment fixes the pretrained TRADES feature extractor except for its final fully connected layer.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 Influence of Integration Time T

  • On CIFAR100 under L2 AutoAttack, SODEF generally performs better at larger integration times T, with no obvious improvement beyond T = 10.The feature extractor is trainable in this experiment.Source: §4.3 · p. 8, Table 4 · p. 9

4.4 Performance Comparison Under AutoAttack

  • 79.10% accuracy for SODEF versus less than 3% for other models under the strongest AutoAttack, with integration time matched across models.The comparison is on CIFAR100 adversarial examples at T = 10.Source: §4.4 · p. 9, Table 5 · p. 9

4.5 Performance Under PGD and FGSM Attacks

  • 45.25% − 1.56% = 43.69% and 45.25% − 1.23% = 44.02% are SODEF’s PGD improvements over ODE and TisODE on MNIST; on CIFAR-10, gains reach 52.38% and 52.54%.SODEF also maintains high accuracy on normal examples on both datasets.Source: §4.5 · pp. 9–10
  • On CIFAR100, the strongest attack reduces vanilla EfficientNet accuracy by 74.6%, while TisODE rises only from 2.39% to 3.44% under PGD and SODEF retains defense capability.The vanilla EfficientNet clean accuracy is 88.3%.Source: §4.5 · p. 10

4.6.1 Impact of Diversity Promotion

  • Fixing the final fully connected layer to the diversity-promoting matrix improves defense performance over using a trainable linear layer.The proposed explanation is that this separates class feature supports and their stable equilibrium points.Source: §4.6.1 · p. 10, Table 8 · p. 10

4.6.2 Transferability Study

  • On CIFAR-10, transfer attacks from ResNet18 reduce accuracy from 68.05% to 59% for FGSM and from 55.59% to 34% for PGD, yet SODEF still outperforms other ODEs without transfer attacks.The authors suggest gradient masking may contribute to transfer attacks deteriorating the defense effect.Source: §4.6.2 · p. 10

5 Conclusion

The conclusion presents SODEF as a neural ODE designed to suppress input perturbations through Lyapunov stability. Its theoretical and empirical findings connect stability constraints and separated class equilibria with adversarial robustness.

5 Conclusion

  • SODEF is compatible with existing networks and can be appended to their final regressor layer to enhance stability against adversarial attacks.The paper reports theoretical and empirical evidence that robustness mainly derives from stability.Source: §5 · p. 10
  • Training constraints ensure the neural ODE layer’s Jacobian eigenvalues have negative real parts, while the final FC layer can maximize distances between class-specific stable equilibria.The latter design applies when each classification class converges to its own equilibrium points.Source: §5 · p. 10

Broader Impact

The broader impact discussion notes that more robust DNN deployment could mitigate adversarial threats while also increasing automation-related employment and social risks. Related-work material distinguishes Lyapunov stability from other stability notions and outlines attack types and parameters.

Broader Impact

  • More robust DNNs could mitigate adversarial threats, but wider automation of high-risk tasks could also bring mass-scale unemployment and social unrest.The discussion connects this concern to tasks where failures could cause harm to human lives or major property losses.Source: Broader Impact · p. 11

Stable Neural Network

  • Some architectures target norm-preserving linearized input-output behavior, whereas this paper pursues Lyapunov stability; prior ODE non-intersection and steady-state constraints do not generally guarantee perturbation robustness.The cited work proposes imposing Lyapunov-stability constraints during training.Source: Stable Neural Network · p. 11

Adversarial Examples

  • Adversarial examples can target image, audio, and point-cloud systems; attacks are categorized as white-box or black-box, and defenses discussed include adversarial training and distillation.The paper defines an adversarial example as a misclassifying input within an Lp-ball of radius ε around the normal example.Source: Adversarial Examples · pp. 11–12
  • FGSM uses a gradient-sign perturbation, while PGD iteratively refines it with smaller gradient steps and clips inputs to the allowed perturbation range.The passages describe FGSM’s loss, perturbation magnitude, and pixelwise update direction.Source: Adversarial Examples · p. 12
  • AutoAttack combines APGD variants with FAB and Square attacks as a parameter-free ensemble for testing adversarial robustness.The ensemble includes white-box and black-box attacks.Source: Adversarial Examples · p. 12

B Attack Parameters

  • Vanilla white-box FGSM and PGD use L∞ perturbations of 0.3 for MNIST and 0.1 for CIFAR10, while AutoAttack uses L2 ε = 0.5 and L∞ ε = 8/255.PGD is iterated 20 times with step size 0.1.Source: B Attack Parameters · p. 12

C Influence of Attack Strength

The attack-strength experiments show that robustness depends on perturbation magnitude, and that SODEF’s feature-space defense exceeds the ODE baseline at a tested high strength. The eigenvalue comparison shows negative real parts for SODEF’s Jacobian eigenvalues, unlike many ODE net eigenvalues.

C Influence of Attack Strength

  • At feature perturbation ε = 0.5, SODEF reaches 59.9% accuracy versus 18.5% for ODE net, while accuracy without an ODE layer falls to zero.The authors also report that accuracy decreases as input FGSM attack strength increases and that SODEF performs well when perturbations are smaller.Source: C Influence of Attack Strength · p. 12, Figure 5 · p. 13

D Eigenvalues Visualization

  • SODEF’s Jacobian eigenvalues all have negative real parts, whereas many ODE net eigenvalues lie in the right plane.The eigenvalue visualization compares ∇fθ(z(0)) for ODE net and SODEF alongside FGSM accuracy across attack strengths.Source: D Eigenvalues Visualization · p. 13, Figure 5 · p. 13

E Classification Accuracy on CIFAR10 using AutoAttack

On CIFAR10, SODEF performs well against each tested attack and the strongest AutoAttack ensemble when integration time is sufficiently large. The evaluation varies integration time under AutoAttack with an L2 norm.

E Classification Accuracy on CIFAR10 using AutoAttack

F Classification Accuracy on CIFAR100 using FGSM and PGD

On CIFAR100, SODEF is reported to be more robust than competitors, especially under stronger attacks. In a separate comparison, fine-tuning the feature extractor with SODEF performs best among the three tested configurations.

F Classification Accuracy on CIFAR100 using FGSM and PGD

  • SODEF is consistently more robust than competitors on CIFAR100, especially when stronger attacks are applied.Table 10 reports classification accuracy on adversarial CIFAR100 examples.Source: F Classification Accuracy on CIFAR100… · p. 14, Table 10 · p. 14
  • The fixed-feature experiment trains SODEF while keeping the pretrained EfficientNet fixed, and compares it with fine-tuned SODEF and the pretrained model alone.The pretrained model is reported to have 88% clean accuracy on CIFAR100.Source: F Classification Accuracy on CIFAR100… · p. 14
  • Fine-tuned SODEF performs best among the three CIFAR100 configurations, while fixed-feature SODEF outperforms the vanilla pretrained model.The comparison covers PGD and FGSM attacks at ε = 0.1; the authors suggest insufficiently diversified z(0) as one possible reason for the fixed-feature model’s lower performance than the fine-tuned model.Source: F Classification Accuracy on CIFAR100… · p. 14, Table 11 · p. 14

G Further Ablation Studies

The ablations examine ODE-block stability, the designed final fully connected layer, and alignment of terminal features with prescribed vectors. Their comparisons associate each design choice with improved robustness.

G Further Ablation Studies

  • Comparing SODEF with Net3 highlights the importance of stability in the ODE block for adversarial robustness.The ablation uses a linear ODE function fθ(z(t)) = −CC⊺z(t), guaranteed asymptotically stable when C has full rank.Source: G Further Ablation Studies · p. 14, Table 13 · p. 15
  • SODEF’s designed fully connected layer is reported superior to the alternatives Net1 and Net2 in the CIFAR10 ablation.Table 13 reports classification accuracy on adversarial CIFAR10 examples generated with 20 PGD iterations.Source: G Further Ablation Studies · p. 14, Table 13 · p. 15
  • Aligning z(T) with a prescribed feature vector, such as a row in V, improves robustness in the comparison of Net1, Net2, and SODEF.Source: G Further Ablation Studies · p. 14

H Further Transferability Studies

Transfer tests compare direct attacks on SODEF with attacks transferred from substitute networks, while solver tests report similar performance across the tested ODE solvers.

H Further Transferability Studies

I Different ODE Solvers

  • Runge-Kutta orders 5 and 2, Euler, Midpoint, and the fourth-order Runge-Kutta 3/8 rule show similar performance in the solver tests.Euler, Midpoint, and the fourth-order 3/8-rule solver use step size 0.05.Source: I Different ODE Solvers · p. 15

J Proofs of Results in Paper

This appendix provides detailed proofs of results stated in the main text. The proofs use a Gram-matrix construction and Whitney extension to establish the stated results and constraints.

J Proofs of Results in Paper

  • Table 16 identifies the evaluation as SODEF classification accuracy under FGSM attack on adversarial CIFAR10 examples across different ODE solvers.The caption gives ε = 0.1; it does not state accuracy values.Source: Table 16 · p. 16

J.1 Proof of Lemma 2

  • Lemma 2 establishes a ≥ 1/(1−k) and attains equality using a diagonally dominant matrix B that is a Gram matrix.The construction uses the fact that a symmetric diagonally dominant real matrix with nonnegative diagonal entries is positive semidefinite.Source: §J.1 · p. 16

J.2 Proof of Corollary 1

  • Corollary 1 follows from the construction of B as the Gram matrix of the vectors.The proof refers to the preceding construction of B.Source: §J.2 · p. 16

J.3 Proof of Lemma 1

  • Lemma 1 follows because the finite set of points is closed, allowing application of the Whitney extension theorem [45].The passage identifies the closedness of the finite set as the basis for this consequence.Source: §J.3 · p. 16

J.4 Proof of Lemma 3

  • Lemma 3 shows that a continuous function vanishing νφ-almost surely must vanish throughout E′, so its points cannot be Lyapunov-stable equilibria.A nonzero value would persist on an open ball of positive measure, contradicting almost-sure vanishing.Source: §J.4 · p. 16

J.5 Proof of Theorem 4

  • Theorem 4 constructs coordinatewise independent dynamics with negative diagonal Jacobians at chosen equilibria, then uses small β and Whitney extension to satisfy the stated constraints.The coordinatewise form satisfies constraint (5); sufficiently small β satisfies constraint (3), and the construction satisfies constraint (4).Source: §J.5 · p. 16
Loading 2110.12976v1…