PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Learning with Jacobian Regularization

Judy Hoffman, Daniel A. Roberts, Sho Yaida

arXiv:1908.02729v1stat.MLcs.LG

TL;DR

Machine-learning models need stability against distribution shifts, input corruption, and adversarial perturbations. The paper introduces Jacobian regularization and an efficient random-projection implementation, finding improved robustness while preserving practical clean-data performance.

  • Problem

    Models must remain reliable under unseen-data shifts and adversarial or corrupted inputs, where instability can make predictions highly vulnerable.

  • Method

    The paper minimizes the input-output Jacobian norm alongside the task loss and approximates it efficiently with random projections compatible with automatic differentiation.

  • Results

    Jacobian regularization enlarges decision cells and improves robustness against random and adversarial perturbations without severely degrading clean-data generalization.

  • Takeaways & Limitations

    The scheme is presented as a task-agnostic, easily implementable addition to existing learning pipelines for improving model stability against input-data corruption.

  • Takeaways & Limitations

    The expected robust-learning outcome depends on sufficient training data and a properly chosen λJR.

Abstract

from arXiv · show

Design of reliable systems must guarantee stability against input perturbations. In machine learning, such guarantee entails preventing overfitting and ensuring robustness of models against corruption of input data. In order to maximize stability, we analyze and develop a computationally efficient implementation of Jacobian regularization that increases classification margins of neural networks. The stabilizing effect of the Jacobian regularizer leads to significant improvements in robustness, as measured against both random and adversarial input perturbations, without severely degrading generalization properties on clean data.

1 Introduction

The paper frames robustness as stability against distribution shifts, sparse sampling, and adversarial perturbations, then proposes Jacobian regularization to enlarge decision cells and improve robustness without severely harming clean-data learning.

  • Motivation: Machine-learning models must withstand shifts between training and unseen test data while resisting adversarial exploitation of instability.The challenge is especially severe under biased training data or sparse sampling of the underlying distribution.
  • Motivation: L2 regularization promotes generalization, but whether it increases neural-network classification margins and stability remains unclear.For linear classifiers and support vector machines, L2 regularization maximizes margins; its effect on neural networks is less certain.
  • Contribution: Jacobian regularization pushes decision boundaries farther from training points, enlarging decision cells and reducing instability.Unlike L2 regularization, which smooths boundaries without necessarily enlarging cells, the Jacobian scheme directly supports larger classification margins.
  • Contribution: The paper aims to make Jacobian regularization generic across architectures, domains, and tasks, including use alongside existing robustness and generalization methods.The authors emphasize ease of integration into existing learning pipelines.
  • Evaluation: The evaluation first checks clean-data learning, then tests robustness to random and adversarial input perturbations.This ordering addresses the possibility that regularization constraints could adversely affect learning on unperturbed inputs.

2 Method

The method minimizes the input-output Jacobian’s Frobenius norm alongside the task loss, using random projections for efficient automatic-differentiation-based computation. This targets local prediction stability while avoiding the output-class-scaled cost of exact computation.

  • Jacobian regularization: For sufficiently small perturbations, the input-output Jacobian governs prediction stability because higher-order Taylor terms can be neglected.Larger Jacobian components correspond to greater prediction instability with respect to input perturbations.
  • Jacobian regularization: The method regularizes learning by minimizing the squared Frobenius norm of the input-output Jacobian alongside the classification objective.The Jacobian captures how output scores change under small input perturbations, and λJR controls the regularizer’s relative importance.
  • Jacobian regularization: For nonlinear models, Jacobian regularization differs from L2 regularization and is expected to enlarge decision cells by reducing prediction variation under input perturbations.The paper contrasts this behavior with L2 regularization, which does not generally produce the same effect for nonlinear networks.
  • Approximate computation: Random projections provide an unbiased approximation to the Jacobian norm with constant-time overhead and negligible solution-quality difference from exact computation.The estimator samples nproj random vectors from the output-space unit sphere and has convergence behavior analyzed theoretically and empirically.
  • Exact computation: The exact Frobenius-norm computation uses output-space basis vectors and requires backpropagating through the model once per output class.Automatic differentiation computes derivatives of contracted outputs, enabling the exact construction and gradient computation.
  • Approximate computation: With mini-batch size |B| = 100, a single projection nearly matches the exact method while reducing computational cost by orders of magnitude.The expected estimator error scales as (nproj|B|)^-1/2 under nearly independent and identically distributed mini-batch samples.

3 Experiments

Experiments evaluate Jacobian regularization on clean-data generalization, unseen-domain transfer, and robustness to random and adversarial perturbations. The method reduces Jacobian norms while retaining or improving generalization and substantially improving robustness.

  • Robustness under data corruption: Jacobian regularization significantly improves robustness to random and adversarial input perturbations.The experiments evaluate corruption using white noise, PGD, and Carlini-Wagner attacks.
  • Experimental scope: The largest observed computational increase is ≈2 times for ResNet-18 on CIFAR-10, while LeNet’ on MNIST was measured on a single NVIDIA GP100.The paper notes that computational efficiency depends on datasets and model architectures.
  • Generalizing within domain: Jacobian regularization reduces the Jacobian norm while retaining test accuracy on clean MNIST data.Table 1 evaluates LeNet’ models trained with varying numbers of samples per class, using confidence intervals across repeated runs.
  • Generalizing to a new domain: Each tested regularizer, including Jacobian regularization, increases accuracy on the unseen USPS domain over an unregularized model.The regularizers may also be combined for stronger generalization effects.
  • Robustness under data corruption: Under white-noise perturbations, Jacobian minimization is the most effective regularizer, while Jacobian regularization alone outperforms adversarial training under stronger attacks.Figure 3 uses LeNet’ trained on MNIST; base models include L2 and dropout regularization.

4 Related Work

The paper situates Jacobian regularization among methods that penalize input derivatives, including double backpropagation and layer-wise Jacobian penalties. Its computational approach addresses the overhead of prior full-Jacobian implementations.

  • Derivative-based regularization: Double backpropagation is an early method that adds squared input derivatives to the loss to reduce the generalization gap.Related approaches subsequently appeared in several forms across the literature.
  • Derivative-based regularization: Prior layer-wise Jacobian regularization can constrain model capacity more strongly than minimizing the input-output Jacobian.The paper links prior reports of degraded clean-test performance and marginal robustness improvements to layer-wise regularization.
  • Computational efficiency: Earlier full-Jacobian implementations incurred computational overhead scaling linearly with the number of output classes, sometimes requiring a layer-wise approximation.The paper states that this computational problem was resolved by its random-projection approach and a contemporaneous approach.
  • Adversarial defenses: Defensive distillation appeared robust to FGSM and PGD under one temperature setting but became fragile when attacked at the same temperature used during training.The paper attributes this observation to a numerical-precision issue reported in prior work.

5 Conclusion

The paper presents Jacobian regularization as a task-agnostic, simply implementable method for improving model stability against input perturbations. Experiments show enlarged decision cells and practical gains in generalization and robustness.

  • Conclusion: Jacobian regularization is presented as a task-agnostic method for improving model stability against input perturbations.The authors state that it can be implemented in open-source automatic differentiation systems.
  • Conclusion: The method enlarges decision cells and improves generalization and robustness in practice.The conclusion frames the scheme as complementary to other machine-learning techniques.

A Gallery of Decision Cells

Figure S2 compares adversarial-hyperplane decision-cell cross sections for an unprotected LeNet’ model and one using Jacobian regularization with adversarial training. Adversarial examples from the unprotected model do not fool the Jacobian-regularized model.

  • Adversarial examples constructed with the unprotected model do not fool the model trained with Jacobian regularization.
  • Figure S2 contrasts MNIST LeNet’ decision-cell cross sections along hyperplanes spanned by FGSM adversarial examples.The test sample is marked black; dark-grey and white-grey dots identify adversarial examples for the two model configurations.

B Additional Details for Efficient Algorithm

The appendix develops an efficient random-projection estimator and analyzes its variance, while also describing a closed-form derivative implementation for multilayer perceptrons.

  • Random-projection estimator: Uniform sphere vectors can be sampled by normalizing a vector whose components are independently drawn from N(0, 1).This sampling procedure supports the random-projection estimator.
  • Estimator variance: The derivation computes the estimator variance using orthogonal-matrix integration and simplifies it after cancellations.
  • Estimator variance: The right-hand side of the variance expression is independent of the Jacobian and therefore independent of model architecture and dataset details.
  • Estimator variance: The random-projection estimate’s relative error decreases with n_proj^-1/2 and is further suppressed when averaging over a mini-batch.
  • Estimator variance: A large-C limit may provide additional variance suppression through structure in Jacobian traces, but this is presented as speculation.
  • Closed-form derivative: A closed-form derivative for multilayer perceptrons bypasses random projections while retaining computational efficiency.The authors provide implementation details intended for practical use and generalization to other models.
  • Cyclopropagation: Cyclopropagation computes Ω^(ℓ) by backpropagating toward the input, forwardpropagating to the output, and backpropagating back to later layers.

D Details for Model Architectures

The appendix specifies LeNet’, DDNet, and modified or standard ResNet-18 architectures used across MNIST, CIFAR-10, and ImageNet experiments, including their regularization conventions.

  • LeNet’: LeNet’ uses two convolutional layers, fully connected layers of widths 120 and 84, dropout, and 10-dimensional MNIST output logits.Its nonlinear activations are hyperbolic tangent.
  • DDNet: DDNet uses four convolutional layers, fully connected layers of widths 256 and 256, dropout, ReLU activations, and 10-dimensional CIFAR-10 output logits.
  • ResNet-18: The CIFAR-10 ResNet-18 is modified for 32-by-32 inputs and uses standard PyTorch parameter initialization.
  • ResNet-18: ImageNet experiments use the standard torchvision ResNet-18 model with standard weight initialization.
  • Regularization: ResNet models typically omit dropout, but the experiments still examine L2 regularization combined with Jacobian regularization.
  • Evaluation: Table 3 evaluates DDNet generalization on CIFAR-10 test data across varying training samples per class and reports reduced Jacobian norms with Jacobian regularization.Errors are 95% confidence intervals over 5 full-training and 15 subsample-training runs.

E Results for CIFAR-10

CIFAR-10 experiments evaluate robustness and generalization for DDNet and ResNet-18 under specified preprocessing, optimization, and regularization settings. Jacobian regularization substantially improves stability and retains robustness gains against white noise and CW attacks, while PGD results are mixed at high degradation when combined with adversarial training.

  • Experimental setup: CIFAR-10 experiments use 32-by-32 color images in ten categories, with 50,000 training and 10,000 test samples.Pixels are rescaled to [−1, 1] after preprocessing.
  • Experimental setup: The experiments use λ_WD = 5·10^-4, p_drop = 0.5, λ_JR = 0.01, and FGSM amplitudes ε_FGSM ∈[0, 0.01].
  • Generalization and stability: Jacobian regularization improves stability by orders of magnitude, and combining it with other regularizers does not compromise this effect.
  • Robustness: Jacobian regularization retains success against white-noise corruption and CW adversarial attacks.
  • Robustness: PGD results are mixed at high degradation when Jacobian regularization is combined with adversarial training.The paper suggests this may reflect PGD’s search behavior and Jacobian regularization’s effect on the input-space loss landscape.
  • Robustness: Figures S3 and S4 report robustness comparisons for DDNet and ResNet-18, with standard deviations estimated over five distinct runs.

F White noise vs. FGSM vs. PGD vs. CW

Attack effectiveness follows CW > PGD > FGSM > white noise, reflecting increasing search complexity. Simple FGSM and PGD attacks can misrepresent decision-cell geometry, while some attack settings introduce additional caveats.

  • CW > PGD > FGSM > white noise in attack effectiveness, respecting the complexity of adversarial-example search methods.
  • Figure S5 compares attack effects on LeNet’ MNIST, DDNet CIFAR-10, and ResNet-18 CIFAR-10 with and without defense.The defended models use Jacobian regularization, adversarial training, L2 regularization, and, except for ResNet-18, dropout.
  • FGSM and PGD can yield erroneous pictures of decision-cell geometry, especially for the closest decision boundary, compared with CW.
  • PGD results include a kink when each pixel is constrained to remain within 32/255 of its original clean value.The authors describe this saturation constraint as unnatural but customary.
  • CW fools almost all LeNet’ MNIST and DDNet CIFAR-10 examples but fails on some ResNet-18 CIFAR-10 examples beyond a distance.The CW hyperparameters were not carefully tuned to resolve this issue.

G Dependence on Jacobian Regularization magnitude

Increasing Jacobian regularization generally improves robustness, but clean-sample performance can degrade by varying amounts. In typical settings, fooling distance can double without much clean-performance degradation.

  • Increasing λJR generally increases model robustness, with varying degradation in clean-sample performance.
  • Typically, fooling distance can double without much degradation in clean-sample performance.
  • Figure S6 evaluates robustness versus λJR across models whose base configurations include L2 regularization and, except for ResNet-18, dropout.Shades indicate standard deviations over 5 distinct runs.

H Results for ImageNet

ImageNet experiments train and evaluate ResNet-18 under specified preprocessing, optimization, and Jacobian-regularization settings. The Jacobian regularizer confers robustness especially against adversarial attacks, but shows no visible white-noise improvement.

  • ImageNet uses ILSVRC images labeled across 1,000 object categories, with training on the training set and evaluation on the validation set.Inputs are standardized and training adds random resized crops and horizontal flips.
  • ResNet-18 is evaluated after 100 epochs with λJR ∈ {0, 0.0001, 0.0003, 0.001} and λWD = 0.0001.Training uses SGD with batch size 256, momentum 0.9, and a learning-rate schedule.
  • PGD uses FGSM iterations with εFGSM = 1/255 and a 16/255 per-pixel saturation constraint in the ImageNet evaluation.CW hyperparameters were unchanged and not fine-tuned.
  • Figure S7 reports ImageNet robustness dependence on λJR for ResNet-18 with L2 regularization in a single run.The CW evaluation uses 10,000 test examples rather than 1,000 to compensate for the lack of multiple runs.
  • The Jacobian regularizer confers robustness to ImageNet models, especially against adversarial attacks, but shows no visible improvement against white-noise perturbations.The authors hypothesize that baseline white-noise robustness may already be strong, but leave this for further investigation.
Loading 1908.02729v1…