PaperlayerView exact revision on arXiv

Source-linked AI summary

Sobolev Training for Neural Networks

Wojciech Marian Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Świrszcz, Razvan Pascanu

arXiv:1706.04859v3cs.LG

TL;DR

Neural-network training often ignores target derivatives even when they are available, leaving potentially useful function information unused. Sobolev Training matches both target values and derivatives, with theory and experiments showing more data-efficient learning and improved performance and generalization across several applications. The method also supports stochastic approximations for expensive Jacobian or Hessian calculations.

  • Problem

    Neural-network approximators often use only input-output pairs although target derivatives may be available, and direct derivative matching has been minimally explored for deep ReLU-based models.

  • Method

    Sobolev Training optimizes neural networks to match a target function’s values and derivatives, using stochastic approximations when Jacobians or Hessians are computationally expensive.

  • Results

    Across regression, Atari policy distillation, and synthetic-gradient applications, Sobolev Training produces improved performance and generalization, including in low-data regimes.

  • Takeaways & Limitations

    Encoding target derivatives in neural-network parameters enables more data-efficient function approximation and supports applications including model distillation and synthetic gradients.

  • Takeaways & Limitations

    Performance can suffer on high-frequency functions in low-data regimes, and label-derivative supervision was omitted from large-scale experiments because TensorFlow did not support differentiating cross entropy with respect to labels.

Abstract

from arXiv · show

At the heart of deep learning we aim to use neural networks as function approximators - training them to produce outputs from inputs in emulation of a ground truth function or data creation process. In many cases we only have access to input-output pairs from the ground truth, however it is becoming more common to have access to derivatives of the target output with respect to the input - for example when the ground truth function is itself a neural network such as in network compression or distillation. Generally these target derivatives are not computed, or are ignored. This paper introduces Sobolev Training for neural networks, which is a method for incorporating these target derivatives in addition the to target values while training. By optimising neural networks to not only approximate the function's outputs but also the function's derivatives we encode additional information about the target function within the parameters of the neural network. Thereby we can improve the quality of our predictors, as well as the data-efficiency and generalization capabilities of our learned function approximation. We provide theoretical justifications for such an approach as well as examples of empirical evidence on three distinct domains: regression on classical optimisation datasets, distilling policies of an agent playing Atari, and on large-scale applications of synthetic gradients. In all three domains the use of Sobolev Training, employing target derivatives in addition to target values, results in models with higher accuracy and stronger generalisation.

1 Introduction

Sobolev Training incorporates target derivatives alongside function values to train neural-network approximators. The paper motivates this approach theoretically and evaluates it across regression, policy distillation, and synthetic-gradient applications.

  • Motivation: Deep neural networks typically learn from input-output pairs, but some target functions also provide observable or computable gradients and higher-order derivatives.This includes cases where the ground-truth function is itself a neural network.
  • Approach: Sobolev Training matches a model’s outputs and derivatives to those of the target function, extending neural-network approximation to Sobolev-space information.The approach is inspired by universal approximation results for function values and derivatives.
  • Related work: Prior derivative-based approaches include Gaussian-process optimization, complexity penalties, handcrafted invariances, dynamical-system estimation, attention distillation, and critic training.The cited literature spans function approximation, reinforcement learning, and related modeling objectives.
  • Research gap: Directly matching target derivatives has been minimally explored for deep, especially modern ReLU-based, networks.The paper positions its method as addressing this gap while encoding derivative information in trainable models.
  • Contributions: The paper contributes theoretical analysis and empirical evaluations in classical optimization regression, Atari policy distillation, and synthetic-gradient training.The authors report improved performance and generalization, particularly in low-data regimes, and a first successful large-scale ImageNet synthetic-gradient model.

2 Sobolev Training

Sobolev Training augments ordinary function fitting with losses on target derivatives. The derivative losses can be optimized with backpropagation, while stochastic projections reduce the cost of high-dimensional derivative calculations.

  • Derivative supervision: Training data can include target function values and j-th order derivatives at the input training points.This extends the usual dataset of input-output pairs with derivative supervision.
  • Derivative supervision: The model is trained by making its derivatives match the corresponding derivatives supplied by the target function.The method incorporates this supervision directly into neural-network training.
  • Objective: Sobolev objectives add derivative-specific losses to the ordinary function-approximation loss.These losses cause the model’s own derivatives to encode derivative information about the target.
  • Optimization: The resulting model remains trainable with backpropagation and off-the-shelf optimizers.Derivative matching is integrated into a standard neural-network training pipeline.
  • Stochastic approximation: Stochastic Sobolev Training uses random projections instead of full Jacobians when output dimensionality or derivative order makes exact computation expensive.Figure 1 contrasts non-stochastic and stochastic order-2 training; the stochastic version avoids adding computational complexity through efficient approximations.

3 Theory and motivation

The theory argues that incorporating derivative information makes function approximation well posed and can reduce the data needed to identify target functions. ReLU networks can approximate C1 functions in Sobolev space S1, while formal sample-complexity results establish gains for several function families.

  • Motivation: Sobolev Training is motivated as a way to reduce sample complexity by matching target-function values and derivatives.The objective is described as well posed and potentially capable of drastically reducing the sample complexity of learning.
  • Illustration: Figure 2 illustrates why derivative information can identify piecewise-linear segments more efficiently than function values alone.The figure contrasts multiple hypotheses consistent with three training points against the single hypothesis consistent with those points plus derivatives.
  • Approximation theory: ReLU and leaky ReLU networks can approximate every C1 function on a compact set in Sobolev space S1 to arbitrary positive error.This establishes that the values and first derivatives of the target can be encoded by a ReLU network model.
  • Finite-data feasibility: For finite training sets, ReLU networks can exactly match arbitrary target values and derivative vectors at the training points.The stated construction achieves zero training loss for both quantities.
  • Sample complexity: Derivative-enriched observations require no more samples than value-only observations under the paper’s definitions of Ksob and Kreg.The paper defines Ksob using agreement on values and derivatives, whereas Kreg uses agreement on values alone.
  • Sample complexity: Ksob is strictly smaller than Kreg for Gaussian densities and element-wise linear piecewise families, with ReLU networks offering an intuitive hyperplane-based analogue.The paper cautions that the proposition does not directly apply to deep ReLU networks because their parameters and hinge positions are not independent.

4 Experimental Results

Across regression, Atari policy distillation, and synthetic gradients, Sobolev Training uses derivative information to improve approximation and generalization, especially when data are scarce. The method also improves synthetic-gradient accuracy, while Ackley’s high-frequency structure exposes a low-data weakness.

  • Artificial Data: Sobolev Training significantly reduced testing error on six of seven optimization benchmarks, often by orders of magnitude versus regular regression.It also captured the functional shape of the Styblinski-Tang function in very low-data regimes.
  • Artificial Data: The benefit was strongest with little data but persisted with 10,000 training examples for a two-dimensional function.This indicates that the effect does not disappear in the high-data regime.
  • Artificial Data: Ackley’s function was the only benchmark where regular regression performed better in the reported setting.Its high-frequency cosine structure can make derivative matching unreliable when training data are insufficient, but Sobolev Training improves once enough points are available.
  • Distillation: In Atari policy distillation, Sobolev Training improved generalization and produced significant gains in the non-Pong games, with minor to no overfitting.The study distilled A3C-trained policies for Pong, Breakout, and Space Invaders into smaller networks using held-out episodes for testing.
  • Synthetic Gradients: Table 1 reports CIFAR-10 accuracy of 79.2% for naive synthetic gradients, 88.5% for an architecture-matched model, 93.5% with Sobolev Training, and 93.2% for a regular critic.The Sobolev synthetic-gradient module adds gradient supervision so predicted gradients remain aligned with the true gradients.
  • Synthetic Gradients: On ImageNet with ResNet50, Sobolev synthetic gradients reached 87.4% top-5 accuracy versus 86.9% for regular critic synthetic gradients and showed smaller standard deviations.The experiments also found loss supervision crucial for synthetic-gradient modules in ResNet-like architectures.

5 Discussion and Conclusion

The paper introduces Sobolev Training as a way to encode target values and derivatives in neural-network approximators. It provides theoretical justification, efficient stochastic training for costly derivatives, and improvements in applications including distillation and meta-optimisation.

  • Sobolev Training incorporates target-function derivatives alongside target values when training neural-network function approximators.The method encodes both kinds of information in the network parameters.
  • Theoretical results establish that ReLU networks can encode both a target function and its derivatives, supporting more data-efficient learning.
  • Stochastic approximations enable efficient training when computing Jacobians or Hessians is computationally expensive.
  • Experiments show significant improvement over classical techniques for model distillation or compression and meta-optimisation applications.The latter includes synthetic gradients and learning-to-learn settings.
  • The work focuses on true derivatives, while curvature- or finite-difference-based signals are proposed as directions for future work.

1 Proofs

The proofs establish approximation and interpolation properties for ReLU or leaky-ReLU networks in Sobolev settings. They also show that derivative information can reduce the number of observations needed to identify functions in specific families.

  • For every positive ε, a single-hidden-layer ReLU or leaky-ReLU network approximates any C1 function on a compact set in Sobolev space S1.
  • Piecewise-linear continuous functions can approximate both a C1 function and its derivative uniformly away from finitely many nondifferentiability points.
  • Sigmoid functions can be approximated by finite combinations of ReLU activations with arbitrary Sobolev accuracy.
  • The proof notes that piecewise-linear nondifferentiability points are finite and therefore form a zero-Lebesgue-measure set.
  • At any finite sample set, a ReLU or leaky-ReLU network can match prescribed function values and derivatives exactly, yielding zero training loss.
  • Sobolev information requires fewer observations than value-only information for the function families F_G and F_PL.The proposition states Ksob(FG) < Kreg(FG) and Ksob(FPL) < Kreg(FPL).

2 Artificial Datasets

The artificial-dataset experiments compare regular neural-network training with Sobolev Training on classical optimisation functions. The figures show each target function, the two model outputs, and gradient vector fields, using uniformly sampled training and test domains.

  • Each figure places the target function on the left and shows regular-training and Sobolev-training models in the corresponding plot regions.
  • Gradient vector fields for each predictor are plotted beneath the function plots.
  • The benchmark functions include Beale, Booth, McCormick, and Styblinski-Tang with their stated analytic definitions.
  • Networks use Adam with learning rate 3e-5; training samples are uniform over the domain, and each test set contains 10,000 uniformly sampled points.

3 Policy Distillation

The policy-distillation experiments use convolutional and linear networks to model Atari agents. Training data consist of 100K policy-generated frames, with time-wise train/test separation across episodes.

  • The agent policies use convolutional layers followed by a 256-unit linear layer and game-specific outputs for Pong, Breakout, or Space Invaders.
  • The larger policy architecture uses three convolutional layers, a 512-unit linear layer, and game-specific outputs.
  • Agents are trained with A3C for 80e6 steps using four-frame grayscale histories, action repeat 4, and 84x84 observations.
  • Distillation data come from 100K frames generated by a trained policy, corresponding to 400K actual steps.
  • Train and test sets are split time-wise so test frames come from different episodes than training frames.
  • Policy training uses Adam with batch size 200 frames and learning rates fitted independently by game and approach between 1e-3 and 1e-5.

4 Synthetic Gradients

Sobolev losses extend synthetic-gradient training by supervising derivatives with respect to labels, with the combined loss shaping predictions across classes. In the reported CIFAR-10 setup, large-scale experiments omitted loss-derivative supervision because TensorFlow did not support differentiating cross-entropy with respect to labels.

  • 4.1 Meaning of Sobolev losses for synthetic gradients: Sobolev losses add supervision for the derivative of the synthetic-gradient model with respect to the label.The true label is used as conditioning, while derivative supervision can be provided separately.
  • 4.1 Meaning of Sobolev losses for synthetic gradients: For log loss, derivative supervision adds penalties that match the model’s log probabilities to the whole-model log probability.The loss supervision aligns the internal prediction for the true label with the whole model’s current prediction.
  • 4.1 Meaning of Sobolev losses for synthetic gradients: Matching label derivatives encourages predictions for all classes to be close to each other, while combining both losses weights the true-label penalty more heavily.The combined objective therefore uses a larger penalty for deviation on the true label than for deviations on the remaining classes.
  • 4.2 Cifar10: Loss-derivative supervision was omitted from the large-scale experiments because TensorFlow did not support differentiating cross-entropy with respect to labels.Toy MNIST experiments showed increased convergence speed and stability when this supervision was included.

5 Gradient-based attention transfer

The paper distinguishes Sobolev training from gradient-based attention transfer: both match predictor and target gradients, but Sobolev training fits the target function while transfer loss adds sensitivity matching to the original objective. Under the stated hypothesis-space condition, a target-equivalent model minimizes Sobolev loss but need not minimize transfer loss.

  • 5 Gradient-based attention transfer: Gradient-based attention transfer adds a sensitivity-matching term to the original prediction minimization problem.The compared transfer cost combines the model loss with a penalty matching input derivatives of the model and target losses.
  • 5 Gradient-based attention transfer: Sobolev training and transfer loss both match predictor and target gradients, but they are not generally the same approach.Sobolev training seeks a closer fit to the target function, whereas transfer loss augments the original minimization objective with sensitivity matching.
  • 5 Gradient-based attention transfer: If the target loss belongs to the model hypothesis space, the corresponding parameter is a minimizer of Sobolev loss but does not have to minimize transfer loss.This formalizes the distinction between matching the target function and adding gradient sensitivity to the transfer objective.
  • 5 Gradient-based attention transfer: The transfer-loss constraint is nonempty for constant models: reducing the constant parameter can improve the original loss while satisfying the sensitivity comparison.The paper uses g(x|θ)=θ with L(p)=∥p∥2 and considers θh=θf/2 as an example.
Loading 1706.04859v3…