PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring and regularizing networks in function space

Ari S. Benjamin, David Rolnick, Konrad Kording

arXiv:1805.08289v3cs.NEcs.LGstat.ML

TL;DR

Neural-network optimization is usually described in parameter space even though learning ultimately changes an input/output function, and the correspondence between the two is not guaranteed. The paper measures function distances in L2 space, compares them with parameter distances during optimization, and applies the metric to forgetting and update constraints. It finds that the correspondence changes during optimization and that direct function-space regularization supports continual-learning and single-task applications, subject to contextual and memory limitations.

  • Problem

    Neural-network analyses usually focus on parameters even though the learning objective is an input/output function whose relation to parameter changes may be unclear.

  • Method

    The paper measures expected output ℓ2 distances in L2 function space, compares them with parameter distances, and uses the metric for function-preserving regularization.

  • Results

    Function and parameter spaces show qualitatively different network trajectories, while direct L2 regularization addresses catastrophic forgetting and constrains single-task update movement.

  • Takeaways & Limitations

    Function-space distances can be measured and regularized directly for multitask learning and supervised optimization rather than relying only on parameter changes.

  • Takeaways & Limitations

    The correspondence depends strongly on context, and working-memory regularization stores old examples despite violating strict online-learning rules.

Abstract

from arXiv · show

To optimize a neural network one often thinks of optimizing its parameters, but it is ultimately a matter of optimizing the function that maps inputs to outputs. Since a change in the parameters might serve as a poor proxy for the change in the function, it is of some concern that primacy is given to parameters but that the correspondence has not been tested. Here, we show that it is simple and computationally feasible to calculate distances between functions in a $L^2$ Hilbert space. We examine how typical networks behave in this space, and compare how parameter $\ell^2$ distances compare to function $L^2$ distances between various points of an optimization trajectory. We find that the two distances are nontrivially related. In particular, the $L^2/\ell^2$ ratio decreases throughout optimization, reaching a steady value around when test error plateaus. We then investigate how the $L^2$ distance could be applied directly to optimization. We first propose that in multitask learning, one can avoid catastrophic forgetting by directly limiting how much the input/output function changes between tasks. Secondly, we propose a new learning rule that constrains the distance a network can travel through $L^2$-space in any one update. This allows new examples to be learned in a way that minimally interferes with what has previously been learned. These applications demonstrate how one can measure and regularize function distances directly, without relying on parameters or local approximations like loss curvature.

1 INTRODUCTION

The paper argues that neural-network analysis should measure functions directly because parameter changes may poorly represent input/output-function changes. It introduces tractable L2 function-space distances and proposes applications to forgetting and supervised learning updates.

  • Neural-network parameters encode functions, but parameter-space analyses are only useful proxies when parameter and function changes correspond closely.
  • L2 function-space distance is computationally feasible because it measures expected output ℓ2 distance on shared inputs using function inference alone.
  • The paper examines whether SGD trajectories, overfitting, and parameter changes correspond to trajectories and changes in function space.
  • For multitask learning, directly regularizing early-task function changes is proposed as an alternative to parameter-based protection against catastrophic forgetting.
  • Hilbert-constrained gradient descent limits each update’s L2 function-space movement, aiming to learn new examples with less interference from prior learning.

2 EXAMINING NETWORKS IN FUNCTION SPACE

The paper estimates function distances from empirical input distributions and compares function-space and parameter-space training trajectories. Networks can move differently in the two spaces, with their relationship changing across optimization and depending on context.

  • Function-space measurement: The L2 norm averages squared output differences over the input distribution, while empirical estimates use batches of examples.The input measure is empirical rather than uniform over all possible inputs, and estimator quality depends on data distribution and function variation.
  • Trajectory comparison: Training snapshots provide parameter and function distance matrices, which are embedded in two dimensions to visualize trajectories.The trajectories use convolutional networks trained on a 5,000-image CIFAR-10 subset, with multidimensional scaling applied to the distance matrices.
  • Trajectory comparison: Random initializations diverge in parameter space yet initially follow similar function-space paths before diverging during late optimization.Despite parameter divergence, the networks fit the training data perfectly and reach approximately 50% test accuracy in the reported experiment.
  • Optimization dynamics: After test error converges around epoch 60, networks continue moving in both spaces, while movement relative to initialization slows.Movement remains large relative to previous iterations and epochs even after test error has converged.
  • Optimization dynamics: The L2/ℓ2 distance relationship changes throughout optimization, with the ratio converging toward a late-optimization value after test error saturates.Early epochs generally show larger L2 changes for a given parameter change, and the changing relationship appears at multiple distance scales.
  • Optimization dynamics: Parameter and function distances lack a consistent positive correlation across trajectory scales and optimization stages.Successive-epoch distances are negatively correlated for most optimization, whereas batch-level correlation is weaker early and stronger late.
  • Scope and robustness: The correspondence between parameter and function distances depends strongly on context, although the changing L2/ℓ2 relationship persists across tested settings.Batch normalization and weight decay alter trends, while analogous MNIST experiments show similar overall behavior.
  • Function-space measurement: A few hundred examples generally suffice to estimate L2 distance within a few percent.

3 APPLICATIONS

The paper applies function-space L2 regularization to catastrophic forgetting and supervised optimization, using stored examples for task retention and constrained updates for learning trajectories. These approaches show distinct benefits, while HCGD's performance depends on task and learning-rate conditions.

  • Catastrophic forgetting: Regularizing L2 distance from earlier tasks uses a working-memory cache of examples and stored outputs to preserve prior input/output functions.The method modifies the new-task loss with the L2 distance between the current function and the function after the earlier task.
  • Catastrophic forgetting: The working-memory approach can be comparable or cheaper in memory than storing old parameters and diagonal matrices, but it violates strict online-learning rules.For the reported large-network setting, the comparison methods stored more memory than the cache.
  • Catastrophic forgetting: The function-space method outperformed retraining on the same cache and was compared with ADAM, EWC, and SI on permuted MNIST.The experiment measured first-task test accuracy while seven subsequent tasks were learned.
  • Constraining changes in L2 during learning: HCGD constrains each optimization update's path length in L2-space by correcting a proposed gradient-based update through an inner optimization loop.The rule first approximates the cost change using the Jacobian and then seeks a corrected update.
  • Function-space regularization and natural gradient: Natural gradient emerges when learning regularizes changes in the output distribution using a second-order KL-divergence approximation and sets λ = 1.The Fisher information metric supplies the local quadratic approximation.
  • Empirical comparison of HCGD: On CIFAR-10 with Squeezenet v1.1, HCGD performed better early at the same learning rate but did not outperform SGD in the final stage.With initial ϵ = 0.3, test accuracy was marginally higher than SGD despite lower training accuracy; HCGD underperformed ADAM on permuted sequential MNIST, although it outperformed SGD.
  • Empirical comparison of HCGD: HCGD outperformed SGD on permuted sequential MNIST, and applying function-space correction to Adam also improved its performance.Using more correction steps improved the recurrent-task result, while HCGD remained below ADAM there.

4 DISCUSSION

The paper argues that function-space distances provide a useful perspective on neural-network behavior and can directly support regularization. Its applications include limiting forgetting across tasks and constraining single-task optimization, although benefits are not universal.

  • Function-space trajectories can differ qualitatively from parameter-space trajectories, so parameter distance is not generally proportional to function distance.
  • L2 function distances can be used to regularize changes in old-task functions during multitask learning.The method estimates distances using stored examples and outputs.
  • The multitask scheme outperforms retraining on the same number of stored examples and may use less memory than EWC or SI for large networks.
  • HCGD constrains movement in L2 function space during single-task optimization, and can improve test performance in recurrent image-classification settings.
  • HCGD does not always improve results, suggesting that SGD may already be stable in some regimes or that other generalization principles matter there.

A CIFAR-10 L2 AND ℓ2 COMPARISON

The L2/ℓ2 relationship changes during optimization, with trends affected by test-error dynamics, Batch Normalization, and weight decay.

  • Epochs with qualitatively different L2/ℓ2 ratios from late optimization coincide with periods when test error changes fastest.
  • Averaging points within each epoch makes the overall L2/ℓ2 trends more apparent.
  • Without Batch Normalization, larger parameter changes produce the same magnitude of L2 changes between updates and epochs.
  • Without Batch Normalization, the L2/ℓ2 ratio between updates changes less across epochs when Batch Normalization is used.
  • Without weight decay, the ℓ2 distance traveled decreases at all three scales, especially late in optimization.

B COMPARING FUNCTION AND PARAMETER SPACES DURING MNIST

On MNIST, most function-space movement occurs early in optimization, while estimating epoch-level L2 distances requires substantial data.

  • Most function-space movement occurs within the first epoch of MNIST optimization.
  • The standard deviation of the L2 estimator is higher for MNIST than for CIFAR-10.

C HCGD DECREASES THE DISTANCE TRAVELED IN L2 SPACE

HCGD is designed to reduce motion through L2-space, and cumulative squared-distance plots compare that motion with SGD during MNIST training.

  • HCGD reduces motion through L2-space by targeting cumulative squared distance traveled during optimization.
  • Cumulative squared distance is computed by summing squared distances between consecutive updates.
  • The cumulative-sum representation makes Brownian motion appear as a linear increase.
  • The MNIST comparison shows that SGD continues to drift through L2-space during overfitting.

D DETAILED HCGD ALGORITHM

The HCGD algorithm augments momentum-based gradient descent with corrective steps based on the L2 gradient.

  • Algorithm 2 implements the Hilbert-constrained gradient-descent procedure specified by Equation 7.
  • The algorithm requires a corrective-step count, overall learning rate, corrective learning rate, and momentum coefficient.
  • Each iteration draws a training batch, computes gradients, updates momentum, and forms a proposed parameter update.
  • The first correction subtracts η(gL2) from the proposed update and adds the same correction to the momentum buffer.
  • Additional corrective steps can be applied when the configured number of corrections exceeds one.

E NATURAL GRADIENT BY GRADIENT DESCENT

The paper adapts natural-gradient optimization through iterative gradient-descent corrections, using Fisher matrix–vector products instead of explicitly inverting the Fisher matrix. The method relies on per-example gradients and faces computational constraints in current deep-learning frameworks.

  • Motivation: Previous natural-gradient methods approximate the inverse Fisher matrix, whereas the proposed approach uses cheaper matrix-vector products within an inner gradient-descent loop.The approach is motivated by the cost of approximately inverting large matrices in very large neural networks.
  • Algorithm: The proposed algorithm starts with an optimizer update and iteratively corrects it toward the natural gradient.A fast diagonal natural-gradient approximation, such as RMSprop or Adagrad, is recommended for the initial update; each correction requires one Fisher matrix–vector product.
  • Algorithm: The corrective update follows Δθ_i+1 = Δθ_i − η(J + λFΔθ_i), with n corrective steps controlling the approximation process.The algorithm pairs a proposed RMSprop update with a configurable corrective loop.
  • Fisher computation: The Fisher matrix need not be stored explicitly because it can be computed from the covariance of per-example gradients.The gradient array G has dimensions corresponding to parameters by examples, and the method requires calculating gradients separately for each example.
  • Limitations: Using aggregated gradients instead of per-example gradients destroys covariance structure and produces a rank-1 Fisher matrix.Efficient computation also requires forward-mode differentiation, which popular deep-learning frameworks do not currently support.
  • Fisher computation: The proper Fisher construction uses gradients from the network’s predictive output distribution, obtained by sampling fictitious targets, or alternatively from unlabeled or validation data.These alternatives are distinguished from using only per-example minibatch gradients, which yield the empirical Fisher.
Loading 1805.08289v3…