PaperlayerView exact revision on arXiv

Source-linked AI summary

Regularisation of Neural Networks by Enforcing Lipschitz Continuity

Henry Gouk, Eibe Frank, Bernhard Pfahringer, Michael J. Cree

arXiv:1804.04368v3stat.MLcs.LG

TL;DR

Neural-network regularisation lacks methods that are both effective and well founded, motivating explicit control of Lipschitz continuity. The paper computes layer-composable Lipschitz upper bounds across multiple norms and trains with constrained optimisation. Experiments find that LCC is effective, particularly with limited data, while its practical scope includes architecture- and norm-dependent behaviour.

  • Problem

    Effective neural-network regularisers are often heuristic, motivating a well-founded method for controlling function variation and generalisation.

  • Method

    The paper computes upper bounds on feed-forward networks’ Lipschitz constants across multiple p-norms and enforces them through constrained training.

  • Results

    Experiments show LCC is effective relative to common regularisers, with larger gains when only a small amount of training data is available.

  • Takeaways & Limitations

    The ℓ∞ norm appears particularly suitable for tabular data, whereas ℓ2 gives the most consistently competitive regularisation performance on natural image datasets.

  • Takeaways & Limitations

    The method uses a shared layer hyperparameter in practice, while selecting different values per layer and allocating capacity efficiently remain open problems.

Abstract

from arXiv · show

We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant---for multiple $p$-norms---of a feed forward neural network composed of commonly used layer types. Our technique is then used to formulate training a neural network with a bounded Lipschitz constant as a constrained optimisation problem that can be solved using projected stochastic gradient methods. Our evaluation study shows that the performance of the resulting models exceeds that of models trained with other common regularisers. We also provide evidence that the hyperparameters are intuitive to tune, demonstrate how the choice of norm for computing the Lipschitz constant impacts the resulting model, and show that the performance gains provided by our method are particularly noticeable when only a small amount of training data is available.

1 Introduction

The paper motivates Lipschitz-based regularisation as a well-founded way to control neural-network variation and improve generalisation. It develops an upper-bound-based approach applicable to feed-forward networks and evaluates its practical benefits.

  • Neural-network regularisation remains difficult because effective methods are heuristic, while adapted ℓ2 penalties are less effective.
  • Lipschitz continuity measures a function’s maximum variation, motivating control of network variation as an inductive bias for generalisation.
  • The method computes an upper bound on a feed-forward network’s Lipschitz constant across multiple p-norms and uses it for regularisation.
  • Layer-wise hyperparameters determine the network-level Lipschitz bound, with experiments reusing one value across multiple layers to accelerate tuning.
  • The experiments report stronger performance than common regularisers, especially when training data are scarce, and show intuitive hyperparameter behaviour.

2 Related work

Related work connects weight magnitude and Lipschitz quantities to neural-network generalisation while highlighting differences between penalty-based and hard-constraint approaches. The paper extends this line beyond ℓ2 norms and considers broader applications.

  • Dropout reduces overfitting by randomly zeroing hidden activations, with extensions that alter weights or learn the dropout rate.
  • Generalisation analyses relate neural-network performance to weight magnitude and products of matrix norms, including spectral and element-wise norms.
  • Prior work questions whether ℓ2 Lipschitz continuity alone guarantees generalisation, while other bounds and robustness theory link Lipschitz constants to generalisation performance.
  • Unlike Yoshida and Miyato’s loss penalty on summed spectral norms, this paper investigates multiple norms and imposes hard constraints on induced matrix norms.
  • Lipschitz constraints have also been applied to GAN discriminators, where specific spectral norms can improve generated-sample quality.

3 Computing the Lipschitz Constant

The paper bounds a feed-forward network’s Lipschitz constant by composing layer-wise bounds, then derives practical computations for common layers and residual architectures. Matrix norms, convolutional operations, and efficient power iterations provide the computational basis.

  • 3 Computing the Lipschitz Constant: A feed-forward network is represented as a composition of activation, linear, or pooling operations whose Lipschitz constants can be combined.
  • 3 Computing the Lipschitz Constant: The network Lipschitz bound is the product of its layer-wise constants, although this modular bound need not be tightest for the complete network.
  • 3.1 Fully Connected Layers: For fully connected layers, the Lipschitz constant is the operator norm of the weight matrix, with efficient forms for p = 1, p = 2, and p = ∞.
  • 3.2 Convolutional Layers: Convolutional layers can be expressed through structured matrices, allowing ℓ1 and ℓ∞ operator norms to be computed from filter coefficients and matrix concatenations.
  • 3.2 Convolutional Layers: The spectral norm of a convolutional layer is computed efficiently by replacing matrix multiplications in the power method with convolution and transposed-convolution operations.
  • 3.3 Activations and Pooling: Common activations and pooling operations are at worst 1-Lipschitz across the considered p-norms, including ReLU, max pooling, and softmax.
  • 3.4 Residual Connections: Residual networks are handled by bounding each residual block as a sum of functions and multiplying the resulting block bounds across the network.

4 Constraining the Lipschitz Constant

The method controls a network’s Lipschitz constant by constraining each layer’s operator norm and optimizing with projected stochastic gradients. It supports multiple p-norms, batch normalization, and efficient norm estimation.

  • 4 Constraining the Lipschitz Constant: The network Lipschitz constant is bounded by the product of the Lipschitz constants of its layers.Constraining each layer separately therefore controls the network-level constant.
  • 4 Constraining the Lipschitz Constant: Projected stochastic gradient training returns infeasible weight matrices to the closest feasible matrices after each update.The projection uses the matrix distance induced by the relevant operator norm.
  • 4 Constraining the Lipschitz Constant: The method works with operator norms for p ∈ {1, 2, ∞}, and Adam and Nesterov-momentum stochastic gradient methods are compatible with it.The paper reports fast convergence with Adam and compatibility with Nesterov momentum.
  • 4 Constraining the Lipschitz Constant: Spectral-norm estimation can underestimate the true norm when the power method has not converged.Slow convergence occurs when the two largest singular values are close, while ℓ1 and ℓ∞ norms are computed exactly in linear time.
  • 4 Constraining the Lipschitz Constant: With batch normalization, the γ parameter must also be constrained using the batch-normalization layer’s operator norm.During minibatch training, a moving-average variance estimate is used for projection.
  • 4 Constraining the Lipschitz Constant: Dropout keeps activation magnitudes approximately similar between training and test time by scaling test-time activations by the retention probability.Standard dropout randomly zeros activations during training.

5 Experiments

The experiments test LCC across architectures and compare it with dropout and spectral decay, while examining hyperparameter sensitivity and performance on CIFAR-10. Results are averaged across independently initialized networks unless stated otherwise.

  • 5 Experiments: The experiments evaluate LCC against related regularisers using held-out accuracy and examine sensitivity to λ hyperparameters and method interactions.The stated questions include comparison with related regularisation methods and hyperparameter behavior.
  • 5 Experiments: Fully connected, VGG-style convolutional, and residual networks are used to test LCC across feed-forward architectures.The implementation passage specifies different optimizers for residual and other networks.
  • 5 Experiments: Reported accuracies are means over five differently seeded networks, with standard deviations indicating robustness to initialization.The code for the experiments is available online.
  • 5 Experiments: CIFAR-10 experiments use VGG19-style and Wide Residual Network architectures, with 10,000 training images reserved for hyperparameter tuning.The dataset contains 60,000 images, including a 50,000-image training set.
  • 5 Experiments: On CIFAR-10, VGG benefits most from LCC-ℓ2, while WRNs show performance similar to dropout and marginally better than spectral decay.Combining dropout with the other methods is not effective on this dataset.

5.2 CIFAR-100

On CIFAR-100, the study uses VGG19-style and WRN architectures and finds a similar regularisation pattern to CIFAR-10, with LCC generally strongest.

  • 5.2 CIFAR-100: CIFAR-100 contains 60,000 images across 100 classes, and the experiments use VGG19-style and Wide Residual Network architectures.Regularisation hyperparameters are optimized specifically for CIFAR-100.
  • 5.2 CIFAR-100: LCC performs best on CIFAR-100, while dropout gives a small improvement over no regularisation and spectral decay is often comparable to no regularisation.The same trend is reported for networks with and without residual connections.
  • 5.2 CIFAR-100: Spectral decay performs noticeably worse than LCC-ℓ2 on CIFAR-100.The passage reports that spectral decay often has performance comparable to no regularisation.
  • 5.2 CIFAR-100: The CIFAR-100 results are reported in Table 2 for VGG and WRN models.The table compares spectral decay, dropout, LCC, and combinations thereof.

5.4 Street View House Numbers

On SVHN, LCC improves accuracy for both VGG and WRN models, whereas spectral decay does not help either architecture. Dropout performs best for VGG when combined with other approaches.

  • 5.4 Street View House Numbers: SVHN contains over 600,000 three-channel 32 × 32 digit images extracted from Google Street View.No data augmentation is performed for this dataset.
  • 5.4 Street View House Numbers: SVHN test-set measurements are reported for each trained model in Table 4.
  • 5.4 Street View House Numbers: LCC improves accuracy in both VGG and WRN models trained on SVHN.The SVHN experiments use images rescaled to [−1, 1] without data augmentation.
  • 5.4 Street View House Numbers: Spectral decay does not improve performance for either VGG or WRN on SVHN.For VGG, dropout combined with other approaches produces the best performance, while dropout alone is not effective.

5.5 Scaled ImageNet Subset (SINS-10)

On SINS-10, the experiments compare LCC with spectral decay, dropout, and their combinations across VGG-style and WRN architectures. LCC significantly improves performance over comparable networks without LCC for both architectures.

  • SINS-10 contains 100,000 ImageNet images in 10 predefined folds, with 9,000 training and 1,000 testing images per fold.
  • The experiments use VGG-style and WRN architectures, with training schedules adjusted because each SINS-10 fold has fewer instances than SVHN.
  • Table 4 compares prediction accuracy for VGG-style and WRN-16-4 networks trained with spectral decay, dropout, LCC, and combinations on SVHN.
  • For both architectures, networks trained with LCC perform statistically significantly better than comparable networks trained without LCC.

5.6 Fully Connected Networks

Fully connected networks are evaluated on numeric-feature classification datasets, with LCC compared alone and in combination with other regularisers. LCC-ℓ∞ achieves the highest mean accuracy on eight of ten datasets, while differences are negligible on the other two.

  • The fully connected experiments use numeric-feature classification datasets collected from the UCI repository.
  • Table 5 reports prediction accuracies for VGG-style networks on SINS-10 and marks statistically significant effects of adding LCC at the 95% confidence level.
  • LCC-ℓ∞ achieves the highest mean accuracy on eight of the 10 datasets.
  • On the remaining two datasets, there is no substantial difference in performance between the regularisation methods.
  • The critical difference diagram shows LCC-ℓ∞ with an average rank just over 1.5, compared with just over 3.5 for no regularisation.

5.7 Sensitivity to λ

Sensitivity analysis scales the layer-specific LCC hyperparameters by a common factor to examine accuracy as regularisation strength changes. Slightly undersized values can severely degrade performance, while optimal scales vary widely across settings.

  • LCC uses up to three λ hyperparameters, one for each parameterised layer type.
  • The sensitivity experiment varies a scalar c multiplying λ⃗ = [λconv, λfc, λbn] and plots resulting test-set accuracy on CIFAR-100 VGG19-style models.
  • Even slightly too-small hyperparameter values cause a massive degradation in performance.
  • Above the optimal value, performance declines slowly until accuracy becomes comparable to an unregularised network.
  • Optimal hyperparameter settings differ vastly across datasets, architectures, and p-norm choices, leaving no typical range for tuning.
  • Table 7 reports mean test-set accuracies from two repetitions of 5-fold cross-validation, with the highest mean accuracy bolded for each dataset.

5.8 Sample Efficiency

LCC improves sample efficiency most visibly when training data are scarce, with its advantage diminishing as more examples become available.

  • 5.8 Sample Efficiency: Approximately 10 percentage points separate LCC-trained VGG networks from a weaker baseline when training uses only 5,000 instances.As training-set size increases, performance differences between methods become smaller.
  • 5.8 Sample Efficiency: The shrinking performance gap reflects increasing reliance on training examples rather than prior knowledge as more data become available.
  • 5.8 Sample Efficiency: Spectral decay performs very well with little data in wide residual networks, but this advantage is lost as training examples grow.Its performance tends toward the unregularised baseline, a pattern also observed on other datasets.

6 Conclusion

The paper concludes that Lipschitz Constant Constraint is a broadly applicable regulariser, especially useful with limited data, while norm choice and approximation affect guarantees and performance.

  • 6 Conclusion: LCC is presented as an effective regulariser for diverse feed-forward architectures, particularly when only a small amount of training data is available.
  • 6 Conclusion: The ℓ∞ norm appears best suited to tabular data, whereas ℓ2 provides the most consistently competitive regularisation performance on natural-image datasets.
  • 6 Conclusion: Few power-method iterations make LCC-ℓ2 only approximately constraining, so ℓ1 or ℓ∞ is preferred when a bound guarantee is required.
  • 6 Conclusion: The experiments suggest that Lipschitz constraints may also benefit GAN discriminator networks, while recurrent-network applications remain unexplored.
  • 6 Conclusion: Using a separate λ for each layer could improve performance, but efficiently allocating modelling capacity per layer remains an open problem.
Loading 1804.04368v3…