PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Margin Deep Networks for Classification

Gamaleldin F. Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, Samy Bengio

arXiv:1803.05598v2stat.MLcs.LG

TL;DR

Deep networks make exact input-space margins difficult to compute, while conventional neural-network margin methods generally enforce them only at the output. The paper introduces a first-order, norm-flexible loss for selected input, hidden, or output layers and reports consistent gains across adversarial, noisy-label, and limited-data tasks.

  • Problem

    Classical large-margin algorithms are mainly suited to shallow models, while deep-network methods commonly enforce margins only at the output layer.

  • Method

    The authors develop a first-order approximate loss that imposes lp-norm margins on any chosen set of input, hidden, or output layers and complements standard regularization.

  • Results

    The margin models consistently outperform conventional-loss baselines across adversarial perturbation, noisy-label, and limited-training-data evaluations.

  • Takeaways & Limitations

    The formulation is architecture-independent, computationally practical, and supports margins throughout deep representations, with hidden-layer margins benefiting adversarial robustness.

Abstract

from arXiv · show

We present a formulation of deep learning that aims at producing a large margin classifier. The notion of margin, minimum distance to a decision boundary, has served as the foundation of several theoretically profound and empirically successful results for both classification and regression tasks. However, most large margin algorithms are applicable only to shallow models with a preset feature representation; and conventional margin methods for neural networks only enforce margin at the output layer. Such methods are therefore not well suited for deep networks. In this work, we propose a novel loss function to impose a margin on any chosen set of layers of a deep network (including input and hidden layers). Our formulation allows choosing any norm on the metric measuring the margin. We demonstrate that the decision boundary obtained by our loss has nice properties compared to standard classification loss functions. Specifically, we show improved empirical results on the MNIST, CIFAR-10 and ImageNet datasets on multiple tasks: generalization from small training sets, corrupted labels, and robustness against adversarial perturbations. The resulting loss is general and complementary to existing data augmentation (such as random/adversarial input transform) and regularization techniques (such as weight decay, dropout, and batch norm).

1 Introduction

Classical large-margin methods are difficult to apply to deep networks, motivating a flexible loss that enforces margins across network layers. The proposed approach improves performance across limited-data, noisy-label, and adversarial settings.

  • Motivation: Deep-network input margins are computationally intractable, although they are relevant to generalization and robustness against input perturbations.A margin γ guarantees that perturbations smaller than γ cannot flip training-point labels.
  • Contribution: The proposed loss uses a first-order margin approximation and applies to arbitrary network architectures while complementing weight decay, dropout, and batch normalization.The formulation supports arbitrary depth, activation functions, convolutions, and residual networks.
  • Illustration: The large-margin toy solution produces a better-shaped decision boundary than cross-entropy in the highlighted region despite both methods reaching 100% training accuracy.The comparison uses a four-layer network trained for 10000 steps.
  • Contribution: The loss supports lp-norm margins on any chosen set of layers, including input and hidden layers, rather than restricting margins to outputs.The authors evaluate this flexibility across multiple datasets, architectures, and tasks.

2 Related Work

Prior deep-network margin methods either target only output-layer margins or have been demonstrated only for linear architectures. Related approaches also include Jacobian-based regularization and methods for limited or noisy data.

  • Deep-network margins: Existing neural-network margin terms encourage margins only at the output layer, limiting their scope within deep models.Cross-entropy does not itself have margin-maximization properties according to the cited prior work.
  • Deep-network margins: A max-margin result for cross-entropy with SGD was demonstrated for linear architectures, making it less directly applicable to deep nonlinear networks.
  • Related objectives: Prior work also explored Jacobian-based regularization and methods addressing generalization, few-shot learning, and noisy labels.These approaches form related context for the paper’s evaluations and motivation.

3 Large Margin Deep Networks

The method defines class-pair decision-boundary distances under arbitrary lp norms, approximates nonlinear distances by linearization, and extends the margin objective to hidden representations. It therefore trains margins at selected layers using a practical loss.

  • Score-based boundaries: The predicted class is the index with the maximal class score, and each class pair defines a decision boundary where their scores tie.
  • Input margin: A point’s distance to a class-pair boundary is the smallest displacement that produces a score tie, measured with any lp norm.The loss penalizes training points that fall short of a target margin from competing class boundaries.
  • Margin loss: For correctly classified points, the penalty covers the shortfall to margin γ; misclassified points additionally pay for reaching the boundary and moving γ beyond it.
  • Approximation: Because exact nonlinear distances are intractable, the method linearizes class scores around zero displacement to obtain a closed-form approximation.The resulting loss is minimized with respect to the network parameters.
  • Approximation: The distance metric uses the dual norm: l1 distances pair with l∞ gradients, l2 with l2, and l∞ with l1.This norm pairing follows the dual-norm relationship.
  • Novelty: The paper distinguishes its contribution from an independently derived distance metric by using that metric in a novel margin loss across multiple hidden layers.The metric coincides with an SVM in the special case of a linear classifier.
  • Margin for Hidden Layers: The same margin formulation is applied to intermediate representations hℓ by replacing the input with layer outputs and assigning each selected layer a margin γℓ.A small ε in the denominator prevents numerical problems in practice.

4 Experiments

Experiments evaluate margin models against cross-entropy across MNIST, CIFAR-10, and ImageNet tasks involving noisy labels, limited data, and adversarial perturbations. Across these settings, all-layer margin models generally outperform cross-entropy, with gains often increasing under more difficult conditions.

  • Experimental setup: The experiments compare cross-entropy, hinge, and margin models using l∞, l1, and l2 norms, with margins applied either at the input layer or across all hidden and output layers.The study uses MNIST, CIFAR-10, and ImageNet architectures, including convolutional, residual, and Inception models.
  • MNIST noisy labels: 96.4% evaluation accuracy at 80% label noise was achieved by the l2 margin model, compared with 93.9% for cross-entropy.Across MNIST noise levels from 0% to 80%, l1 and l2 margin models outperform cross-entropy, while l∞ is slightly worse; all-layer models outperform input-only models.
  • MNIST generalization: 3.7% was the all-layer l∞ margin model’s advantage over cross-entropy when training used only 68 MNIST samples.All-layer margin models outperform cross-entropy, hinge, and the cited Bayesian active-learning method across the tested training-set sizes, with the advantage increasing as data decreases.
  • MNIST adversarial perturbations: 90% accuracy versus 67% for cross-entropy was obtained at ϵ = 0.1 against MNIST black-box IFGSM attacks.Margin models significantly outperform cross-entropy in white-box and black-box FGSM and IFGSM settings; all-layer models outperform input-only margin models.
  • CIFAR-10 noisy and limited data: 4% to 10% was the consistent CIFAR-10 advantage of l∞ and l2 margin models over cross-entropy across label-noise levels from 0% to 80%.For limited-data CIFAR-10, l1 and l∞ models also outperform cross-entropy, including a 2.5% l1 advantage at 5% and 1% of the data.
  • Adversarial perturbations: 44% white-box and 59% black-box accuracy was achieved by the ImageNet margin model at ϵ = 0.1, versus 33% white-box accuracy for cross-entropy.On CIFAR-10, the l1 model improves over cross-entropy by about 14% against a cross-entropy black-box attack at ϵ = 0.1; on ImageNet, margin models consistently outperform cross-entropy across tested attack settings.

5 Discussion

The proposed loss supports large margins at input, hidden, or output layers under multiple norm choices, while remaining architecture- and domain-independent and complementary to standard techniques.

  • The loss can define margins on input, hidden, or output layers using l∞, l1, or l2 distances.
  • The formulation is independent of network architecture and input domain.
  • Margin-loss models perform well across practical scenarios on standard datasets compared with baselines.
  • The loss complements existing data augmentation and regularization techniques.

A Derivation of Equation (7) in paper

The derivation characterizes the minimum p-norm perturbation satisfying a linear constraint and connects the resulting margin formulation to linear classifiers and SVMs.

  • The optimization minimizes ||δ||p subject to the linear constraint aTδ = b.
  • The minimum distance is d = |b| ||a||*, where ||·||* is the dual norm.
  • For a linear classifier, the formulation coincides with an SVM.
  • Scaling redundancy in (w, b) is removed in SVMs by requiring |wT x + b| ≥ 1.
  • The linear margin is the smallest decision-boundary distance, γ = mink d̃k = 1/||w||2.
  • IFGSM repeatedly updates inputs with signed loss gradients and clips each pixel within an ϵ-sized neighborhood.

E CIFAR-10 - Additional Results

Additional CIFAR-10 results evaluate model performance against FGSM attacks in both black-box and white-box settings.

  • Figure 11 reports CIFAR-10 model performance against FGSM black-box and white-box attacks.

F MNIST Model Architecture and Hyperparameter Details

The MNIST experiments use a four-hidden-layer network with convolutional and fully connected components, while accompanying figures cover attacks and Gaussian-noise robustness.

  • The network has four hidden layers: two convolutional layers followed by two fully connected layers.The convolutional filters are 5 × 5 with 32 and 64 units; each fully connected hidden layer has size 512.
  • The experiments sweep dropout, weight decay, and margin-strength values.Dropout is 0 or 0.2, weight decay is 0 or 0.005, and γl is 200 or 1000.
  • Figure 9 covers MNIST performance under white-box and black-box FGSM attacks.
  • Figure 10 covers MNIST performance under input Gaussian noise with varying standard deviations.

G CIFAR-10 Model Architecture and Hyperparameter Details

The CIFAR-10 experiments use a depth-58, k=10 residual architecture without dropout, with separate learning rates and swept margin-strength values.

  • Architecture: The model uses the depth-58, k=10 architecture from Zagoruyko and Komodakis (2016), comprising three sets of nine residual units.It begins with a 3 × 3 convolutional layer containing 16 units.
  • Regularization: No dropout is used in the CIFAR-10 model.
  • Optimization: The learning rate is 0.001 for margin training and 0.01 for cross-entropy and hinge losses.The decay factor is 0.9, applied every 2000 or 20000 steps based on a hyperparameter sweep.
  • Margin setting: The margin coefficient γ_l is selected from 5000, 10000, or 20000, using the same value at all layers.

H Imagenet Model Architecture and Hyperparameter Details

The ImageNet experiments follow the architecture of Szegedy et al. (2016), using RMSProp and auxiliary losses for margin training.

  • Architecture: The ImageNet experiments follow the architecture described by Szegedy et al. (2016).
  • Optimization: RMSProp is used for optimization in all ImageNet experiments.
  • Margin training: Margin training adds cross-entropy loss with weight 0.1 and an auxiliary loss in the network’s middle.The auxiliary loss follows the approach suggested in the cited paper.

I Evolution of distance metric

On CIFAR-10, the proposed margin training produces larger mean distances to the decision boundary than cross-entropy across input, hidden, and output layers. For the test set at the input layer, the margin distance is about 100 versus about 70 for cross-entropy.

  • Experimental setup: The distance approximation is averaged over mini-batches during 50,000 training steps on CIFAR-10 using 100% of the data.Training, validation, and test distances are plotted separately.
  • Input layer: Margin training achieves a higher mean distance to the boundary than cross-entropy at the input layer.
  • Hidden layer 7: Margin training achieves a higher mean distance to the boundary than cross-entropy at hidden layer 7.
  • Output layer: Margin training achieves a higher mean distance to the boundary than cross-entropy at the output layer.
  • Input-layer result: About 100 versus about 70: the test-set input-layer mean distance is higher for margin training than for cross-entropy.For cross-entropy, the reported quantity is only a measured distance, not the proposed margin objective.
Loading 1803.05598v2…