PaperlayerView exact revision on arXiv

Source-linked AI summary

mixup: Beyond Empirical Risk Minimization

Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz

arXiv:1710.09412v2cs.LGstat.ML

TL;DR

Large neural networks trained by empirical risk minimization can memorize training data and behave undesirably outside it, including fragility to adversarial examples. mixup trains on convex combinations of examples and labels, improving generalization across image, speech, and tabular tasks while reducing memorization and adversarial sensitivity.

  • Problem

    Empirical risk minimization can let large neural networks memorize finite training data, producing undesirable behavior outside the training examples and fragility to adversarial perturbations.

  • Method

    mixup augments training with virtual examples formed by random convex combinations of raw inputs and their one-hot labels, encouraging linear behavior between examples.

  • Results

    mixup improves generalization across image, speech, and tabular datasets while reducing memorization of corrupt labels, adversarial sensitivity, and adversarial-training instability.

  • Takeaways & Limitations

    The data-agnostic method integrates into existing training pipelines with little or no computational overhead and broadens robustness benefits across supervised data types.

  • Takeaways & Limitations

    The paper lacks a good theory for selecting mixup’s bias-variance trade-off sweet spot, and training-error sensitivity to α varies across datasets and model capacities.

Abstract

from arXiv · show

Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.

1 INTRODUCTION

The introduction argues that ERM is ill-suited to overparameterized neural networks because it permits memorization and unstable predictions, motivating mixup as a simple, data-agnostic augmentation method. Mixup interpolates examples and labels, improving performance and robustness across multiple modalities and stabilizing GAN training.

  • Motivation: ERM’s suitability is challenged because large neural networks can memorize training data despite regularization and change predictions drastically on nearby examples.Classical learning theory guarantees ERM convergence when model size does not grow with the number of training examples, unlike current neural networks.
  • Method: Mixup is a simple, data-agnostic augmentation routine that constructs virtual examples by convexly combining randomly selected inputs and their one-hot labels.It uses ˜x = λx_i + (1 − λ)x_j and ˜y = λy_i + (1 − λ)y_j, with λ ∈ [0, 1].
  • Method: Mixup extends the training distribution by encoding the prior that linear feature interpolations should produce linear target interpolations, with minimal computational overhead.The method can be implemented in a few lines of code.
  • Results: Mixup achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet-2012 image classification datasets.The introduction also reports improved generalization on speech and tabular data.
  • Results: Mixup increases robustness to corrupt labels and adversarial examples, while also stabilizing generative adversarial network training.The introduction presents these findings alongside broader generalization improvements across image, speech, and tabular datasets.
  • Results: Ablation studies indicate that mixup outperforms related methods and that each design choice contributes to final performance.The experiments examine the effects of various mixup design choices.

2 FROM EMPIRICAL RISK MINIMIZATION TO mixup

The section motivates mixup as a vicinal-risk alternative to ERM, which can memorize finite training data, and defines it by interpolating pairs of examples and labels. This interpolation encourages simpler, smoother linear behavior between training examples while recovering ERM as α →0.

  • From ERM to VRM: ERM minimizes empirical risk over a finite set of examples, allowing sufficiently large neural networks to memorize training data.The resulting behavior outside the training examples is undesirable.
  • From ERM to VRM: VRM instead approximates the unknown distribution with a vicinal distribution that generates virtual feature-target pairs near training examples.Gaussian vicinities provide one example, equivalent to augmenting data with additive Gaussian noise.
  • mixup: mixup forms virtual pairs by convexly interpolating two randomly selected examples and their labels, with λ sampled from Beta(α, α).The hyper-parameter α controls interpolation strength, and mixup recovers ERM as α →0.
  • What is mixup doing?: mixup acts as data augmentation that encourages the model to behave linearly between training examples, reducing undesirable oscillations outside them.This linearity is presented as a simple inductive bias consistent with Occam’s razor.
  • What is mixup doing?: mixup produces class-to-class decision boundaries that transition linearly, yielding a smoother estimate of uncertainty.The comparison uses models with the same architecture, training procedure, evaluation points, and CIFAR-10 training data.

3 EXPERIMENTS · 3.1 IMAGENET CLASSIFICATION · 3.2 CIFAR-10 AND CIFAR-100

Experiments on ImageNet-2012, CIFAR-10, and CIFAR-100 evaluate mixup against ERM across several state-of-the-art architectures. Mixup improves performance, with larger benefits for higher-capacity or longer-trained ImageNet models and significant gains on both CIFAR datasets.

  • 3.1 IMAGENET CLASSIFICATION: ImageNet-2012 contains 1.3 million training images, 50,000 validation images, and 1,000 classes; evaluation uses a 224 × 224 central crop.Training uses scale and aspect-ratio distortions, random crops, and horizontal flips.
  • 3.1 IMAGENET CLASSIFICATION: All experiments use data-parallel distributed training in Caffe2 with minibatches of 1,024 and a staged learning-rate schedule.The learning rate rises from 0.1 to 0.4 over five epochs, then decays by 10 after specified milestones.
  • 3.1 IMAGENET CLASSIFICATION: α ∈[0.1, 0.4] improves performance over ERM, whereas large α causes underfitting.Higher-capacity models and longer training runs benefit most from mixup.
  • 3.1 IMAGENET CLASSIFICATION: 0.5% to 0.6% improvement over ERM is achieved by 90-epoch mixup variants of ResNet-101 and ResNeXt-101, compared with 0.2% for ResNet-50.These results illustrate larger gains for higher-capacity architectures.
  • 3.2 CIFAR-10 AND CIFAR-100: CIFAR-10 and CIFAR-100 experiments compare ERM and mixup using PreAct ResNet-18, WideResNet-28-10, and DenseNet.The DenseNet growth rate is changed to 40 to follow the cited implementation.
  • 3.2 CIFAR-10 AND CIFAR-100: In both CIFAR-10 and CIFAR-100, mixup-trained models significantly outperform their ERM analogues.The results are summarized in Figure 3a.
  • 3.2 CIFAR-10 AND CIFAR-100: Mixup and ERM converge at a similar speed to their best test errors on the CIFAR experiments.DenseNet’s differing reported performance may reflect its 300-epoch training and additional learning-rate decays at epochs 150 and 225.

3.3 SPEECH DATA · 3.4 MEMORIZATION OF CORRUPTED LABELS · 3.5 ROBUSTNESS TO ADVERSARIAL EXAMPLES

The experiments examine mixup on speech recognition, corrupted-label memorization, and adversarial robustness. Across these settings, the paper evaluates implementation choices, memorization behavior, and attack robustness relative to ERM.

  • 3.3 SPEECH DATA: The Google commands dataset contains 65,000 one-second utterances from 30 voice-command classes spoken by a few thousand speakers.The experiment uses normalized spectrograms sampled at 16 kHz and zero-padded to 160 × 101.
  • 3.3 SPEECH DATA: Mixup is applied at the spectrogram level immediately before the network receives the speech data.The passage notes that mixup could reasonably be applied at either waveform or spectrogram level.
  • 3.3 SPEECH DATA: Speech experiments compare LeNet and VGG-11 models trained for 30 epochs with Adam, minibatches of 100, and a five-epoch mixup warm-up.Training starts at 3 × 10^-3 and divides the learning rate by 10 every 10 epochs.
  • 3.4 MEMORIZATION OF CORRUPTED LABELS: The corrupted-label experiments compare ERM and mixup while testing whether stronger interpolation makes memorizing random labels more difficult.The hypothesis is that interpolations between real examples are easier to learn than interpolations involving random labels.
  • 3.4 MEMORIZATION OF CORRUPTED LABELS: The evaluation records best and final test errors after 200 epochs, plus final training errors on real and corrupted labels.The passage states that ERM begins overfitting corrupted labels as the learning rate becomes smaller than 0.01, while dropout can help with probabilities such as 0.7 or 0.8.
  • 3.5 ROBUSTNESS TO ADVERSARIAL EXAMPLES: ERM models are fragile to adversarial examples, which add tiny, visually imperceptible perturbations generated by ascending the loss gradient with respect to legitimate inputs.The passage frames adversarial robustness as an active research problem.
  • 3.5 ROBUSTNESS TO ADVERSARIAL EXAMPLES: Mixup improves adversarial robustness without the significant computational overhead associated with Jacobian penalties or adversarial-example augmentation.The paper assesses three ResNet-101 models on ImageNet-2012, including two ERM models and one mixup model.
  • 3.5 ROBUSTNESS TO ADVERSARIAL EXAMPLES: 2.7 times more robustness is reported for mixup than ERM under FGSM white-box attacks by Top-1 error; black-box FGSM improves by 1.25 times and black-box I-FGSM by about 40%.Both methods remain non-robust to white-box I-FGSM attacks.

3.6 TABULAR DATA

On six UCI classification problems, mixup was evaluated with fully connected neural networks and improved average test error.

  • Experimental setup: Experiments covered six arbitrary classification problems drawn from the UCI dataset.The study examined mixup on non-image data.
  • Experimental setup: The models were fully connected, with two hidden layers of 128 ReLU units, trained using Adam for 10 epochs with mini-batches of size 16.Adam used default hyper-parameters.
  • Results: Mixup improved the average test error across the tabular-data experiments.This result is reported in Table 4.

3.7 STABILIZATION OF GENERATIVE ADVERSARIAL NETWORKS (GANS)

Mixup is proposed to stabilize GAN training by regularizing discriminator gradients and smoothing its behavior between real and generated samples. The GAN mixup formulation trains the discriminator on interpolated samples with interpolated targets, and toy experiments illustrate this stabilizing effect.

  • Motivation: GAN training is notoriously difficult because the discriminator can provide the generator with vanishing gradients.GANs train a generator and discriminator competitively to model a distribution.
  • Motivation: Mixup should stabilize GAN training by regularizing discriminator gradients, while discriminator smoothness supports more stable generator training.The passage relates this mechanism to the regularization behavior of a binary classifier.
  • Mixup formulation: The GAN mixup objective evaluates binary cross-entropy on interpolated real and generated samples with interpolation-weighted targets.The formulation is d E x,z,λ ℓ(d(λx + (1 −λ)g(z)), λ).
  • Experiments: Figure 5 illustrates mixup’s stabilizing effect when fully connected GANs model two toy datasets.The networks use three hidden layers of 512 ReLU units and train for 20,000 mini-batches of size 128 with Adam.

3.8 ABLATION STUDIES

The ablation study compares mixup design choices, including input versus latent interpolation, pairing strategies, class restrictions, and label targets. Mixup performs best overall, while its preferred regularization strength and interpolation location reveal how its regularizing effect changes.

  • Design choices: The ablations compare raw-input and latent-representation interpolation, random-pair and nearest-neighbor mixing, all-class and same-class mixing, and alternative label targets.The study also includes label smoothing and Gaussian noise as comparison methods.
  • Main findings: Mixup is the best tested data-augmentation method and significantly outperforms the second-best method, mixing inputs with label smoothing.The reported results use median test errors from the final 10 epochs.
  • Regularization: For ERM, large weight decay performs better, whereas mixup prefers small weight decay, confirming mixup’s regularization effects.The compared settings are 10−4 and 5 × 10−4.
  • Regularization: The advantage of large weight decay increases when interpolation moves to higher latent layers, indicating weaker regularization there.Latent interpolation is evaluated before residual blocks and before the uppermost average-pooling plus fully connected layer.
  • Input interpolation: Among input-interpolation variants, mixing random pairs from all classes, AC + RP, provides the strongest regularization effect.AC denotes mixing across all classes, while RP denotes random pairing.

4 RELATED WORK

Prior deep-learning data augmentation commonly relies on substantial domain knowledge to design transformations. Related interpolation methods are more limited than mixup, which combines augmentation and regularization benefits while linking transformed data linearly to supervision.

  • Data augmentation: Data augmentation underlies successful deep-learning applications, but typically uses substantial domain knowledge to design transformations that improve generalization.Examples include image-classification transformations such as rotation, translation, cropping, resizing, and flipping.
  • Interpolation methods: Earlier interpolation methods augment nearest neighbors within a class at the input or feature level, without accounting for corresponding label changes.Chawla et al. (2002) target rare classes in imbalanced datasets, while DeVries & Taylor (2017) use same-class interpolation and extrapolation.
  • Mixup: Mixup retains benefits of prior augmentation and regularization schemes without their drawbacks, requiring little domain knowledge and preventing ground-truth labels from overly dominating supervision.Unlike the cited approaches, mixup establishes a linear relationship between data augmentation and the supervision signal.

5 DISCUSSION

The discussion characterizes mixup as a simple, data-agnostic vicinal risk minimization method that improves generalization and robustness across several domains. It also identifies an unresolved bias–variance trade-off and directions for extending interpolation beyond current supervised tasks.

  • Contributions: Mixup constructs virtual training examples by linearly interpolating random training examples and their labels, with little or no computational overhead.The method can be incorporated into existing training pipelines with only a few lines of code.
  • Empirical findings: Mixup improves generalization on ImageNet, CIFAR, speech, and tabular datasets while combating corrupt-label memorization, adversarial sensitivity, and adversarial-training instability.These findings come from the paper’s extensive evaluation across state-of-the-art models and multiple dataset types.
  • Limitations: As α increases, training error on real data rises while the generalization gap decreases, supporting mixup’s implicit control of model complexity.The discussion notes that the appropriate bias–variance trade-off, or “sweet spot,” still lacks a satisfactory theory.
  • Future work: Future work could extend interpolation methods to regression, structured prediction such as image segmentation, and learning settings beyond supervised learning.The authors describe regression as straightforward to generalize, whereas structured prediction is less obvious.

A. Veit, 2017. URL https://github.com/andreasveit.

This section lists related work, including P. Warden’s 2017 speech commands dataset reference and S. Xie et al.’s work on aggregated residual transformations.

  • Related work: P. Warden, 2017, is cited with a URL for the speech commands dataset.URL: https://research.googleblog.com/2017/08/ launching-speech-commands-dataset.html.
  • Related work: S. Xie, R. Girshick, P. Doll´ar, Z. Tu, and K. He are cited for “Aggregated residual transformations for deep neural networks.”

S. Zagoruyko and N. Komodakis. Wide residual networks. BMVC, 2016a.

This section lists related works by Zagoruyko and Komodakis, Zhang and colleagues, and Zhong and colleagues.

  • Related work: Zagoruyko and Komodakis are cited for Wide residual networks (BMVC, 2016a), with a related 2016b URL listed.The URL points to the wide-residual-networks GitHub repository.
  • Related work: Zhang, Bengio, Hardt, Recht, and Vinyals are cited for Understanding deep learning requires rethinking generalization (ICLR, 2017).The title spans two consecutive passages.
  • Related work: Zhong, Zheng, Kang, Li, and Yang are cited for Random erasing data augmentation (arXiv, 2017).This is listed as an arXiv work.
Loading 1710.09412v2…