PaperlayerView exact revision on arXiv

Source-linked AI summary

MixUp as Locally Linear Out-Of-Manifold Regularization

Hongyu Guo, Yongyi Mao, Richong Zhang

arXiv:1809.02499v3cs.LGcs.AIstat.ML

TL;DR

MixUp is effective but its mechanism and interpolation-policy tuning are not well explained. The paper analyzes it as out-of-manifold regularization, identifies manifold intrusion, and introduces AdaMixUp, which learns mixing policies adaptively; experiments report improvements over standard MixUp and current deep classification models.

  • Problem

    MixUp’s effectiveness is established mainly empirically, while the role and tuning of its global interpolation parameter α remain insufficiently explained.

  • Method

    The paper models MixUp as out-of-manifold regularization with local linearity constraints and proposes AdaMixUp to learn policy regions that avoid manifold intrusion.

  • Results

    AdaMixUp significantly improves current deep classification models and outperforms standard MixUp by a large margin across several image-classification tasks.

  • Takeaways & Limitations

    MixUp’s useful policy region can be understood as a region avoiding manifold intrusion, while AdaMixUp automatically learns such regions from data.

  • Takeaways & Limitations

    Manifold intrusion makes MixUp’s local-linearity constraint fail on a nonzero-measure policy region when mixed points collide with manifold examples.

Abstract

from arXiv · show

MixUp is a recently proposed data-augmentation scheme, which linearly interpolates a random pair of training examples and correspondingly the one-hot representations of their labels. Training deep neural networks with such additional data is shown capable of significantly improving the predictive accuracy of the current art. The power of MixUp, however, is primarily established empirically and its working and effectiveness have not been explained in any depth. In this paper, we develop an understanding for MixUp as a form of "out-of-manifold regularization", which imposes certain "local linearity" constraints on the model's input space beyond the data manifold. This analysis enables us to identify a limitation of MixUp, which we call "manifold intrusion". In a nutshell, manifold intrusion in MixUp is a form of under-fitting resulting from conflicts between the synthetic labels of the mixed-up examples and the labels of original training data. Such a phenomenon usually happens when the parameters controlling the generation of mixing policies are not sufficiently fine-tuned on the training data. To address this issue, we propose a novel adaptive version of MixUp, where the mixing policies are automatically learned from the data using an additional network and objective function designed to avoid manifold intrusion. The proposed regularizer, AdaMixUp, is empirically evaluated on several benchmark datasets. Extensive experiments demonstrate that AdaMixUp improves upon MixUp when applied to the current art of deep classification models.

Introduction

MixUp improves deep-network accuracy empirically, but its mechanism and hyper-parameter α have been insufficiently explained. The paper interprets MixUp as out-of-manifold regularization, identifies manifold intrusion, and proposes adaptive AdaMixUp to avoid it.

  • MixUp linearly interpolates training examples and their labels to generate synthetic training data that can improve predictive accuracy.
  • MixUp’s effectiveness is established mainly empirically, while its global interpolation parameter α must be tuned by trial and error.
  • The paper formulates MixUp as data-dependent regularization that imposes local linearity constraints beyond the data manifold.
  • Manifold intrusion occurs when a mixed example collides with a real example but receives a conflicting soft label, causing under-fitting and degraded performance.
  • AdaMixUp generalizes mixing to higher folds and learns policy regions from data to automatically avoid manifold intrusion.

Preliminaries

The paper defines classification over a vector input space containing a data manifold of valid examples. Training seeks a model that maps inputs to predictive label distributions while optimizing a data-dependent loss.

  • Each example lies in a vector space X, while labels belong to a set Y and classifiers output predictive distributions over Y.
  • The data manifold M is the subset of X containing valid examples that can be associated with labels.
  • Under the Basic Hypothesis, every x ∈ M has a unique label g(x), whereas g is undefined outside M.
  • The training set D is a labeled subset of the data manifold M.
  • A mixing policy is a probability distribution used as coefficients for convex combinations, and a policy region is a nonzero-measure subset of the relevant simplex.
  • Learning selects a model H from a hypothesis space by optimizing a loss L_D(H) dependent on the training data.

Regularization

Regularization constrains model capacity or the hypothesis space, either independently of data or with respect to training examples. The paper positions MixUp as data-dependent regularization that constrains behavior outside the data manifold.

  • Regularization reduces model capacity or the space H to address overfitting and poor generalization.
  • Data-independent regularization constrains functions directly without allowing the training data to enter the regularization term.
  • Data augmentation adds presumed in-manifold examples and trains on their labels, thereby imposing data-dependent constraints on the model.
  • Adversarial training similarly adds examples assumed to share labels with the corresponding training inputs.
  • MixUp differs by introducing constraints on the label function outside the manifold M.

Regularization

The paper frames MixUp as data-dependent regularization that imposes local linearity constraints beyond the data manifold. It identifies manifold intrusion as a source of under-fitting and proposes AdaMixUp, which learns mixing policies from data to avoid intrusion while extending MixUp to higher-fold mixtures.

  • Regularization: MixUp can be viewed as data-dependent regularization because it augments training with synthetic examples formed from data-dependent mixing policies.The synthetic example is paired with the correspondingly mixed training target.
  • Regularization: MixUp constrains models by requiring local linearity for convex mixtures of training examples and their soft labels.The standard formulation uses two examples, while higher-fold local linearity generalizes the constraint.
  • Manifold intrusion: Manifold intrusion occurs when a mixed example lands on the data manifold but receives a soft label conflicting with the real example's label.This makes the MixUp constraint incompatible with the original training condition.
  • Manifold intrusion: Intrusion can empty the intersection between fitting the original data and satisfying MixUp, producing under-fitting and high prediction bias.Without intrusion, MixUp instead reduces the hypothesis space while retaining compatible classifiers, lowering prediction variance and over-fitting.
  • Adaptive MixUp (AdaMixUp): The original MixUp hyper-parameter search can be understood as trial-and-error discovery of the largest intrusion-free policy region.The paper generalizes this idea by learning policy regions from data with a policy region generator.
  • Adaptive MixUp (AdaMixUp): AdaMixUp extends mixing from two-fold to higher-fold policies and jointly regularizes models while penalizing intrusion to prevent over-regularization.For kmax > 2, the generalized hypothesis imposes additional constraints that further regularize the model and improve generalization.

Implementation of AdaMixUp

AdaMixUp recursively constructs higher-fold mixing policies using a single network π2, while Fold-2 policies generate adaptive coefficient intervals and an intrusion discriminator supports training.

  • Recursive policy construction: A single network π2 recursively expresses Λk(X) for all input fold numbers k.This avoids constructing kmax − 1 separate policy networks.
  • Fold-2 policy generation: For Fold-2 mixing, π2 returns α and ∆ defining the coefficient interval (α, α + ∆), then samples γ within that interval.The network generates α, ∆, and a discarded third softmax value whose sum is one.
  • Higher-fold generation: Higher-fold samples are generated by repeatedly mixing one example from Dnew with one original example from D using a policy region from π2.Dnew contains original and previously generated mixed examples.
  • Differentiable sampling: The policy sampler uses reparametrization, drawing ǫ uniformly and computing γ := ∆·ǫ + α so gradients can back-propagate through sampling.This implements drawing γ from (α, α + ∆).
  • Joint training: The intrusion discriminator ϕ classifies original versus mixed examples and is trained jointly with π2 through alternating discriminator, classifier, and intrusion-loss optimization.ϕ shares all but the final layers with the classifier and ends in a logistic binary classifier.

Experiments

AdaMixUp is evaluated across eight image-classification benchmarks using CNN and ResNet-18 baselines. It outperforms both Baseline and MixUp across all testing datasets, while learned policies and training behavior vary across tasks.

  • Data sets and settings: AdaMixUp is evaluated on eight benchmarks, reporting top-1 and top-5 error rates across datasets ranging from MNIST to ImageNet-R.The experiments use reduced Cifar10-S and Cifar100-S variants and omit SVHN’s additional samples.
  • Experimental comparisons: The study compares Fold-2 AdaMixUp with baseline CNN or ResNet-18 networks and MixUp applied to those baselines.MNIST and Fashion use three-layer CNNs; the other six tasks use ResNet-18.
  • Predictive performance: AdaMixUp outperforms both the Baseline and MixUp on all eight testing datasets, with at least 5.7% relative error reduction over the Baseline.The relative reduction exceeds 30% on SVHN and Cifar10.
  • Predictive performance: With default α, MixUp fails to improve baseline accuracy on MNIST, Cifar10-S, Cifar100-S, and ImageNet-R, suggesting over-regularization or under-fitting.These comparisons are reported in Table 1’s error-rate results.
  • Learned policies: Learned α varies from 0.4 to 0.9 across datasets, while ∆ ranges from 0.010 to 0.035 and intrusion losses remain close to zero.The reported losses suggest the datasets support Fold-2 AdaMixUp under the proposed policy-generator structure.
  • Training characteristics: On Cifar10, the policy generator stabilizes around 40K iterations with mean α = 0.48 and mean ∆ = 0.03.Figure 4 tracks the means of α and ∆ during Fold-2 training.
  • Training characteristics: On Cifar10, the mixing-policy range stabilizes around 40K iterations, coinciding with a drop and subsequent stabilization in classification error.Figure 5 pairs the mean γ trajectory with training and testing error rates.

2 AdaMixUp

AdaMixUp improves as the mixing fold increases, while its performance depends on model capacity, sample size, and the Intrusion Discriminator. However, higher-fold mixing makes policy-region learning more difficult, and hidden-layer interpolation substantially worsens error rates.

  • Impact of Mixing Fold: AdaMixUp performance improves when the mixing fold increases from 2 to higher values, reflecting stronger regularization constraints.Further increases may eventually cause diminishing gains or under-fitting.
  • Impact of Mixing Fold: Higher-fold AdaMixUp produces larger intrusion losses, while high α2 values indicate that mixing often occurs only slightly.The Policy Region Generator has greater difficulty finding regions that avoid intrusion.
  • Sensitivity to Model Capacity: Fold-2 AdaMixUp benefits from increasing ResNet-18 filter counts, whereas the baseline improves mainly before reaching half the base filter count.The comparison uses Cifar100 test error rates across filter-count settings.
  • Sensitivity to Training Samples: More training samples help both ResNet-18 and AdaMixUp, while the accuracy gap between the methods widens with sample size.The experiment varies Cifar100 training-set size from 20% to 80%.
  • Benefit of the Intrusion Discriminator: Removing the Intrusion Discriminator preserves improvement over baseline but yields substantially lower accuracy than the full AdaMixUp system.The ablation is evaluated on Cifar10 and Cifar100 with ResNet-18.
  • Interpolating on Hidden Layer: Hidden-layer interpolation dramatically increases Fold-2 AdaMixUp error rates to 22.21% on Cifar100 and 4.94% on Cifar10.The corresponding input-space error rates are 20.97% and 3.52%, respectively; intrusion losses are around 0.49.

Related Work

Prior augmentation methods rely on domain knowledge, particular architectures, feedback signals, or user-predefined mixing policies. AdaMixUp instead learns mixing policy regions automatically and supports mixing multiple images.

  • Related Work: Common augmentation methods depend on domain knowledge, specific network architectures, or feedback signals to search augmentation strategies.These requirements motivate a method using fewer such assumptions.
  • Related Work: Interpolation-based approaches rely on correct user-predefined mixing policies and typically interpolate along lines connecting sample pairs.AdaMixUp automatically learns policy regions and benefits from mixing multiple images.

Conclusions and Outlook

The paper frames MixUp and AdaMixUp as out-of-manifold regularization and addresses manifold intrusion by learning higher-fold policy regions. Its justification remains primarily qualitative, leaving quantitative generalization analysis and broader regularization designs for future work.

  • Conclusions and Outlook: AdaMixUp generalizes MixUp to higher-fold policies and automatically learns policy regions that avoid manifold intrusion.The paper presents this as an out-of-manifold regularization perspective on MixUp’s effectiveness.
  • Conclusions and Outlook: The paper’s justification is primarily qualitative, and it calls for quantitative characterization of AdaMixUp’s generalization capability.The authors also identify broader possibilities beyond local linearization for new regularization techniques.
Loading 1809.02499v3…