PaperlayerView exact revision on arXiv

Source-linked AI summary

TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation

Samuel G. Müller, Frank Hutter

arXiv:2103.10158v2cs.CVcs.LG

TL;DR

Automatic augmentation methods can require costly policy search and complex task-specific designs. This paper introduces TrivialAugment, which samples one augmentation and its strength uniformly without learned policy parameters. It performs as well as or better than prior methods in almost all image-classification tasks, while object detection remains outside its out-of-the-box scope.

  • Problem

    Automatic augmentation policy search can be more expensive than training, motivating simpler methods that avoid task-specific policy learning.

  • Method

    TrivialAugment uniformly samples one augmentation and one strength for each image without learning a task-specific policy.

  • Results

    TrivialAugment performs as well as or better than previous methods in almost all image-classification tasks.

  • Takeaways & Limitations

    The results support not overlooking simple augmentation solutions, while the paper provides practical recommendations and reproducible code.

  • Takeaways & Limitations

    TrivialAugment does not work out-of-the-box for object detection and requires tuning there; broader task applicability requires further study.

Abstract

from arXiv · show

Automatic augmentation methods have recently become a crucial pillar for strong model performance in vision tasks. While existing automatic augmentation methods need to trade off simplicity, cost and performance, we present a most simple baseline, TrivialAugment, that outperforms previous methods for almost free. TrivialAugment is parameter-free and only applies a single augmentation to each image. Thus, TrivialAugment's effectiveness is very unexpected to us and we performed very thorough experiments to study its performance. First, we compare TrivialAugment to previous state-of-the-art methods in a variety of image classification scenarios. Then, we perform multiple ablation studies with different augmentation spaces, augmentation methods and setups to understand the crucial requirements for its performance. Additionally, we provide a simple interface to facilitate the widespread adoption of automatic augmentation methods, as well as our full code base for reproducibility. Since our work reveals a stagnation in many parts of automatic augmentation research, we end with a short proposal of best practices for sustained future progress in automatic augmentation methods.

1. Introduction

Data augmentation expands image-classification datasets with likely class-preserving transformations, while automatic methods seek effective policies but can make search costly. TrivialAugment challenges the need for task-specific policy learning with a simple baseline and unified comparisons.

  • Motivation: Data augmentation generates novel, likely class-preserving images to improve model generalization, but effective transformations require domain knowledge.Small image rotations illustrate how augmentations can preserve categories while encouraging human-like generalization.
  • Motivation: Automatic augmentation methods design augmentation policies automatically and have improved performance across tasks.
  • Motivation: Policy search can cost more than training, despite applying the resulting augmentation policy cheaply.
  • TrivialAugment: TrivialAugment challenges task-specific policy fitting with a baseline that performs comparably to more expensive methods without learning a policy per task.It applies no combinations of augmentations.
  • Contributions: The paper analyzes minimal requirements, evaluates methods in many setups with unified open-source code, and provides practical recommendations and code.

2. Related Work

Prior automatic augmentation methods vary widely in search cost, policy parameterization, and augmentation complexity. TrivialAugment is presented as a parameter-free, simpler alternative that applies one augmentation strategy and outperforms AugMix on the shared ImageNet comparison.

  • Common search space: Automatic augmentation methods operate over prespecified augmentations and discrete strength settings, then learn how to use them.The considered strength set is {0, . . . , 30}.
  • Previous methods: Prior methods range from expensive reinforcement-learning and evolutionary searches to cheaper approaches with restricted policy spaces.The related-work review orders methods by total computational cost.
  • Previous methods: RandAugment reduces task-specific tuning to augmentation strength m and augmentation count n, but still combines multiple augmentations per image.
  • TrivialAugment: TrivialAugment is parameter-free like UniformAugment but simpler, and is reported to perform better than comparatively cheap augmentation strategies.
  • Visualization: Figure 2 depicts deterministic augmented crosses and uniform sampling across them in a two-class dataset separated by a decision boundary.
  • Comparison: AugMix targets robustness through augmentation chains, mixed outputs, and consistency loss, whereas TrivialAugment outperforms it on the evaluated ResNet-50 ImageNet test metric.

3. TrivialAugment

TrivialAugment samples one augmentation and one strength uniformly at random for each image, producing a parameter-free procedure rather than complex stochastic augmentation combinations. Its setup also retains task-specific choices in the augmentation space and strength range.

  • Augmentation representation: Each augmentation is modeled as a function a mapping image x and discrete strength m to an augmented image, although some augmentations ignore m.Most augmentations use m to control distortion strength.
  • Procedure: TrivialAugment takes an image and augmentation set A, samples one augmentation uniformly, samples m uniformly from {0, . . . , 30}, and returns a(x, m).The procedure is summarized in Algorithm 1.
  • Procedure: Unlike previous multi-augmentation methods, TrivialAugment applies only one augmentation per image and views the resulting dataset distribution as an average over augmentation-specific distributions.
  • Practical setup: The method retains a task-dependent augmentation set A and strength range, so its parameter-free sampling procedure does not eliminate all setup choices.The supplied text also notes that choosing n and m per task can be difficult for new tasks.

4. Experiments

The experiments compare TrivialAugment with prior methods across datasets, models, training setups, compute budgets, and augmentation-space ablations. TA is generally among the strongest methods, while its performance depends on task-specific augmentation-space and strength choices.

  • Comparison to Published Results: TA performs as well or better than previous methods in almost all tasks, with SVHN as the only reported exception.The authors note that the SVHN difference may reflect training-pipeline differences and could not reproduce RandAugment’s published SVHN Core performance.
  • Comparison to Published Results: TA outperformed all other methods on ImageNet for both top-1 and top-5 accuracy.With image width 224, TA also achieved 77.97 ± .21 top-1 and 93.98 ± .07 top-5 accuracy, exceeding the previously best methods in that setup.
  • Comparison of Reproduced Results in a Fixed Training Setup: TA (Wide) was best across all Wide-ResNet-28-10 benchmarks, while TA (RA) was best on both Wide-ResNet-40-2 benchmarks in the reproduced comparison.This evaluation used reproductions in a unified codebase to reduce setup-related confounding factors.
  • TA with Different Hand-Picked Augmentation Spaces: TA’s performance was relatively stable across augmentation spaces, although the full space performed worst on both SVHN Core and CIFAR-10 for Wide-ResNet-28-10.The authors report that selecting the augmentation space more carefully could further improve results for particular tasks.
  • Ablation Studies: Performance decreases slowly as augmentation subsets shrink, and three strengths can suffice, but the best strength subset varies substantially across datasets.CIFAR-100 benefits from larger strength sets, whereas SVHN Core improves when only strength {30} is considered; mixtures of strong and weak augmentations appear useful.

5. Automatic Augmentation Methods in Practice

The authors emphasize that automatic augmentation methods should be practical because their impact depends on application to new settings. They provide practical guidance and software to support adoption and reproducibility.

  • Practicality matters because automatic augmentation methods realize their impact when applied to new setups and problems.The authors base this motivation on evaluating many settings and augmentation methods.
  • The authors compile application-oriented learnings for augmentation methods in Appendix F.
  • A one-file Python library implements RA, UA, and TA and supports all augmentation spaces considered in the work.The library can transform PIL images with TA and is intended to simplify practical use.

6. Best Practices Proposal for Research

The paper proposes best practices for sustainable automatic augmentation research, centered on reproducibility, fair comparisons, and uncertainty reporting. These practices address implementation differences and confounding experimental setups.

  • The authors found many published methods difficult to reimplement and often observed performance similar to the simple TA baseline under the original setups.
  • Sharing code improves beginner access and helps align experimental setups, reducing reproducibility problems caused by implementation differences.
  • Methods should be compared under the same setup, training budget, and augmentation space, or prior results should be reproduced with differences reported.
  • Confidence intervals should be reported to distinguish outperforming from performing comparably.

7. Limitations

TA worked across the tested image-classification settings but did not work out of the box for object detection. Object detection required tuning, and other computer-vision tasks remain insufficiently studied.

  • TA did not fail in the tested image-classification settings.
  • TA does not work out of the box for object-detection setups and requires tuning for that task.
  • The authors wholeheartedly recommend TA for image classification, while application to other computer-vision tasks requires further study.

8. Conclusion

The conclusion presents TA as a simple baseline that exposes overlooked simplicity in automatic augmentation. It also identifies randomness in augmentation strengths as important and emphasizes the need to evaluate practical alternatives carefully.

  • Conclusion: TA provides a crucial baseline that had been missing from automatic augmentation methods.
  • Conclusion: Despite many complicated policy-search methods, TA was overlooked even though it performs comparably or better.
  • Conclusion: The paper argues that simple solutions should not be overlooked when studying automatic augmentation methods.
  • Conclusion: Randomness in the chosen augmentation strengths appears very important for good performance.

B. Comparison of Different Methods on the Same Augmentation Space

Using identical augmentation spaces, TrivialAugment’s improvements generalize across augmentation methods and spaces, including non-standard image-classification datasets where it outperforms brute-force tuned RandAugment.

  • Same augmentation space: TA’s improvements generalize across augmentation spaces and methods when compared using each method’s exact augmentation space.The comparison uses reproduced results for each method on the augmentation space from its original paper.
  • Same augmentation space: On CIFAR-10 in the AA augmentation space, TA reaches 97.55 ± .06 versus AA’s 97.31 ± .22.
  • Non-standard datasets: TA continues to perform well on Occ. CIFAR-10 and Stanford Cars, outperforming even brute-force tuned RA in these settings.The experiments evaluate WRN-28-10 on Occ. CIFAR-10 and RN-50 on Stanford Cars.

D. Evaluation on EfficientNet-B1

The paper additionally evaluates augmentation methods with EfficientNet-B1 on ImageNet, using five reruns to compare average performance.

  • Evaluation setup: EfficientNet-B1 is evaluated on ImageNet following the original ImageNet setup closely.UA and RA are reimplemented because the compared methods do not evaluate this task.
  • Evaluation setup: Average performance is reported across five reruns for each augmentation method.

E. Approximation of the Compute Costs for Different Methods

The compute-cost analysis estimates method overhead on an RTX 2080 Ti, showing that search-based methods add substantial costs while TA requires only standard training.

  • Cost estimation: The comparison estimates costs on an RTX 2080 Ti and includes only models that the authors ran.For CIFAR-100, the analysis considers Wide-ResNets and Shake-Shake-26-2x96d.
  • Cost estimation: One full CIFAR-10 epoch is estimated at 101 seconds for Wide-ResNet-28-10 and 83 seconds for Shake-Shake-26-2x96d.
  • Search-based methods: AA’s search cost is estimated at 1600 hours before adding standard training time.The estimate uses 15,000 evaluations of Wide-ResNet-40-2 on CIFAR-10-derived training workloads.
  • Search-based methods: Fast AA is estimated to cost more than one full training, so its compute cost is approximated as one training.
  • Parameter-free methods: UA and TA incur no search costs, making total cost equal to a single training.The total is computed as #epochs · costperepoch.
  • Search-based methods: The analysis models additional overhead for RA, OHL, AWS, and Adv. AA using their reported search, worker, epoch, or augmented-batch settings.Examples include RA’s 5·9/10 search factor, OHL’s 8 · 300/200 factor, and AWS’s 500-policy assumption.

F. Recommendations for the Application of Automatic Augmentation Methods

The paper recommends choosing augmentation methods according to whether the model–dataset setup is established in prior literature or novel, while emphasizing reproducibility information.

  • Reproducibility: Table 12 compares reproducibility through policy availability, training-code availability, and search or meta-training code availability.
  • Standard model and dataset: For a standard model and dataset combination, the authors recommend using the best published method with published code and policies.
  • Novel model or dataset: For a novel model or dataset, they recommend trying both the best model on a similar task and a parameter-free baseline.They identify UniformAugment and TrivialAugment as examples of parameter-free baselines.
  • Novel model or dataset: Parameter-free baselines are expected to generalize to new tasks because they generalized across standard automatic-augmentation benchmarks without tuning.
Loading 2103.10158v2…