Source-linked AI summary
SuperMix: Supervising the Mixing Data Augmentation
Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, Nasser M. Nasrabadi
TL;DR
Existing mixing augmentations can disregard salient image regions and produce inaccurate labels, while training data quality and quantity matter for generalization. SuperMix uses supervised masks to combine salient regions while preserving realistic image priors, and its reported evaluations show comparable classification performance to automated augmentation methods and improved knowledge distillation.
Problem
Existing mixing methods can disregard salient regions, suppress visual patterns, and produce inaccurate pseudo-labels.
Source: §1 · p. 1Method
SuperMix uses supervised mixing masks to combine salient regions from multiple input images while constraining the mixtures to retain salient features and realistic image priors.
Source: §1 · pp. 1–2Results
SuperMix enhances knowledge-distillation performance and is comparable to automated augmentation methods; combining it with those methods notably improves model generalization.
Source: §5 · p. 8Takeaways & Limitations
Mixed images intrinsically induce smooth predictions, helping reveal teacher-model knowledge in knowledge distillation.
Source: §1 · p. 2
Abstract
from arXiv · showhide
This paper presents a supervised mixing augmentation method termed SuperMix, which exploits the salient regions within input images to construct mixed training samples. SuperMix is designed to obtain mixed images rich in visual features and complying with realistic image priors. To enhance the efficiency of the algorithm, we develop a variant of the Newton iterative method, $65\times$ faster than gradient descent on this problem. We validate the effectiveness of SuperMix through extensive evaluations and ablation studies on two tasks of object classification and knowledge distillation. On the classification task, SuperMix provides comparable performance to the advanced augmentation methods, such as AutoAugment and RandAugment. In particular, combining SuperMix with RandAugment achieves 78.2\% top-1 accuracy on ImageNet with ResNet50. On the distillation task, solely classifying images mixed using the teacher's knowledge achieves comparable performance to the state-of-the-art distillation methods. Furthermore, on average, incorporating mixed images into the distillation objective improves the performance by 3.4\% and 3.1\% on CIFAR-100 and ImageNet, respectively. {\it The code is available at https://github.com/alldbi/SuperMix}.
1. Introduction
SuperMix combines salient regions from multiple images to create mixed training data, addressing limitations in existing mixing methods. The paper formalizes supervised mixing, constrains masks to preserve salient features and realistic image structure, and introduces a faster optimization algorithm.
1. Introduction
- Data augmentation enlarges training sets to support generalization, while prior mixing methods can ignore salient regions, fail to scale, suppress visual patterns, or use inaccurate pseudo-labels.Source: §1 · p. 1
- SuperMix uses salient regions from input images to construct more advantageous mixed training data, with supervision from self-training or a teacher guiding a student.Source: §1 · pp. 1–2
- The paper formalizes supervised mixing with masks that associate each output pixel with input-image locations and constrains solutions to retain salient features and realistic image priors.Source: §1 · p. 2
- 65× speed-up over SGD on ImageNet is achieved by the modified Newton iterative algorithm developed for large-scale SuperMix optimization.Source: §1 · p. 2
- Mixed images intrinsically induce smooth predictions, helping reveal teacher-model knowledge in knowledge distillation.Source: §1 · p. 2
2. Related work
Prior augmentation work includes single-image transformations and automated policy search, while mixing methods combine multiple images. The paper contrasts SuperMix with approaches that mix without saliency guidance or scale poorly.
2. Related work
- Automated policy methods search for image transformations, while most augmentation methods focus on transforming single images rather than combining them.Source: §2 · p. 2
- Smart Augmentation merges same-class images but requires an additional concurrently trained DNN, limiting scalability and the diversity of merged patterns.Source: §2 · p. 2
- MixUp and CutMix can average or replace salient regions with insignificant features, and their unsupervised mixed-image labels may be inaccurate; SuperMix uses salient regions and realistic-prior constraints.Source: §2 · p. 2
3. Supervised Mixing Augmentation
SuperMix combines images locally with spatial mixing masks and uses teacher predictions to guide masks toward mixed images that retain salient input information. Its optimization balances teacher-label agreement with smoothness and sparsity, using a Newton-based update to make mask estimation efficient.
3.1. Mixing function
- SuperMix generalizes local image mixing with spatial masks, recovering MixUp and CutMix under their respective uniform-weight and cropped-region conditions.Each mask assigns a weight to each spatial location, and the weights across input images sum to one.Source: §3.1 · pp. 2–3
3.2. Supervised mixing
- Teacher predictions guide masks so the mixed image reflects salient input regions while matching a target soft label formed from Dirichlet-sampled class weights.The objective also penalizes mask roughness and lack of sparsity to encourage spatially smooth masks and reduce averaging across images at each location.Source: §3.2 · p. 3
3.3. Optimization Method
- A Newton-based method with smooth projection updates masks using smoothed gradients, optimizing teacher-label agreement and sparsity while preserving mask smoothness.Masks are optimized at a down-sampled resolution and up-sampled before mixing; iterations stop when the teacher's top-k predicted classes match the input samples' predicted classes.Source: §3.3 · p. 4
4. Experiments
Experiments evaluate SuperMix for object classification and knowledge distillation on CIFAR-100 and ImageNet. The results cover benchmark comparisons, ablations, execution time, and mixed-image representations.
4.1. Object classification
- SuperMix is competitive with automated augmentation in four of five classification experiments, and adding RandAugment improves all tested settings.The cited table reports CIFAR-100 Top-1 and ImageNet Top-1/Top-5 accuracy.Source: §4.1 · p. 5, Table 1 · p. 5
4.2. Knowledge Distillation
- On CIFAR-100 and ImageNet, teacher-supervised mixed data supports strong distillation: CE consistently outperforms prior methods, and supervised mixing outperforms blind mixing.On ImageNet, combining mixed data with the original distillation objective further enhances performance.Source: §4.2 · p. 6, Table 2 · p. 6, Table 3 · p. 7
4.3. Ablation studies
- Distillation plateaus at 5 × 10^5 mixed samples; SuperMix favors k = 2 on CIFAR-100 but k = 3 on ImageNet, with α = 3 and λs ≈ 30 selected.For MixUp and CutMix, k > 2 degrades distillation; increasing sparsity promotes preservation of important features until its loss begins to dominate the KL loss.Source: §4.3 · p. 7
4.4. Execution time
- SuperMix is at least 65× faster than SGD and at least 19× faster than the same algorithm when it includes Ls.The comparison attributes the latter speedup to SP directly satisfying the spatial smoothness condition.Source: Table 4 · p. 7, §4.4 · p. 8
4.5. Embedding space evaluations
- SuperMix embeddings overlap less with original-data representations and cluster near the center; model predictions on mixed images are smoother, most so for SuperMix.The paper interprets the embedding pattern as evidence that SuperMix data encompass more novel structure and are harder for the model to classify.Source: §4.5 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8
5. Conclusion
SuperMix combines salient regions from multiple images to create unseen training samples. Experiments validate its effectiveness and efficiency, with reported gains for knowledge distillation and improved generalization when combined with automated augmentation methods.
5. Conclusion
- SuperMix combines salient regions from multiple images to produce unseen training samples.Source: §5 · p. 8
- Extensive experiments, evaluations, and ablation studies validate SuperMix’s effectiveness and efficiency.Source: §5 · p. 8
- SuperMix enhances state-of-the-art knowledge distillation, performs comparably to automated augmentation methods, and notably improves generalization when combined with them.Source: §5 · p. 8