Source-linked AI summary
Dataset Condensation with Differentiable Siamese Augmentation
Bo Zhao, Hakan Bilen
TL;DR
Large-scale training is computationally expensive, and existing condensation or augmentation strategies do not adequately support small synthetic sets trained with augmentation. The paper introduces Differentiable Siamese Augmentation, which jointly applies shared transformations to real and synthetic data during condensation. It reports about 7% improvement on CIFAR10/100 over state-of-the-art methods and promising results in continual learning and neural architecture search.
Problem
Large datasets make deep-network training computationally expensive, while synthetic images are not necessarily optimized for use with data augmentation.
Method
Differentiable Siamese Augmentation applies the same randomly sampled transformation to real and synthetic data and backpropagates through differentiable transformations while learning synthetic images.
Results
About 7% improvement on CIFAR10/100 over state-of-the-art methods is reported across multiple image-classification benchmarks.
Takeaways & Limitations
The learned synthetic training set can be used with data augmentation and shows promising results for continual learning and neural architecture search.
Takeaways & Limitations
Using minibatches from multiple classes leads to slower convergence because gradient averaging makes image/class correspondence harder to retrieve.
Abstract
from arXiv · showhide
In many machine learning problems, large-scale datasets have become the de-facto standard to train state-of-the-art deep networks at the price of heavy computation load. In this paper, we focus on condensing large training sets into significantly smaller synthetic sets which can be used to train deep neural networks from scratch with minimum drop in performance. Inspired from the recent training set synthesis methods, we propose Differentiable Siamese Augmentation that enables effective use of data augmentation to synthesize more informative synthetic images and thus achieves better performance when training networks with augmentations. Experiments on multiple image classification benchmarks demonstrate that the proposed method obtains substantial gains over the state-of-the-art, 7% improvements on CIFAR10 and CIFAR100 datasets. We show with only less than 1% data that our method achieves 99.6%, 94.9%, 88.5%, 71.5% relative performance on MNIST, FashionMNIST, SVHN, CIFAR10 respectively. We also explore the use of our method in continual learning and neural architecture search, and show promising results.
1. Introduction
Large datasets and models deliver strong performance but impose substantial computational costs, while coreset selection cannot reliably preserve distributed information. The paper addresses this by learning synthetic datasets with Differentiable Siamese Augmentation, which improves augmentation-compatible condensation and benchmark performance.
- Computational Motivation: 12.3k TPU days are required to train EfficientNet-L2 on JFT-300M, illustrating the computational burden of large-scale training.The paper estimates the corresponding training energy at about 3 × 10^7 J.
- Limitations of Coresets: Coreset methods use heuristics without guaranteed downstream-task optimality and may fail when information is uniformly distributed across the dataset.This motivates generating synthetic training sets rather than selecting existing examples.
- Augmentation Challenge: Naively augmenting synthesized images can yield negligible gains because synthetic images differ from natural images and are not optimized for transformations.The paper therefore treats combining synthesis and augmentation as requiring a principled design.
- Proposed Approach: Differentiable Siamese Augmentation applies the same randomly sampled transformation to real and synthetic data while allowing gradients to reach synthetic images.This transfers augmented information from real images and supports later augmentation across network architectures.
- Reported Results: Around 7% improvement on CIFAR10/100 is reported over state-of-the-art methods, alongside promising applications to continual learning and neural architecture search.The paper validates the method on multiple image-classification benchmarks.
2. Related Work
The related work spans dataset condensation, data augmentation, auto-augmentation, GANs, and differentiable augmentation. DSA differs by using a shared transformation for real and synthetic images while synthesizing training data rather than searching for an augmentation policy or generating realistic images.
- Data Augmentation: Data augmentation expands effective training sets with semantic-preserving transformations such as cropping, cutout, flipping, scaling, and rotation.Mixup and CutMix are cited as more elaborate augmentation strategies.
- Auto-augmentation: Auto-augmentation methods automatically search for augmentation strategies through adversarial optimization, reinforcement learning, or learned transformation parameters.The paper instead focuses on synthesizing training data.
- GANs: GANs generally generate realistic novel images by fooling a discriminator network.This objective differs from dataset condensation for downstream training.
- Differentiable Siamese Augmentation: Figure 1 depicts DSA matching network-parameter gradients from identically augmented real and synthetic minibatches to update synthetic images.The shared example uses the same rotation degree for both batches.
- Differentiable Augmentation: Differentiable augmentation has been applied to GAN training, but typically applies transformations independently to real and fake images.DSA uses a Siamese strategy that shares the transformation across real and synthetic data.
3. Method
The method condenses a large training set by matching training behavior on synthetic and real data, then integrates differentiable, shared augmentation into this objective. Its algorithm repeatedly samples transformations and class-specific minibatches, while multi-class minibatches can slow convergence.
- Dataset Condensation Review: Dataset Condensation learns a much smaller synthetic set whose trained network aims to achieve comparable generalization performance to a network trained on the full set.The formulation compares losses on synthetic and real datasets under the same network parameters.
- Dataset Condensation Review: The condensation objective replaces nested optimization with approximate equality of real and synthetic parameter updates at each iteration from the same initialization.A single network parameterization is used to simplify learning.
- Dataset Condensation Review: The matching discrepancy D is defined as a sum of cosine distances between gradients of weights associated with each output node at each layer.This gradient discrepancy drives synthetic-data optimization.
- Differentiable Siamese Augmentation: Post-hoc augmentation provides negligible gains, so the method applies semantic-preserving transformations while learning synthetic images.The transformation family includes cropping, color jittering, and flipping, with parameters for synthetic and real sets.
- Differentiable Siamese Augmentation: Siamese augmentation samples one transformation and applies it consistently across the real and synthetic minibatch pair, avoiding ambiguous gradient matching from independently sampled transformations.Independent crops could force one synthetic region to match gradients from different real-image regions, producing averaging and information loss.
- Differentiable Siamese Augmentation: Differentiable transformations enable backpropagation of the matching-loss gradient to the synthetic images.Standard image augmentations are implemented as differentiable layers for this purpose.
- Training Algorithm: The training algorithm randomly reinitializes network parameters across outer iterations and updates class-specific synthetic data using sampled transformations and gradient matching.Inner-loop updates use real and synthetic minibatches from one class, while model parameters can also be trained on augmented synthetic data.
- Training Algorithm: Multi-class minibatches slow convergence because averaging gradients across classes makes image/class correspondence harder to retrieve.This is presented as a limitation of the training procedure.
4. Experiments
Experiments evaluate DSA across image-classification benchmarks, augmentation schemes, continual learning, and neural architecture search. DSA generally improves condensed-data performance, with especially large gains on CIFAR datasets and useful transfer across architectures and tasks.
- Benchmark results: DSA achieves the best performance in most settings and improves over DC by 7.2% and 6.7% on CIFAR10 with 10 and 50 images per class.It can perform comparably or worse than DC with only 1 image per class.
- Benchmark results: 99.2% accuracy on MNIST with 50 images per class approaches the 99.6% upper bound using less than 1% of the data.The performance gap to the upper bound is larger on more challenging datasets.
- Synthetic-set visualization: Synthetic images capture diverse appearances, writing styles, viewpoints, and backgrounds, and are more recognizable and more similar to real images than earlier methods' images.The visualization uses 10 synthetic images per class for MNIST and CIFAR10.
- Ablation study: Siamese augmentation with cropping improves CIFAR10 performance by 3.6% over no augmentation and by 2.2% over cropping synthetic data only during testing.Applying transformations only to real or synthetic images, or independently during synthesis, performs worse than no augmentation.
- Continual learning and neural architecture search: DSA outperforms herding and DC in continual learning, reaching 94.4% testing accuracy after three tasks without knowledge distillation.This surpasses DC by 1.4% and herding by 3.7%; for neural architecture search, DSA attains 0.76 correlation at 1.2% of the full-data time cost.
5. Discussion
The discussion attributes DSA’s stronger synthetic data to Siamese augmentation acting as a regularizer that matches real and synthetic gradients across transformations. Synthetic images are then used to train networks without post-processing, with training illustrated from noise and real-image initialization.
- 5.1. Why Does DSA Work?: DSA is hypothesized to regularize high-dimensional synthetic data and reduce its overfitting to the real training set.The analysis reformulates the objective over multiple randomly sampled augmentations.
- 5.1. Why Does DSA Work?: Multiple transformations force the synthetic set to match gradients from the real set under sampled augmentations, making optimization harder and less prone to overfitting.
- 5.1. Why Does DSA Work?: The learning and rendering process illustrates two-class CIFAR10 synthetic data initialized from random noise and real images.
- 5.1. Why Does DSA Work?: At iteration k = 1000, synthetic-data gradient magnitudes quickly vanish without DSA, producing very small updates, whereas DSA keeps synthetic images learnable.The comparison is reported for 10 images per class on CIFAR10 with and without DSA.
- 5.1. Why Does DSA Work?: Synthetic images are initialized from normalized real training images of the corresponding category, optimized with the condensation objective, and then used without post-processing.
6. Conclusion
The paper concludes that DSA enables synthetic training sets to work effectively with data augmentation, improving performance over prior methods. It also reports promising applications in continual learning and neural architecture search.
- 6. Conclusion: DSA enables synthetic training sets to be used with data augmentation when training deep neural networks.
- 6. Conclusion: About 7% improvement on CIFAR10/100 is reported over state-of-the-art methods.
- 6. Conclusion: DSA also shows promising results in continual learning and neural architecture search.