PaperlayerView exact revision on arXiv

Source-linked AI summary

Dataset Condensation via Efficient Synthetic-Data Parameterization

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, Hyun Oh Song

arXiv:2205.14959v2cs.LG

TL;DR

Massive-data training is costly, while existing condensation methods struggle to represent regular data efficiently and optimize their synthetic datasets. The paper introduces a regularity-aware multi-formation framework and improved gradient-matching optimization, reporting state-of-the-art performance across multiple datasets and domains, including 10∼20%p gains in experimental settings.

  • Problem

    Existing condensation methods directly optimize synthetic data without data regularity, limiting representability under storage constraints and leaving surrogate-objective challenges unresolved.

  • Method

    The paper uses differentiable multi-formation to generate multiple regularized synthetic data under fixed storage and trains optimization networks on real data with stronger regularization.

  • Results

    10∼20%p performance improvements over state-of-the-art methods are reported across datasets and domains including ImageNet and Speech Commands.

  • Takeaways & Limitations

    The unified algorithm achieves state-of-the-art condensed-data performance across settings including speech and continual learning.

  • Takeaways & Limitations

    Alternative parameterizations such as neural generators may require more storage or expensive synthesis computation.

Abstract

from arXiv · show

The great success of machine learning with massive amounts of data comes at a price of huge computation costs and storage for training and tuning. Recent studies on dataset condensation attempt to reduce the dependence on such massive data by synthesizing a compact training dataset. However, the existing approaches have fundamental limitations in optimization due to the limited representability of synthetic datasets without considering any data regularity characteristics. To this end, we propose a novel condensation framework that generates multiple synthetic data with a limited storage budget via efficient parameterization considering data regularity. We further analyze the shortcomings of the existing gradient matching-based condensation methods and develop an effective optimization technique for improving the condensation of training data information. We propose a unified algorithm that drastically improves the quality of condensed data against the current state-of-the-art on CIFAR-10, ImageNet, and Speech Commands.

1. Introduction

Massive datasets enable deep learning but impose substantial computational, environmental, and storage costs. Dataset condensation addresses this by synthesizing compact training data, while this work improves optimization through regularity-aware parameterization and multi-formation.

  • Massive real-world datasets drive deep-learning success but create large computational and environmental costs for training, tuning, and architecture search.
  • Coreset selection relies on heuristics and representative original samples, whereas condensation synthesizes compact datasets for improved storage efficiency.
  • Existing methods optimize synthetic data elements directly without regularity constraints, limiting representability and the number of data synthesized under fixed storage.
  • The proposed multi-formation process creates multiple synthetic data under the same storage constraints, imposing regularity while enlarging the optimized dataset.
  • 10∼20%p performance improvements over state-of-the-art methods are reported across datasets and domains including ImageNet and Speech Commands.

2. Preliminary

Dataset condensation seeks a compact surrogate whose trained models resemble those trained on the original data. Prior work commonly replaces direct performance optimization with gradient or feature matching, but these surrogate objectives retain unresolved challenges.

  • Dataset condensation constructs a compact surrogate dataset whose trained models should perform similarly to models trained on the original training set.
  • Existing surrogate optimization objectives remain insufficiently explained as proxies for the ultimate model-performance objective.
  • Gradient matching optimizes synthetic data by aligning network gradients computed on synthetic and real datasets during training.
  • The class-wise gradient-matching method uses alternating updates and has been extended with differentiable siamese augmentation for synthetic-data optimization.
  • Feature matching accelerates optimization by matching hidden features, but it exhibits performance degradation compared with gradient matching.

3. Multi-Formation Framework

The multi-formation framework improves synthetic-data optimization by increasing data multiplicity under fixed storage while preserving natural-data regularity. Its theoretical analysis formalizes when the enlarged formation space can achieve a no-worse dataset-distance optimum.

  • 3.1. Observation: More synthetic data reduce matching loss, motivating increased data multiplicity under a fixed storage budget.
  • 3.2. Multi-Formation: Existing methods directly match and use stored synthetic data, whereas multi-formation maps each stored data element to multiple deterministic training examples.
  • 3.2. Multi-Formation: The framework matches the original data using f(S) and evaluates models on the resulting enlarged dataset without increasing stored data.
  • 3.2. Multi-Formation: Differentiable multi-formation enables end-to-end gradient descent, using local interpolation to preserve spatial or temporal locality in natural data.
  • 3.3. Theoretical Analysis: The theoretical model assumes regular natural data lie in a subspace N of the m-dimensional data space.
  • 3.3. Theoretical Analysis: A dataset distance measure is defined through symmetry, replacement by a closer point, and non-increasing distance when target data points are added.
  • 3.3. Theoretical Analysis: Under N n′ ⊆Mf, Proposition 1 gives min S∈Rn×m D(f(S), T ) ≤ min S∈Rn×m D(S, T ).
  • 3.3. Theoretical Analysis: The proposition states that multi-formation attains a dataset at least as close to the original under any qualifying distance measure, with relaxed analysis provided separately.

4. Improved Optimization Techniques

The paper interprets gradient matching as saliency-weighted local-feature matching, identifies gradient vanishing and coupled optimization as limitations, and proposes real-data-trained networks, stronger augmentation, and alternating optimization to improve condensation.

  • Interpretation: Gradient matching can be viewed as saliency-weighted average local-feature matching because convolutional weight gradients are weighted sums of local hidden features.The weighting reflects loss sensitivity, while averaging compares features globally across locations.
  • Interpretation: Saliency weighting targets label-relevant information, while global local-feature matching can help when target objects are spatially non-aligned.The paper conjectures these properties explain gradient matching’s advantage over feature matching, especially for datasets such as ImageNet.
  • Problems and Solutions: DSA’s gradients vanish on synthetic data during training because the small condensed set overfits, while gradients on real data increase.This creates undesirable distance-based matching between synthetic and real data sources.
  • Problems and Solutions: Using networks trained on real data decouples synthetic-data optimization from network weights and alleviates gradient vanishing caused by overfitting.The larger real dataset reduces the dependence of the network on the condensed set.
  • Problems and Solutions: Stronger regularization combines augmentation sequences and CutMix, enabling distance-based objectives for better distillation of learning information.CutMix supplies soft-label training that addresses neural-network overconfidence.
  • Problems and Solutions: Condensation performs best near 10 epochs, when early-training gradients have larger norms and more aligned intra-class directions.The paper uses this early phase because it contains more useful information to distill and reduces training cost.
  • Algorithm: The unified algorithm alternates updates to the synthetic dataset and network weights, periodically reinitializing the network to preserve the early-training regime.It combines multi-formation with alternating optimization and can initialize networks randomly or from briefly pretrained weights.

5. Experimental Results

Experiments across CIFAR-10, ImageNet-subset, and Speech Commands evaluate condensed data under multiple models and controlled training costs. IDC and IDC-I consistently outperform baselines, while multi-formation improves cross-architecture transfer and dataset-specific factor choices matter.

  • Condensed Dataset Evaluation: IDC and IDC-I significantly outperform condensation baselines across CIFAR-10, ImageNet-subset, and Speech Commands evaluations.The experiments compare multiple datasets, compression settings, and test-model architectures.
  • Condensed Dataset Evaluation: IDC still consistently outperforms baselines when training steps and evaluation costs are matched.The comparison reduces IDC's training epochs and aligns gradient-descent steps across methods.
  • Condensed Dataset Evaluation: On ImageNet-subset, IDC and IDC-I outperform baselines across class counts, compression ratios, and test models, whereas some methods transfer poorly across architectures.IDC remains superior regardless of test-model type, unlike methods whose performance varies substantially by architecture.
  • Condensed Dataset Evaluation: On Mini Speech Commands, IDC achieves performance close to full-dataset training while outperforming baseline methods by large margins.The dataset contains 8,000 one-second clips across 8 command classes, represented as 64 × 64 magnitude spectrograms.
  • Analysis: Multi-formation factor 2 is optimal for CIFAR-10, whereas factors 3–4 are optimal for ImageNet.Higher factors produce more synthetic data at lower resolution, creating a dataset-dependent resolution–quantity trade-off.
  • Condensed Dataset Evaluation: Multi-formation improves performance by over 10%p on average and enables condensation with small networks for use on deeper networks.Experiments compare ConvNet-3 and ResNet-10, where DSA and DM do not maintain performance under network transfer.

6. Related Work

Prior work reduces dataset size through selection or synthesis, but the paper emphasizes storage-efficient parameterization and notes a limitation of its same-shape design.

  • Coreset selection constructs compact representative subsets but depends on heuristics and assumes representative original samples exist.
  • Earlier synthetic-dataset methods optimized training performance or used generative modeling without fully addressing storage efficiency.
  • The paper constrains condensation variables to the original training-data shape, enabling an intuitive formation function with negligible computation and storage overhead.
  • Alternative structures such as dictionary coding or neural-network generators may require more storage or expensive synthesis computation.

7. Conclusion

The paper addresses optimization difficulties in dataset condensation with a multi-formation framework and techniques targeting gradient matching, reporting state-of-the-art performance across several settings.

  • The proposed multi-formation process defines an enlarged and regularized optimization space for synthetic data.
  • The paper analyzes shortcomings of existing gradient matching and provides effective solutions for condensation optimization.
  • The resulting algorithm achieves state-of-the-art condensed-data performance across image, speech, and continual-learning experiments.

A.1. Proofs

The proofs formalize why multi-formation can improve dataset-distance objectives: regularity-based projection and increased synthetic-data count can reduce distance, subject to representability assumptions.

  • A dataset distance measure is defined through nonnegativity, identity and symmetry, replacement monotonicity, and augmentation monotonicity properties.
  • If N^n′ ⊆ M_f, then min_S D(f(S), T) ≤ min_S D(S, T), so the formed space cannot worsen the optimum distance.
  • Projection of synthetic points onto the regularity subspace makes each point no farther from every original point under the stated assumptions.
  • D(¯S_n, T) ≤ D(¯S_n−1, T) ≤ ... ≤ D(S, T), establishing nonincreasing dataset distance during partial projection.
  • Multi-formation increases the number of synthetic data from n to n′, and the augmented optimum lies in the regularity space under the proof assumptions.
  • With an ε-cover, the improvement is bounded by the gain G from increasing data count minus representability loss ε.

B. Datasets

The experiments use an ImageNet subset for large-scale image evaluation and Mini Speech Commands, an eight-class audio dataset split into training and test sets.

  • The ImageNet subset uses 100 classes, with the first 10 classes selected for ImageNet-10 experiments.
  • Mini Speech Commands contains one-second audio clips from eight command classes, with 1,000 samples per class.
  • Mini Speech Commands is randomly split using seven-eighths for training and one-eighth for testing.

C. Implementation Details

The experiments use fixed inner-loop and dataset-specific learning-rate settings, standardized evaluation procedures, and continual-learning protocols. Additional studies examine alternative multi-formation functions and their computational trade-offs.

  • Implementation settings: All experiments fix the number of inner iterations at M = 100, while learning rates and objectives vary by dataset.CIFAR-10 uses λ = 0.005, η = 0.01, and MSE; other datasets use L1 matching objectives with dataset-specific rates.
  • Evaluation protocol: CIFAR-10 evaluation trains models for 1,000 epochs with DSA augmentation and CutMix, while ImageNet uses random resize-crop, CutMix, and convergence-based training.ImageNet training lasts 2,000 epochs for 10 images per class and 1,500 epochs for 20 images per class.
  • Baseline evaluation: Baseline evaluations use author-released datasets or implementations, with CutMix omitted for DSA because it degraded performance.For DM, the authors implemented the method and obtained 65.6 on CIFAR-10 ConvNet-3 with 50 images per class, versus 63.0 reported previously.
  • Continual learning: The continual-learning experiments use distillation and continuous weight loading for class-incremental learning, and evaluate SVHN → MNIST → USPS task-incremental learning with exemplars.The task-incremental setting prohibits previously seen datasets except for a few exemplars and reports significant gains over baselines.
  • Multi-formation variants: The study also evaluates multi-scale and learnable multi-formation functions as alternatives to the default formation process.Multi-scale formation shares condensed data elements across synthetic data, while learnable formation uses FSRCNN with additional computation and storage overhead.

E.1. Experimental Results on Other Datasets

Across additional datasets, storage budgets, augmentations, architectures, and condensation objectives, the experiments generally favor the proposed multi-formation strategy, with benefits depending on the available budget and representation.

  • Other datasets: The methods consistently outperform baselines on SVHN, MNIST, FashionMNIST, and CIFAR-10, including the 1 img/cls setting.At low compression, multi-formation improves performance by up to 30%p on SVHN; its effect diminishes at FashionMNIST with 50 images per class.
  • Formation functions: The multi-scale formation function outperforms the uniform formation function on CIFAR-10.The comparison is reported in Table 8 as a test-performance evaluation of the two formation functions.
  • Augmentation ablation: Strong augmentation contributes about 1%p, compared with about 14%p from multi-formation and 9%p from algorithmic development.The ablation indicates that the algorithm does not mainly rely on augmentation.
  • Larger storage budgets: IDC outperforms other methods through 200 images per class but underperforms at 500 images per class, while IDC-I outperforms baselines in all tested storage settings.The results indicate diminishing returns from increasing synthetic data when storage is sufficient to represent original-data diversity.
  • Architecture effects: Simple convolutional networks outperform deeper ResNet architectures as condensation networks, and ConvNet-3 outperforms ConvNet-4 in the reported comparison.The architecture study includes width, depth, ResNet-10, and average-pooling variants.
  • Generality: Applying multi-formation to DM consistently improves DM performance across various CIFAR-10 test models.This supports the reported general applicability of the multi-formation framework to another feature-matching condensation method.
  • ZCA whitening: ZCA whitening mildly degrades IDC performance on CIFAR-10.The authors speculate that removing pixel correlation is unsuitable for IDC’s upsampling process.

F. Visual Examples

The visual examples compare synthetic and real samples across several datasets and show the condensed datasets produced by IDC under compact storage budgets. The examples emphasize class structure and dataset coverage.

  • Real versus synthetic images: Figures 10–13 compare IDC’s synthetic samples with real training samples on CIFAR-10, ImageNet, MNIST, and SVHN.The real samples were used to initialize the synthetic data, and the synthesized data is described as more class-representative.
  • Condensed datasets: Figures 14–17 show full IDC condensed datasets for MNIST, CIFAR-10, SVHN, and ImageNet-10 under a budget of 10 images per class.The ImageNet-10 examples use a multi-formation factor of 2 and 224 × 224-pixel images per class.
  • CIFAR-10 visualization: The CIFAR-10 condensed-data figure organizes each row by class, covering airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.The figure caption identifies the class labels in row order.
Loading 2205.14959v2…