PaperlayerView exact revision on arXiv

Source-linked AI summary

Dataset Condensation with Gradient Matching

Bo Zhao, Konda Reddy Mopuri, Hakan Bilen

arXiv:2006.05929v3cs.CVcs.LG

TL;DR

Large datasets impose substantial storage and training costs, while existing selection methods rely on heuristics and representative examples. Dataset Condensation learns a small synthetic set by matching gradients from original and condensed data, and it outperforms prior methods across vision benchmarks while supporting efficient continual learning and neural architecture search.

  • Problem

    Large datasets make storage, preprocessing, and model training expensive, while existing coreset methods lack guaranteed downstream optimality and representative samples.

  • Method

    Dataset Condensation learns synthetic samples by minimizing the distance between network-loss gradients computed on original and condensed training sets.

  • Results

    The method outperforms Dataset Distillation and coreset baselines across computer-vision benchmarks; with 50 MNIST images per class, it reaches 98.8% accuracy versus 99.6% using 6000.

  • Takeaways & Limitations

    Condensed images reduce dataset memory and training costs, while supporting training across different deep networks and applications in continual learning and neural architecture search.

  • Takeaways & Limitations

    The original nested-loop formulation does not scale to large models or accurate inner-loop optimization with many steps because it requires expensive computation-graph unrolling.

Abstract

from arXiv · show

As the state-of-the-art machine learning methods in many fields rely on larger datasets, storing datasets and training models on them become significantly more expensive. This paper proposes a training set synthesis technique for data-efficient learning, called Dataset Condensation, that learns to condense large dataset into a small set of informative synthetic samples for training deep neural networks from scratch. We formulate this goal as a gradient matching problem between the gradients of deep neural network weights that are trained on the original and our synthetic data. We rigorously evaluate its performance in several computer vision benchmarks and demonstrate that it significantly outperforms the state-of-the-art methods. Finally we explore the use of our method in continual learning and neural architecture search and report promising gains when limited memory and computations are available.

1 INTRODUCTION

Dataset Condensation addresses the storage and training burden of large datasets by learning a small synthetic set optimized for downstream neural-network performance. It matches gradients between models trained on original and condensed data, outperforming prior approaches across computer-vision benchmarks.

  • Motivation: Large-scale datasets make storage, preprocessing, and model training burdensome, motivating more data-efficient training sets.The passage describes datasets containing millions of samples and the infrastructure required to process and train on them.
  • Motivation: Heuristic coreset methods may lack downstream-task optimality and representative samples, while Dataset Distillation learns informative images from the training data.The cited related-work passage identifies both shortcomings and positions Dataset Distillation as addressing them.
  • Approach: Dataset Condensation learns a small synthetic set whose trained network achieves comparable performance and a close parameter-space solution to training on the large dataset.The method is designed to optimize synthetic samples for downstream generalization rather than restrict them to original examples.
  • Approach: The method formulates condensation as minimizing the difference between gradients computed on the large training set and the learnable condensed set.This gradient-matching objective updates synthetic samples using gradients of the network parameters.
  • Contributions: Across multiple computer-vision benchmarks, the method outperforms Dataset Distillation and coreset methods by a wide margin.The paper also investigates whether one synthetic set can train models across different neural-network architectures.

2 METHOD

Dataset Condensation learns a much smaller synthetic training set whose models approximate models trained on the original data. The method avoids expensive parameter-optimization unrolling by matching real and synthetic gradients, with curriculum guidance and layerwise matching for scalability.

  • 2.1 DATASET CONDENSATION: A dataset T of labeled images is condensed into a synthetic set S with |S| ≪ |T|, then models trained on S are evaluated on real data.The synthetic set includes images and labels, and is intended to preserve generalization performance relative to training on T.
  • 2.1 DATASET CONDENSATION: The target is comparable real-data generalization between models trained on the original and condensed sets.The method seeks Ex∼PD[ℓ(φθT(x), y)] ≃ Ex∼PD[ℓ(φθS(x), y)].
  • 2.2 DATASET CONDENSATION WITH PARAMETER MATCHING: Parameter matching can be computationally expensive because it requires an inner optimization, while incomplete optimization trades speed against reaching the optimal solution.Large models and many optimization steps make recursive or inner-loop procedures difficult to scale.
  • 2.2 DATASET CONDENSATION WITH PARAMETER MATCHING: Parameter matching seeks synthetic samples whose trained network approaches the original network in parameter space, using local smoothness to relate similar weights to similar mappings.The formulation considers random initialization distributions rather than only one fixed initialization.
  • 2.3 DATASET CONDENSATION WITH CURRICULUM GRADIENT MATCHING: Curriculum gradient matching decomposes the objective into subproblems so synthetic parameters follow the training trajectory of the original-data parameters, not only its endpoint.This provides more guided optimization while constraining the optimization dynamics.
  • 2.3 DATASET CONDENSATION WITH CURRICULUM GRADIENT MATCHING: The final method minimizes the distance between gradients from real and synthetic losses, eliminating recursive unrolling and improving speed, memory efficiency, and scalability.At each iteration, randomly initialized networks compute both losses and gradients, then synthetic samples are updated to match them.
  • 2.3 DATASET CONDENSATION WITH CURRICULUM GRADIENT MATCHING: Gradient matching uses layerwise losses that group gradient vectors by output node instead of flattening tensors across all layers.The gradients correspond to fully connected and convolutional weights, and the paper reports this as a better distance for gradient matching.

3 EXPERIMENTS

Experiments evaluate Dataset Condensation across benchmark datasets, architectures, baselines, and applications. The method generally improves data efficiency, transfers across architectures, and reduces computational or memory demands.

  • Benchmark evaluation: Experiments cover MNIST, FashionMNIST, SVHN, and CIFAR10 using MLP, ConvNet, LeNet, AlexNet, VGG-11, and ResNet-18.The evaluation includes classification, cross-architecture generalization, continual learning, and neural architecture search.
  • Experimental setup: The condensation pipeline first learns synthetic images and then trains classifiers from scratch on them, with 1, 10, or 50 images per class.The two stages may use different model architectures, and coreset baselines select samples during the first stage.
  • Benchmark results: 98.8% accuracy with 50 images per class on MNIST approaches the 99.6% whole-dataset upper bound, while gaps are larger on SVHN and CIFAR10.The whole-dataset result uses 6,000 training images per class, two orders of magnitude more than the condensed setting.
  • Benchmark results: The method significantly outperforms Dataset Distillation, including 5% higher accuracy with 1 synthetic sample per class than DD with 10 samples per class.On CIFAR10, it also trains 2 times faster and requires 50% less memory; MNIST standard deviation is 0.6% versus DD’s 8.1%.
  • Benchmark results: Increasing condensed images improves accuracy across all four benchmarks and narrows the upper-bound gap especially on MNIST and FashionMNIST.The method also outperforms Herding by a large margin in all tested image-count settings.
  • Applications: In neural architecture search, the method reaches 84.5% testing performance and 0.79 rank correlation while reducing search time from 8604.3 to 18.8 minutes.Storage decreases from 5 × 10^4 to 1 × 10^2 images, and synthetic-image training takes around 50 minutes for K = 500.

4 CONCLUSION

The paper introduces Dataset Condensation to synthesize informative images that are more data-efficient than original or prior-method samples. These architecture-independent images reduce dataset memory and support efficient training in continual learning and neural architecture search.

  • Dataset Condensation learns a small set of informative synthetic images for training neural networks.
  • The synthetic images are significantly more data-efficient than the same number of original images and samples produced by the previous method.
  • The condensed images are not architecture dependent and can train different deep networks.
  • The method can lower dataset memory usage and efficiently train numerous networks in continual learning and neural architecture search.

A IMPLEMENTATION DETAILS

The experiments use a standardized dataset-condensation setup with fixed optimization choices, class-wise gradient matching, and task-specific protocols for continual learning and neural architecture search. In NAS, the condensed proxy dataset produces substantially stronger performance correlation with whole-dataset training than early stopping.

  • Dataset condensation: Dataset condensation tunes six hyperparameters covering outer-loop and inner-loop steps, synthetic-sample optimization, and model-weight optimization.The default settings are K = 1000, ηS = 0.1, ηθ = 0.01, ςS = 1, with SGD; MLP synthesis uses ηS = 0.01.
  • Dataset condensation: Class-wise real and synthetic mini-batch pairs compute matching losses separately and update each class’s synthetic images by back-propagation.The class-wise updates can be performed in parallel when sufficient computational resources are available.
  • Dataset condensation: Experiments use standard train/test splits, with data augmentation restricted to specified MNIST and CIFAR10 comparisons.The dataset train/test statistics are reported in Table T5.
  • Dataset condensation: Batch Normalization is used while learning condensed images, then replaced by Instance Normalization when training VGG and ResNet networks on condensed sets.Running statistics are estimated from many real training examples and frozen before synthetic-image updates.
  • Continual learning: Continual learning evaluates task-incremental learning across SVHN, MNIST, and USPS after reshaping images to 32×32 RGB and testing on growing balanced sets.The evaluation uses 2,000, 4,000, and 6,000 test images across the three stages.
  • Neural Architecture Search: NAS searches 720 ConvNets using varied widths, depths, normalization, activation, and pooling choices, ranking candidates trained on a CIFAR10 proxy dataset.Five thousand CIFAR10 training images are used as the validation set; the condensed proxy achieves correlation 0.79 versus 0.42 for early stopping.

B FURTHER ANALYSIS

Further analyses examine architectural choices, optimization sensitivity, gradient-distance metrics, and the qualitative structure of condensed images. The results generally favor the proposed gradient metric and show that synthetic sets can preserve diverse class variation.

  • Ablation study on activation functions: Activation functions are evaluated separately during condensed-image learning and classifier training in MNIST with one image per class.The study compares Sigmoid, ReLU, and LeakyReLu across the two stages.
  • Ablation study on pooling functions: Pooling ablations compare no pooling, average pooling, and max pooling for one-image-per-class MNIST condensation.The analysis states that both pooling options outperform no pooling when used in the second stage.
  • Ablation study on normalization functions: Cross-normalization experiments report accuracy for five normalization choices during condensed-set learning and classifier training.The accompanying analysis states that normalization has little influence on learning the condensed set, while the training-stage choice matters more.
  • Ablation study on network depth and width: Depth and width are varied independently between networks that learn condensed images and networks trained to classify with them.The experiments report cross-depth and cross-width accuracy in Tables T9 and T10.
  • Ablation study on hyper-parameters: Performance is not sensitive to hyper-parameter selection, with similar optimal K and T values across datasets and defaults of K = 1000 and T = 10.The same general hyper-parameter settings are used across datasets.
  • Ablation study on gradient distance metric: The proposed gradient distance metric is reported as more effective and robust than Euclidean and Cosine alternatives, especially on complex architectures.Table T11 compares the metrics across architectures including LeNet, AlexNet, VGG, and ResNet.
  • Further qualitative analysis: With ten images per class, condensed images cover main class variations, whereas one-image sets resemble class prototypes.This qualitative pattern is illustrated for MNIST, FashionMNIST, SVHN, and CIFAR10.

C COMPARISON TO MORE BASELINES

The method is compared with optimal random coresets, cGANs, conventional coreset methods, Label Distillation, and DD. Across these comparisons, the condensed sets are reported to outperform selected baselines and retain broader generalization evidence.

  • Optimal random selection: The condensed set surpasses the selected Top 10 of 1000 optimal random coresets by a large margin on all four datasets.The comparison also includes Top 1000, Top 100, and Top 10 coresets ranked by performance.
  • Generative model: The baseline comparison includes conditional GANs alongside optimal random selection using ConvNet training and testing across four datasets.The cGAN generates class-conditioned images from random noise and labels.
  • Analysis of coreset performances: K-Center and Forgetting perform worse than Random and Herding because they target other settings and tend to select hard, often outlier samples.The analysis links these selected outliers to confused training when only a small number of images is retained.
  • Generalization comparison: The generalization comparison trains ten condensed images per class with LeNet and tests them across multiple architectures against DD.Table T14 reports that the proposed method has better generalization ability.
  • Performance on CIFAR100: On the more challenging CIFAR100 benchmark, the method achieves testing accuracies of 12.8% and 25.2% in the reported comparisons.CIFAR100 has ten times as many classes as the other benchmarks and uses the same hyper-parameters.

D FURTHER COMPARISON TO DD (WANG ET AL., 2018)

Compared with Dataset Distillation, the proposed method generalizes better across network architectures, produces more interpretable synthetic images, and substantially reduces training resources.

  • Generalization ability comparison: The condensed set produced by our method achieves good classification performance across MLP, ConvNet, LeNet, AlexNet, VGG11, and ResNet18 architectures.The comparison uses 10 synthesized images per class learned with LeNet on MNIST.
  • Qualitative comparison: Our method produces more interpretable and realistic synthetic images than DD on MNIST and CIFAR10.DD’s MNIST images are noisy, while its CIFAR10 images lack clear class structure; ours are visually meaningful and diverse.
  • Training memory and time: Our method reduces memory use by approximately 17% for MNIST and 55% for CIFAR10 compared with DD.The comparison uses one NVIDIA GTX1080-Ti GPU in the 10 images/class setting.
  • Training memory and time: Our method reduces training time by approximately 71% for MNIST and 51% for CIFAR10 compared with DD.The reductions result from decoupling model weights from previous states instead of maintaining DD’s recursive computation graph.
  • Training memory and time: Smaller K, T, and real-image batch-size hyper-parameters can further reduce training time and memory with a slight performance decline.

E EXTENDED RELATED WORK

The paper situates Dataset Condensation among extensions of Dataset Distillation, zero-shot knowledge distillation, and privacy-oriented synthetic-data methods.

  • Variations of Dataset Distillation: Several methods extend Dataset Distillation through soft labels or generators, but their reported improvements over DD are minor compared with those of the proposed method.
  • Zero-shot Knowledge Distillation: Unlike zero-shot knowledge distillation, the proposed method does not require a pretrained teacher model to provide features and labels.
  • Data Privacy & Federated Learning: Synthetic datasets have been explored for protecting medical-data privacy and reducing communication rounds in federated learning.
Loading 2006.05929v3…