Source-linked AI summary
Dataset Distillation
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, Alexei A. Efros
TL;DR
Large datasets contain knowledge that may be difficult to use efficiently, raising whether they can be compressed into a few synthetic samples without matching the natural-image distribution. The paper optimizes synthetic data through differentiable training dynamics, extending the approach across initialization settings and objectives. It compresses 60,000 MNIST images into 10 images achieving 94% test accuracy versus 99% with full training, while also supporting adaptation and poisoning applications.
Problem
The paper asks whether the knowledge of large training datasets can be compressed into a small synthetic set that need not follow the real-data distribution.
Method
The method optimizes synthetic image pixels through differentiable network training, with extensions for random initializations, iterative multi-epoch training, and alternative objectives.
Results
60,000 MNIST training images compressed into 10 synthetic images train LeNet to 94% test accuracy versus 99% with the original dataset, and experiments cover multiple datasets and settings.
Takeaways & Limitations
A few distilled images can provide dataset-scale knowledge for rapid training, pre-trained-model adaptation, and malicious data-poisoning objectives.
Takeaways & Limitations
The method is sensitive to the distribution of initializations, and distilled data optimized for one initialization may not generalize well to others.
Abstract
from arXiv · showhide
Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few gradient descent steps, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.
1 INTRODUCTION
Dataset distillation compresses large training datasets into a few synthetic images optimized for rapid model training. The method supports fixed and random initializations, multiple training objectives, and extensions to fine-tuning and poisoning.
- Core idea: Dataset distillation keeps the model fixed while encoding an entire training dataset into a small number of synthetic training images.Unlike network distillation, the approach distills dataset knowledge rather than model knowledge.
- Core idea: 60,000 MNIST images become 10 synthetic images that train LeNet to 94% test accuracy, versus 99% with the full dataset.This result uses one synthetic image per class and a fixed network initialization.
- Motivation: Distilled images can load a network with dataset-scale knowledge more efficiently than conventional training requiring tens of thousands of gradient-descent steps.The paper frames this as both a scientific question about dataset compressibility and a practical efficiency benefit.
- Optimization: The optimization differentiates network weights through training on synthetic data and directly updates distilled-image pixels for fast training in a few gradient steps.The initial formulation requires access to the network’s initial weights.
- Optimization: The method is extended to random initializations by optimizing images over an initialization distribution and to multiple epochs through iterative distilled-image sequences.A simple linear analysis also derives a lower bound on the distilled-data size needed to match full-dataset performance.
- Evaluation: Dataset distillation is evaluated across fixed, random, fixed pre-trained, and random pre-trained weights, using classification and malicious-poisoning objectives on multiple datasets.The paper also studies fast fine-tuning of pre-trained models and data-poisoning attacks.
2 RELATED WORK
The paper distinguishes dataset distillation from methods that compress models, select representative examples, optimize hyperparameters, or visualize learned representations. Its distinctive perspective is to understand and compress full datasets into synthetic samples.
- Knowledge distillation: Network distillation compresses knowledge from multiple trained networks into one model, whereas dataset distillation compresses an entire dataset into synthetic training images.Data-free knowledge distillation also optimizes synthetic samples, but matches teacher activation statistics instead.
- Dataset reduction: Dataset pruning, core-set construction, and instance selection summarize data by retaining or labeling valuable real examples rather than synthesizing training images.These approaches reduce the dataset through subset selection or active labeling.
- Optimization: Gradient-based hyperparameter optimization backpropagates through optimization steps, but this work focuses on learning synthetic training data using training-set information.The shared optimization-through-training perspective differs in the object being optimized.
- Understanding datasets: Model-interpretability methods visualize learned models, while dataset distillation investigates intrinsic properties of training data by reducing full datasets to a few synthetic samples.The paper presents dataset distillation as a new perspective for understanding datasets.
3 APPROACH
Dataset distillation learns a much smaller synthetic dataset and learning rate that can update a fixed or sampled-initialization model toward performance on the original data. The approach extends across multiple update steps, initializations, pretrained models, and learning objectives, while analysis identifies limits for arbitrary initializations.
- 3.1 OPTIMIZING DISTILLED DATA: The method learns M ≪ N synthetic data points and a learning rate so one gradient step can improve performance on the real test set.The optimized synthetic data and learning rate are evaluated by updating the model and measuring loss over the original training data.
- 3.1 OPTIMIZING DISTILLED DATA: The optimization differentiates through the model update and fixes discrete components, such as class labels, rather than learning them.The updated parameters are obtained from the distilled data and learning rate, after which the resulting loss is optimized with gradient-based methods.
- 3.2 DISTILLATION FOR RANDOM INITIALIZATIONS: Distilled data optimized for one initialization often fail to generalize to other initializations, motivating optimization over a distribution of initial weights.The paper reports that fixed-initialization distilled data may look like noise because they encode both the dataset and a particular network initialization.
- 3.3 ANALYSIS OF A SIMPLE LINEAR CASE: In a quadratic-loss linear model, distilled data can attain the full-dataset global minimum for any initialization, but the analysis implies M ≥ D.The lower bound follows under the assumption that the original feature columns are independent, making the result restrictive for high-dimensional datasets.
- 3.4 MULTIPLE STEPS AND EPOCHS: Using multiple gradient steps and epochs is more effective than using one step on neural networks when the total distilled-data amount is fixed.Learning rates are not tied across epochs because later epochs often use smaller learning rates.
- 3.5 DISTILLATION WITH DIFFERENT INITIALIZATIONS: The framework also supports fast adaptation from pretrained weights and malicious objectives that make classifiers forget one category while retaining high accuracy on others.For poisoning, the method trains distilled images across a distribution of models and applies the attack in one iteration with few data.
4 EXPERIMENTS
Experiments show that distilled images support effective training and adaptation across initialization settings, datasets, objectives, and training schedules, often outperforming baselines.
- Fixed initialization: 10 distilled images raise MNIST accuracy from 12.90% to 93.76% with fixed initialization, while 100 images raise CIFAR10 accuracy from 8.82% to 54.03%.Both settings use access to the initial network weights.
- Training schedules: More gradient descent steps significantly improve results, while increasing epochs produces a similar but slower improvement.The reported schedule uses 10 gradient descent steps across 3 epochs, with 100 total images.
- Baseline comparison: Our method outperforms all baselines on CIFAR10 and most baselines on MNIST under both fixed and random initialization.Table 1 compares methods using 100 images total, except for the average-real-images baseline, which reuses images across steps.
- Pre-trained models: For digit-domain adaptation, the method outperforms few-shot domain adaptation and other baselines in most settings using 100 distilled images.Experiments adapt models among MNIST, USPS, and SVHN.
- Pre-trained models: Using one distilled image per category, the method significantly outperforms baselines and matches fine-tuning on full datasets for PASCAL-VOC and CUB-200.The experiments adapt AlexNet models pre-trained on ImageNet.
- Malicious objective: With random pre-trained weights and a malicious objective, the method significantly outperforms all baselines on CIFAR10, while performing similarly to some baselines on MNIST.All methods use one gradient descent step and modified labels for the poisoning comparison.
5 DISCUSSION
The paper presents dataset distillation as compressing an entire training dataset into a few synthetic images for rapid training. It also extends the approach to pre-trained-model adaptation and malicious data poisoning, while identifying sensitivity to initialization distributions.
- Dataset distillation compresses the knowledge of entire training data into a few synthetic training images.
- A network can reach high performance using a small number of distilled images and several gradient descent steps.
- The method extends to adapting pre-trained models to new datasets and performing a malicious data-poisoning attack.
- The current method is sensitive to the distribution of initializations, motivating investigation of alternative initialization strategies.
SUPPLEMENTARY MATERIAL
The supplementary material documents optimization and evaluation settings, including dropout removal, learning-rate choices, sampled initial weights, and real-image baselines.
- Dropout is disabled because its randomness and computational cost interfere with distillation.
- Distilled learning rates are initialized between 0.001 and 0.02 by task, while Adam uses a learning rate of 0.001.
- For random initialization and random pre-trained weights, each optimization step samples 4 to 16 initial weights.
- Real-image baselines include randomly sampled images, optimized real-image sets, and k-means cluster centroids.