PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey of Dataset Distillation

Shiye Lei, Dacheng Tao

arXiv:2301.05603v4cs.LG

TL;DR

Dataset distillation addresses the difficulty of training on rapidly growing datasets with limited computing resources. The survey organizes DD methods into meta-learning and data matching frameworks, reviews algorithms and applications, and compares their performance. It reports that DD methods outperform random selection and coreset selection, while identifying challenges including complex label spaces and high-resolution data.

  • Problem

    Rapid data growth creates training-efficiency and storage challenges when computing resources are limited.

  • Method

    The survey presents a hierarchical review of dataset distillation frameworks, algorithms, factorized methods, applications, performance comparisons, limitations, and future directions.

  • Results

    Dataset distillation methods outperform random pick and coreset selection by large margins, and factorized distillation can largely improve test accuracy.

  • Takeaways & Limitations

    The survey provides a systematic framework for understanding DD development and identifying directions for future studies.

  • Takeaways & Limitations

    DD remains constrained by complex label spaces, which make high-dimensional synthetic labels difficult to adjust efficiently.

Abstract

from arXiv · show

Deep learning technology has developed unprecedentedly in the last decade and has become the primary choice in many application domains. This progress is mainly attributed to a systematic collaboration in which rapidly growing computing resources encourage advanced algorithms to deal with massive data. However, it has gradually become challenging to handle the unlimited growth of data with limited computing power. To this end, diverse approaches are proposed to improve data processing efficiency. Dataset distillation, a dataset reduction method, addresses this problem by synthesizing a small typical dataset from substantial data and has attracted much attention from the deep learning community. Existing dataset distillation methods can be taxonomized into meta-learning and data matching frameworks according to whether they explicitly mimic the performance of target data. Although dataset distillation has shown surprising performance in compressing datasets, there are still several limitations such as distilling high-resolution data or data with complex label spaces. This paper provides a holistic understanding of dataset distillation from multiple aspects, including distillation frameworks and algorithms, factorized dataset distillation, performance comparison, and applications. Finally, we discuss challenges and promising directions to further promote future studies on dataset distillation.

1 INTRODUCTION

Dataset distillation addresses the mismatch between rapidly growing data and limited computing resources by synthesizing small datasets that can support comparable or better model performance. This survey organizes methods into meta-learning and data matching frameworks and reviews algorithms, applications, comparisons, limitations, and future directions.

  • Motivation: Dataset distillation synthesizes a small dataset from large data to address training-efficiency and storage challenges caused by data growth.Unlike coreset selection, it removes the restriction that retained examples must remain unmodified.
  • Motivation: Models trained on distilled datasets can achieve better generalization performance than models trained on uneditable coresets.
  • Taxonomy: DD methods are grouped into meta-learning and data matching frameworks according to whether they explicitly mimic target-data performance.These frameworks support finer-grained classifications of DD techniques.
  • Survey scope: The survey covers DD algorithms, factorized distillation, applications, performance comparisons, limitations, and promising future directions.Its hierarchical taxonomy first separates meta-learning and data matching before introducing finer divisions.

2 BACKGROUND

The background defines dataset distillation as extracting a large target dataset’s knowledge into a much smaller synthetic set while training models through empirical-risk minimization. It also introduces gradient-based training, initialization robustness, and the relationship between empirical and expected risk.

  • Risk definitions: Because the data-generating distribution is unknown, empirical risk is used as a practical estimate of expected risk.
  • Training setup: The training algorithm alg returns parameters learned by empirical risk minimization from dataset T and initialization θ(0).
  • Training setup: Gradient descent iteratively updates neural-network parameters by following the gradient of empirical risk.
  • Dataset distillation objective: Dataset distillation extracts knowledge from target dataset T into synthetic dataset S, where n ≪m, while seeking comparable generalization performance.
  • Robustness: Many DD methods average objectives over parameter initializations to improve robustness across different initial networks.

3 META-LEARNING FRAMEWORK

The meta-learning framework treats the distilled dataset as a hyperparameter optimized through an inner training loop and an outer target-data risk objective. Its main variants include BPTT and KRR, which trade iterative neural-network optimization for closed-form kernel solutions and differ in efficiency and transferability.

  • Meta-learning formulation: Dataset distillation updates synthetic data in an outer loop while an inner loop trains model parameters on that synthetic dataset.The outer objective minimizes risk on the target dataset, causing the synthetic set to converge toward an optimum.
  • Framework variants: The meta-learning framework is classified into BPTT and KRR according to the model and optimization method used in the inner loop.BPTT uses neural-network gradient descent, whereas KRR uses regression with a kernel model.
  • Backpropagation Through Time Approach: BPTT recursively backpropagates through intermediate parameter states, making it computationally expensive and memory demanding.Its inefficiency arises from unrolling the recursive computation graph over the parameter trajectory.
  • Kernel Ridge Regression Approach: KRR bypasses recursive meta-gradient computation by solving the inner regression problem in closed form and backpropagating through the kernel function.The distilled dataset is updated through the kernel-based meta-gradient rather than an iterative parameter trajectory.
  • Discussion: KRR improves efficiency but can lose performance when transferred to neural networks because it uses simple kernels, while BPTT can drop on more complicated architectures.The survey notes that kernel approaches remain feasible for wide neural networks because infinite-width neural-network training matches kernel dynamics.

4 DATA MATCHING FRAMEWORK

The data matching framework distills datasets by aligning informative byproducts of target and synthetic data, including gradients, parameters, and features. Its variants trade memory and computational efficiency against distribution coverage, scalability, and preservation of discriminative information.

  • Framework overview: Data matching aligns target and synthetic datasets in informative spaces such as gradients, parameters, or features rather than directly extracting target information.Randomly sampled batches can replace full datasets to reduce memory use and accelerate convergence.
  • Gradient matching approach: Gradient matching uses induced gradients as a proxy for matching training trajectories and model-parameter influence.Layer-wise and class-wise variants measure gradient differences to improve practical convergence.
  • Gradient matching approach: Class-wise gradient matching can overlook class-discriminative features, especially in fine-grained datasets dominated by class-common features.Improved objectives add between-class gradient signals to capture contrastive information.
  • Discussion: Compared with gradient matching, distribution matching can more comprehensively cover the real CIFAR-10 feature distribution.The comparison is shown through a T-SNE visualization of the first CIFAR-10 class.
  • Trajectory matching approach: Trajectory matching directly aligns long-range parameter trajectories by training models on target and synthetic datasets from buffered expert initializations.It avoids second-order gradient computation but requires unrolling N SGD updates, increasing memory burden and impeding scalability.
  • Distribution matching approach: Distribution matching aligns synthetic and target data in feature space using a pretrained feature extractor and avoids model-parameter dependence and bilevel optimization.It requires less memory but empirically underperforms gradient and trajectory matching, while potentially wasting budget on less informative features.

5 FACTORIZED DATASET DISTILLATION

Factorized dataset distillation learns compact latent codes and decoders instead of directly optimizing synthetic images. Code-based, decoder-based, and code-decoder methods generate informative synthetic data while supporting shared representations and large-scale settings.

  • Overview: Factorized dataset distillation optimizes latent codes and corresponding decoders to implicitly generate synthetic datasets from low-dimensional representations.The approach is motivated by the possibility that high-dimensional images lie on low-dimensional manifolds recoverable through decoders.
  • Code-based DD: Code-based distillation learns low-dimensional codes or latent vectors that a generator uses to produce informative synthetic images.Some methods initialize codes by inverting real examples into a pretrained GAN and then optimize them with distribution matching.
  • Decoder-based DD: Decoder-based distillation learns a generator or decoder that produces synthetic images from random noise or other inputs, updating the generator through meta-gradients.The generative teaching network generates images from random noise conditioned on labels.
  • Code-decoder DD: Code-decoder distillation jointly trains codes and decoders, including methods that combine learned bases with addressing functions to generate class-specific synthetic images.A hallucinator can encode a basis, apply an affine transformation, and decode the result into a synthetic image.
  • Advantages and scalability: The number of learnable parameters can be independent of real image resolution and class count, supporting large-scale distillation.This property is identified as promising for large-scale distillation problems.
  • Advantages and scalability: Factorized distillation offers more compact representations and shared representations across classes, which consequently improve distillation performance.The code-decoder factorization is compatible with the survey’s other distillation approaches.

6 PERFORMANCE COMPARISON

The survey compares dataset distillation methods across multiple image datasets and evaluates both standard performance and transfer to unseen architectures. It reports strong gains over random selection and coresets, while factorized methods improve cross-architecture transferability.

  • Standard Benchmark: The comparison covers MNIST, FashionMNIST, SVHN, CIFAR-10/100, Tiny ImageNet, and ImageNet-1K, with blanks retained where methods lack meaningful or scalable evaluations.The survey also compares random selection, Herding-based coresets, and the original target dataset.
  • Standard Benchmark: Dataset distillation methods outperform random selection and coreset selection by large margins.The coreset baseline uses Herding, and distilled data are evaluated with the same three-block ConvNet for fairness.
  • Standard Benchmark: KRR and trajectory matching achieve advanced performance on CIFAR-10/100 and Tiny ImageNet among nonfactorized methods.These datasets provide the larger-scale comparison highlighted by the survey.
  • Standard Benchmark: Factorized distillation using latent codes and decoders largely improves distilled-data test accuracy, with KFS showing the best overall performance across CIFAR-10/100 and Tiny ImageNet.Factorized methods are marked with an asterisk in the comparison tables.
  • Cross-Architecture Transferability: Nonfactorized methods show a significant performance drop on unseen architectures, and distilled data do not consistently perform best across architectures.The survey evaluates ConvNet-distilled data on ResNet-10/18/152, DenseNet-121, and ViT.
  • Cross-Architecture Transferability: IDC and KFS have better cross-architecture transferability because they suffer smaller accuracy drops on unseen architectures.The survey emphasizes multiple-architecture evaluation because no standard architecture set exists for this comparison.

7 DATA MODALITIES

Dataset distillation extends beyond images to graphs and text, requiring representations or optimization strategies suited to discrete structures and data. Graph distillation faces scalability and applicability constraints tied to adjacency matrices.

  • Data Modalities: Dataset distillation has been applied to multimodal data, including graphs and text, beyond the image data that dominate prior work.The survey treats graph and text distillation as extensions to non-image modalities.
  • Graph Data: Graph distillation compresses a graph T = {A, X, Y} into a smaller synthetic graph S = {A′, X′, Y′} with N′ ≪ N nodes for node classification.The graph contains adjacency, node-feature, and node-label components, and gradient matching is used in the described approach.
  • Graph Data: Because A′ has quadratically growing parameters, an MLP-based model predicts graph structure from node features to improve scalability.The model uses the implicit correlation between graph structure and node features.
  • Graph Data: For discrete graph classification, adjacency entries are modeled probabilistically with Bernoulli variables and made differentiable through reparameterization.The learnable parameter Ω is transformed through a sigmoid function before synthetic adjacency optimization.
  • Text Data: Text distillation converts discrete text into continuous embedding matrices, while soft labels can be optimized using attention-probability matching for BERT fine-tuning.The described objective decreases KL divergence between distilled-data and model attention probabilities.

8 APPLICATION

Dataset distillation is applied where training efficiency, storage, privacy, or data sharing are constrained. The survey covers continual learning, neural architecture search, explainability, visual design, and medical-image sharing.

  • Continual Learning: Dataset distillation supports continual learning by compressing data for limited replay memory under changing data distributions.Frequent memory updates impose strict efficiency requirements on distillation algorithms.
  • Continual Learning: Distillation can learn stable dataset sequences whose trained networks avoid catastrophic forgetting.This complements replay-based approaches to continual learning.
  • Neural Architecture Search: In neural architecture search, distilled data can serve as a proxy dataset for training candidate architectures and estimating their generalization ranking.NAS otherwise trains thousands of candidates on a small proxy of the original dataset to save time.
  • Federated Learning: The survey connects dataset distillation with federated learning, where clients train on exclusive data and transmit gradients or parameters to a central server.The connection is motivated by distributed training and private data protection.
  • Other Applications: Small synthetic datasets support explainability by making it easier to measure how examples influence test predictions.Shared synthetic images can act as a bridge connecting training and testing examples.
  • Other Applications: Dataset distillation has been used for visual design by extracting representative textures and modeling outfit compatibility.Representative textures are generated by randomly cropping synthetic images during distillation.
  • Medical Applications: Medical-image studies report advantages for image transfer and patient-information anonymization, enabling lower-cost, lower-risk data sharing for collaborative diagnosis systems.The cited empirical studies concern gastric X-ray images.

9 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies scalability, complex labels, multimodal data, theory, privacy, robustness, and cross-architecture transfer as important challenges or directions for dataset distillation. It also highlights optimization objectives and model augmentation as routes toward improved performance and broader applicability.

  • Scalability: 2.5 billion hyperparameters are required for 1000-class ImageNet at 224 × 224 resolution and 50 images per class, illustrating dataset distillation’s scalability problem.The burden comes from simultaneously optimizing pixels and other numerous hyperparameters.
  • Multimodal dataset distillation: Video distillation remains challenging because spatial and temporal information are entangled, despite progress on image, graph, text, tabular, and recommendation data.The survey describes multimodal applications beyond these modalities as underexplored.
  • DD with complex labels: High-dimensional label spaces in detection and segmentation make enumerated assignment impractical and complicate joint optimization of synthetic labels and inputs.This limitation is especially pronounced beyond single-label classification, where labels are integer-valued.
  • Theory of dataset distillation: Few studies explain dataset distillation theoretically, although theory could clarify dataset knowledge and relate synthetic-dataset size to test-error bounds.The survey argues that such theory can guide development rather than encourage blind performance improvement.
  • Optimization objective of DD: Factorized codes and decoders outperform direct optimization, motivating research on better information carriers and generative-model representations.The survey also points to optimizing 3D data to summarize information from 2D images as a possible direction.
  • Cross-architecture transferability: Cross-architecture performance drops because distilled data depend on the network architecture used during distillation, while more realistic synthetic data can improve transfer.Better transfer would allow synthetic datasets to train models too expensive to include in the distillation process.
  • Trustworthy dataset distillation: Future work should address privacy, robustness, calibration, and out-of-distribution detection so distilled datasets support more trustworthy deployment.The survey notes privacy-evaluation disputes, adversarially robust distilled datasets, over-confident predictions, and pseudo-outliers for OoD detection.
Loading 2301.05603v4…