PaperlayerView exact revision on arXiv

Source-linked AI summary

Dataset Distillation: A Comprehensive Review

Ruonan Yu, Songhua Liu, Xinchao Wang

arXiv:2301.07014v3cs.LG

TL;DR

Large-scale training data impose storage, transmission, computation, privacy, and copyright burdens, motivating dataset distillation. This review formalizes DD, organizes methods into a common taxonomy and framework, and examines applications and challenges. It reports that current methods approach original-data performance in some settings while retaining substantial computational and cross-architecture generalization challenges.

  • Problem

    Large-scale datasets burden storage, transmission, and training, while raw-data use raises privacy and copyright concerns.

  • Method

    The paper comprehensively reviews DD, classifies methods into performance, parameter, and distribution matching, and abstracts a common algorithmic framework.

  • Results

    Current methods have achieved reasonable performance, with FRePo often attaining state-of-the-art results and MTT often achieving second-best performance.

  • Takeaways & Limitations

    Dataset distillation has been applied across research fields, but practical use remains bounded by computational cost and limited transferability across heterogeneous architectures.

  • Takeaways & Limitations

    Synthetic datasets generated by existing methods have not reached homogeneous performance on heterogeneous models, and normalization-layer choice significantly affects transferability.

Abstract

from arXiv · show

Recent success of deep learning is largely attributed to the sheer amount of data used for training deep neural networks.Despite the unprecedented success, the massive data, unfortunately, significantly increases the burden on storage and transmission and further gives rise to a cumbersome model training process. Besides, relying on the raw data for training \emph{per se} yields concerns about privacy and copyright. To alleviate these shortcomings, dataset distillation~(DD), also known as dataset condensation (DC), was introduced and has recently attracted much research attention in the community. Given an original dataset, DD aims to derive a much smaller dataset containing synthetic samples, based on which the trained models yield performance comparable with those trained on the original dataset. In this paper, we give a comprehensive review and summary of recent advances in DD and its application. We first introduce the task formally and propose an overall algorithmic framework followed by all existing DD methods. Next, we provide a systematic taxonomy of current methodologies in this area, and discuss their theoretical interconnections. We also present current challenges in DD through extensive experiments and envision possible directions for future works.

1 INTRODUCTION

Dataset distillation addresses the storage, transmission, computation, privacy, and copyright burdens of large-scale training data by generating small informative synthetic datasets. This review organizes the field, presents a common framework, and discusses applications, challenges, and future directions.

  • Motivation: Large-scale datasets create substantial burdens for data collection, storage, transmission, preprocessing, and model-training computation.Training may require thousands of GPU hours for satisfactory performance.
  • Motivation: Core-set methods reduce dataset size by selecting representative original samples, whereas dataset distillation generates synthetic samples.The two approaches share size-reduction goals but differ in whether retained data are raw or synthesized.
  • Dataset distillation: Dataset distillation aims to produce a small informative dataset whose trained models achieve test performance similar to models trained on the original dataset.
  • Review scope: The review comprehensively summarizes dataset-distillation research and its applications.
  • Review scope: It classifies methods into performance matching, parameter matching, and distribution matching according to their optimization objectives.The review also discusses relationships among these categories.
  • Review scope: The paper abstracts key components into an overall algorithmic framework, evaluates challenges, and proposes future research directions.The article also covers applications and experimental evaluation of existing methods.

2 RELATED WORKS

Dataset distillation is related to knowledge distillation, core-set selection, generative modeling, and hyperparameter optimization, but differs from each in its inputs, outputs, or optimization target. These connections clarify DD’s conceptual foundations while preserving its distinct focus on synthetic datasets.

  • Knowledge Distillation: Knowledge distillation transfers knowledge from a large teacher network to a smaller student network, whereas dataset distillation targets a lightweight dataset.
  • Knowledge Distillation: Dataset distillation and knowledge distillation are conceptually related but technically orthogonal.Data-free knowledge distillation takes a teacher model as input, while DD takes an original dataset as input.
  • Core-set Selection: Core-set selection retains valuable or representative raw samples, while dataset distillation synthesizes new data.Core-set optimization commonly relies on heuristic strategies because finding the best subset is NP-Hard.
  • Generative Models: Generative models relate to DD through distribution matching and through parameterizing synthetic samples with latent codes.Latent codes can represent knowledge more compactly than raw samples and improve storage efficiency.
  • Hyperparameter Optimization: Dataset distillation can be formulated as hyperparameter optimization when synthetic samples are treated as high-dimensional hyperparameters.Both use gradient-based optimization, but their focus and final goals differ.

3 DATASET DISTILLATION METHODS

The canonical dataset-distillation problem learns a small synthetic dataset from a large real dataset so that models trained on both achieve comparable performance. The review formalizes this objective and presents a general framework spanning existing methods.

  • Problem Definition: Dataset distillation extracts knowledge from a large dataset into a much smaller synthetic dataset with comparable model performance.
  • Problem Definition: The canonical problem learns synthetic data from real image-label pairs so models trained on the synthetic set perform comparably to those trained on the original.
  • Problem Definition: The dataset-distillation objective L determines how the synthetic dataset is optimized relative to the original dataset.
  • General Framework: The general framework takes an original dataset T as input and returns a synthetic dataset S.
  • General Framework: Synthetic dataset S is initialized using random values, real samples, or core-set samples before iterative optimization.
  • General Framework: The framework updates S through an objective L(S, T), using performance matching, parameter matching, distribution matching, or variants.

3.2 General Workflow of Dataset Distillation

Dataset-distillation methods alternate between training neural networks and computing an objective through those networks while optimizing the synthetic dataset. Initialization and network-refresh choices affect convergence, performance, and generalizability.

  • Workflow: Current DD methods alternately train neural networks and compute the dataset-distillation objective through those networks.These two steps support optimization of the synthetic dataset against the real dataset.
  • Initialization: Synthetic dataset initialization can strongly affect condensation convergence and final performance.Common choices include Gaussian noise, randomly selected real samples, and core-set samples.
  • Optimization Loop: During iterative optimization, neural networks and synthetic data are updated alternately.Networks may be randomly initialized or loaded from cached checkpoints.
  • Optimization Loop: Networks are periodically refreshed during optimization to reduce overfitting and improve the generalizability of the resulting synthetic dataset.

3.3 Optimization Objectives in DD

DD methods use three mainstream optimization objectives: performance matching, parameter matching, and distribution matching. The review also explains their computational trade-offs and theoretical relationships, including equivalences between several objectives.

  • Performance Matching: Performance matching uses bilevel optimization: models update on synthetic data, then validation loss on real data is backpropagated through the inner training loop.The framework optimizes synthetic data so models trained on it achieve low loss on the original dataset.
  • Performance Matching: Kernel ridge regression replaces expensive multi-step inner training with convex optimization and a closed-form linear-model solution.NTK-based methods approximate wide-network training, while RFAD uses an empirical NNGP kernel and cross-entropy label fitting for classification.
  • Parameter Matching: Parameter matching encourages networks trained on synthetic and real data to have consistent parameters, using either single-step gradients or multi-step training trajectories.Single-step matching is more memory-efficient than meta-learning, whereas multi-step matching addresses errors accumulated over repeated synthetic-data updates.
  • Parameter Matching: Gradient matching can overfit because synthetic-data gradients rapidly lose magnitude, motivating an added Euclidean-distance term alongside cosine distance.The cited limitation concerns gradient norms reaching zero after only a few descent steps, making angle matching less meaningful.
  • Parameter Matching: Multi-step parameter matching yields better performance than its single-step counterpart, while FTD reduces accumulated trajectory error through flat regularization.FTD makes target networks more robust to weight perturbations by producing flatter training trajectories.
  • Distribution Matching and Connections: For kernel ridge regression, performance matching is equivalent to optimal parameter matching, also called infinity-step parameter matching.This equivalence follows because the relevant matrix is constant with respect to synthetic-data optimization.
  • Distribution Matching and Connections: Distribution matching directly minimizes distances between synthetic and real feature distributions, and its first- and second-order objectives connect theoretically to gradient matching.First-moment matching is approximately equivalent to class-wise gradient matching, while second-order matching optimizes an upper bound involving means and correlations.

3.4 Synthetic Data Parameterization

Synthetic data parameterization compresses dataset information into codes, memories, decoders, generators, or reduced-resolution representations while retaining differentiable optimization. These designs increase data efficiency under storage constraints by separating shared and sample-specific knowledge or expanding combinations of learned components.

  • Code-based parameterization: Synthetic datasets can be represented by compact codes mapped to raw-data format through a differentiable function, enabling end-to-end updates of codes and mapping parameters.The generation process remains differentiable, so gradients can optimize both the synthetic representation and its decoder.
  • Augmentation: Differentiable siamese augmentation applies identical differentiable transformations to synthetic and real samples within each iteration to improve data efficiency and generalizability.The policies include crop, cutout, flip, scale, rotate, and color jitter.
  • Resolution and storage: IDC increases the number of synthetic images within a fixed storage budget by reducing resolution and using multi-formation, motivated by dataset size dominating resolution.The parameterization uses downsampled image formats to trade resolution for more stored samples.
  • Generators and latent vectors: Generator-based parameterizations use shared generators to capture common patterns and latent vectors to encode sample-specific information.Latent vectors generally provide a lower-dimensional representation than raw data.
  • Addressable memories: Addressing matrices learn a shared memory representation accessed through learnable matrices, so synthetic-dataset size need not scale linearly with the number of classes.The associated optimization uses performance matching with momentum in the inner loops.
  • Decoders: Interchanging |Z| latent codes with |Φ| decoders can produce up to |Z| × |Φ| synthetic samples, increasing data efficiency exponentially.The construction combines latent codes and decoders arbitrarily.

3.5 Label Distillation Methods.

Label distillation methods relax the common one-hot-label assumption by learning labels directly or using teacher-predicted soft labels. Reported studies find that learnable labels can improve performance, and labels alone can sometimes yield satisfactory results.

  • Learnable labels: Learnable labels can improve dataset-distillation performance compared with fixing labels as one-hot vectors.Several studies learn synthetic labels alongside synthetic images.
  • Label-only distillation: Learning labels without learning images can achieve satisfactory performance in some dataset-distillation settings.This result indicates that label optimization can itself preserve useful training information.
  • Soft labels: Teacher models trained on real datasets can provide soft labels for synthetic samples, and these labels are reported to be significant.The approach replaces fixed one-hot labels with teacher-predicted supervision.

4 APPLICATIONS

Dataset distillation applications use compact synthetic data to reduce memory, communication, or evaluation costs across continual learning, federated learning, neural architecture search, privacy, robustness, backdoor analysis, graph learning, and knowledge condensation. These applications adapt distillation to the constraints of each domain, including rehearsal buffers, client transmission, proxy evaluation, and graph structure.

  • Continual learning: Continual-learning methods use synthetic datasets and parameterizations to preserve prior-task knowledge within limited memory.Approaches include partitioning and upsampling, shared addressing memories, weighted combinations of historical images, shared components, and generative models.
  • Federated learning: Federated-learning methods transmit locally generated synthetic datasets instead of large model updates to reduce upload communication costs.Other work also reduces communication epochs by generating synthetic data from global model trajectories on the server.
  • Neural architecture search: Synthetic datasets can serve as small proxy sets for accelerating neural architecture search, while generative teaching networks aim to generalize across learner architectures and initializations.The proxy-set use is reported as feasible in several works.
  • Privacy: Dataset distillation has been applied to privacy protection by transmitting synthetic data, analyzing connections with differential privacy, and testing irreversibility using L2 and LPIPS similarity metrics.Some methods also use distilled labels for remote training.
  • Robustness: Distillation-based robust optimization trains optimized data rather than model parameters and is reported to provide transferability and satisfactory robustness against PGD attacks.The approach combines adversarial training with KIP.
  • Backdoor attacks: DOORPING attacks during dataset distillation by repeatedly optimizing triggers before synthetic-data updates, targeting defenses at model, input, and dataset levels.This contrasts with traditional backdoor injection after the original data is prepared.
  • Graph learning: Graph distillation compresses graph structure and node attributes, and feature-only optimization can avoid quadratic computational growth with the number of synthetic graph nodes.The method exploits connections between node features and graph structure.
  • Knowledge distillation: Knowledge condensation alternates sample condensation with model distillation, selecting sample value iteratively from student-model feedback.The motivation is to account for changing student capacity rather than transferring knowledge from the whole sample space throughout training.

5 EXPERIMENTS

The experiments compare representative dataset distillation methods across distillation performance, cross-architecture transferability, runtime, and GPU memory. FRePo generally performs best, while methods exhibit distinct accuracy, generalization, and efficiency trade-offs.

  • Distillation Performance: FRePo achieves state-of-the-art performance in most cases, especially on CIFAR-10, CIFAR-100, and Tiny-ImageNet, while MTT is often second-best.DD reaches state-of-the-art performance on MNIST and Fashion-MNIST at selected image-per-class settings; DM often performs worse than other methods.
  • Cross Architecture Generalization: Cross-architecture transfer degrades for most methods without normalization, whereas FRePo is comparatively robust.Instance normalization appears important in several methods but may harm transferability when normalization differs between training and evaluation architectures.
  • Run-Time Evaluation: DSA’s augmentation slightly increases runtime but can significantly improve distillation performance, particularly at low image-per-class settings.At small image-per-class values, both JAX and PyTorch FRePo versions are significantly faster than other methods; the advantage changes as image-per-class increases.
  • Peak GPU Memory Usage: DM requires the least GPU memory, whereas MTT requires the most and encounters out-of-memory at 50 images per class.As image-per-class increases, JAX FRePo shows increasing memory-efficiency advantages.
  • Empirical Studies: DD’s improved performance comes with relatively poor generalizability and considerable runtime and GPU-memory requirements.The study attributes the performance improvement to adding momentum.
  • Empirical Studies: DSA improves distillation performance and generalizability over DC without substantially increasing runtime or memory requirements.
  • Empirical Studies: MTT has the second-best overall performance but requires substantial running time and memory because gradient computation is unrolled through backpropagation.
  • Empirical Studies: FRePo delivers state-of-the-art performance and training cost when image-per-class is small, but loses comparable performance as image-per-class increases.

6 CHALLENGES AND POSSIBLE IMPROVEMENTS

Dataset distillation faces scaling, computational, transferability, task-coverage, and security challenges despite ongoing methodological progress. Experiments indicate that increasing compression ratios, dataset complexity, or network complexity can reduce effectiveness and efficiency.

  • Computational Efficiency: Generating synthetic data is typically expensive because updating it requires backpropagation through unrolled computational graphs, while MTT still needs hundreds of pretrained teacher models.KRR-based methods avoid extensive inner-loop training through convex optimization and closed-form solutions for linear models.
  • Scaling Up: Existing DD methods are difficult to scale because larger IPC, datasets, and networks demand more information, computation, memory, and adaptability.The review identifies larger synthetic datasets, ImageNet-scale data, and complex models as three distinct scaling barriers.
  • Larger Compression Ratios: At higher IPC, accuracy improves more slowly, DD methods lose their advantage over selection methods beyond 200 IPC, and selection can become slightly better.The comparison is reported for CIFAR-10 across Random selection, K-Center, DD, DC, DSA, DM, MTT, and FRePo-JAX.
  • Generalization across Different Architectures: Synthetic datasets transfer imperfectly across heterogeneous architectures, especially when optimization strongly binds them to one network or uses different normalization layers.Unbinding networks and synthetic data can help, but DM has relatively poor performance and IDC adds runtime for real-data network training.
  • Future Applications and Security: DD research remains concentrated on classification, while semantic segmentation, object detection, machine translation, and multimodal applications are identified as future directions.The review also notes that security and privacy have been comparatively neglected, including the reported DOORPING backdoor attack during distillation.

7 CONCLUSION

The paper reviews dataset distillation as a way to synthesize small datasets that support performance similar to training on large original datasets. It organizes methods, analyzes their connections, abstracts a common framework, surveys applications, and uses experiments to identify unresolved challenges.

  • The review synthesizes dataset distillation methods that seek similar model performance from a small synthetic dataset given a large original dataset.
  • It classifies methods into performance matching, parameter matching, and distribution matching, and discusses their theoretical connections.
  • The paper abstracts a common algorithmic framework, surveys applications including continual learning, neural architecture search, and privacy, and experimentally compares accuracy, time efficiency, and scalability.
  • The comparisons indicate critical challenges in dataset distillation that remain for future research.

APPENDIX A TAXONOMY OF EXISTING DD METHODS

The taxonomy analyzes dataset distillation methods through four aspects: optimization objective, network-update strategy, synthetic-data parameterization, and label-learning strategy.

  • A dataset distillation method can be analyzed by its optimization objective, fashion of updating networks, synthetic data parameterization, and fashion of learning labels.

APPENDIX B PERFORMANCE OF EXISTING DD METHODS

The appendix compiles performance results for existing dataset distillation methods across benchmark datasets and IPC settings. It presents taxonomy details and combines official-paper results, DC-Benchmark results, and the authors’ experiments.

  • Performance results are listed for MNIST, Fashion-MNIST, SVHN, CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K at different IPC values.
  • The reported results combine values from official papers, DC-Benchmark, and the authors’ experiments.
  • The appendix provides a tabular taxonomy of existing dataset distillation methods, with “v” denoting a variant of an optimization objective.
Loading 2301.07014v3…