PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm

Peng Sun, Bei Shi, Daiwei Yu, Tao Lin

arXiv:2312.03526v2cs.CVcs.AIcs.LG

TL;DR

Dataset distillation methods face computational burden and architecture overfitting, while large-scale applications call for realism, diversity, and efficiency. RDED selects and stitches realistic patches without optimization; on full ImageNet-1K at IPC = 10, it reaches 42% top-1 validation accuracy with ResNet-18 and surpasses the SOTA method by 2× in performance.

  • Problem

    Existing methods incur high computational burden from bi-level optimization, and synthetic images can overfit to the optimization architecture, hindering generalization.

    Source: §1 · p. 1

  • Method

    RDED selects realistic patches from original data and stitches them into synthetic images to support diversity and realism without optimization.

    Source: §1 · p. 2

  • Results

    42% top-1 validation accuracy with ResNet-18 on full ImageNet-1K at IPC = 10, with performance surpassing the SOTA method by 2×.

    Source: §6 · p. 8

  • Takeaways & Limitations

    RDED is presented as well suited to large-scale, high-resolution datasets while pursuing realism and diversity.

    Source: §1 · p. 2

    Prior approaches retain tradeoffs: bi-level optimization is computationally burdensome, while CoreSet methods limit information representation.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Contemporary machine learning requires training large neural networks on massive datasets and thus faces the challenges of high computational demands. Dataset distillation, as a recent emerging strategy, aims to compress real-world datasets for efficient training. However, this line of research currently struggle with large-scale and high-resolution datasets, hindering its practicality and feasibility. To this end, we re-examine the existing dataset distillation methods and identify three properties required for large-scale real-world applications, namely, realism, diversity, and efficiency. As a remedy, we propose RDED, a novel computationally-efficient yet effective data distillation paradigm, to enable both diversity and realism of the distilled data. Extensive empirical results over various neural architectures and datasets demonstrate the advancement of RDED: we can distill the full ImageNet-1K to a small dataset comprising 10 images per class within 7 minutes, achieving a notable 42% top-1 accuracy with ResNet-18 on a single RTX-4090 GPU (while the SOTA only achieves 21% but requires 6 hours).

1. Introduction

Dataset distillation compresses datasets for efficient model training, but existing methods face computational burdens and realism-related generalization challenges. RDED uses realistic patches to build diverse synthetic images without optimization, and experiments report improved accuracy and speed over prior work.

1. Introduction

  • Existing bi-level optimization methods are computationally costly, and architecture-specific non-realistic images can hinder generalization to other architectures.Prior work reports a trade-off: realism aids cross-architecture generalization but can hurt distillation performance, while expressiveness may impede transfer.Source: §1 · pp. 1–2
  • RDED combines realistic patches from original data into synthetic images to pursue diversity and realism without optimization.The authors describe the paradigm as efficient for large-scale, high-resolution datasets.Source: §1 · p. 2
  • RDED achieves twice the top-1 validation accuracy of SRe2L while operating 52 times faster, according to the authors.The contribution summary points to Figure 1 for this comparison.Source: §1 · p. 2
  • The paper identifies realism, diversity, and efficiency as properties needed for effective distillation on large-scale, high-resolution datasets.It introduces diversity ratio and realism score, backed by V-information theory [42].Source: §1 · p. 2

2. Related Work

Prior dataset distillation approaches use optimization, regularization, or sample selection to reduce training data. The paper describes trade-offs in computational cost, cross-architecture generalization, and information retained.

2. Related Work

  • Bi-level methods match gradients [18] [49] [51], features [38], distributions [52], or training trajectories [1] [5] [8] [13] [45], but synthetic data can overfit to an architecture.Trajectory-matching methods have shown strong benchmark performance at low IPC.Source: §2 · p. 2
  • GLaD integrates a generative prior to improve generalization across distillation methods, while prior-regularized bi-level methods retain computational and memory challenges [2] [5].Source: §2 · p. 2
  • Uni-level kernel ridge-regression methods can reduce training costs [53] and improve performance [23], but matrix inversion makes scaling to larger IPC challenging.Yin et al. [44] decouple bi-level optimization into two single-level learning procedures.Source: §2 · p. 2
  • CoreSet methods select representative labeled samples using importance measures such as forgetting score [37], memorization [10], EL2N [27], and diverse ensembles [25].Source: §2 · p. 2

3. On the Limits of Dataset Distillation

This section defines dataset distillation and evaluates conventional approaches against the desired properties of diversity, realism, and efficiency. Its review highlights different limitations across optimization-based and sample-selection paradigms.

3.1. Preliminary

  • Dataset distillation seeks a much smaller labeled dataset whose trained models perform within an acceptable deviation from models trained on the original dataset.The distilled set S is generated from the original set T by an algorithm A.Source: §3.1 · p. 2
  • On ImageNet-Fruits, the figure compares four methods across pineapple, banana, pomegranate, and fig; the caption says only RDED and GLaD generally achieve superior realism and diversity together.Source: Figure 2 · p. 3
  • The paper defines diversity as broad sample and label coverage, realism as accurate prediction across observer models, and efficiency as scalable computation and memory use.The definition frames realism as avoiding features or annotations tailored to a specific model.Source: §3.1 · p. 3

3.2. Pitfalls of Conventional Dataset Distillation

  • Conventional bi-level methods can create non-realistic high-resolution patterns, overfit their training architecture, and incur high computational burden [1] [51] [2] [5] [44].Source: §3.2 · pp. 3–4
  • Prior regularization improves realism and cross-architecture generalization [2], but inherits bi-level methods’ low efficiency and remains unsuitable for large-scale datasets.Source: §3.2 · p. 4
  • SRe2L alleviates efficiency and realism challenges for large-scale, high-resolution data, yet its synthesis from partial pretrained-model information limits diversity [43].Source: §3.2 · p. 4
  • CoreSet selection efficiently retains realistic images but can lose information representation and diversity, leading to severely degraded performance [34] [35] [27].Source: §3.2 · p. 4

4. Methodology

RDED frames dataset distillation around diversity, realism, and efficiency, using two stages to select source patches and reconstruct them as distilled images. It uses V-information-based proxies for diversity and realism, then relabels reconstructed regions with an observer.

4.1. Enhancing Data Diversity and Realism

  • The method approximates diversity with a V-information preservation ratio and realism with a score, decoupling the two objectives for computational feasibility.The diversity term favors uncertainty in representations without side information, while the realism term favors predictability given the image.Source: §4.1 · p. 4
  • RDED first extracts realistic key patches from the original dataset, then compresses them into distilled images and relabels them.The figure illustrates patch ranking and selection in Stage 1, followed by N-patch image reconstruction and observer relabeling in Stage 2.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2. Extracting Key Patches from Original Dataset

  • For each original image, RDED crops candidate patches and keeps the patch with the highest observer-based realism score.The score compares the observer’s prediction for a patch with its human-annotated label.Source: §4.2 · p. 5
  • A second, class-level selection prunes key patches by score, while uniform random preselection reduces scanning cost and score-selection bias.The class-level stage addresses redundancy that pixel-space extraction alone does not remove.Source: §4.2 · p. 5

4.3. Information Reconstruction of Patches

  • RDED randomly combines N selected patches without replacement into full-resolution distilled images because smaller patches alone can leave pixel space sparse.The selected patches are reconstructed into images matching the original image resolution.Source: §4.3 · pp. 5–6
  • Because crops may depict objects unlike their original labels, RDED relabels regions of reconstructed images with observer-generated soft labels.This is intended to provide informative labels for the multi-patch distilled images.Source: §4.3 · p. 6

5. Experiment

The experiments compare RDED with state-of-the-art methods across datasets and neural architectures, then examine efficiency, cross-architecture generalization, and design choices.

5. Experiment

  • The experiments assess RDED against state-of-the-art methods across diverse datasets and neural architectures and include ablation studies.Source: §5 · p. 6

5.1. Experimental Setting

  • Evaluation spans CIFAR-10/100, Tiny-ImageNet, full ImageNet-1K and ImageNet-1K subsets at IPC 1, 10, and 50; cross-architecture evaluation trains other architectures from distilled data and records validation accuracy.Source: §5.1 · p. 6

5.2. Main Results

  • At IPC 10 and 50, RDED significantly outperforms or matches methods on high-resolution data, while IPC 1 and smaller CIFAR sets weaken; CIFAR-100 IPC 50 is especially strong.Source: Table 2 · p. 7, §5.2 · p. 7

5.3. Efficiency Comparison

  • RDED shows superior synthesis efficiency; scoring is the main memory-intensive step, runs in parallel across mini-batch images, and distillation time depends on teacher-model size.Source: Table 3 · p. 7, §5.3 · pp. 7–8

5.4. Cross-architecture Generalization

  • RDED has robust cross-architecture generalization versus SRe2L, which the authors associate with realistic images and insensitivity to teacher-model variation.Source: Table 4 · p. 8, §5.4 · p. 8

5.5. Ablation Study

  • When |T′c| equals N × IPC, performance drops abruptly as Stage 1 becomes uniform random sampling; values near 300 balance diversity and realism.Source: Figure 4 · p. 8, §5.5 · p. 8
  • N=4 yields the highest validation performance on three selected datasets; increasing N boosts diversity but lowers source-patch resolution and realism.Source: Figure 4 · p. 8, §5.5 · p. 8

6. Conclusion

RDED distills all of ImageNet-1K to IPC = 10 and achieves 42% top-1 validation accuracy with ResNet-18, while surpassing the SOTA method by 2× in performance.

6. Conclusion

  • 42% top-1 validation accuracy with ResNet-18 is achieved on an IPC = 10 dataset distilled from all of ImageNet-1K, with performance 2× the SOTA method.The method is described as optimization-free and efficient.Source: §6 · p. 8

Supplementary Material

The supplementary material compares RDED and SRe2L distilled images, then reviews predictive families and V-information concepts used in the paper.

A. Distilled Images Comparison

  • RDED images show greater feature diversity within and across images in each class than SRe2L images, while retaining high realism.The comparison identifies SRe2L as limited in diverse features within each image and in diversity and realism within each class.Source: A. Distilled Images Comparison · p. 11

B.1. Definitions

  • A predictive family is a collection of permissible predictive models, and optional ignorance lets an agent disregard supplied side information.The observer models may be constrained by computational or statistical limits.Source: §B.1 · p. 11
  • V-information estimation can deviate from its true value on finite datasets; its stated properties include non-negativity, independence, and monotonicity.The passage notes that less complex predictive families and larger datasets yield more precise PAC bounds.Source: §B.1 · p. 11

B.2. Intuition of V-information on Distilled Dataset

The paper uses V-information to frame realism and diversity in distilled data, then describes how those goals inform practical objectives for the distilled set.

B.2. Intuition of V-information on Distilled Dataset

  • For real-world datasets, maximizing V-information is intractable, but a finite original dataset allows an optimal observer family trained on it to assess distilled-data realism.The passage attributes the difficulty to boundless information sources and constrained observers.Source: §B.2 · p. 11
  • The finite information in the original dataset bounds the diversity of distilled data, unlike the challenge of bounding diversity in real-world datasets.Source: §B.2 · p. 11
  • With a fixed observer family, unrealistic images or labels make label prediction harder or less accurate, motivating realism and diversity in both inputs and outputs.The passages describe encrypted or noisy image features and labels as examples.Source: §B.2 · pp. 11–12
  • The paper summarizes the joint goal for distilled inputs and outputs as maximizing V-information between images and labels.Source: §B.2 · p. 12

B.3. Maximizing V-information in Practice

  • The practical diversity analysis bounds distilled-set entropy by original-dataset entropy; maximizing label entropy can make the stated ratio one and the entropies equal.Source: §B.3 · p. 12
  • For realism, the method minimizes conditional V-entropy by selecting samples with low observer disagreement loss and assigning the teacher observer's prediction as the label.Source: §B.3 · p. 13

C. Detailed Implementation

The implementation uses pre-trained observers and a patch-selection, image-reconstruction, and relabeling pipeline, with stated training augmentations and default experiment settings.

C.1. Pre-training Observer Models

  • Pre-trained observer models vary by dataset, including ResNet-18 for ImageNet variants and ConvNets of different depths for several smaller datasets.The passage also lists modified ResNet-18 for CIFAR-10, CIFAR-100, and Tiny-ImageNet.Source: §C.1 · p. 13

C.2. Implementing RDED algorithm.

  • One RDED distillation process can produce datasets at IPC = 50, 10, and 1.Source: §C.2 · p. 13
  • For each class, 300 images are preselected, five random crops per image are scored for realism, and the highest-scoring patch represents each image.The realism score is the negative loss of the observer model's prediction against the image label.Source: §C.2 · p. 13
  • The method selects top-ranked patches according to the target IPC, then concatenates four distinct patches per high-resolution distilled image.For IPC = 50, the passage specifies 200 key patches; the IPC = 10 and IPC = 1 tasks select top-40 and top-4 patches.Source: §C.2 · p. 13
  • Distilled images receive region-level soft labels generated from random crops and aggregated across each image.The passage attributes this relabeling procedure to SRe2L and uses Fast Knowledge Distillation.Source: §C.2 · p. 13

C.3. Training on Distilled Dataset

  • Training uses RandomCropResize and CutMix augmentations.Source: §C.3 · p. 13

D. Experiment

  • Experiments default to ResNet-18, IPC = 10, and 300 preselected images per class; high-resolution images use four patches, while resolutions below 64 × 64 use one.Source: D. Experiment · p. 13

D.1. Multisize Dataset Distillation

RDED supports distillation at adaptable IPC without additional computational overhead and outperforms MDC on larger distilled datasets. Direct CoreSet selection methods perform poorly for dataset distillation.

D.1. Multisize Dataset Distillation

  • MDC unifies multiple condensation processes to produce datasets of varying sizes, avoiding extra condensation runs and reusing condensed images to reduce storage.RDED also supports adaptable IPC without additional computational overhead.Source: §D.1 · pp. 13–14
  • RDED synthesizes distilled datasets with adaptable IPC without additional computational overhead and outperforms MDC on larger distilled datasets.Table 6 evaluates both methods at IPC = 50, with IPC = 10 and IPC = 1 datasets as subsets; Conv-3 is used for distillation and evaluation.Source: §D.1 · p. 14, Table 6 · p. 14

D.2. CoreSet Selection Baselines

  • Random, Herding, and K-Means CoreSet selection methods perform catastrophically poorly when used directly for dataset distillation.Table 7 compares these three selection strategies.Source: §D.2 · p. 14, Table 7 · p. 14

D.3. Cross-architecture Generalization

The evaluation extends RDED to architectures without batch normalization and reports stronger performance than SRe2L, including across substantial architecture differences.

D.3. Cross-architecture Generalization

  • RDED outperforms SRe2L in cross-architecture generalization evaluations using architectures that lack batch normalization [16] [44], including under substantial architectural disparities such as transfer from ResNet-18 to Swin-V2-Tiny (Table 5).Source: §D.3 · p. 14
  • The evaluation includes neural network architectures that lack batch normalization.The authors describe this extension as an assessment of RDED's cross-architecture generalization.Source: §D.3 · p. 14
  • RDED remains effective under substantial architectural disparity, including transfer from ResNet-18 to Swin-V2-Tiny.The passage presents this as a notable cross-architecture example.Source: §D.3 · p. 14

D.4. Detailed Ablation Study

The ablations examine parameter choices, the contribution of RDED's four techniques, and realism-score-based patch selection.

D.4. Detailed Ablation Study

  • |T′c| = 300 and N = 1 are suitable configurations for low-resolution datasets.The study varies pre-selected subset size |T′c| and patches per distilled image N on Tiny-ImageNet, CIFAR-10, and CIFAR-100.Source: §D.4 · p. 14, Figure 5 · p. 14
  • Ablations identify all four RDED techniques—EKP, CCI, IR, and LR—as essential to the final performance.The table omits Images Reconstruction for low-resolution datasets under N = 1, where that step is impractical; the authors hypothesize LR helps produce informative and aligned labels.Source: §D.4 · p. 15, Table 8 · p. 15
  • Realism-score-based patch selection consistently outperforms alternative approaches except on CIFAR-10.The comparison covers three widely used CoreSet selection methods; the authors suggest more realistic selections may help the observer model reconstruct realistic labels.Source: §D.4 · p. 15, Table 9 · p. 15

Related papers

Loading 2312.03526v2…