PaperlayerView exact revision on arXiv

Source-linked AI summary

DREAM: Efficient Dataset Distillation by Representative Matching

Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, Yang You

arXiv:2302.14416v3cs.CV

TL;DR

Optimization-based dataset distillation uses matching objectives to synthesize compact datasets, but original-image selection is commonly random and can overlook evenness and diversity. DREAM matches representative images selected by clustering; it reaches comparable performance in fewer than one eighth the iterations and can surpass state-of-the-art methods with sufficient training. The authors note that computation remains burdensome as image size and class count grow.

  • Problem

    Optimization-based distillation still requires substantial training time, while random image selection may overlook sample-distribution evenness and diversity.

    Source: §3.1 · p. 3, §3.2 · p. 4

  • Method

    DREAM clusters samples within each class and matches representative sub-cluster centers, with a similar clustering strategy used to initialize synthetic images.

    Source: §1 · p. 2

  • Results

    DREAM reaches comparable performance in less than one eighth the iterations of baseline methods and, given sufficient iterations, surpasses state-of-the-art methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DREAM can be plugged into popular dataset-distillation frameworks and reduces training iterations by more than eight times without performance drop.

    Source: §5 · p. 9

    Computation remains large as image size and class count increase, making ultra-large-scale datasets such as ImageNet difficult to handle.

    Source: §6 · p. 9

Abstract

from arXiv · show

Dataset distillation aims to synthesize small datasets with little information loss from original large-scale ones for reducing storage and training costs. Recent state-of-the-art methods mainly constrain the sample synthesis process by matching synthetic images and the original ones regarding gradients, embedding distributions, or training trajectories. Although there are various matching objectives, currently the strategy for selecting original images is limited to naive random sampling. We argue that random sampling overlooks the evenness of the selected sample distribution, which may result in noisy or biased matching targets. Besides, the sample diversity is also not constrained by random sampling. These factors together lead to optimization instability in the distilling process and degrade the training efficiency. Accordingly, we propose a novel matching strategy named as \textbf{D}ataset distillation by \textbf{RE}present\textbf{A}tive \textbf{M}atching (DREAM), where only representative original images are selected for matching. DREAM is able to be easily plugged into popular dataset distillation frameworks and reduce the distilling iterations by more than 8 times without performance drop. Given sufficient training time, DREAM further provides significant improvements and achieves state-of-the-art performances.

1. Introduction

Dataset distillation reduces large datasets to smaller surrogates, but optimization-based methods still require substantial training time and commonly select matching images at random. DREAM matches representative images and reports faster distillation with comparable performance in fewer iterations, as well as stronger results given sufficient training.

1. Introduction

  • Random selection can overweight boundary samples with larger gradients and fail to ensure batch diversity, contributing to unstable optimization and lower training efficiency.The paper identifies uneven sample distribution and unconstrained within-batch diversity as the underlying issues.Source: §1 · p. 2
  • DREAM selects representative original images for matching, using class-level clustering to choose centers that cover the distribution evenly and represent surrounding samples.It also uses a similar clustering strategy to initialize synthetic images, which the paper says further accelerates training.Source: §1 · p. 2
  • DREAM takes less than one eighth of the iterations to reach performance comparable to baseline methods, and with sufficient iterations surpasses state-of-the-art methods.The strategy is described as easily integrated into popular dataset-distillation frameworks.Source: §1 · p. 2

2. Related Works

The paper reviews coreset and optimization-based dataset distillation, then develops DREAM as a representative-sample matching strategy. It motivates the strategy through problems with random matching and describes clustering-based selection and initialization.

2.1. Dataset Distillation

  • Coreset methods select original samples but may struggle to preserve rich information at high compression, while optimization-based methods synthesize images using matching objectives.Prior optimization-based work matches gradients, embedding distributions, or training trajectories; the paper focuses on improving its training efficiency.Source: §2.1 · pp. 2–3

2.2. Clustering

  • The paper introduces K-Means, DBSCAN, and hierarchical clustering as approaches with different cluster-count, shape, and cluster-size properties.DBSCAN can form clusters of any shape, but cluster sizes may be unstable, outliers excluded, or nearby clusters merged.Source: §2.2 · p. 3

3. Method

  • DREAM aims to make optimization-based dataset distillation more stable and efficient by matching representative original images.The method section proceeds from distillation preliminaries and training-efficiency observations to the representative-matching design.Source: §3 · p. 3

3.1. Preliminaries

  • Optimization-based distillation initializes a small surrogate set and updates it to match objectives on original data; gradient matching compares synthetic-set and original-data gradients across training stages.The matching objective and model training alternate, using mini-batches of original images with the same class labels.Source: §3.1 · p. 3

3.2. Observations on Training Efficiency

  • Boundary samples can dominate gradient matching, while random mini-batches show high, fluctuating MMD and uneven coverage, yielding biased supervision and reduced diversity.The paper links these issues to unstable optimization and lower distillation efficiency.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Representative Matching

  • DREAM uses K-Means within each class to form a predefined number of sub-clusters and matches their centers, which cover the class space evenly while retaining diversity.It clusters periodically during training and initializes synthetic images from cluster-center samples to accelerate convergence.Source: §3.3 · p. 4

4. Experiments

Across the evaluated benchmarks, DREAM consistently surpasses other state-of-the-art methods; its full method also reaches baseline performance in more than eight times fewer iterations. Ablations and visualizations examine its components, matching behavior, and continual-learning performance.

4.1. Datasets and Implementation Details

  • Experiments evaluate distilled datasets on original test images using top-1 accuracy across six datasets, with ConvNet-3 as the default network and IDC as the baseline.The default setup uses 1,200 matching iterations and reports means and standard deviations over five experiments.Source: §4.1 · p. 5

4.3. Ablation Study and Analysis

  • The full DREAM combines representative matching and clustering-based initialization, reaching baseline performance in more than 8× fewer iterations and extending to other distillation frameworks.Representative matching substantially improves final performance; initialization offers an early lead and stable boosts, while similar effects appear in other frameworks.Source: §4.3 · p. 5, Table 3 · p. 6
  • DREAM surpasses compared methods in absolute performance and performance drop when the distilled dataset is applied on an unseen architecture; the dataset is distilled on model D and validated on another model T.Source: §4.3 · pp. 5–6, Table 4 · p. 6

4.2. Comparison with State-of-the-art Methods

  • DREAM consistently surpasses state-of-the-art methods across datasets and IPC settings, with particular robustness at small IPC; Tables 1 and 2 report the comparisons.The evaluation uses ConvNet-3 for distillation, including the TinyImageNet comparison.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.3. Ablation Study and Analysis

  • With 128 sub-clusters and one center sample per cluster, gradient supervision is optimal; clustering every iteration performs best, while interval 10 balances accuracy and cost.A larger N=256 approaches random sampling, while N=32 centers tend toward lower-gradient regions; interval 20 causes an obvious top-1 accuracy degradation.Source: §4.3 · p. 7, Figure 3 · p. 7
  • On ImageNet-1K, DREAM shows excellent performance against TESLA under the same experimental setting.The results are reported in Table 6.Source: §4.3 · p. 7, Table 6 · p. 7

4.4. Visualizations

  • DREAM yields smaller gradient differences, more even class coverage than random sampling, and distilled images with clearer categorical characteristics and greater variety.The gradient comparison includes DC, DSA, and IDC; the image comparison is against DC.Source: §4.4 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

4.5. Application on Continual Learning

  • In five-step CIFAR-100 class-incremental learning, DREAM maintains performance advantages throughout training, with the gap widening as learned classes increase.Distillation uses ConvNet-3 and validation uses ResNet-10.Source: §4.5 · p. 9

5. Conclusion

DREAM selects representative original images to address dataset-distillation training inefficiency. It reduces training iterations by more than 8 times without performance drop, while improving optimization stability and final performance.

5. Conclusion

  • DREAM reduces training iterations by more than 8 times without performance drop while improving optimization stability and final performance.The conclusion also reports improved generalization capability and says efficient matching allows future work to design more complicated matching metrics.Source: §5 · p. 9

6. Limitations and Future Works

DREAM improves the efficiency of optimization-based dataset distillation, but computational burden remains a constraint as image size and class number increase.

6. Limitations and Future Works

  • Computational burden remains large as image size and class number increase, despite DREAM's efficiency gains for optimization-based dataset distillation.Source: §6 · p. 9
  • Optimization-based dataset distillation methods remain difficult to apply to ultra-large-scale datasets such as ImageNet, even with DREAM's efficiency improvement.Source: §6 · p. 9
  • The authors identify more resource-friendly dataset-distillation methods as future work.Source: §6 · p. 9

7. Comparisons with More Methods

Across MNIST, SVHN, CIFAR-10, and CIFAR100, DREAM achieves state-of-the-art results in most tested settings, with important exceptions in comparisons against RFAD and HaBa.

7. Comparisons with More Methods

  • DREAM achieves state-of-the-art results in most tested settings across MNIST, SVHN, CIFAR-10, and CIFAR100.The experiments use 1, 10, and 50 images per class.Source: §7 · p. 12
  • At IPC=1 on CIFAR-10, DREAM does not outperform RFAD; otherwise, it distills better synthetic images than RFAD using a 128-channel rather than 1024-channel ConvNet.Source: §7 · p. 12
  • HaBa performs better at IPC=10 on CIFAR-10 and IPC=1 on CIFAR-100, while DREAM performs better in the other reported circumstances.Source: §7 · p. 12
  • DREAM's performance gap over other methods is especially large at small IPC, supporting the effectiveness of matching representative samples.Source: §7 · p. 12

8. Accuracy Curve Visualization

On CIFAR10 with 10 images per class, applying DREAM to DC and DSA preserves their original performance with fewer iterations. Additional training continues to improve performance.

8. Accuracy Curve Visualization

  • DREAM reaches the original performance using one fifth of the iterations for DC and one tenth for DSA.The experiments use 10 images per class on CIFAR10.Source: §8 · p. 12
  • Further training time brings continuous performance improvement after DREAM is applied to DC and DSA.Source: §8 · p. 12
  • The experiments report stable performance improvements and describe DREAM as easily applicable to popular dataset-distillation frameworks.Source: §8 · p. 12

9. DREAM on Distribution Matching

DREAM is applied to embedding distribution matching, where random sampling can shift average features and undermine matching efficiency. On CIFAR10 with 10 images per class, DREAM substantially improves IDC’s performance and reaches it in fewer than one tenth as many iterations.

9. DREAM on Distribution Matching

  • DREAM’s distribution-matching application constrains optimization using feature distributions from synthetic and original images.The section introduces distribution matching as an additional setting beyond gradient matching.Source: §9 · p. 12
  • Random sampling can shift average features in distribution matching, while DREAM selects even and diverse original images to reduce those shifts and provide consistent supervision.Boundary samples shift the average feature in distribution matching; ξ represents averaging features in the channel dimension.Source: §9 · p. 12
  • On CIFAR10 with 10 images per class, DREAM improves IDC’s performance by a large margin and reaches it in less than one tenth of the original iterations.The original IDC performs much worse than gradient matching in this setting; DREAM reverses that comparison.Source: §9 · p. 12

10. Distilled Dataset Visualization

The visualizations indicate that DREAM produces distilled images with clearer categorical characteristics and greater variety, alongside better performance. Additional distilled-image examples are provided for MNIST, FashionMNIST, and SVHN.

10. Distilled Dataset Visualization

  • DREAM makes distilled images show more obvious categorical characteristics and greater variety, improvements associated with better performance.The paper describes image quality from these two perspectives.Source: §10 · p. 12
  • Additional distilled-image visualizations are provided for MNIST, FashionMNIST, and SVHN.Source: §10 · p. 12

11. Differences from Related Works

DREAM addresses matching efficiency through representative-image selection, whereas RFAD reduces KIP’s kernel-matrix calculation complexity. The paper presents DREAM as complementary to these efficiency approaches and applicable to both gradient and embedding distribution matching.

11. Differences from Related Works

  • RFAD reduces KIP kernel-matrix calculation complexity using random feature approximation, while DREAM targets matching efficiency through representative-image selection.The paper states that the approaches do not conflict.Source: §11 · p. 12
  • DREAM improves matching efficiency through representative-image sampling for both gradient matching and embedding distribution matching.The paper says DREAM can be plugged into other distillation methods to reduce required training iterations.Source: §11 · p. 12

12. Clustering Analysis

The clustering analysis reports that DREAM’s added clustering cost is outweighed by its lower iteration requirement: CIFAR-10 saves more than 70% of time, while CIFAR-100’s clustering overhead is negligible. The section also points to stable efficiency and performance improvements and improved distilled-image quality and diversity.

12. Clustering Analysis

  • For CIFAR-10, each clustering process takes 1s every 10 inner loops, averaging 0.1s of clustering time per loop.Matching and image updating cost 0.2s per inner loop.Source: §12 · p. 12
  • Table 7 reports top-1 accuracy for test models trained on distilled synthetic images across multiple datasets using ConvNet-3.Source: Table 7 · p. 13
  • Figure 7 describes stable performance and efficiency improvements from applying DREAM.Source: Figure 7 · p. 13
  • DREAM saves more than 70% of time, needing one tenth to one fifth the iterations to obtain original performance, while average inner-loop time rises from 0.2s originally to 0.3s.Source: §12 · p. 13
  • On CIFAR-100, extra clustering time is one twentieth of the original image-updating time and is described as negligible.Source: §12 · p. 13
  • Figure 8 describes improved image quality and sample diversity from applying DREAM.Source: Figure 8 · p. 14
  • Example distilled-image visualizations are shown for MNIST, FashionMNIST, and SVHN.Source: Figure 9 · p. 14

Related papers

Loading 2302.14416v3…