PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching

Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, Yang You

arXiv:2310.05773v2cs.CV

TL;DR

Dataset distillation has struggled to preserve full-dataset performance as synthetic-set size grows, because previous methods become ineffective at high IPC. This paper aligns the difficulty of generated patterns with synthetic-set size through trajectory matching, achieving lossless distillation at substantial compression ratios while remaining limited to its backbone network.

  • Problem

    Previous dataset-distillation methods lose effectiveness as synthetic-set size increases, leaving a gap from full real-data performance.

  • Method

    The method controls pattern difficulty by matching trajectory segments aligned with the synthetic dataset size, while sequentially learning easy and hard patterns.

  • Results

    The method achieves lossless distillation on ConvNet by reducing CIFAR-10 and CIFAR-100 to 1/5 and Tiny ImageNet to 1/10 of their original sizes without performance loss.

  • Takeaways & Limitations

    Aligning pattern difficulty with synthetic-set size keeps trajectory-matching distillation effective in both low- and high-IPC settings.

  • Takeaways & Limitations

    Losslessness holds only for the distillation backbone network; performance drops remain when evaluating the distilled data with other networks.

Abstract

from arXiv · show

The ultimate goal of Dataset Distillation is to synthesize a small synthetic dataset such that a model trained on this synthetic set will perform equally well as a model trained on the full, real dataset. Until now, no method of Dataset Distillation has reached this completely lossless goal, in part due to the fact that previous methods only remain effective when the total number of synthetic samples is extremely small. Since only so much information can be contained in such a small number of samples, it seems that to achieve truly loss dataset distillation, we must develop a distillation method that remains effective as the size of the synthetic dataset grows. In this work, we present such an algorithm and elucidate why existing methods fail to generate larger, high-quality synthetic sets. Current state-of-the-art methods rely on trajectory-matching, or optimizing the synthetic data to induce similar long-term training dynamics as the real data. We empirically find that the training stage of the trajectories we choose to match (i.e., early or late) greatly affects the effectiveness of the distilled dataset. Specifically, early trajectories (where the teacher network learns easy patterns) work well for a low-cardinality synthetic set since there are fewer examples wherein to distribute the necessary information. Conversely, late trajectories (where the teacher network learns hard patterns) provide better signals for larger synthetic sets since there are now enough samples to represent the necessary complex patterns. Based on our findings, we propose to align the difficulty of the generated patterns with the size of the synthetic dataset. In doing so, we successfully scale trajectory matching-based methods to larger synthetic datasets, achieving lossless dataset distillation for the very first time. Code and distilled datasets are available at https://gzyaftermath.github.io/DATM.

1 INTRODUCTION

Dataset distillation seeks compact synthetic datasets that preserve real-data performance, but prior methods degrade as synthetic-set size grows. The paper attributes this to a mismatch between pattern difficulty and dataset size, and proposes aligning them to achieve lossless distillation.

  • Prior methods perform well with very few images per class but remain separated from full real-data performance.
  • As IPC increases, previous distillation methods can become less effective than random selection.
  • Easy patterns suit low-IPC sets, whereas hard patterns better cover the real distribution when the synthetic set is sufficiently large.
  • The method aligns pattern difficulty with synthetic-set size and sequentially learns easy and hard patterns for stable soft-label optimization.
  • The approach achieves state-of-the-art performance across low- and high-IPC settings and losslessly distills CIFAR-10, CIFAR-100, and Tiny ImageNet on ConvNet.It reduces CIFAR-10 and CIFAR-100 to 1/5, and Tiny ImageNet to 1/10, without performance loss.

2 PRELIMINARY

Dataset distillation trains a smaller synthetic dataset to match the test performance of models trained on real data. Trajectory matching instead optimizes synthetic data so models trained on it follow training trajectories produced by real-data training.

  • Dataset distillation synthesizes Dsyn so models trained on it achieve similar test performance to models trained on Dreal.
  • Trajectory matching matches training trajectories from surrogate models optimized over the real and synthetic datasets.
  • Expert trajectories provide the start and target parameters for matching, with target pairs sampled using a preset interval M.
  • The synthetic trajectory is produced by inner optimization using cross-entropy loss and a trainable learning rate α over N preset steps.

3 METHOD

The method aligns trajectory-matching difficulty with synthetic-set size: early trajectories favor small datasets, while later trajectories better support larger ones. DATM controls this difficulty through trajectory-range selection and stabilizes soft-label learning with dedicated initialization and optimization considerations.

  • 3.1 EXPLORATION: Matching early trajectories works better for small synthetic datasets, whereas matching late trajectories performs better as the synthetic set grows.The paper associates early trajectories with easy patterns and later trajectories with hard patterns.
  • 3.1 EXPLORATION: Late-trajectory matching performs poorly at low IPC, while early-trajectory matching undermines performance as high-IPC distillation continues.Matching all trajectories is also reported as ineffective.
  • 3.2 DIFFICULTY-ALIGNED TRAJECTORY MATCHING: DATM calibrates generated-pattern difficulty to dataset size by restricting the trajectory-matching range.The method is designed to avoid patterns that are too easy or too difficult across low- and high-IPC settings.
  • 3.2 DIFFICULTY-ALIGNED TRAJECTORY MATCHING: Soft labels enrich the synthetic dataset, but their optimization introduces mislabeling and instability challenges.DATM initializes labels using correctly classified real samples and identifies low-IPC instability when labels are optimized.
  • 3.2 DIFFICULTY-ALIGNED TRAJECTORY MATCHING: A floating upper bound starts small and gradually increases, progressively expanding the matched trajectory range during distillation.This schedule first generates easier patterns and later permits harder ones.
  • 3.2 DIFFICULTY-ALIGNED TRAJECTORY MATCHING: Each iteration samples start and target parameters from expert trajectories, computes the matching loss, and updates synthetic images and soft labels.The updated labels are represented as softmax-transformed logits.

4 EXPERIMENTS

Experiments show that the method remains effective at high IPC, generalizes across architectures, and benefits from its initialization, soft-label, and sequential-generation components. It distills CIFAR-10 and CIFAR-100 to one-fifth and Tiny ImageNet to one-tenth their original sizes without performance loss.

  • Main results: The method outperforms other methods with the same architecture in all evaluated settings except CIFAR-10 with IPC=1, remaining effective as IPC increases.Previous methods saturate as the distillation ratio approaches 20% and often match or underperform random selection above 10% on CIFAR-10.
  • Main results: CIFAR-10 and CIFAR-100 are distilled to 1/5, and Tiny ImageNet to 1/10, their original sizes without performance drops.
  • Main results: The distilled datasets perform well on unseen networks, while ablations show that every proposed component provides non-trivial improvement.Without sequential generation, optimizing soft labels becomes unstable and can cause poor performance or program crashes.
  • Ablation: Learning soft labels during distillation improves consistency between synthetic data and labels, whereas directly assigning soft labels can hurt performance on problems with few categories.Learning labels also reduces sensitivity to the model used to initialize the labels.
  • Ablation: Initializing synthetic data from correctly classified samples significantly speeds optimization because the selected data are easier for networks to learn.The initialization gap gradually narrows during distillation, but remains useful as a speed-up technique.

5 EXTENSION

The extension studies how trajectory choice, IPC, and distillation-network capacity affect the synthetic images and optimization. Late trajectories produce detailed hard-pattern images and make high-IPC distillation computationally efficient, while stronger backbones become relatively more suitable as IPC increases.

  • 5.1 VISUALIZATION: Matching early trajectories produces easier patterns with blurred details and stronger color changes, whereas late trajectories produce finer details useful for hard samples.The visual comparison uses CIFAR-10 with IPC=50 and matches either steps 0–10 or 30–40.
  • 5.1 VISUALIZATION: At extremely low IPC, synthetic images collapse toward color and vague shape information, while increasing IPC keeps images closer to their initialization.
  • 5.1 VISUALIZATION: IPC=1000 requires 20x more data than IPC=50 but costs only 1.05 times more because high-IPC optimization matches late trajectories and converges faster.
  • 5.2 DISTILLATION WITH DIFFERENT NETWORKS: ConvNet’s low capacity favors easy-pattern generation at low IPC, while stronger networks should perform relatively better when high IPC makes hard patterns important.The paper identifies the relationship between backbone capacity and pattern difficulty as a perspective on ConvNet’s effectiveness for distillation.

6 RELATED WORK

Related work organizes dataset distillation around bi-level optimization and matching-based objectives. Kernel-based methods simplify the inner optimization with NTK, while matching methods align gradients, features, distributions, or training trajectories.

  • Dataset Distillation: Dataset distillation seeks to compress a large dataset into a small one without causing performance drops, naturally forming a bi-level optimization problem.
  • Kernel-based methods: Kernel-based methods use NTK kernel ridge regression to obtain a closed-form inner solution, reducing distillation to single-level optimization but retaining heavy matrix-inversion costs.
  • Matching-based methods: Matching-based methods minimize metrics between surrogate models trained on synthetic and original data, covering gradients, features, distributions, and trajectories.

7 CONCLUSION AND DISCUSSION

The paper aligns generated-pattern difficulty with synthetic-dataset size, preserving effectiveness across low- and high-IPC settings. It reports lossless distillation for the distillation backbone network, while cross-network performance gaps and scaling cost remain unresolved.

  • Aligning pattern difficulty with synthetic-dataset size keeps distillation effective in both low- and high-IPC settings.
  • The method achieves lossless dataset distillation for the first time.
  • Losslessness is limited to the distillation backbone network, with performance drops on other networks.
  • Scaling trajectory-matching methods to large datasets remains difficult because of their high distillation cost.

A.1 SOFT LABEL DISTRIBUTION

The standard deviation of soft labels after softmax tends to increase throughout distillation across all datasets. This increase reflects stronger suppression of non-target labels rather than greater diversity among soft-label values.

  • Soft-label standard deviation after softmax tends to increase as distillation proceeds across all datasets.
  • The increase does not result from greater diversity among soft-label values, but from suppressing non-target-category labels.

A.2 SOFT LABEL INITIALIZATION

Stable soft-label optimization requires initialization from class probabilities produced by a pretrained model. One-hot initialization, direct optimization, and simply adding softmax did not stabilize distillation.

  • Initializing soft labels with pretrained-model softmax class probabilities enables stable optimization during distillation.
  • One-hot initialization, direct value optimization, and adding a softmax layer failed to stabilize distillation.
  • The pre-softmax logit distribution matters because equal post-softmax labels can produce different backpropagation gradients.
  • Experiments identify appropriate pre-softmax initialization as crucial for maintaining stable distillation.

A.2.1 STABABILITY

Sequentially generating easy and hard patterns stabilizes distillation sufficiently to learn soft labels by helping surrogate models match expert trajectories. Increasing inner-optimization updates can also provide stabilization.

  • Sequential easy-then-hard pattern generation stabilizes soft-label learning during distillation.
  • The sequence helps surrogate models learn more easy patterns within finite inner-optimization steps and limited samples, improving expert-trajectory matching.
  • Increasing inner-optimization update times can also stabilize distillation.

A.2.2 SOFT LABEL OPTIMIZATION

Optimizing soft labels improves trajectory matching and synthetic-dataset performance, while longer synthetic optimization can especially benefit soft-label and high-IPC settings. The broader analyses show that trajectory difficulty must be matched to dataset size, because early and late trajectories favor different IPC regimes.

  • Soft Label Optimization: Optimizing soft labels lowers matching loss below one-hot labels and improves distilled-dataset performance.Unoptimized soft labels are harder for surrogate models to reconcile with expert trajectories.
  • Synthetic Steps: Larger synthetic steps N provide greater performance gains when soft labels are used, because inner optimization improves trajectory matching.The surrogate benefits less from longer optimization when soft labels are absent because the synthetic dataset contains less information.
  • Synthetic Steps: For IPC=1000, setting N=80 performs best while matching trajectories over 80K synthetic images against targets obtained from 100K real images.The passage reports that trajectory matching can still improve training performance even when synthetic and real trajectory lengths are similar.
  • High-IPC Behavior: Previous FTD distillation degrades training performance as IPC increases, and hyperparameter tuning only slows the degradation.The reported settings varied learning rate, batch size, synthetic steps, and the upper bound of the sample range.
  • Soft Label Optimization: Learning soft labels during distillation enables surrogate models to match expert trajectories better than fixed label choices.Figure 8 compares one-hot, unoptimized soft, and optimized soft labels using matching-loss and performance logs.
  • Difficulty Alignment: Early trajectories are easier to match, whereas late trajectories can also reduce early-trajectory loss in high-IPC settings but harm basic-pattern learning when IPC is small.Late phases emphasize hard patterns, while limited synthetic data benefits from preserving easy patterns.
Loading 2310.05773v2…