PaperlayerView exact revision on arXiv

Source-linked AI summary

Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation

Jiawei Du, Yidi Jiang, Vincent Y. F. Tan, Joey Tianyi Zhou, Haizhou Li

arXiv:2211.11004v3cs.LGcs.AIcs.CV

TL;DR

Large-scale datasets enable deep learning but impose high computational and storage costs, motivating dataset distillation; gradient-matching methods additionally face accumulated trajectory error from training–evaluation discrepancies. The paper proposes Flat Trajectory Distillation, which regularizes training toward a flat trajectory, and reports improved performance across datasets, architectures, and downstream tasks.

  • Problem

    Gradient-matching dataset-distillation methods suffer accumulated trajectory error because training and evaluation use discrepant starting weights.

  • Method

    Flat Trajectory Distillation regularizes training on real data toward a flat trajectory robust to weight perturbations, without increasing the information distilled.

  • Results

    FTD outperforms vanilla trajectory matching across CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet, with reported improvements including 3.5% on CIFAR-100 at 10 images per class.

  • Takeaways & Limitations

    FTD mitigates accumulated trajectory error and generalizes across architectures and downstream dataset-distillation tasks, including neural architecture search.

  • Takeaways & Limitations

    Existing gradient-matching methods use simplified teacher-trajectory optimization, and accumulated error may limit the total number of teacher-trajectory training epochs.

Abstract

from arXiv · show

Model-based deep learning has achieved astounding successes due in part to the availability of large-scale real-world data. However, processing such massive amounts of data comes at a considerable cost in terms of computations, storage, training and the search for good neural architectures. Dataset distillation has thus recently come to the fore. This paradigm involves distilling information from large real-world datasets into tiny and compact synthetic datasets such that processing the latter ideally yields similar performances as the former. State-of-the-art methods primarily rely on learning the synthetic dataset by matching the gradients obtained during training between the real and synthetic data. However, these gradient-matching methods suffer from the so-called accumulated trajectory error caused by the discrepancy between the distillation and subsequent evaluation. To mitigate the adverse impact of this accumulated trajectory error, we propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7% on a subset of images of the ImageNet dataset with higher resolution images. We also validate the effectiveness and generalizability of our method with datasets of different resolutions and demonstrate its applicability to neural architecture search. Code is available at https://github.com/AngusDujw/FTD-distillation.

1. Introduction

Dataset distillation addresses the cost of large-scale data by synthesizing tiny datasets, but gradient-matching methods suffer accumulated trajectory error from training–evaluation discrepancies. Flat Trajectory Distillation regularizes training toward a flat trajectory to improve robustness and performance without increasing distilled information.

  • Large-scale deep learning data creates substantial costs in computation, storage, training, hyperparameter tuning, and architectural search.
  • Dataset distillation synthesizes a tiny dataset whose trained model aims to match the performance of a model trained on real data.The approach can expedite model training and reduce costs, with applications including continual learning, neural architecture search, and privacy-preserving tasks.
  • Gradient-matching methods learn synthetic data by matching gradients from real and synthetic datasets, improving over earlier dataset-distillation optimization.
  • Training–evaluation starting-point discrepancies produce accumulated trajectory error that grows across optimization iterations and harms subsequent starting weights.During training, synthetic-data trajectories reproduce real-data trajectories from real-updated weights; during evaluation, weights are continually updated by synthetic data.
  • Flat Trajectory Distillation regularizes real-data training toward a flat trajectory, making synthetic-data weights more robust to accumulated-error perturbations without increasing distilled information.The method is compatible with gradient-matching methods and supports cross-architecture scenarios.
  • 43.2% accuracy with 10 images per class and 50.7% with 50 images per class are reported on CIFAR-100.The passage introduces these results as examples of performance achieved by the proposed method, but the comparison baseline is truncated.

2. Preliminaries and Related Work

Dataset distillation seeks a much smaller synthetic dataset that preserves performance on the real-data distribution. Related methods optimize synthetic data directly or match real and synthetic training gradients across one or multiple steps.

  • Dataset distillation synthesizes a dataset S much smaller than real dataset T so models trained on S achieve comparable performance over the real data distribution.
  • Each synthetic class contains ipc images, giving |S| = ipc × C with |S| ≪ |T|.
  • The synthetic-data objective uses an empirical loss, with cross-entropy as the loss function in this paper.
  • DD directly optimizes synthetic data against the real training loss, whereas DC improves distillation by matching informative gradients from real and synthetic data.DD uses a bilevel optimization formulation; DC explicitly optimizes the synthetic dataset.
  • In the gradient-matching objective, randomly sampled initial weights are updated with real data, and a cosine-similarity-based distance measures discrepancies between gradient matrices.
  • Gradient-matching methods regularize synthetic-data distillation by matching gradients over either single steps, as in DC, or multiple steps, as in MTT.

3. Methodology

Gradient-matching methods match student updates to teacher trajectories during distillation, but evaluation reuses recurrent student weights, causing accumulated trajectory error. FTD addresses this mismatch by regularizing the teacher trajectory to be flat and robust to weight perturbations.

  • 3.1. Matching Training Trajectories (MTT): Gradient-matching methods match short or multi-step student trajectories to teacher trajectories during distillation.MTT matches accumulated gradients over several steps, while its optimizer maps sampled initial weights toward an approximately optimal parameter space.
  • 3.1. Matching Training Trajectories (MTT): Teacher trajectories are prerecorded on real data, partitioned into segments, and used to guide synthetic-data optimization against student trajectory segments.The figure’s pipeline records teacher parameters, matches student and teacher segments, then evaluates the entire student trajectory and its accumulated error.
  • 3.2. Accumulated Trajectory Error: During evaluation, each segment starts from the previous student segment’s final weights rather than the corresponding teacher weights.This initialization discrepancy produces accumulated trajectory error that propagates across later segments and degrades final performance.
  • 3.3. Flat Trajectory helps reduce the accumulated trajectory error: The proposed remedy minimizes initialization error by making the synthetic-data optimization robust to perturbations of the weights.A first approach adds Gaussian noise during initialization, but its effectiveness depends on images per class and can degrade performance when ipc is small.
  • 3.3. Flat Trajectory helps reduce the accumulated trajectory error: FTD regularizes the teacher trajectory toward a flat trajectory so standard optimization produces synthetic-data trajectories robust to weight perturbations.The method seeks a flat teacher trajectory by reducing loss-landscape sharpness, using GSAM to optimize the teacher trajectory efficiently.

4. Experiments

Experiments evaluate FTD across resolutions, architectures, ablations, and neural architecture search, showing improved performance and generalization over established distillation baselines.

  • CIFAR and Tiny ImageNet: FTD surpasses all baselines on CIFAR-10, CIFAR-100, and Tiny ImageNet, with improvements of 2.2% on CIFAR-10 and 3.5% on CIFAR-100.The reported gains use ipc = 50 for CIFAR-10 and ipc = 10 for CIFAR-100.
  • ImageNet Subsets: FTD improves performance on high-resolution ImageNet subsets, exceeding MTT by more than 4.5% on ImageNette at ipc = 1 and 10.The ImageNet subsets use 128×128 images and are described as more challenging because of their higher resolution.
  • Cross-Architecture Generalization: FTD generalizes across architectures when distilled with ConvNet-3 and evaluated on ResNet, VGG, and AlexNet using CIFAR-10 with ipc = 50.The experiments address performance degradation observed when evaluation uses an architecture different from the distillation network.
  • Ablation and Parameter Studies: Using SAM to reach a flat minimum does not improve synthetic-data generalization, supporting FTD’s focus on suppressing accumulated trajectory error instead.The comparison tests ρ ∈ {0.005, 0.01, 0.03, 0.05, 0.1} on CIFAR-100 with ipc = 10.
  • Ablation and Parameter Studies: FTD’s buffer-phase regularization contributes most to performance gains, while EMA improves evaluation accuracy but is not the primary driver.FTD without EMA still significantly surpasses MTT.
  • Neural Architecture Search: For neural architecture search, FTD produces a stronger proxy dataset, achieving 0.87 top-5 rank correlation versus 0.41 for MTT.The real-dataset reference correlation is 1.0.

5. Conclusion and Future Work

FTD mitigates accumulated trajectory error through flat trajectory distillation and generalizes across architectures and downstream dataset-distillation tasks. The authors identify non-state-of-the-art teacher trajectories as a limitation requiring further research.

  • FTD mitigates accumulated trajectory error and yields significant performance gains.
  • FTD generalizes across architectures and downstream tasks, including neural architecture search.
  • Existing teacher trajectories do not represent the state-of-the-art because their optimization is simplified to improve distillation convergence.
  • Accumulated trajectory error may limit the total training epochs of teacher trajectories, motivating further research.

A.1. Exploring the Accumulated Trajectory Error

Experiments demonstrate that accumulated trajectory error arises during evaluation and that FTD reduces it relative to MTT. Additional ablations examine initialization discrepancy using teacher-trajectory weights.

  • Exploring the Accumulated Trajectory Error: FTD reduces the evaluation-phase loss difference below MTT and makes it tend to converge.
  • Exploring the Accumulated Trajectory Error: The evaluation-phase loss difference quantifies how closely student and teacher trajectories match across epochs.
  • Exploring the Accumulated Trajectory Error: MTT’s evaluation loss difference accumulates and exceeds its distillation-phase loss difference, demonstrating accumulated trajectory error.
  • Exploring the Accumulated Trajectory Error: Initialization-discrepancy ablations train networks from teacher-trajectory weights at different start epochs.

A.2. Exploring the Flat Trajectory

The analysis explains why flat minima do not automatically improve synthetic-dataset generalization: matching error disperses the synthetic posterior and loosens the relevant bound. Experiments support this explanation.

  • Exploring the Flat Trajectory: The theoretical analysis assumes Gaussian prior, likelihood, posterior, and matching-error distributions.
  • Exploring the Flat Trajectory: Matching error makes the synthetic posterior more dispersed than the posterior trained on real data.
  • Exploring the Flat Trajectory: The generalization bound measures the closeness between test and training losses.
  • Exploring the Flat Trajectory: The resulting synthetic-dataset generalization error is greater than the real-dataset error under the stated analysis.
  • Exploring the Flat Trajectory: Experiments verify that flat minima in the synthetic-dataset setting do not improve generalization because the bound is loose.

A.3.1 Parameter Study

FTD regularizes only the buffer-phase training trajectory by combining vanilla and perturbed-loss gradients, with α and ρ selected through grid searches. A parameter study evaluates ρ on CIFAR-100.

  • A.3.1 Parameter Study: ρ = 0.01 achieves the best improvement in the CIFAR-100 parameter study, differing from the suggested ρ = 0.05.
  • A.3.1 Parameter Study: FTD regularizes buffer-phase training to obtain a flat teacher trajectory through a minimax optimization.
  • A.3.1 Parameter Study: ρ controls perturbation amplitude and therefore affects the flatness of the teacher trajectories.
  • A.3.1 Parameter Study: The update gradient combines perturbed-loss and vanilla-loss gradients as g = α · gS+L + (1 −α) · gL.
  • A.3.1 Parameter Study: The hyperparameters α and ρ are selected by grid searches on a CIFAR-10 validation set.

A.3.4 Neural Architecture Search.

The neural architecture search evaluates 720 ConvNet candidates trained on a proxy dataset and ranks them by test performance. Additional visualizations cover synthetic datasets across multiple resolutions and dataset types.

  • A.3.4 Neural Architecture Search.: 720 ConvNet candidates vary width, depth, normalization, activation, and pooling, then are trained with the proxy dataset and ranked by test performance.Widths are {32, 64, 128, 256}; depths are {1, 2, 3, 4}.
  • A.3.4 Neural Architecture Search.: Synthetic-image visualizations include 32 × 32 CIFAR-10, 64 × 64 Tiny ImageNet, and 128 × 128 ImageNette datasets with ipc = 1.These visualizations are shown in Figures 6–8.
  • A.3.4 Neural Architecture Search.: Additional visualizations show parts of synthetic images from the 32 × 32 CIFAR-100 dataset with ipc = 10.The CIFAR-100 visualizations are shown in Figure 9.

B. More Related Work

Related work develops dataset distillation through end-to-end training, gradient matching, and trajectory matching, while this paper uses flat-trajectory guidance to mitigate accumulated trajectory errors. The section also connects this strategy to loss-landscape geometry and sharpness-aware optimization.

  • Dataset Distillation: Dataset distillation seeks a much smaller synthetic dataset that performs almost as well as the original dataset.End-to-end approaches can require enormous computation and memory, inexact relaxations, or numerous unrolled iterations that cause training instabilities.
  • Dataset Distillation: Trajectory matching reduces error accumulation by matching synthetic-data parameter segments with long-range real-data training trajectory segments, but segment-specific accumulation remains inevitable.The proposed strategy further mitigates these errors using a flat teacher trajectory inspired by Sharpness-aware Minimization.
  • The geometry of the loss landscape: Sharpness-aware methods provide a low-cost way to optimize the Hessian spectrum, which is used here to minimize accumulated trajectory error.The paper relates Hessian-spectrum geometry to loss-landscape sharpness and network generalization abilities.
  • Additional visualizations: Figures 6–9 visualize synthetic datasets spanning CIFAR-10, Tiny ImageNet, ImageNette, and CIFAR-100 at different resolutions and images-per-class settings.The figures include ipc = 1 for the first three datasets and ipc = 10 for CIFAR-100.
Loading 2211.11004v3…