PaperlayerView exact revision on arXiv

Source-linked AI summary

DataDAM: Efficient Dataset Distillation with Attention Matching

Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, Konstantinos N. Plataniotis

arXiv:2310.00093v3cs.CVcs.LG

TL;DR

Dataset distillation seeks smaller synthetic training sets that preserve real-data information and competitive test performance, but prior methods can be computationally expensive and produce biased or weaker representations. DataDAM matches spatial attention maps and last-layer feature distributions using randomly initialized networks, avoiding pre-trained parameters and bi-level optimization. It reports state-of-the-art benchmark performance with up to a 100x reduction in training costs and benefits for continual learning and neural architecture search.

  • Problem

    Dataset distillation must reduce training cost while preserving the distribution, discriminative information, and test performance of larger real datasets.

  • Method

    DataDAM learns synthetic images by matching spatial attention maps and last-layer feature distributions from randomly initialized neural networks.

  • Results

    DataDAM achieves state-of-the-art results across multiple benchmark settings while offering up to a 100x reduction in training costs and cross-architecture generalization.

  • Takeaways & Limitations

    The distilled data support downstream continual learning and neural architecture search through improved memory efficiency and a more representative proxy dataset.

  • Takeaways & Limitations

    DataDAM is limited to convolutional networks, struggles with cross-architecture generalization to Vision Transformers, and must be re-optimized when the distillation ratio changes.

Abstract

from arXiv · show

Researchers have long tried to minimize training costs in deep learning while maintaining strong generalization across diverse datasets. Emerging research on dataset distillation aims to reduce training costs by creating a small synthetic set that contains the information of a larger real dataset and ultimately achieves test accuracy equivalent to a model trained on the whole dataset. Unfortunately, the synthetic data generated by previous methods are not guaranteed to distribute and discriminate as well as the original training data, and they incur significant computational costs. Despite promising results, there still exists a significant performance gap between models trained on condensed synthetic sets and those trained on the whole dataset. In this paper, we address these challenges using efficient Dataset Distillation with Attention Matching (DataDAM), achieving state-of-the-art performance while reducing training costs. Specifically, we learn synthetic images by matching the spatial attention maps of real and synthetic data generated by different layers within a family of randomly initialized neural networks. Our method outperforms the prior methods on several datasets, including CIFAR10/100, TinyImageNet, ImageNet-1K, and subsets of ImageNet-1K across most of the settings, and achieves improvements of up to 6.5% and 4.1% on CIFAR100 and ImageNet-1K, respectively. We also show that our high-quality distilled images have practical benefits for downstream applications, such as continual learning and neural architecture search.

1. Introduction

DataDAM targets the computational cost and representation biases of dataset distillation by matching attention and feature distributions in randomly initialized networks. It reports state-of-the-art performance, lower training costs, and downstream benefits for continual learning and neural architecture search.

  • Motivation: Dataset distillation compresses large training datasets into smaller synthetic sets intended to preserve competitive test performance while supporting continual learning, neural architecture search, federated learning, and privacy-preserving tasks.This data-centric alternative addresses the cost of training, tuning, and architecture search.
  • Motivation: Prior methods incur bi-level optimization, second-order computation, memory demands, biased images, or weaker scalability, while distribution matching can underperform despite avoiding inner-loop optimization.CAFE may generate biased images and incur significant time costs, whereas DataDAM is presented as avoiding these issues.
  • Method: DataDAM combines spatial attention matching with last-layer feature alignment to approximate real-data distributions without pre-trained parameters or bi-level optimization.Multiple randomly initialized neural networks provide representations for aligning discriminative feature maps and feature distributions.
  • Contributions: Up to a 100x reduction in training costs is reported alongside state-of-the-art results across computer-vision benchmark settings and cross-architecture generalization.The method is evaluated on datasets with different resolutions.
  • Applications: DataDAM’s distilled data are reported to improve continual-learning memory efficiency and accelerate neural architecture search through a more representative proxy dataset.These are presented as downstream applications of the distilled set.

2. Related Work

Related work spans dataset-distillation objectives, heuristic coreset selection, and attention mechanisms. DataDAM applies spatial attention matching to dataset distillation while addressing computational and representation limitations identified in prior approaches.

  • Dataset Distillation: Earlier dataset-distillation methods use bi-level optimization, gradient or trajectory matching, and feature or distribution matching, but face computational, memory, bias, or scalability limitations.DataDAM instead matches intermediate spatial attention maps and is described as reducing memory costs while outperforming most existing methods on standard benchmarks.
  • Coreset Selection: Coreset selection relies on heuristics such as random sampling, herding, and K-Center, whereas DataDAM learns synthetic data beyond a subset of original samples.Heuristic selection may be suboptimal when dataset information is not concentrated in a few samples.
  • Attention Mechanism: Attention mechanisms have improved representation or compression in other settings, but this paper introduces spatial attention matching to approximate real-dataset distributions for dataset distillation.The proposed module targets informative feature maps in the dataset-distillation setting.

3. Methodology

DataDAM distills datasets by matching spatial attention representations from real and synthetic images across randomly initialized networks, with a complementary embedding-distribution regularizer. The method learns fixed-label synthetic samples by optimizing attention-matching and MMD losses with SGD and momentum.

  • 3. Methodology: DataDAM represents images with spatial attention maps from early, intermediate, and final network layers, capturing low-, mid-, and high-level information across randomly initialized networks.The network family provides a more comprehensive representation than any single randomly initialized network.
  • 3.1. Dataset Distillation with Attention Matching: The SAM module aggregates absolute feature-map activations across channels to emphasize informative spatial regions before comparing real and synthetic data.Attention maps are generated from every layer except the last layer.
  • 3.1. Dataset Distillation with Attention Matching: DataDAM minimizes normalized spatial-attention differences between real and synthetic class-wise batches using the loss LSAM across layers and randomly initialized networks.The method samples real and synthetic batches for each class and estimates expectations empirically when ground-truth distributions are unavailable.
  • 3.1. Dataset Distillation with Attention Matching: A class-wise MMD regularizer LMMD aligns mean vectors of final-layer embeddings to reduce residual discrepancies between synthetic and real datasets.The final-layer features provide high-level abstract semantic information that complements spatial attention matching.
  • 3.1. Dataset Distillation with Attention Matching: The synthetic dataset is optimized with L = LSAM + λLMMD using SGD with momentum, while each synthetic sample keeps a fixed label throughout training.Initialization may use random noise, randomly sampled real images, or K-Center clustering.

4. Experiments

DataDAM is evaluated across datasets, architectures, ablations, visualizations, and downstream applications. It generally improves distillation performance while reducing computational cost and supporting continual learning and neural architecture search.

  • 4.2. Comparison to State-of-the-art Methods: DataDAM consistently outperforms baselines across low-, medium-, and high-resolution datasets, including ImageNet-1K, where it achieves 2.0%, 2.2%, 6.3%, and 15.5% accuracy at 1, 2, 10, and 50 IPC.On CIFAR10/100 with IPC50, it improves over DM by 4.0% and 5.8%, respectively.
  • 4.2. Comparison to State-of-the-art Methods: When transferred from ConvNet to unseen architectures, DataDAM exceeds DM by 3.8% on AlexNet and 7.4% on VGG-11.The authors attribute remaining improvements to robust synthetic images with reduced architectural bias.
  • 4.2. Comparison to State-of-the-art Methods: DataDAM reduces runtime by almost two orders of magnitude relative to most state-of-the-art methods while using randomly initialized networks without inner-loop training.The comparison also reports up to a 100x reduction in training costs and cross-architecture generalization.
  • 4.3. Ablation Studies: For DataDAM, under IPC50, performance is robust to random and K-Center initialization with only a minute gap, whereas Gaussian-noise initialization reduces performance; under CIFAR10 IPC10, increasing λ improves performance up to a certain point.
  • 4.3. Ablation Studies: The joint use of spatial attention matching and complementary MMD loss produces the strongest results, whereas either component alone underperforms the combination.MMD alone misses multiple representation levels, while attention matching alone remains improvable; feature-map transfer also shows a significant gap.
  • 4.3. Ablation Studies: Matching spatial attention maps across intermediate layers is important because individual layers alone cannot capture the multiple information levels of the real data distribution.The network-distribution study further reports robustness across varied random initializations, attributed to transferring discriminative attention information rather than full feature statistics.
  • 4.4. Visualization: t-SNE visualizations show that DataDAM and DM produce more equalized synthetic-data distributions, whereas DSA and CAFE are biased toward cluster edges.The authors connect distribution preservation to applications where biased representations can create discrimination risks.
  • 4.5. Applications: DataDAM reaches 39.7% final accuracy in both 5-step and 10-step CIFAR100 continual learning, versus 34.4% and 34.7% for DM.In NAS, it selects a model achieving 89.0% accuracy versus 89.2% using original data and obtains the highest search-space correlation at 0.72.

5. Conclusion and Limitations

DataDAM combines spatial attention matching and last-layer feature alignment to capture informative and discriminative information while lowering CNN training costs. Its main limitation is restricted cross-architecture generalization and the need to re-optimize when the distillation ratio changes.

  • DataDAM uses spatial attention matching and last-layer feature alignment across layers of randomly initialized networks to capture informative, discriminative dataset information.
  • The method achieves strong generalization across datasets and lowers CNN training costs while supporting continual learning and neural architecture search applications.
  • DataDAM is limited to convolutional networks, with cross-architecture generalization to Vision Transformers remaining challenging.
  • All data distillation methods, including DataDAM, must be re-optimized when the distillation ratio changes, limiting efficiency in some applications.

6.1. Implementation Details

The experiments cover datasets from 32×32 CIFAR images through 128×128 ImageNet subsets, using standardized preprocessing and architecture-specific ConvNets. Comparisons reuse common evaluation settings, publicly available distilled data, and documented hyperparameters.

  • 6.1.1 Datasets: Experiments span CIFAR10/100, TinyImageNet, ImageNet-1K, and 128×128 ImageNet subsets including ImageNette, ImageWoof, and ImageSquawk.
  • 6.1.2 Data Preprocessing: Standardized differentiable augmentations are used across datasets, with ZCA whitening applied only to CIFAR10/100 because full-size ZCA is computationally expensive at higher resolutions.
  • 6.1.3 Implementations of Prior Works: Baseline comparisons use publicly available distilled data, shared ConvNet architectures and preprocessing, with limited reproduction on high-resolution datasets when prior methods do not scale.
  • 6.1.4 Hyperparameters: The study documents hyperparameters, uses SGD, resolution-specific three-, four-, or five-layer ConvNets, and learns synthetic sets from mini-batches of 256 real images per class.

6.2. Additional Results and Further Analysis

Additional experiments show DataDAM produces informative synthetic images, remains robust across several design choices, and provides useful proxy rankings for neural architecture search. Its advantages are supported by comparisons with generative, distribution-matching, and competing distillation methods.

  • 6.2.1 Comparison to More Baselines: DataDAM significantly outperforms popular generative models on CIFAR10, whose images are designed for realistic appearance rather than training efficiency.
  • 6.2.1 Comparison to More Baselines: DataDAM surpasses vanilla pixel-space MMD because feature extractors provide information from multiple representation levels for more informative synthetic images.
  • 6.2.2 More Ablation Studies: Gaussian-noise initialization converges more slowly and underperforms Random and K-Center, yet still transfers real-data information and beats KIP and DSA.
  • 6.2.2 More Ablation Studies: Evaluation augmentation materially affects performance, with DSA performing best and no augmentation substantially reducing results on CIFAR10 with IPC 10.
  • 6.2.2 More Ablation Studies: MSE gives the best LSAM results, while loss choices remain broadly effective and a suitable configuration improves ablation performance by up to 2.0%.
  • 6.2.2 More Ablation Studies: L2 normalization is most effective, but both normalization schemes work well; omitting normalization causes significant degradation by increasing optimization overshooting risk.
  • 6.2.3 More Experiments and Analysis on Neural Architecture Search: DataDAM gives a 0.72 performance correlation across 720 ConvNet architectures and generally outperforms competing proxy sets among the top 20% of the search space.
  • 6.2.3 More Experiments and Analysis on Neural Architecture Search: On NAS-Bench-201, DataDAM obtains a small positive correlation while most methods are negatively correlated, although distilled data still poses challenges for model design.

6.3. Additional Visualizations and Analysis

Visual analyses indicate that DataDAM preserves a balanced data distribution and that synthetic-image appearance reflects the selected architecture, loss components, matching layers, and initialization. Additional figures extend these observations across datasets and configurations.

  • 6.3.1 More Analysis on Data Distribution: DataDAM, like DM, spreads synthetic features across the CIFAR10 distribution, whereas DC, DSA, and CAFE show cluster-boundary bias and higher false-positive rates.
  • 6.3.2 Extended Visualizations of Synthetic Images: Synthetic images encode architectural inductive bias: ConvNet produces natural-looking images, while VGG-11 yields brighter, higher-contrast images than natural data.
  • 6.3.2 Extended Visualizations of Synthetic Images: DataDAM images combine the visual effects of LSAM and LMMD, while LSAM produces brighter, more contrasted images by matching discriminative feature-map information.
  • 6.3.2 Extended Visualizations of Synthetic Images: Matching intermediate and final layers produces brighter, higher-contrast images because attention maps and final-layer embeddings convey distinct distributional information.
  • 6.3.2 Extended Visualizations of Synthetic Images: Random and K-Center initializations yield similar images, whereas Gaussian-noise images differ but still retain class information, supporting robustness to initialization.
  • 6.3.2 Extended Visualizations of Synthetic Images: Additional distilled-image visualizations cover CIFAR10, CIFAR100, TinyImageNet, ImageNet-1K, ImageNette, ImageWoof, and ImageSquawk; CIFAR10 and CIFAR100 include IPC 10 and 50 examples.
Loading 2310.00093v3…