PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation

Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, Yang Gao

arXiv:2404.00563v1cs.CV

TL;DR

Distribution matching dataset distillation is efficient but can scatter same-class features and represent distributions only through means. The paper adds class centralization and covariance matching constraints, improving performance across benchmarks while retaining cross-architecture robustness.

  • Problem

    Distribution matching methods can reduce class discrimination through dispersed same-class features and incompletely match distributions by focusing only on mean features.

  • Method

    The method adds class centralization and covariance matching constraints to distribution matching, using covariance information to capture inter-feature relationships.

  • Results

    The constraints improve performance by up to 6.6% on CIFAR10 and limit cross-architecture performance reduction to 1.7% across four architectures.

  • Takeaways & Limitations

    The two plug-and-play constraints enhance class discrimination and feature distribution matching across multiple distribution matching methods and benchmark datasets.

  • Takeaways & Limitations

    The method with DM does not provide IPC=1 results because it requires more than one sample, whereas IDM remains compatible at IPC=1.

Abstract

from arXiv · show

Dataset distillation has emerged as a promising approach in deep learning, enabling efficient training with small synthetic datasets derived from larger real ones. Particularly, distribution matching-based distillation methods attract attention thanks to its effectiveness and low computational cost. However, these methods face two primary limitations: the dispersed feature distribution within the same class in synthetic datasets, reducing class discrimination, and an exclusive focus on mean feature consistency, lacking precision and comprehensiveness. To address these challenges, we introduce two novel constraints: a class centralization constraint and a covariance matching constraint. The class centralization constraint aims to enhance class discrimination by more closely clustering samples within classes. The covariance matching constraint seeks to achieve more accurate feature distribution matching between real and synthetic datasets through local feature covariance matrices, particularly beneficial when sample sizes are much smaller than the number of features. Experiments demonstrate notable improvements with these constraints, yielding performance boosts of up to 6.6% on CIFAR10, 2.9% on SVHN, 2.5% on CIFAR100, and 2.5% on TinyImageNet, compared to the state-of-the-art relevant methods. In addition, our method maintains robust performance in cross-architecture settings, with a maximum performance drop of 1.7% on four architectures. Code is available at https://github.com/VincenDen/IID.

1. Introduction

Distribution matching distillation reduces computational cost but can produce dispersed same-class features and incomplete mean-only distribution matching. The paper addresses these issues with class centralization and covariance matching constraints, improving benchmark performance and cross-architecture robustness.

  • 1. Introduction: Distribution matching methods [41] match real and synthetic feature distributions while avoiding the higher computational cost of gradient- and trajectory-matching approaches.The related methods include DM, CAFE [32], and DataDAM [25].
  • 1. Introduction: Class centralization clusters same-class synthetic features to improve class discrimination, especially when images per class are limited.The paper motivates this constraint using dispersed within-class features in distribution matching datasets.
  • 1. Introduction: Covariance matching complements mean alignment by representing inter-feature relationships for more comprehensive feature distribution matching.The constraint targets the limitations of mean-only matching in distribution matching methods.
  • 1. Introduction: Up to 6.6% performance gain on CIFAR10 accompanies the proposed constraints, with only 1.7% maximum reduction across four cross-architecture evaluations.The method was evaluated on SVHN, CIFAR10, CIFAR100, and Tiny-Imagenet.
  • 1. Introduction: Both constraints are plug-and-play, are applied to DM and IDM, and surpass recent baselines across multiple benchmark datasets.The contribution summary reports substantial improvements and state-of-the-art performance across the evaluated benchmarks.

2. Related Works

Dataset distillation research progressed from coreset selection and meta-learning toward distribution matching, which reduces computational cost by aligning embedding-space feature distributions. Related distribution matching methods refine this objective through layer alignment, augmentation, regularization, or attention matching, while the paper identifies remaining class-discrimination and distribution-completeness gaps.

  • 2. Related Works: Earlier coreset selection methods choose representative samples but struggle with scalability on large datasets and often deliver suboptimal performance.Examples include random selection, Herding, K-Center [27], and forgetting-based selection [30].
  • 2. Related Works: Dataset distillation condenses large datasets into much smaller informative datasets and supports applications including neural architecture search, continual learning, and privacy protection.The field was introduced through a meta-learning approach by Wang et al. [33].
  • 2. Related Works: Distribution matching methods reduce computational cost by matching real and synthetic feature distributions in embedding space.DM, CAFE [32], and DataDAM [25] are presented as distribution matching approaches.
  • 2. Related Works: Existing distribution matching methods retain dispersed same-class features and focus mainly on mean consistency, leaving distribution matching incomplete.The paper proposes two plug-and-play constraints targeting these limitations.

3. Method

The method augments distribution matching with class centralization and covariance matching constraints to improve class discrimination and feature-distribution alignment in small synthetic datasets.

  • Preliminaries: Dataset distillation optimizes a much smaller synthetic dataset so a model trained from scratch on it performs comparably to one trained on the real dataset.Distribution matching methods pursue this through feature-distribution alignment, with DM minimizing maximum mean discrepancy between synthetic and real data.
  • Class centralization constraint: Class centralization clusters synthetic samples within each class to reduce scattered features and clarify class boundaries.A smaller β encourages tighter within-class clustering.
  • Class centralization constraint: The class centralization constraint uses ResNet18 features, which the experiments find more effective for differentiating class features than the baseline feature network.Using different networks also enhances cross-architecture generalization.
  • Covariance matching constraint: Covariance matching complements mean matching by aligning local feature covariance matrices between corresponding real and synthetic classes.The method reshapes each feature into d × hw local descriptors, producing d × d covariance matrices for matching.
  • Covariance matching constraint: Reshaping features reduces dimensionality and avoids high-dimensional vector computations, enabling more accurate covariance estimation when samples are fewer than feature dimensions.The synthetic and real covariance matrices are computed from class-wise local feature descriptors and their means.
  • Overall framework: The proposed constraints are plug-and-play additions to distribution matching methods, with separate objectives for DM and IDM baselines.Their weighting parameters combine the new losses with the corresponding baseline distribution-matching loss.

4. Experiments

Across benchmark datasets, the proposed constraints improve distribution-matching dataset distillation and preserve strong cross-architecture performance. Ablations and sensitivity studies show gains from both constraints, with concentrated features and relatively efficient convergence.

  • Performance comparison: Up to 6.6% improvement over DM on CIFAR10 accompanies gains of 2.9% on SVHN and 2.5% on TinyImageNet across tested IPC settings.The method also improves over IDM on CIFAR10 and CIFAR100, while achieving state-of-the-art accuracies on CIFAR10, CIFAR100, and TinyImageNet.
  • Cross-architecture generalization: Cross-architecture testing on CIFAR10 records accuracies of 69.0%, 67.3%, 67.3%, and 68.3% across ConvNet, AlexNet, VGG11, and ResNet18, with less than 1.7% performance drop.At IPC=10, performance loss stays below 3.1% on AlexNet, VGG11, and ResNet18, whereas KIP loses 10.8% from ConvNet to ResNet18.
  • Analysis of cluster constraint threshold: Performance is optimal for β below 0.7, where smaller β values produce more concentrated per-class feature distributions and higher class discrimination.Larger β values make class features more dispersed, while smaller values improve performance without significantly compromising generalization.
  • Effectiveness of each component: Using both constraints concurrently improves DM by 6.6% on CIFAR10 and 2.5% on CIFAR100, exceeding the gains from either constraint alone.The class centralization and covariance matching constraints individually improve DM by 3.7% and 3.1% on CIFAR10, and by 1.4% and 1.5% on CIFAR100, respectively.
  • Evaluation of weighting parameter λ: Varying λCC from 0.01 to 0.09 changes performance by 1.47%, while varying λCM from 0.005 to 0.015 changes it by 1.1%.Larger λCC and λCM values exert a more pronounced impact on the optimization objective.
  • Number of iterations required for convergence: The method reaches peak performance in fewer than 2,000 iterations for IPC=10 and about 3,000 iterations for IPC=50, compared with roughly 20,000 iterations for previous methods.Accuracy grows quickly during early training, allowing shorter training when lower performance is acceptable.

5. Conclusion

The paper addresses insufficient class discrimination and incomplete distribution matching in dataset distillation with two constraints targeting inter-sample and inter-feature relations. Experiments show superiority across resolutions and cross-architecture settings while examining compression ratios needed to retain full-dataset performance.

  • 5. Conclusion: The class centralization and covariance matching constraints address dispersed within-class features and incomplete distribution matching, respectively.Class centralization clusters samples toward class centers, while covariance matching aligns inter-feature relationships between real and synthetic datasets.
  • 5. Conclusion: The method demonstrates significant superiority over previous methods across various resolutions and excels in cross-architecture scenarios.
  • 5. Conclusion: The study explores larger compression ratios to determine the ratio required to maintain performance comparable to the full dataset.
Loading 2404.00563v1…