PaperlayerView exact revision on arXiv

Source-linked AI summary

Dataset Distillation with Neural Characteristic Function: A Minmax Perspective

Shaobo Wang, Yicun Yang, Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, Linfeng Zhang

arXiv:2502.20653v1cs.CVcs.AIcs.LG

TL;DR

Existing distribution-matching metrics such as MSE may inadequately capture distributional differences because they neglect semantic information. Neural Characteristic Function Matching reformulates distribution matching as minmax optimization, maximizing a characteristic-function discrepancy while minimizing it for synthetic data. NCFM outperforms state-of-the-art baselines on CIFAR-10/100 and achieves a 20.5% improvement on ImageSquawk over MTT [6] at 10 IPC.

  • Problem

    Existing distribution-matching metrics such as MSE may inadequately capture distributional differences because they neglect semantic information.

  • Method

    Neural Characteristic Function Matching reformulates distribution matching as minmax optimization, maximizing a characteristic-function discrepancy while minimizing it for synthetic data.

  • Results

    NCFM outperforms state-of-the-art baselines on CIFAR-10/100 and achieves a 20.5% improvement on ImageSquawk over MTT [6] at 10 IPC.

  • Takeaways & Limitations

    The results support NCFM as an efficient and scalable approach for dataset distillation across varied image resolutions and compression settings.

Abstract

from arXiv · show

Dataset distillation has emerged as a powerful approach for reducing data requirements in deep learning. Among various methods, distribution matching-based approaches stand out for their balance of computational efficiency and strong performance. However, existing distance metrics used in distribution matching often fail to accurately capture distributional differences, leading to unreliable measures of discrepancy. In this paper, we reformulate dataset distillation as a minmax optimization problem and introduce Neural Characteristic Function Discrepancy (NCFD), a comprehensive and theoretically grounded metric for measuring distributional differences. NCFD leverages the Characteristic Function (CF) to encapsulate full distributional information, employing a neural network to optimize the sampling strategy for the CF's frequency arguments, thereby maximizing the discrepancy to enhance distance estimation. Simultaneously, we minimize the difference between real and synthetic data under this optimized NCFD measure. Our approach, termed Neural Characteristic Function Matching (\mymethod{}), inherently aligns the phase and amplitude of neural features in the complex plane for both real and synthetic data, achieving a balance between realism and diversity in synthetic samples. Experiments demonstrate that our method achieves significant performance gains over state-of-the-art methods on both low- and high-resolution datasets. Notably, we achieve a 20.5\% accuracy boost on ImageSquawk. Our method also reduces GPU memory usage by over 300$\times$ and achieves 20$\times$ faster processing speeds compared to state-of-the-art methods. To the best of our knowledge, this is the first work to achieve lossless compression of CIFAR-100 on a single NVIDIA 2080 Ti GPU using only 2.3 GB of memory.

1. Introduction

Dataset distillation addresses limited-data and storage constraints by condensing real datasets into smaller synthetic sets, while distribution-matching methods offer efficient, strong performance but depend on inadequate discrepancy metrics. The paper proposes NCFM, a minmax framework that adaptively learns Neural Characteristic Function Discrepancy to align neural-feature amplitude and phase, balancing synthetic-data realism and diversity.

  • Motivation: Dataset distillation condenses large real datasets into smaller synthetic datasets to reduce data requirements and address storage constraints.The introduction frames effective learning with limited data as especially important when datasets become too large to store.
  • Background: Distribution matching is attractive because it avoids nested optimization and, at 50 images per class on CIFAR-10, achieves higher test accuracy than gradient matching with one-tenth the computation time.These methods are positioned as a practical balance between performance and computational efficiency.
  • Problem: MSE-based matching compares point-wise features in Euclidean space and misses semantic structure in high-dimensional manifolds, while MMD may fail to capture overall distributional alignment.These limitations can produce suboptimal synthesized image quality and motivate a more comprehensive discrepancy measure.
  • Method: NCFM reformulates distribution matching as adversarial minmax optimization: a sampling network maximizes real–synthetic discrepancy while synthetic data minimizes it.This adaptively learns a discrepancy metric rather than relying on rigid fixed metrics such as MSE or MMD.
  • Method: NCFM aligns the amplitude and phase of neural features in the complex plane, balancing realism and diversity in synthetic data.An auxiliary network embeds features, while a lightweight sampling network dynamically adjusts Characteristic Function sampling with a scale mixture of normals.

2. Related Work

Prior dataset distillation work includes distribution- and feature-matching methods, with DM balancing performance and computational efficiency without nested model optimization. Although CFD has been used for generative modeling [1] [27], prior work had not applied it to dataset distillation.

  • DM is regarded as an efficient dataset distillation method that balances performance and computational efficiency without nested model optimization.
  • Dataset distillation methods based on distribution and feature matching are categorized into point-wise and moment-wise matching.
  • Although CFD has been adopted for generative modeling [1] [27], no prior work had considered CFD for dataset distillation.

3. Preliminaries: Distribution Matching

Distribution Matching minimizes discrepancies between real and synthetic data distributions as an alternative to bi-level, gradient-matching, and trajectory-matching methods. Existing feature- and moment-based measures face limitations, including semantic blindness in MSE and insufficient rigor in empirical MMD estimates.

  • Distribution Matching: Distribution Matching replaces traditional bi-level optimization with discrepancy minimization between real and synthetic data, using feature point matching and moment matching.It is positioned against gradient matching [24] [54] [56] and trajectory matching [6] [9] [12] [16].
  • Feature Point Matching: MSE compares extracted features directly in Euclidean space but neglects important semantic information when measuring distribution differences.The formulation uses a feature extractor f applied to samples from the real distribution D and synthetic distribution ˜D.
  • Moment Matching: MMD aligns high-order moments in a feature-space RKHS, but its effectiveness depends on carefully selecting a kernel function K [53, 55, 57].The kernel is associated with the feature extractor f and a function class F.
  • Moment Matching: Empirical moment matching uses LMMD = ∥E_x∼D[f(x)] − E_˜x∼˜D[f(˜x)]∥2, but lacks MMD’s theoretically required maximal upper bound on discrepancy.Most DM-based methods [10] [57] directly align moments in feature space through this approximation.

4. Adversarial Distribution Matching

This section reformulates distribution matching as minmax optimization and introduces NCFD, a characteristic-function-based metric whose frequency sampling is learned adversarially. NCFM minimizes this metric for synthetic data while balancing phase and amplitude information in feature space.

  • Characteristic Function Discrepancy: CFD is a valid distance metric because characteristic functions uniquely determine distributions and satisfy nonnegativity, symmetry, and the triangle inequality.The empirical characteristic function weakly converges to the population characteristic function, supporting its use as a distributional proxy.
  • Characteristic Function Discrepancy: The CFD decomposes into amplitude differences that capture distribution scale and phase differences that encode data centres important for realism.NCFM introduces α to balance amplitude and phase information in the NCFD objective.
  • Neural Characteristic Function Discrepancy: A parameterized network ψ learns the frequency-argument distribution, extending CFD in feature space while enabling efficient discrepancy estimation.The sampling distribution uses a scale mixture of zero-mean Gaussians, and more sampled frequency arguments improve empirical-CF approximation and synthetic-data quality.
  • Neural Characteristic Function Matching: NCFM maximizes NCFD over a neural sampling network ψ while minimizing it over the synthetic distribution, producing a robust discrepancy metric for distribution matching.The feature extractor maps inputs to a latent space, and the minmax objective jointly learns the discrepancy and synthetic data distribution.
  • Implementation: The feature extractor combines pretrained and randomly initialized models through β-blending to remain moderately diverse and discriminative during distillation.At each distillation step, β is selected within (0, 1) to blend checkpoints from specified model subsets.

5. Experiments

NCFM outperforms state-of-the-art dataset distillation baselines across standard and higher-resolution benchmarks, including a 20.5% gain on ImageSquawk at 10 IPC. Ablations show that its minmax sampling network, amplitude-phase alignment, and frequency sampling strategy materially improve synthetic-data quality and accuracy.

  • CIFAR-10/100 and Tiny ImageNet: NCFM outperforms all evaluated state-of-the-art baselines on CIFAR-10/100 and Tiny ImageNet, improving over DM by 23.5% and 23.0% on CIFAR-10 and CIFAR-100 at 1 IPC.The evaluation includes gradient-, kernel-, distribution-, and trajectory-matching methods, including MTT [6].
  • Higher-resolution Datasets: 20.5% improvement on ImageSquawk at 10 IPC over MTT [6] demonstrates NCFM’s strong performance on higher-resolution ImageNet subsets.The method also uses about 1.9 GB of GPU memory on CIFAR-100, indicating scalability under high IPC conditions.
  • Cross-Architecture Generalization: NCFM generalizes across AlexNet, VGG-11 [42], and ResNet-18 [17], extending its evaluation beyond the ConvNet used for condensation.The cross-architecture evaluation is conducted on CIFAR-10 using synthetic data condensed with a 3-layer network.
  • Sampling Network Ablation: Adding the sampling network ψ improves performance by 3.2% on CIFAR-10 at 50 IPC, 2.6% on Tiny ImageNet at 1 IPC, and 10.1% at 10 IPC.It also yields gains of 2.8% on ImageMeow and 2.0% on ImageSquawk without additional data curation.
  • NCFD Ablations: Balancing amplitude and phase alignment improves performance, while increasing sampled frequency arguments raises CIFAR-10 accuracy from 62% to approximately 67% at 10 IPC.The frequency-count improvement occurs when increasing sampled arguments from 16 to 1024, after which gains diminish.

6. Discussion

The discussion frames CFD as an adaptively learned characteristic-kernel formulation that generalizes fixed-kernel MMD for distribution alignment. It also explains CFD’s computational advantage: linear-time scaling with sampled frequency arguments makes it faster and more scalable than MMD.

  • CF as Well-Behaved Kernels in the MMD Metric: NCFM adaptively learns the frequency distribution F_T(t), enabling flexible characteristic-kernel selection for optimal distribution alignment beyond MMD’s fixed kernels.Mixtures of Gaussian distributions within the characteristic-function framework produce well-defined characteristic kernels.
  • CF as Well-Behaved Kernels in the MMD Metric: MMD becomes a special case of CFD when only specific moments are matched, explaining NCFM’s minimal memory overhead as IPC grows.The correspondence follows when MMD uses a characteristic kernel whose structure aligns with CFD.
  • Computational Advantage of CFD over MMD: CFD operates in linear time relative to the number of sampled frequency arguments, whereas approximate MMD computation requires quadratic time in the number of samples.This efficiency makes CFD substantially faster and more scalable than MMD, particularly for large-scale datasets.

7. Conclusion

The work reformulates distribution matching for dataset distillation as a minmax problem and introduces NCFD, a theoretically grounded metric based on characteristic functions. NCFD dynamically aligns phase and amplitude information to balance realism and diversity in synthetic data.

  • 7. Conclusion: The method reformulates dataset distillation’s distribution matching as a minmax optimization problem and introduces Neural Characteristic Function Discrepancy (NCFD).NCFD is designed to maximize separability between real and synthetic data and is theoretically grounded.
  • 7. Conclusion: NCFD leverages the Characteristic Function to dynamically align phase and amplitude information in the complex plane.This alignment supports measuring distributional differences between real and synthetic data.
  • 7. Conclusion: The approach balances realism and diversity in synthetic data through its characteristic-function-based discrepancy measure.
Loading 2502.20653v1…