PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding and Enhancing Informativeness and Utility in Dataset Distillation

Shaobo Wang, Yantai Yang, Guo Chen, Peiru Li, Kaixin Li, Yufa Zhou, Zhaorun Chen, Linfeng Zhang

arXiv:2601.21296v1cs.LGcs.AI

TL;DR

Dataset distillation lacks a principled account of how information in original data relates to selected synthetic samples, while existing approaches also face efficiency and interpretability challenges. The paper defines Informativeness and Utility, introduces InfoUtil to optimize them using Shapley Value attribution and Gradient Norm-based selection, and reports gains including 6.1% on ImageNet-1K over the previous state of the art.

  • Problem

    Existing dataset-distillation approaches face efficiency constraints and lack a solid theoretical foundation for interpreting sample selection and the relationship between original and distilled data.

  • Method

    InfoUtil defines Optimal Dataset Distillation using Informativeness and Utility, then combines Shapley Value attribution for informative content with Gradient Norm-based selection for utility.

  • Results

    6.1% improvement over the previous state-of-the-art approach on ImageNet-1K is reported, alongside superior or comparable performance across IPC settings and cross-architecture gains.

  • Takeaways & Limitations

    InfoUtil provides a theoretically grounded dataset-distillation framework that jointly targets informative content and training utility.

Abstract

from arXiv · show

Dataset Distillation (DD) seeks to create a compact dataset from a large, real-world dataset. While recent methods often rely on heuristic approaches to balance efficiency and quality, the fundamental relationship between original and synthetic data remains underexplored. This paper revisits knowledge distillation-based dataset distillation within a solid theoretical framework. We introduce the concepts of Informativeness and Utility, capturing crucial information within a sample and essential samples in the training set, respectively. Building on these principles, we define optimal dataset distillation mathematically. We then present InfoUtil, a framework that balances informativeness and utility in synthesizing the distilled dataset. InfoUtil incorporates two key components: (1) game-theoretic informativeness maximization using Shapley Value attribution to extract key information from samples, and (2) principled utility maximization by selecting globally influential samples based on Gradient Norm. These components ensure that the distilled dataset is both informative and utility-optimized. Experiments demonstrate that our method achieves a 6.1\% performance improvement over the previous state-of-the-art approach on ImageNet-1K dataset using ResNet-18.

1 INTRODUCTION

Dataset distillation compresses large datasets into small synthetic sets, but existing methods face efficiency and interpretability challenges. InfoUtil addresses these issues by grounding distillation in Informativeness and Utility and combining Shapley Value attribution with Gradient Norm-based selection.

  • Dataset distillation synthesizes a smaller representative dataset intended to match models trained on the original dataset.
  • Matching-based methods can require substantial GPU memory and time, limiting their scalability to larger datasets such as ImageNet-1K.
  • Knowledge distillation-based methods may perform well but lack a solid theoretical foundation for interpreting sample selection and the relationship between real and distilled data.
  • Optimal Dataset Distillation formalizes Informativeness as essential information within samples and Utility as sample importance for model training.
  • InfoUtil maximizes informativeness with Shapley Value attribution and utility by selecting influential samples using Gradient Norm.
  • 6.1% improvement over the previous state-of-the-art approach on ImageNet-1K demonstrates InfoUtil’s reported performance gain.

2 PRELIMINARIES

Dataset distillation seeks a much smaller synthetic dataset whose trained model performs comparably to one trained on the original data. In knowledge distillation-based methods, a teacher model first learns the original dataset’s information and then synthesizes the distilled dataset.

  • Dataset distillation synthesizes a dataset eD with m ≪ n samples from an original dataset D.
  • The distilled dataset should enable comparable, potentially lossless, performance to training on D when evaluated on a held-out test set.
  • The distilled dataset influences test loss through the optimization trajectory of model parameters trained on eD.
  • Knowledge distillation-based DD first transfers information from D into a teacher model and then uses that teacher to synthesize eD.

3 METHOD

The method defines optimal dataset distillation by jointly maximizing sample informativeness and dataset utility, then operationalizes these objectives in a two-step InfoUtil pipeline. Shapley Value attribution compresses informative sample content, while gradient-norm scoring selects influential compressed samples.

  • Optimal Dataset Distillation: Informativeness is optimized by finding a d′-sized mask whose masked sample preserves performance similarity with the original sample.The mask s is binary, has |s| = d′, and forms the compressed input s ◦ x through element-wise multiplication.
  • Optimal Dataset Distillation: Utility measures how strongly removing a training example changes gradient flow, using the worst-case effect across relevant examples.Gradient flow is the instantaneous loss-change rate during training, providing a continuous-time view of training dynamics.
  • Optimal Dataset Distillation: Optimal dataset distillation jointly maximizes informativeness and utility on the test dataset.Informativeness concerns essential information within samples, while utility concerns sample importance for model training.
  • InfoUtil Pipeline: InfoUtil implements the formulation through game-theoretic informativeness maximization followed by principled utility maximization.The first step extracts informative content using feature attribution; the second step selects samples to maximize dataset utility.
  • InfoUtil Pipeline: Shapley Value attribution identifies informative regions, while gradient-norm upper bounds enable efficient selection of the most influential compressed samples.Kernel-based estimation reduces Shapley computation, and top gradient-norm scores produce the final distilled set of size m ≪ n.

4 EXPERIMENTS

Experiments evaluate InfoUtil across datasets, architectures, IPC settings, downstream tasks, efficiency, and component ablations. Results show strong performance, cross-architecture generalization, substantial efficiency gains, and benefits from utility, informativeness, noise, and Shapley-based attribution.

  • Experimental settings: Experiments use CIFAR-10/100, Tiny-ImageNet, ImageNet-1K, and ImageNet subsets with multiple neural architectures.Evaluation follows standard dataset-distillation settings across lower- and higher-resolution datasets.
  • Main results: 16% higher accuracy on ImageNet-100 and 6.1% higher accuracy on ImageNet-1K than RDED in the reported settings.The ImageNet-100 result uses ResNet-101 at IPC=10; the ImageNet-1K result uses ResNet-18 at IPC=1.
  • Main results: 10% better performance than SRe2L and RDED in the VGG-11 teacher versus Swin-V2-Tiny student setting.Additional ImageNet-1K comparisons report consistent outperformance across ResNet-18/101 architectures and IPC settings.
  • Efficiency analysis: 50× lower time and 100× smaller memory than TESLA across all distillation stages.Efficiency is measured on a single NVIDIA A100; ImageNet-21K distillation completes in 5.83 hours.
  • Downstream tasks: InfoUtil consistently surpasses RDED across all stages of a five-step ImageNette continual-learning task.The experiments use 50 IPC and repeat results across five varied class orders.
  • Ablation studies: Utility maximization improves performance by 4.6% on ImageNette and 1.5% on ImageNet-1K, while combining both components adds 1.2% and 0.7%, respectively.The combined method reaches 86.2% on ImageNette and 44.2% on ImageNet-1K in the reported ablations.
  • Ablation studies: Removing noise causes substantial degradation, while Shapley-based attribution reaches 43.88% at IPC=10 and exceeds Grad-CAM by 13.49%.The ablations associate noise with diversity and Shapley Value with theoretically principled patch attribution.

5 DISCUSSION

The paper adapts soft-label generation to dataset size: early-stage teachers support small IPC settings, while fully pretrained teachers perform better at larger IPC values.

  • For small IPC settings, the method extracts soft labels from a model at the 10-th epoch to leverage high-entropy, diverse information.This strategy is intended for limited-data regimes.
  • For large IPC settings, the method uses fully pretrained RDED networks to obtain low-entropy, precise labels.
  • At IPC = 1, 10-th-epoch soft labels outperform labels from pretrained networks.
  • At IPC = 10 or IPC = 50, labels from fully pretrained networks yield superior results.

6 CONCLUSION

The paper presents InfoUtil as a principled dataset-distillation framework grounded in Informativeness and Utility, combining Shapley attribution with gradient norm-based sample selection. It also identifies scalability and robustness on more complex datasets as future directions.

  • InfoUtil models optimal dataset distillation through Informativeness, which captures critical information within samples, and Utility, which captures essential samples for effective training.
  • The framework combines game-theoretic informativeness maximization using Shapley value attribution with utility maximization based on gradient norms.
  • The paper reports superior dataset-distillation performance and cross-architecture generalization for InfoUtil.
  • Future work targets more complex and diverse datasets, scalability, and robustness in real-world applications.
  • Attribution methods reveal each input variable’s contribution to a black-box model’s final prediction, motivating their use for extracting key information in dataset distillation.

B DETAILED IMPLEMENTATION

The implementation computes Shapley-based informativeness, adds noise during patch selection, and then uses gradient norms to select utility-maximizing samples for synthesis. It also describes teacher-model settings, coreset comparisons, and evaluation tables.

  • Computation of Informativeness: InfoUtil uses PyTorch and Captum’s KernelShap implementation to compute Shapley values that guide data refinement.
  • Computation of Informativeness: During the first four croppings, Gaussian noise is added to pooled Shapley heatmaps to reduce repeated cropping at the same location.The fifth cropping retains the original Shapley values.
  • Computation of Informativeness: In most scenarios, each image is divided into a 4 × 4 patch grid, and the highest-Shapley patch center determines the cropping center.
  • InfoUtil Pipeline: The pipeline computes each candidate patch’s gradient norm and selects the top-k × IPC samples for utility maximization.
  • InfoUtil Pipeline: Selected patches are combined into synthetic images and assigned soft labels produced by teacher models before forming the distilled dataset.
  • Teacher-Model Settings: For IPC = 1 and 10, teacher models are trained for 10 epochs, whereas IPC = 50 uses fully converged teacher models across the reported settings.
  • Coreset Comparison: InfoUtil is evaluated against coreset selection methods, including Random, Herding, Forgetting, and K-Means, across ConvNet and ResNet-18 settings.

C.2 PERFORMANCE ON LARGE-SCALE DATASETS

On large-scale datasets, the paper evaluates InfoUtil with ResNet-18 and compares it with coreset selection while controlling for teacher soft-labeling protocols. InfoUtil substantially outperforms the reported coreset baseline on ImageNet-1K.

  • The evaluation uses ResNet-18 on challenging large-scale datasets and compares InfoUtil with K-Means coreset selection.
  • On ImageNet-1K at IPC = 10, InfoUtil achieves 44.2%, compared with 5.8% for Herding.The reported InfoUtil result is nearly 7.6 times higher than the best coreset method.
  • The study unifies the teacher protocol to isolate InfoUtil’s contribution from potential advantages of soft-labeling strategies.

D.1 CONTROLLED EXPERIMENT WITH FULLY CONVERGED TEACHER

The controlled ImageWoof experiment fixes the teacher model across RDED and InfoUtil, isolating differences in their data-synthesis mechanisms. InfoUtil maintains an advantage across student architectures and IPC settings.

  • Controlled experimental setup: The experiment uses the exact same Fully Converged Teacher for RDED and InfoUtil across IPC settings of 1, 10, and 50.This controls for differences in teacher-model convergence stages.
  • Results: InfoUtil consistently outperforms RDED across ConvNet, ResNet-18, and ResNet-101 under every tested architecture and IPC configuration.The results are reported in Table 11.
  • Results: 7.9% improvement is observed for ResNet-101 at IPC = 50.This is identified as a particularly substantial gain in the controlled setting.

E PROOFS OF SHAPLEY VALUE AXIOMS

The paper formalizes Shapley-value attribution for feature informativeness by modeling neural-network inputs as players in a cooperative game. It then establishes the four Shapley axioms: linearity, dummy, symmetry, and efficiency.

  • Game-theoretic formulation: The neural network acts as a characteristic function, input variables are players, binary masks represent coalitions, and coalition predictions provide rewards.This defines the cooperative-game interpretation used for feature attribution.
  • Game-theoretic formulation: Shapley values assign each input variable an attribution based on its contribution across coalitions represented by binary masks.The displayed definition uses e_i for the i-th coordinate and s for active variables.
  • Shapley axioms: Linearity states that the attribution for a merged game equals the sum of the attributions from the component games.The proof substitutes the merged characteristic function into the Shapley-value formula.
  • Shapley axioms: The dummy axiom assigns a player’s standalone contribution when that player has no interactions with others.The condition requires adding the player to any coalition to contribute only its individual reward.
  • Shapley axioms: Symmetry gives equal Shapley values to players that contribute equally in every coalition.The proof constructs a bijection between corresponding coalitions.
  • Shapley axioms: Efficiency states that the total game reward relative to the empty input equals the sum of all players’ Shapley values.The proof derives this by summing contributions and telescoping intermediate terms.

F PROOFS OF THEOREMS

The theorem section establishes that sample utility is bounded by gradient norm under bounded-gradient assumptions. The bound is uniform over training examples and remains controlled when the learning rate is small.

  • Utility bound: Theorem 1 states that utility is bounded by a constant times the sample’s gradient norm.The utility function is the one defined for dataset pruning.
  • Proof strategy: The proof derives the bound by applying the chain rule to gradient flow and then using the Cauchy–Schwarz inequality.The resulting inner-product term is bounded by the product of gradient norms.
  • Assumptions and scope: The constant c is independent of the current measured data and depends on an upper bound for gradient norms together with the learning rate.Taking the maximum over training examples yields the stated utility bound.
  • Assumptions and scope: A small learning rate keeps c controlled rather than excessively large.The argument assumes the model has converged sufficiently for gradient norms to have an upper bound.

G ADDITIONAL VISUALIZATIONS OF SYNTHETIC DATA

The paper adds visualizations of InfoUtil’s synthesized ImageNet-1K data across several categories and compares its images with other state-of-the-art methods. The comparison emphasizes preservation of meaningful visual content and reduced attention to irrelevant regions.

  • Visual comparisons: InfoUtil is compared with SRe2L and RDED using synthetic ImageNet-1K images at IPC = 10.The visual comparisons cover the categories shown in Figures 6–9.
  • Visual comparisons: InfoUtil preserves more realistic details and color consistency than optimization-based SRe2L.This comparison concerns the visual quality of synthesized images.
  • Visual comparisons: Compared with optimization-free RDED, InfoUtil emphasizes key semantic details while reducing focus on irrelevant areas.The figures include Welsh Springer Spaniel, schooner, indigo bunting, and Siamese cat categories.
Loading 2601.21296v1…