PaperlayerView exact revision on arXiv

Source-linked AI summary

Dataset Distillation by Influence Matching

Haoru Tan, Wang Wang, Sitong Wu, Xiuzhe Wu, Yangtian Sun, Chirui Chang, Shaofeng Zhang, Xiaojuan Qi

arXiv:2607.16859v1cs.CV

TL;DR

Dataset distillation methods often match intermediate training signals without ensuring alignment with the final trained model. Inf-Match instead matches sample influence on converged parameters, achieving leading results across image classification and vision-language retrieval benchmarks, including 31.5% on Tiny-ImageNet at IPC=10, 4.7% above NCFM.

  • Problem

    Existing process-alignment proxies do not guarantee outcome alignment, while influence estimators are computationally costly and often assume convex losses unsuitable for deep networks.

  • Method

    Inf-Match learns synthetic data by matching its influence on final optimized parameters to the real dataset using a differentiable linear-time estimator from optimization unrolling and first-order Taylor approximation.

  • Results

    Inf-Match achieves the best performance across classification and vision-language distillation benchmarks, reaching 31.5% on Tiny-ImageNet (IPC=10), +4.7% over NCFM.

  • Takeaways & Limitations

    The results support outcome alignment as an effective alternative to process matching for dataset distillation across image and vision-language tasks.

  • Takeaways & Limitations

    The estimator’s approximation-error bound assumes Lipschitz-continuous gradients, bounded network-parameter gradients, and a bounded maximum learning rate.

Abstract

from arXiv · show

We revisit dataset distillation from an outcome-centric perspective. Rather than aligning process surrogates (per-step gradients or training trajectories), Influence Matching (Inf-Match) aligns the final outcome of training: it learns a compact synthetic set whose effect on the converged parameters matches that of the full dataset. Concretely, we introduce a fully differentiable, sample-level influence estimator that quantifies parameter shifts from adding or removing data, without time-consuming inverse-Hessian products or convexity assumptions. The estimator runs in linear time by unrolling the optimization dynamics and applying a first-order Taylor approximation. We then learn the synthetic set by minimizing the mismatch between its influence and that of the real dataset, yielding outcome alignment rather than heuristic process imitation. Inf-Match delivers the best accuracy across standard classification benchmarks. For instance, on Tiny-ImageNet (IPC=10), Inf-Match attains 31.5\%, a +4.7\% improvement over NCFM. Beyond classification, Inf-Match scales to vision-language distillation on Flickr30K, outperforming strong process-matching baselines. For instance, with 200 to 1000 synthetic samples, our method achieved a leading impressive average on image/text retrieval tasks, higher than NCFM by 2.5\%. The code will be released via https://github.com/hrtan/infmatch.

1. Introduction

The introduction motivates dataset distillation as a response to the storage, transmission, and training costs of massive visual datasets, and identifies outcome influence matching as the unresolved alternative to process alignment. It presents Inf-Match, a differentiable sample-influence framework, and reports leading results across image and vision-language benchmarks.

  • Motivation: Modern visual datasets reach tens or hundreds of millions of samples, creating prohibitive storage, transmission, and training costs.Dataset distillation addresses these costs by synthesizing a small, informative dataset that preserves knowledge from the larger one.
  • Problem: Process-matching methods align intermediate gradients or optimization paths, but these proxies do not guarantee alignment with the full dataset’s final influence on learned parameters.The introduction frames direct outcome-influence matching as the central open question for dataset distillation.
  • Method: Inf-Match shifts dataset distillation from process alignment to outcome alignment using a differentiable sample-influence estimator for final optimized parameters.The estimator requires neither convexity assumptions nor inverse-Hessian computation.
  • Results: 31.5% on Tiny-ImageNet (IPC=10) represents a +4.7% improvement over NCFM.Inf-Match consistently achieves the best performance across CIFAR-10, CIFAR-100, and Tiny-ImageNet and all reported IPC settings.
  • Results: 2.5% higher than NCFM on image/text retrieval tasks with 200 to 1000 synthetic samples demonstrates Inf-Match’s vision-language scalability.With 200 synthetic samples, it also exceeds DATM on image-to-text retrieval by 1.3%.

2. Related Works

Prior dataset distillation methods mainly use feature matching or process matching as proxy objectives, while other work incorporates diversity, authenticity, generative models, meta-gradients, and kernels. Influence estimation links training data to post-training performance but remains limited by convexity assumptions and costly inverse-Hessian-gradient products.

  • Feature matching: Feature-matching methods align synthetic and real data in latent feature space, including distribution matching, attention-map matching, and MMD minimization.Distribution Matching targets data distributions, Datadam matches spatial attention maps across layers, and M3D minimizes maximum mean discrepancy.
  • Process matching: Process-matching methods align intermediate training states, chiefly through gradient matching or training-trajectory matching.DSA extends gradient matching with differentiable augmentation to synthesize more informative images.
  • Others: Other dataset-distillation methods address data diversity, synthesis authenticity, generative models, meta-gradients, and kernel-based formulations.These approaches broaden dataset distillation beyond feature and process matching.
  • Influence estimation: Influence estimation connects individual training samples to post-training model performance, commonly using leave-one-out retraining to measure performance changes.Scalable estimators and refined influence functions have been proposed to reduce retraining costs and improve group-impact precision.
  • Influence estimation: Influence-estimation methods face convexity assumptions and computational bottlenecks from inverse-Hessian-gradient products, limiting scalability to large models and datasets.The required convexity of the loss with respect to parameters is frequently unmet.

3. Preliminaries

Dataset distillation learns a compact synthetic set whose training outcome matches that of the full dataset through a bilevel optimization problem. Because this problem is difficult, existing methods often use heuristic proxy objectives whose intermediate-signal matching can create an optimization gap.

  • Dataset distillation objective: Dataset distillation seeks a compact synthetic set S that produces a model performing comparably to one trained on the real dataset D.The setup uses a real dataset D, network parameters θ, and task loss L(·, ·).
  • Bilevel formulation: The bilevel formulation trains the network on S at the inner level and updates S at the outer level to minimize loss on D.The learned set induces a final model whose performance closely mirrors training on the full dataset.
  • Proxy objectives: Because bilevel optimization is difficult, methods often replace it with proxy objectives such as trajectory matching or gradient alignment.These proxies attempt to make synthetic data mimic intermediate training signals, including gradients and parameter states.
  • Proxy objectives: Matching intermediate training signals relies on the fragile assumption that similar gradients or parameter states will produce the same final outcome as the full dataset.The passage identifies this assumption as a source of an optimization gap.

4. Method

Inf-Match distills data by matching the influence of synthetic samples to the full dataset’s effect on converged model parameters. It uses a differentiable, efficient estimator based on unrolled optimization and first-order approximation, producing outcome alignment without retraining.

  • Influence Definition: Inf-Match defines removal influence as the final-parameter difference caused by excluding a sample or subset from the training dataset.Addition influence analogously measures the final-parameter shift caused by introducing external samples.
  • Data Influence Estimator: The influence estimator avoids convexity assumptions and inverse-Hessian products by unrolling optimization dynamics and applying a first-order Taylor approximation.It leverages checkpoints along the SGD trajectory and efficiently approximates Hessian-gradient products.
  • Data Influence Estimator: O(p) is the estimator’s computational complexity, where p denotes the number of model parameters.The method also provides a theoretical approximation-error bound and supports efficient implementation in deep-learning frameworks.
  • Error Bound: The error bound scales polynomially with training steps T rather than exponentially, while depending jointly on the Lipschitz constant ℓ and gradient norm g.Empirical results indicate that estimated influence closely correlates with exact influence despite the bound being worst-case.
  • Influence Matching Objective: Inf-Match learns a synthetic set S by minimizing the residual between synthetic-data addition influence and real-data removal influence.This objective aligns parameters trained on S with those trained on D, directly targeting final-model outcomes without retraining.
  • Inf-Match Algorithm: The pipeline initializes S with real images and soft labels, then jointly updates images and labels using sampled real and synthetic minibatches.Random minibatch estimation improves efficiency while preserving unbiased gradient estimates.

5. Experiments · 5.1. Experimental Settings

The experiments evaluate Inf-Match on image-classification and vision-language datasets, comparing it with established distillation and selection baselines. The settings vary compression levels, architectures, and training configurations, with repeated evaluations for robustness.

  • 5. Experiments: The evaluation covers image classification and vision-language datasets, followed by experimental results and an ablation study of key method components.
  • 5.1. Experimental Settings: Baselines include DD, GM, MTT, DM, IDM, DATM, TESLA, G-VBSM, NCFM, Herding, and random selection.
  • 5.1. Experimental Settings: Experiments default to ConvNet architectures with three convolutional blocks, or four for Tiny ImageNet, plus a linear classifier.
  • 5.1. Experimental Settings: Cross-architecture comparisons use LeNet, AlexNet, VGG11, and ResNet18, while vision-language tasks use trainable Vision Transformer and frozen BERT backbones.
  • 5.1. Experimental Settings: Image-classification experiments use IPC values of 1, 10, and 50, whereas vision-language experiments use synthetic datasets containing 100, 500, or 1000 samples.
  • 5.1. Experimental Settings: Each experiment learns from synthetic data, trains a model on it, and evaluates performance on the real test set using PyTorch.
  • 5.1. Experimental Settings: Training uses batch size 50 and SGD-M with momentum 0.9; synthetic images and soft labels use learning rates 50.0 and 7.0, respectively.
  • 5.1. Experimental Settings: Each experiment is independently repeated 10 times to support robust and fair evaluation.

5.2. Image Classification

Inf-Match achieves leading image-classification results across CIFAR-10, CIFAR-100, and Tiny-ImageNet configurations, supporting its outcome-alignment strategy. It also shows strong cross-architecture generalization on CIFAR-100 and surpasses selection-based methods.

  • Benchmark results: Inf-Match consistently achieves leading results across CIFAR-10, CIFAR-100, and Tiny-ImageNet for every tested Images Per Class configuration.The results are presented as evidence for the effectiveness and robustness of the outcome-alignment strategy.
  • Benchmark results: 72.5% at IPC=10 is Inf-Match’s highest reported CIFAR-10 accuracy in the provided passage.The passage states that Inf-Match achieves the highest accuracy in all CIFAR-10 settings.
  • Benchmark results: 31.5% at IPC=10 and 33.8% at IPC=50 on Tiny-ImageNet exceed NCFM by approximately 4.7% and 4.2%, respectively.Tiny-ImageNet shows the most pronounced gains among the reported benchmarks.
  • Cross-architecture generalization: Cross-architecture evaluation on CIFAR-100 with IPC=50 shows that distillation schemes significantly surpass random and Herding selection methods.The assessment evaluates the cross-network generalization capability of synthetic data generated by the method.

5.3. Vision-language Datasets

The vision-language experiments evaluate dataset distillation on Flickr30K using pretrained unimodal encoders and compare Inf-Match with BTM. Inf-Match consistently outperformed competing methods, including on image-to-text and text-to-image retrieval examples.

  • 5.3. Vision-language Datasets: Flickr30K contains 31,783 images of daily activities and scenes, each paired with five textual descriptions.Experiments use Normalizer-free ResNet as the vision encoder and BERT as the text encoder.
  • 5.3. Vision-language Datasets: The model uses trainable randomly initialized linear projections after each encoder and a loss encouraging similarity between paired images and texts.BTM is selected as the baseline and uses a bi-trajectory matching loss for vision-language dataset distillation.
  • 5.3. Vision-language Datasets: 7.4% on image-to-text retrieval tasks with 200 samples exceeded DATM at 1.3%.The passage reports this as an example of Inf-Match outperforming the next-best method.
  • 5.3. Vision-language Datasets: 14.6% on the text-to-image setting with 500 samples exceeded DATM at 14.1%.Inf-Match was reported as leading the comparison in this setting.

5.4. Ablation study

The ablation on CIFAR-100 (IPC=50) shows that adding Inf-Match components progressively improves accuracy, reaching 57.4% when fully integrated. The complete method outperforms DATM and NCFM, supporting the contribution of its outcome-alignment components.

  • Component ablations: On CIFAR-100 (IPC=50), Real-data Initialization raises accuracy to 53.7%, Sampling Schedule to 55.0%, and Learnable Labels to 54.6%.These results show the effect of sequentially incorporating individual components.
  • Full configuration: 57.4% is the final accuracy when all Inf-Match components are integrated.The full configuration produces the strongest ablation result.
  • Baseline comparison: 55.0% for DATM and 54.7% for NCFM are surpassed by the fully integrated method.The ablation study selects NCFM, DATM, DM, GM, and MTT as baselines.

5.5. Learning process visualization

Figure 5 compares learning processes from synthetic data, showing that Inf-Match directly optimizes the original problem rather than using a heuristic proxy. Although it converges more slowly than MTT, it ultimately achieves significantly better performance.

  • Learning process visualization: Figure 5 visualizes the learning process from synthetic data during algorithm execution.The visualization compares performance as iterations progress.
  • Learning process visualization: Inf-Match directly optimizes the original problem instead of relying on the heuristic-based proxy task used by MTT.The comparison is made against MTT [7].
  • Learning process visualization: Despite a slower convergence rate than MTT, Inf-Match ultimately achieves significantly better performance.This conclusion is based on performance trajectories as iterations progress.

5.6. Feature space visualization

Feature-space visualization of CIFAR-100 Wolf samples at IPC=10 shows that Inf-Match produces a more balanced synthetic distribution than DM and MTT, whose samples are overly concentrated near distribution edges.

  • Feature space visualization: At IPC=10, the CIFAR-100 Wolf feature-space visualization compares Inf-Match with DM and MTT.White scatter points denote projections of synthetic samples in the embedding space.
  • Feature space visualization: DM’s synthetic samples tend to concentrate in high-density regions when matching feature distributions at small IPC settings.This concentration occurs near the distribution’s edges.
  • Feature space visualization: Inf-Match produces a more balanced representation than MTT and DM in the visualized feature space.The comparison is presented as an illustration of why the method performs well for dataset distillation.

6. Conclusion

Inf-Match reframes dataset distillation around rigorous outcome alignment, using a differentiable sample influence estimator to quantify real and synthetic data contributions to final model parameters. Experiments show superior performance over state-of-the-art distillation methods.

  • Inf-Match replaces heuristic process alignment with rigorous outcome alignment as the dataset distillation objective.
  • The fully differentiable sample influence estimator efficiently and accurately quantifies real and synthetic datasets’ contributions to final model parameters.
  • Extensive experiments demonstrate superior performance over state-of-the-art dataset distillation methods.
Loading 2607.16859v1…