PaperlayerView exact revision on arXiv

Source-linked AI summary

Dataset Distillation via Factorization

Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, Xinchao Wang

arXiv:2210.16774v1cs.CVcs.LG

TL;DR

Conventional dataset distillation overlooks relationships among synthetic samples, limiting information efficiency and potentially causing information loss or overfitting. HaBa factorizes datasets into hallucination networks and bases, adds adversarial contrastive constraints, and serves as a plug-and-play strategy for existing DD baselines. It improves downstream performance while using substantially fewer compressed parameters and improving cross-architecture generalization.

  • Problem

    Conventional DD treats synthetic samples independently, ignoring inter-instance relationships and risking information loss and overfitting in downstream models.

  • Method

    HaBa factorizes datasets into hallucination networks and bases, combines them to generate samples, and uses adversarial contrastive constraints to increase diversity.

  • Results

    Using only 35% of memory for storage, HaBa significantly improves downstream models and remains compatible with different DD configurations.

  • Takeaways & Limitations

    HaBa provides a versatile factorization strategy that improves the data efficiency and downstream utility of distilled datasets.

  • Takeaways & Limitations

    Online pairwise hallucinator-basis combination slightly increases training time and GPU-memory cost, and HaBa may inherit baseline limitations.

Abstract

from arXiv · show

In this paper, we study \xw{dataset distillation (DD)}, from a novel perspective and introduce a \emph{dataset factorization} approach, termed \emph{HaBa}, which is a plug-and-play strategy portable to any existing DD baseline. Unlike conventional DD approaches that aim to produce distilled and representative samples, \emph{HaBa} explores decomposing a dataset into two components: data \emph{Ha}llucination networks and \emph{Ba}ses, where the latter is fed into the former to reconstruct image samples. The flexible combinations between bases and hallucination networks, therefore, equip the distilled data with exponential informativeness gain, which largely increase the representation capability of distilled datasets. To furthermore increase the data efficiency of compression results, we further introduce a pair of adversarial contrastive constraints on the resultant hallucination networks and bases, which increase the diversity of generated images and inject more discriminant information into the factorization. Extensive comparisons and experiments demonstrate that our method can yield significant improvement on downstream classification tasks compared with previous state of the arts, while reducing the total number of compressed parameters by up to 65\%. Moreover, distilled datasets by our approach also achieve \textasciitilde10\% higher accuracy than baseline methods in cross-architecture generalization. Our code is available \href{https://github.com/Huage001/DatasetFactorization}{here}.

1 Introduction

Conventional dataset distillation can lose information by treating synthetic samples independently. HaBa factorizes distilled data into hallucinators and bases, using their combinations and adversarial constraints to improve efficiency, diversity, and downstream performance.

  • Motivation: Conventional DD treats synthetic samples independently, ignoring inter-instance relationships and risking information loss and overfitting in downstream models.The issue is especially pronounced when few distilled samples represent models with many parameters.
  • Dataset factorization: HaBa factorizes a dataset into hallucination networks and bases, whose arbitrary pairings can represent |H|×|B| images.Hallucinators take bases as input and output hallucinated images while learning relationships among original samples.
  • Adversarial constraints: Adversarial contrastive constraints promote diversity among images generated by different hallucinators from a common basis.The objective minimizes image correlation while an adversary maximizes it, increasing useful information.
  • Results: HaBa is compatible with existing DD training objectives and yields consistent improvements while reducing compressed parameters by up to 65%.The method is presented as a plug-and-play strategy across DD baselines.
  • Contributions: HaBa introduces dataset factorization and adversarial contrastive objectives as its two main contributions.Together, they target the representation and information efficiency of distilled datasets.

2 Related Works

Prior dataset distillation methods optimize synthetic data through training effects or distribution matching, but relatively few address the data efficiency of distilled samples. HaBa instead factorizes datasets into hallucination networks and bases, while related IDC uses a complementary parameterization strategy.

  • Dataset distillation: Dataset distillation compresses data by optimizing a smaller synthetic dataset to replace the original dataset for downstream training.This differs from coreset selection, which directly selects samples from the raw dataset.
  • Prior approaches: Existing DD methods optimize synthetic images through bilevel objectives, learnable labels, gradient matching, or distribution matching.Examples include MMD-based constraints and feature-space alignment between synthetic and real datasets.
  • Data efficiency: Few prior works specifically improve the data efficiency of distilled samples, and augmentation operations such as crop, flip, scale, and rotation do not encode target-dataset information.HaBa addresses this gap through factorization into hallucination networks and bases.
  • Related efficient parameterization: IDC and HaBa both improve synthetic-parameter efficiency, but IDC can be viewed as a special HaBa case and the techniques are described as orthogonal.IDC uses parameter-free upsampling with smaller bases, whereas HaBa focuses on hallucinator-basis factorization.

3 Methods

HaBa reformulates dataset distillation as factorization into bases and hallucination networks, whose combinations generate training images. Its adversarial contrastive training promotes diversity while retaining compatibility with existing distillation objectives and end-to-end optimization.

  • 3 Methods: HaBa decomposes dataset distillation into bases and hallucination networks, redefining the task as a hallucinator-basis factorization problem.Traditional methods treat synthetic samples independently, whereas HaBa models relationships among samples through shared factorized components.
  • 3 Methods: Each basis–hallucinator pair produces an image online, with the basis label reused as the generated sample’s label.The i-th basis is sent to the j-th hallucinator to create the training pair (x̃_ij, ỹ_ij).
  • 3 Methods: Bases need not match original image dimensions, while hallucinators transform them into images with the original spatial and channel dimensions.This makes the factorized basis representation more flexible than conventional synthetic samples.
  • 3 Methods: Hallucinators use an encoder, affine feature transformation, and decoder to generate images from bases.The transformation applies element-wise scale σ and shift μ to encoded features before decoding.
  • 3 Methods: Adversarial contrastive constraints train hallucinators and bases to increase diversity among images generated from a common basis.The feature extractor minimizes inter-image divergence, while the factorized components maximize it; supervised contrastive and task losses provide task-specific signals.
  • 3 Methods: The factorization is compatible with multiple dataset-distillation losses and is optimized iteratively through differentiable, end-to-end updates.The default trajectory-matching loss compares updates using synthetic and real data, while gradients reach sampled hallucinators and bases.

4 Experiments

Experiments evaluate HaBa across standard benchmarks, DD baselines, architectures, continual learning, and design choices. Results show improved data efficiency, cross-architecture performance, and storage efficiency, with several ablations clarifying useful configurations.

  • Experimental Setup: Experiments use SVHN, CIFAR10, and CIFAR100 image-classification benchmarks with 32 × 32 RGB images.The datasets contain 10, 10, and 100 classes respectively.
  • Comparisons: HaBa is compared with state-of-the-art DD methods at 1, 10, and 50 images per class using storage-matched bases and hallucinators.The comparison protocol sets bases per class to IPC minus 1 when IPC exceeds 1, accounting for hallucinator storage.
  • Comparisons: 70.27% test accuracy versus 65.92% for MTT illustrates HaBa’s visual and quantitative advantage in factorized results.The reported comparison is from CIFAR10 visualization results.
  • Comparisons: HaBa consistently improves DC, DM, and MTT pipelines on CIFAR10, supporting factorization as a general strategy for DD data efficiency.The evaluations train and test on ConvNet while keeping storage costs approximately comparable.
  • Cross-Architecture Performance: 17.57% is the maximum reported cross-architecture accuracy gain when ConvNet-trained synthetic datasets are evaluated on ResNet, VGG, and AlexNet.The authors attribute the improvement to increased data diversity and informative feature capture.
  • Ablation Studies: Single-channel bases reduce memory cost by nearly 2/3 without substantially hurting performance, while three times more bases can improve accuracy at fixed memory.The latter effect is especially strong when BPC is small.
  • Ablation Studies: More hallucinators help when BPC is small, but performance stops improving beyond 10 hallucinators at BPC values of 10 or 50.Sparse hallucinator sampling can make joint optimization more difficult when the hallucinator set is large.

5 Conclusions, Limitations, and Future Works

The paper concludes that HaBa improves dataset distillation through hallucinator-basis factorization and adversarial contrastive constraints. It also acknowledges modest training-resource overhead, inherited baseline limitations, and class-wise factorization as future work.

  • Conclusions: HaBa uses hallucinators to encode relations among original samples, while adversarial contrastive constraints diversify the knowledge captured by different hallucinators.The method improves downstream models trained on synthetic datasets using 35% of the memory cost.
  • Limitations: HaBa’s online pairwise hallucinator-basis combinations slightly increase training time and GPU memory compared with its baseline.The authors note that lightweight hallucinators reduce this overhead but do not eliminate it.
  • Limitations: When image counts are already large, further increasing them may provide limited performance gains because HaBa can inherit baseline-method limitations.This limitation concerns the scope inherited from the underlying DD baseline.
  • Future Works: Future work could explore class-wise hallucinator sharing and more advanced dataset factorization.The proposed direction allows sharing across selected classes rather than uniformly across all classes.

Checklist

The checklist records affirmative responses for contribution scope, limitations, reproducibility materials, training details, repeated experiments, compute reporting, and asset licensing. Several ethics and human-subject items are marked not applicable.

  • Author Checklist: The authors confirm that they described their contributions, scope, and limitations.The limitations discussion is referred to in the supplement.
  • Reproducibility: The paper reports code, data, and reproduction instructions, along with training details and hyperparameter-selection information.The checklist points readers to the supplement and Section 4.1 for these materials.
  • Resources and Assets: The authors report compute resources and cite existing assets, licenses, and newly included assets.The checklist refers to Section 4.1 for compute details and confirms asset documentation.
  • Ethics: Potential negative societal impacts and human-subject requirements are marked not applicable.The checklist also marks participant risks, compensation, consent, and personally identifiable information items as not applicable.

Appendix A Algorithm Details

The appendix algorithm takes an original dataset and HaBa hyperparameters, then jointly samples, combines, and updates hallucinators, bases, and a feature extractor to produce distilled components.

  • Algorithm Overview: Algorithm 1 outputs a set of hallucinators H and bases B as the distilled result.The pipeline is designed to equip the distilled dataset with downstream performance similar to the original data.

Appendix B More Results

Appendix experiments show that HaBa improves performance across datasets, modalities, corruption settings, and cross-architecture tests, while configuration choices trade off accuracy, diversity, storage, and downstream speed.

  • Low-Resolution Data: HaBa consistently improves over baselines on MNIST and FashionMNIST, including settings where dataset-distillation performance is largely saturated.The comparison is reported in Table 7, with IPC, BPC, and compression ratio as evaluation quantities.
  • Cross-Architecture Generalization: HaBa outperforms the baseline in almost all ImageNet-subset experiments and achieves superior performance in every cross-architecture setting.The exceptions occur in several same-architecture experiments with small IPC and BPC.
  • Configuration and Efficiency: Deeper hallucinators generally improve performance, but one nonlinear block offers the preferred speed–accuracy trade-off because additional depth adds nonnegligible downstream latency.Under small and medium budgets, depth 2 nearly consistently outperforms depth 1; deeper networks are recommended only when training overhead is acceptable.
  • Configuration and Efficiency: Configuration effects depend on storage budget: one-channel bases help at small budgets, whereas three-channel bases and more hallucinators become preferable at larger budgets.At larger budgets, performance becomes less sensitive to configuration, while more hallucinators can slightly improve diversity and results.
  • Factorization Analysis: Independent hallucinator networks outperform shared encoder–decoder components, while t-SNE visualizations indicate that factorized reconstructions represent the original distribution with less information loss.The visualization comparison uses equal total parameter counts, supporting improved data efficiency under the same storage budget.
  • Speech Domain: HaBa yields consistent downstream accuracy gains on Mini Speech Commands, extending the factorization approach beyond image data.The experiments use 8,000 audio clips across 8 command classes and adopt IDC as the baseline.
  • Robustness to Corruption: HaBa remains more robust than the baseline across all CIFAR10-C corruption levels, while the distilled-versus-original accuracy gap narrows as corruption severity increases.CIFAR10-C evaluates mean accuracy across 19 corruption types and five severity levels.
Loading 2210.16774v1…