PaperlayerView exact revision on arXiv

Source-linked AI summary

Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment

Jiawei Du, Xin Zhang, Juncheng Hu, Wenxin Huang, Joey Tianyi Zhou

arXiv:2409.17612v3cs.LGcs.CV

TL;DR

Isolated synthesis can be efficient but risks losing representativeness and diversity, and random initialization contributes little to diversity. DWA dynamically adjusts teacher weights for each synthetic-data batch to promote diversity. It outperforms SRe2L across ImageNet-1K settings and validation models, with a reported 16.6% advantage at one setting.

  • Problem

    Individually synthesized instances can miss holistic data characteristics, while random initialization contributes only marginally to diversity.

    Source: §1 · p. 2

  • Method

    DWA dynamically and directionally adjusts teacher weights for each synthetic-data batch to promote diversity while retaining informative supervision.

    Source: §1 · p. 2

  • Results

    DWA outperforms SRe2L across ImageNet-1K ipc settings and validation models, by 16.6% at ipc = 10 using ResNet-18.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    DWA enhances diversity in isolated dataset synthesis with less than 0.1% additional computation.

    Source: §1 · p. 2

    Increasing the variance of the random distribution used for weight adjustment can introduce unexpected noise and bottleneck overall performance.

    Source: §6 · p. 9

Abstract

from arXiv · show

The sharp increase in data-related expenses has motivated research into condensing datasets while retaining the most informative features. Dataset distillation has thus recently come to the fore. This paradigm generates synthetic datasets that are representative enough to replace the original dataset in training a neural network. To avoid redundancy in these synthetic datasets, it is crucial that each element contains unique features and remains diverse from others during the synthesis stage. In this paper, we provide a thorough theoretical and empirical analysis of diversity within synthesized datasets. We argue that enhancing diversity can improve the parallelizable yet isolated synthesizing approach. Specifically, we introduce a novel method that employs dynamic and directed weight adjustment techniques to modulate the synthesis process, thereby maximizing the representativeness and diversity of each synthetic instance. Our method ensures that each batch of synthetic data mirrors the characteristics of a large, varying subset of the original dataset. Extensive experiments across multiple datasets, including CIFAR, Tiny-ImageNet, and ImageNet-1K, demonstrate the superior performance of our method, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense. Our code is available at https://github.com/AngusDujw/Diversity-Driven-Synthesis.https://github.com/AngusDujw/Diversity-Driven-Synthesis.

1 Introduction

Dataset distillation can reduce data-related costs, but isolated synthesis raises concerns about whether each instance remains representative and diverse. The paper analyzes this issue and proposes dynamic, directed weight adjustment to improve diversity and performance with negligible added computation.

1 Introduction

  • Dataset distillation synthesizes a compact dataset to retain informative features and aim for performance comparable to training on the original data, unlike methods that select original examples.The motivation includes reducing data-related costs and supporting more efficient storage and processing.Source: §1 · p. 1
  • Random initialization contributes only marginally to diversity in isolated synthesis, while the BN loss plays a practical role in enhancing it.The analysis identifies the BN loss variance regularizer as promoting diversity and its mean regularizer as constraining it.Source: §1 · p. 2
  • SRe2L individually synthesizes instances to reduce GPU memory and computation, but isolated synthesis can miss holistic data characteristics needed for representativeness and diversity.The paper notes that individual synthesis makes optimization parallelizable and GPU-memory use flexible.Source: §1 · p. 2
  • DWA combines a decoupled emphasis on the BN loss variance term with dynamic, directed teacher-weight perturbations to promote diversity while retaining informative supervision.The perturbation direction reflects characteristics of a large subset that varies with each synthetic-data batch; added computation is below 0.1%.Source: §1 · p. 2
  • The method is evaluated on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K, with the authors reporting superior performance and enhanced diversity.The paper highlights ImageNet-1K as a large-scale evaluation.Source: §1 · p. 2

2 Preliminaries

The preliminaries formulate dataset distillation as learning a compact synthetic set whose trained model approaches the original-data model's test performance. They also describe SRe2L's instance-wise synthesis and the BN loss's normalization and diversity roles.

Notation and Objective

  • The formulation maps real and synthetic samples into latent space with the feature extractor and denotes the full network as a feature extractor followed by a classifier.The feature extractor and classifier use the corresponding parameters from θ.Source: §2 · p. 3
  • Dataset distillation seeks a compact synthetic set whose trained model achieves test performance comparable to a model trained on the original dataset.The objective is expressed as approximate equality between expected losses on data drawn from the natural distribution.Source: §2 · pp. 2–3

Synthesizing S

  • SRe2L optimizes each synthetic instance using a BN loss whose coefficient is λ, aligning synthetic and real data normalization distributions while also contributing to synthetic-set diversity.The passage identifies the diversity role as an additional aspect of the BN loss beyond its normalization objective.Source: §2 · p. 3

3 Methodology

The methodology analyzes how batch-normalization loss and weight perturbations affect diversity when synthetic instances are optimized individually. It develops Directed Weight Adjustment (DWA) to increase diversity while limiting noise during synthesis.

3.1 Batch Normalization Loss Enhances Diversity of S

  • The BN loss's variance component encourages synthetic instances to differ, while its mean component aligns them with the representative centroid of the original data in latent space.For individually optimized instances, diversity is determined by the variance regularizer; simply increasing the shared coefficient also emphasizes the mean regularizer, so the method decouples the variance coefficient.Source: §3.1 · p. 4
  • Real training instances initialize synthesis because Gaussian pixel-space initialization can cluster synthetic instances in latent space instead of distributing them uniformly around the representative instance.The initialization choice is inspired by MTT [1].Source: §3.1 · p. 4

3.2 Random Perturbation on θT Helps Improve Diversity

  • Random perturbations to θT increase the variance of the resulting synthetic dataset compared with synthesis without that perturbation.The analysis links perturbation variance to the resulting change in synthetic instances.Source: §3.2 · p. 5

3.3 Directed Weight Adjustment on θT

  • DWA directs weight perturbations using a small random subset of the training data, decreasing loss on the remaining data while increasing loss on the subset.The method uses the subset to guide the adjustment and avoid unanticipated noise.Source: §3.3 · p. 5

4 Experiments

Experiments compare DWA with distillation baselines on CIFAR-10/100, Tiny-ImageNet, and ImageNet-1K, then test its diversity mechanism, perturbation settings, and cross-architecture generalization.

4 Experiments

  • Experiments use Top-1 validation accuracy after training models from scratch on distilled data, with ResNet-18 as the default distillation backbone unless otherwise specified.Comparisons span CIFAR-10/100, Tiny-ImageNet, and ImageNet-1K, using multiple network architectures and seven distillation baselines.Source: §4 · p. 6

4.1 Results & Discussions

  • DWA surpasses SRe2L across all considered CIFAR-10/100 settings, with accuracy improvements greater than 5% and 8%, respectively, and over 10% performance enhancement over MTT at ipc = 50 on CIFAR-100.The CIFAR comparisons use the Table 1 settings and baselines described in the experiment setup.Source: §4.1 · p. 6, Table 1 · p. 6
  • For goldfish-class distilled images, DWA produces visibly greater instance diversity than SRe2L, illustrating a broader range of captured features.Figure 2 presents SRe2L in panel (a) and DWA in panel (b).Source: Figure 2 · p. 7
  • On Tiny-ImageNet and ImageNet-1K, DWA significantly outperforms SRe2L across all ipc settings and validation models, including a 16.6% advantage at ImageNet-1K ipc = 10 with ResNet-18.Table 2 covers the large-scale comparisons; MTT results marked † use a 10-class ImageNet-1K subset.Source: Table 2 · p. 6, §4.1 · p. 7

4.2 Ablation Study

  • Emphasizing the decoupled variance term outperforms jointly emphasizing mean and variance for both SRe2L and DWA, with the reported optimal decoupled coefficient at 0.11.The coefficient analysis varies λvar on CIFAR-100 with ResNet-18; Figure 3 shows the compared conditions.Source: §4.2 · p. 7, Figure 3 · p. 7
  • The directed adjustment ablation shows random weight adjustment can reduce performance versus no adjustment at ipc = 10, while diversity is more effective as synthetic instances increase, as reflected at ipc = 50.Table 3 compares no, random, and directed weight adjustment on CIFAR-100 across architectures.Source: §4.2 · p. 8, Table 3 · p. 8
  • The perturbation grid varies step count K and magnitude ρ; experiments use K = 12 and ρ = 15e−3 across datasets, with authors noting tuning may suit particular datasets and networks.The reported grid indicates a positive impact from perturbation without meticulous manual parameter tuning.Source: §4.2 · p. 8, Figure 5 · p. 8
  • DWA-synthesized CIFAR-100 data generalizes across architectures; at ipc = 50, it beats SRe2L by 14.2%, 15.8%, and 17.6% on ShuffleNetV2, EfficientNet-B0, and ConvNet-128, respectively.These three architectures were not involved in the data synthesis phase; Table 4 reports cross-architecture performance for distilled sets using ResNet-18 and ConvNet-128.Source: Table 4 · p. 8, §4.2 · p. 9

5 Related Works

Dataset distillation synthesizes a compact dataset through bi-level optimization of model parameters and image pixels, with methods differing in their matching objectives and use of diversity cues. SRe2L uses batch-normalization statistics to scale distillation to large datasets, while other approaches explicitly target diversity.

5 Related Works

  • Table 5 reports cross-architecture performance for ImageNet-1K distilled using ResNet-18.Source: Table 5 · p. 9
  • Dataset distillation uses performance, gradient, distribution, and trajectory matching, while SRe2L uses a three-step paradigm and BN running statistics to extend distillation to ImageNet-1K.The cited work describes DD as bi-level optimization over model parameters and image pixels; Dream uses cluster centers, and SRe2L uses representation variance as a diversity cue.Source: §5 · p. 9
  • Dream uses cluster centers to induce synthesis, whereas SRe2L relies on variance of representations in pretrained weights for diversity.Source: §5 · p. 9

6 Conclusion

The paper concludes that Directed Weight Adjustment (DWA) can enhance diversity in dataset distillation by customizing weight adjustments for each synthetic mini-batch. It also identifies random-distribution sampling as a limitation because higher variance can add noise and bottleneck performance.

6 Conclusion

  • DWA customizes weight adjustments for each synthetic mini-batch, and experiments on ImageNet-1K particularly confirm its superior performance.The paper reports that random initialization contributes minimally to ensuring each instance captures unique knowledge from the original dataset.Source: §6 · p. 9
  • Higher variance in the random distribution used to adjust weights can introduce unexpected noise and bottleneck overall performance.Source: §6 · p. 9
  • Future work could synthesize instances sequentially so later instances consciously distinguish themselves from earlier ones.Source: §6 · p. 9

A.1 Minimizing Lmean and Lvar can be contradictory

The appendix shows that minimizing the mean and variance regularizers can produce contradictory gradient directions for some synthetic instances. Which instances face this conflict depends on the sign of a product of mean and variance mismatches.

A.1 Minimizing Lmean and Lvar can be contradictory

  • The appendix derives gradients for Lmean and Lvar and substitutes them to test whether their minimizing directions can oppose one another for an instance.Source: §A.1 · p. 14
  • When R > 0, instances below the synthetic-set mean face contradictory objectives; when R < 0, instances above that mean do.R is the product of the synthetic-versus-target mean and variance differences.Source: §A.1 · p. 14

A.2.1 Hyper-parameter Settings

The hyper-parameter settings are listed separately for CIFAR-10/100, Tiny-ImageNet, and ImageNet-1K. The method maintains consistency with SRe2L for a fair comparison.

A.1 Hyper-parameter Settings

A.2.2 Feature Distance Calculation

The appendix defines feature-distance measurement for diversity and reports DWA results on a BN-adapted ViT, continual learning, and generation overhead.

A.2.2 Feature Distance Calculation

  • Dfea measures distilled-dataset diversity using synthetic-instance latent representations from the last convolutional layer.The appendix describes the calculation as class-wise feature distance.Source: §A.2.2 · p. 15

A.2.3 Generalization to Vision Transformer-based Models

  • For DeiT-Tiny on ImageNet-1K, DWA with the ViT-BN workaround outperforms the baseline.The workaround replaces LayerNorm with BN and adds BN between feed-forward linear layers; DWA is not directly applicable to models without BN.Source: §A.2.3 · pp. 15–16, Table 9 · p. 16

A.2.4 Application to Downstream Tasks

  • DWA significantly outperforms SRe2L across all five CIFAR-100 class-incremental stages under a 20-images-per-class memory constraint.The authors report superior knowledge retention as new classes are introduced.Source: §A.2.4 · p. 16, Table 10 · p. 16

A.2.5 Computational Overhead of Distillation

  • DWA incurs a 7.32% increase in computational overhead for ResNet-18 on CIFAR-100; its K = 12 extra forward-backward passes are small beside 1,000 generation iterations per ipc.The added propagations arise from directed weight perturbation before generating each ipc.Source: §A.2.5 · p. 16, Table 11 · p. 16

Related papers

Loading 2409.17612v3…