PaperlayerView exact revision on arXiv

Source-linked AI summary

PSM: Dataset Distillation Based on Precise Statistical Matching by Difficulty

Hongxu Ma, Guang Li, Shijie Wang, Dongzhan Zhou, Suorong Yang, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui Wang

arXiv:2609.34299v1cs.CV

TL;DR

Decoupled dataset distillation uses global BN statistics that capture average features but do not provide batch-specific difficulty supervision. PSM groups samples by difficulty and adjusts teacher statistics and distilled-sample initialization; experiments report broader difficulty coverage and improved student performance in most evaluated settings.

  • Problem

    Global BN statistics mainly capture average feature distributions, while their batch-index-independent target cannot explicitly provide batch-specific difficulty supervision.

    Source: §1 · p. 1, §4.1 · p. 4

  • Method

    PSM ranks and groups original images by GPS, then uses SUA for group-specific teacher BN statistics and ISS to initialize distilled data from the corresponding group.

    Source: §1 · p. 2

  • Results

    PSM broadens distilled-sample difficulty and improves performance in most evaluated settings; on CIFAR-10 with IPC = 10, ResNet50 achieves a Top-1 accuracy of 52.7%, outperforming FADRM by 8.9%.

    Source: §1 · p. 2, §5.2.1 · p. 6

  • Takeaways & Limitations

    PSM better preserves the difficulty distribution of the original data while improving student performance across the reported experiments.

    Source: §6 · p. 9

    Existing decoupled methods’ batch-index-independent statistical target cannot explicitly provide batch-specific difficulty supervision.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Dataset distillation (DD) condenses a large original dataset into a small distilled dataset with high training utility. Decoupled statistical matching methods substantially reduce distillation time and memory overhead while achieving strong performance. However, they typically supervise all distilled samples using running statistics estimated from the entire original dataset. These statistics mainly capture the average feature distribution while overlooking differences in sample difficulty, limiting their ability to characterize the difficulty structure of the original data. To address this issue, we propose Precise Statistical Matching (PSM) by difficulty. After pretraining, PSM uses the Global Precision Score (GPS) to estimate image difficulty, ranks the samples within each class, and partitions each class into IPC (images per class) difficulty groups. During distillation, Statistics Updated Again (SUA) updates the teacher's batch normalization (BN) running statistics through forward passes on original samples from each group, providing difficulty-specific supervision for the corresponding distilled batch. Meanwhile, Initial Sample Screening (ISS) initializes distilled samples using original images from the corresponding difficulty group, providing an effective starting point for precise matching. Experiments across multiple datasets and model architectures demonstrate that PSM broadens the difficulty range of distilled samples and improves downstream performance in most evaluated settings. Code will be released.

1 INTRODUCTION

Dataset distillation can reduce the costs of training on large datasets, but global BN statistics in decoupled methods capture average features and overlook difficulty differences. PSM groups samples by difficulty and uses group-specific supervision and initialization; experiments report broader difficulty coverage and improved performance.

1 INTRODUCTION

  • Global BN statistics mainly capture average feature distributions, smoothing differences between easy and hard samples and limiting representation of the original difficulty range.The introduction links this limitation to the nearly constant difficulty observed in data distilled by FADRM.Source: §1 · p. 1, Figure 1 · p. 2
  • GPS ranks original images by difficulty using statistics at teacher BN inputs, then partitions each class into IPC difficulty groups.A higher GPS indicates greater deviation from the average feature distribution and greater difficulty.Source: §1 · p. 2
  • SUA updates teacher BN statistics using original samples from the corresponding group, while ISS initializes distilled data with images from that group.The two components provide group-specific supervision and starting points for matching.Source: §1 · p. 2
  • Across multiple datasets and model architectures, PSM broadens the difficulty range of distilled samples and improves final performance.The experiments report these outcomes without claiming gains in every setting.Source: §1 · p. 2

2 RELATED WORK

Related work frames dataset distillation as compressing a large dataset while retaining training utility, and surveys several families of approaches.

2 RELATED WORK

  • Dataset distillation compresses a large dataset into a small set, aiming to reduce storage and training costs while retaining comparable performance.Source: §2 · p. 2
  • The related-work taxonomy includes gradient matching, distribution matching, and trajectory matching methods.Source: §2 · p. 2
  • The taxonomy also includes decoupled and generative distillation, with further related work discussed in Appendix A.Source: §2 · p. 2

3 PRELIMINARIES

The preliminaries define dataset distillation by its goal of matching full-dataset training performance under a sample budget. They then describe decoupled distillation and the BN statistics that guide its optimization.

3.1 DATASET DISTILLATION

  • Dataset distillation constructs a set of M samples whose trained models achieve performance comparable to models trained on the original set.Distilled-set quality is typically measured by student test performance on the original test set.Source: §3.1 · p. 3

3.2 DECOUPLED DATASET DISTILLATION

  • Decoupled distillation separates teacher pretraining from data optimization, keeping the pretrained teacher fixed while optimizing distilled data.The teacher’s BN running statistics accumulated during pretraining guide distillation.Source: §3.2 · p. 3
  • During data optimization, classification loss enforces semantic consistency while BN statistics matching aligns distilled-batch statistics with teacher running statistics.The stated loss definitions identify cross-entropy, the statistics-matching weight, and the BN layers and channel statistics involved.Source: §3.2 · p. 3
  • After distillation, teacher class-probability distributions provide soft labels that preserve semantic relationships across classes.The student is then trained on distilled data and these labels for richer supervision.Source: §3.2 · p. 3

3.3 BATCH NORMALIZATION

  • BN normalizes features using batch means and variances, while exponential moving averages update its running statistics during training.The update parameter ρ controls the statistics update; larger values give more weight to the current batch.Source: §3.3 · p. 3

4 METHOD

PSM replaces homogeneous global-statistics supervision with difficulty-specific groups, using GPS to rank samples, SUA to adapt BN statistics, and ISS to initialize distilled data from matching groups. The supplied method passages report that GPS tracks teacher classification difficulty and ISS improves final distillation performance.

4.1 LIMITATIONS OF DECOUPLED DATASET DISTILLATION

  • Global BN statistics give every distilled batch the same target, so supervision does not explicitly encode batch-specific difficulty and makes the original difficulty structure difficult to preserve.Existing decoupled methods use the pretrained teacher’s global BN statistics as the target for every batch.Source: §4.1 · pp. 3–4

4.2 DATASET DISTILLATION BASED ON PRECISE STATISTICAL MATCHING BY DIFFICULTY

  • The PSM pipeline applies GPS ranking, group-specific SUA supervision, and matching-group ISS initialization to corresponding batches.Different batches consequently match statistics for progressively varied difficulty groups.Source: §4.2 · p. 4, Figure 2 · p. 4

4.2.1 GPS: GLOBAL PRECISION SCORE

  • PSM ranks each class’s samples by GPS and divides them into IPC groups from easy to hard; cross-entropy loss generally increases along the GPS ranking.GPS compares per-image statistics at teacher BN-layer inputs with corresponding running statistics; larger GPS indicates greater difficulty.Source: §4.2.1 · pp. 4–5

4.2.2 SUA: STATISTICS UPDATED AGAIN

  • SUA forwards samples from each difficulty group through the teacher to adapt BN running statistics, which then supervise the corresponding distilled batch.Under independent forward batches, SUA statistics converge in expectation to those of the current difficulty group.Source: §4.2.2 · p. 5

4.2.3 ISS: INITIAL SAMPLE SCREENING

  • ISS initializes each distilled batch with real images from its corresponding difficulty group, aligning initialization with statistical supervision and reducing initial statistical bias.Experiments show that ISS significantly improves final distillation performance.Source: §4.2.3 · pp. 5–6

5 EXPERIMENTS

Across the evaluated settings, PSM improves performance in most comparisons while preserving a broader difficulty range than FADRM. The experiments also assess cross-architecture generalization, GPS, and the contributions of SUA and ISS.

5.1 EXPERIMENTAL SETUP

  • Experiments cover low- and high-resolution benchmarks, compare PSM with decoupled and other distillation baselines, and use matched FADRM settings plus a four-teacher PSM+ variant.Students train for 1000 epochs on Tiny-ImageNet with IPC = 1 and CIFAR-10/100, and 300 epochs in other settings.Source: §5.1 · p. 6, §5.2.1 · p. 6

5.2.1 MAIN RESULTS

  • PSM outperforms its corresponding baselines in most settings; on CIFAR-10 with IPC = 10, it reaches 52.7% Top-1 accuracy with ResNet50, 8.9% above FADRM.On ImageWoof, performance degradation occurs mainly with ResNet50 students at IPC = 1/10; the authors suggest limited architecture compatibility as one possible explanation.Source: §5.2.1 · p. 6, Table 1 · p. 7

5.2.2 DIFFICULTY AND EFFICIENCY ANALYSIS.

  • PSM’s distilled difficulty follows the original data’s trend across a broad range, while FADRM’s difficulty remains nearly constant; GPS and SUA add limited overhead relative to distillation.GPS and SUA require much less time than distillation because they involve no gradient computation; memory use is similar on small datasets, while distillation uses substantially more on Tiny-ImageNet.Source: §5.2.2 · pp. 6–7, Figure 3 · p. 7

5.2.3 CROSS-ARCHITECTURE GENERALIZATION.

  • PSM outperforms FADRM in most small-scale dataset and student-architecture settings, including by 6.5% on ImageNette with ResNet101; it also performs well with ResNet and DeiT on large-scale datasets.Source: §5.2.3 · p. 8, Table 2 · p. 8, Table 3 · p. 8

5.3 ABLATION STUDY

  • GPS achieves the highest Top-1 accuracy across datasets among the compared difficulty metrics, including Forgetting Score, Confidence Score, and Logits.Source: §5.3 · p. 8, Table 4 · p. 8
  • The best ISS screening choice and SUA forward-epoch ratio vary by dataset: Back performs best on CIFAR-10/100 and ImageWoof, while Front performs best on ImageNette.The reported best accuracies occur at rfwd = 0.9/1.0 for CIFAR-10/100 and ImageNette, and rfwd = 0.4 for ImageWoof; the authors attribute the varying optimum to dataset distribution.Source: §5.3 · p. 8, Figure 4 · p. 9
  • SUA and ISS individually improve several datasets but hurt ImageWoof, whereas using both achieves the best results across all datasets.SUA controls difficulty through supervision and ISS through initialization; the authors describe the mechanisms as complementary.Source: §5.3 · pp. 8–9, Table 5 · p. 8

5.4 VISUALIZATION

  • On ImageNette, ISS or SUA introduces clearer difficulty variation than FADRM, and combining them yields the clearest variation and highest Spearman’s ρ.Source: §5.4 · p. 9, Figure 5 · p. 9

6 CONCLUSION

PSM addresses difficulty variation that decoupled distillation methods do not fully capture, using difficulty-ranked groups to provide group-specific statistical supervision and initialization. The conclusion reports broader difficulty coverage and improved student performance, while identifying extreme IPC and fine-grained datasets as future-work settings.

6 CONCLUSION

  • PSM uses GPS to rank and group original data by difficulty, then applies SUA for group-specific BN supervision and ISS for group-matched initialization.The conclusion reports that this approach broadens the difficulty range of distilled samples and improves student performance.Source: §6 · p. 9
  • PSM’s performance under extreme settings such as IPC = 1 and on datasets with different distributions, including fine-grained datasets, remains for future work.The conclusion also identifies studying parameter effects on those dataset distributions as future work.Source: §6 · p. 9

A ADDITIONAL RELATED WORK

  • Decoupled methods are reported to overlook original-data difficulty structure, which PSM targets by preserving a broader difficulty range in distilled data.The conclusion reports improved student performance alongside this broader coverage.Source: §6 · p. 9, A ADDITIONAL RELATED WORK · p. 14

B THEORETICAL ANALYSIS

The analysis examines whether GPS tracks teacher-perceived classification difficulty and whether SUA statistics approach those of a difficulty group. GPS generally rises with cross-entropy loss, while the expected SUA discrepancy decreases geometrically with forward-batch updates.

B.1 THE EMPIRICAL RELATIONSHIP BETWEEN GPS AND CLASSIFICATION DIFFICULTY

  • The analysis defines classification difficulty using the teacher’s cross-entropy loss on the ground-truth label.A larger loss indicates lower teacher confidence in that label and greater classification difficulty.Source: §B.1 · p. 14
  • Cross-entropy loss generally increases with GPS, supporting GPS as a way to rank and group samples from easy to difficult.The analysis uses a pretrained ResNet18 teacher on CIFAR-10/100 and ImageNette/Woof; Figure 6 depicts the GPS–classification difficulty relationship.Source: §B.1 · p. 14, Figure 6 · p. 15

B.2 THE EXPECTED CONVERGENCE OF STATISTICS

  • SUA’s expected discrepancy from current-group statistics decreases geometrically with forward-batch updates for running means and variances across all BN layers.The resulting statistics provide the statistical matching signal for the corresponding distillation batch.Source: §B.2 · p. 15

C.1 COMPARISON WITH OTHER SOTA METHODS

The comparison reports that PSM preserves a broader difficulty structure and achieves the best results in most settings. The evaluation compares representative distillation methods using ResNet18 students, while noting that teacher architectures differ among methods.

C.1 COMPARISON WITH OTHER SOTA METHODS

  • Random reaches 75.8% on ImageNette with IPC = 50, and coreset methods generally approach DD methods as IPC increases.The comparison includes coreset, gradient, distribution, trajectory, generative, and decoupled methods; teacher architectures vary across methods.Source: §C.1 · p. 15, Table 6 · p. 16
  • PSM achieves the best results in most settings and preserves a broader difficulty structure than other evaluated methods.Table 6 reports top-1 accuracy (%) comparisons with ResNet18 as the student; the passage describes coreset methods as generally lower, while DD methods perform better in most settings.Source: §C.1 · pp. 15–16, Table 6 · p. 16
  • The paper attributes difficulty-structure preservation to PSM’s broader coverage, whereas other methods’ aggregated supervision is described as making internal difficulty structure difficult to capture.The passage states that as dataset scale increases, difficulty differences are more easily smoothed by aggregated objectives.Source: §C.1 · p. 16

C.2 MORE ABLATION STUDY

The ablations examine forward-pass epoch ratios across architectures and datasets, and initialization patch grids. They select ratios by reported accuracy and use a 1 × 1 patch grid for experiments.

C.2 MORE ABLATION STUDY

  • For ImageNette at IPC = 10, optimal rfwd values are 1.0 (69.1%) for DenseNet121, 0.7 (54.2%) for MobileNetV2, and 0.5 (50.3%) for ShuffleNetV2.The text says models with more parameters require more forward passes to better capture the current difficulty group’s distribution; due to small performance gaps, the ResNet18 optimal ratio is used for other settings.Source: §C.2 · pp. 16–17
  • On Tiny-ImageNet and ImageNet-1K, rfwd = 0.9 gives the highest reported accuracies, 48.1% and 48.8%, respectively, at IPC = 10.The experiments use ResNet18 as teacher and student, and adopt rfwd = 0.9 for other experiments on those datasets.Source: §C.2 · p. 17
  • The Tiny-ImageNet and ImageNet-1K ablation varies rfwd only; ISS screening methods are not further investigated there due to limited computational resources.The comparison is at IPC = 10 with ResNet18 as both teacher and student.Source: §C.2 · p. 17
  • The 1 × 1 patch grid performs best on all evaluated datasets, and the paper uses it for all experiments.The comparison uses ResNet18 teacher and student models with IPC = 10; the text notes that 2 × 2 patches may lose local-region information, more evidently on high-resolution datasets.Source: §C.2 · p. 17, Table 8 · p. 17

D IMPLEMENTATION DETAILS

The implementation details specify teacher pretraining, difficulty-group distillation, and soft-label generation; visualizations compare the difficulty range of samples generated by FADRM and PSM.

D.1 PRETRAINING.

  • Teacher models use the same pretraining epochs and configuration per dataset; ImageNet-1K uses official PyTorch pretrained weights.GPS is computed after pretraining, with batch size unchanged, β = 1.0, and ϵ = 1 × 10−6.Source: §D.1 · p. 18

D.2 DISTILLATION.

  • When IPC = 1, the dataset forms one difficulty group and only ISS applies; PSM and FADRM(+) perform similarly on most datasets.SUA forward-pass batch size matches the pretraining batch size to reduce statistical bias.Source: §D.2 · p. 18

D.3 SOFT LABEL GENERATION.

  • Soft labels come from the original pretrained teachers, with equal teacher weights for FADRM+ and PSM+.This choice maintains consistency in the soft-label distribution.Source: §D.3 · p. 19

E MORE VISUALIZATION

Loading 2609.34299v1…