PaperlayerView exact revision on arXiv

Source-linked AI summary

Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective

Zeyuan Yin, Eric Xing, Zhiqiang Shen

arXiv:2306.13092v3cs.CVcs.AIcs.LG

TL;DR

Large-scale dataset condensation is hindered by the computational cost of bilevel optimization, especially for large datasets and high-resolution images. SRe2L decouples model training and synthetic-data optimization through Squeeze, Recover, and Relabel stages, achieving strong results on Tiny-ImageNet and ImageNet-1K. The method reaches 60.8% accuracy on full ImageNet-1K at 224×224 resolution with IPC 50, while maintaining reasonable computational costs.

  • Problem

    Prior condensation methods mainly handled small, low-resolution, or partial ImageNet datasets because bilevel optimization incurred prohibitive computational expense.

  • Method

    SRe2L decouples model training and synthetic-data optimization through sequential Squeeze, Recover, and Relabel stages, including batch-normalization-statistics alignment during recovery.

  • Results

    60.8% accuracy is achieved on full ImageNet-1K at 224×224 resolution with IPC 50, outperforming previous methods by a large margin.

  • Takeaways & Limitations

    SRe2L demonstrates practical large-scale dataset condensation on Tiny-ImageNet and ImageNet-1K while supporting high-resolution synthesis and reasonable computational costs.

  • Takeaways & Limitations

    A performance disparity remains between the condensed and original full datasets, and soft labels require extra storage.

Abstract

from arXiv · show

We present a new dataset condensation framework termed Squeeze, Recover and Relabel (SRe$^2$L) that decouples the bilevel optimization of model and synthetic data during training, to handle varying scales of datasets, model architectures and image resolutions for efficient dataset condensation. The proposed method demonstrates flexibility across diverse dataset scales and exhibits multiple advantages in terms of arbitrary resolutions of synthesized images, low training cost and memory consumption with high-resolution synthesis, and the ability to scale up to arbitrary evaluation network architectures. Extensive experiments are conducted on Tiny-ImageNet and full ImageNet-1K datasets. Under 50 IPC, our approach achieves the highest 42.5% and 60.8% validation accuracy on Tiny-ImageNet and ImageNet-1K, outperforming all previous state-of-the-art methods by margins of 14.5% and 32.9%, respectively. Our approach also surpasses MTT in terms of speed by approximately 52$\times$ (ConvNet-4) and 16$\times$ (ResNet-18) faster with less memory consumption of 11.6$\times$ and 6.4$\times$ during data synthesis. Our code and condensed datasets of 50, 200 IPC with 4K recovery budget are available at https://github.com/VILA-Lab/SRe2L.

1 Introduction

SRe2L introduces a three-stage framework that decouples synthetic-data synthesis from real-data input and separates inner- and outer-loop optimization, targeting large-scale dataset condensation. It supports high-resolution ImageNet-1K condensation with reduced computational demands and reports strong results on Tiny-ImageNet and ImageNet-1K.

  • Motivation: Prior condensation methods primarily targeted small, low-resolution, or partial ImageNet datasets because bilevel optimization required expensive unrolled iterations.The limitation involved computational expense from alternating model and synthetic-data updates.
  • Method: The recovery stage aligns synthetic data with batch-normalization statistics from a model trained on the original dataset, without using real data during synthesis.The authors contrast this dataset-wide statistic alignment with batch-level feature matching and report reduced computational cost versus bilevel training.
  • Advantages: SRe2L supports large-resolution condensation, Tiny-ImageNet and ImageNet-1K, and direct use of pretrained large models containing batch-normalization layers.These properties are presented as practical advantages that can reduce training overhead.
  • Method: SRe2L uses sequential Squeeze, Recover, and Relabel stages for large-scale dataset condensation.The framework separates model training, synthetic-data recovery, and label alignment.
  • Analysis: The study includes ablations of data augmentation, recovery regularization, and teacher alternatives for relabeling.These analyses examine components of original-data compression, data recovery, and synthetic-data relabeling.
  • Results: 60.8% accuracy is achieved on full ImageNet-1K at 224×224 resolution with IPC 50, outperforming previous methods by a large margin.The result is reported alongside reasonable computational cost and memory use.

2 Approach

SRe2L reformulates dataset condensation as a decoupled Squeeze–Recover–Relabel pipeline, separating model training from synthetic-data optimization to improve scalability across datasets, resolutions, and architectures.

  • Objective: Dataset condensation seeks a small synthetic dataset that preserves crucial information from a much larger labeled dataset.The condensed set is optimized to reduce the performance gap between models trained on synthetic and full data.
  • Decoupling: Joint bilevel methods incur high computation and memory costs because each outer update unrolls inner-loop training while transferring bias from real data.SRe2L instead decouples condensed-data optimization from neural-network training.
  • Stage-1 Squeeze: Squeeze extracts information from original data into a neural network, providing a pretrained model for subsequent synthetic-data recovery.The stage evaluates data augmentations and training strategies while abstracting original images into learned representations.
  • Stage-2 Recover: Recover reconstructs images by optimizing synthetic data against batch-normalization statistics, class labels, and regularization while freezing the pretrained model.Global BN statistics from the original dataset provide a more comprehensive alignment target than per-batch feature matching.
  • Stage-2 Recover: Multi-crop optimization randomly crops and resizes synthesized images to 224×224 during recovery, increasing the informational content available for later training.The strategy mirrors RandomResizedCrop used during model training and updates only a cropped region per iteration.
  • Stage-3 Relabel: Relabel trains models on synthetic crops paired with corresponding soft labels, improving alignment between condensed data and labels.The paper reports that this stage is crucial and significantly improves trained-model performance.
  • Scalability: The decoupled design reduces the need to feed real data through iterative synthesis and training, lowering computational and memory overhead.Its stated scope includes large datasets, high-resolution condensation, and broader network architectures.

3 Experiments

Experiments evaluate SRe2L across large-scale datasets, recovery and relabeling choices, synthetic-image properties, and cross-architecture performance. The method achieves strong results on Tiny-ImageNet and ImageNet-1K while showing sensitivity to training budgets, augmentation, recovery models, and relabeling settings.

  • Experiment Setting: SRe2L is evaluated on Tiny-ImageNet and full ImageNet-1K using ResNet-18, ResNet-50, ResNet-101, ViT-Tiny, and BN-ViT-Tiny target models.Condensed datasets are evaluated by training models from scratch and measuring validation accuracy on real validation data.
  • Squeezing Analysis: 50 iterations is adopted as the default Tiny-ImageNet squeezing budget because longer squeezing reduces recovered-model performance.Data augmentation during squeezing likewise decreases recovered-data accuracy and makes recovery more difficult.
  • Recovering Analysis: Cropped synthetic images distribute category-relevant miniature regions across the image and are more beneficial for soft-label model training.Multi-crop optimization also improves validation accuracy compared with non-crop settings.
  • Recovering Analysis: Longer recovery budgets improve classification accuracy, while recovering from ResNet-50 performs worse than recovering from ResNet-18 under the same iterative setting and budget.The ablation uses recovery budgets of 0.5k, 1k, 2k, and 4k iterations.
  • Relabeling Analysis: Relabeling with a teacher close or identical to the recovery model gives better accuracy, while larger architectural disparity increases Top-1 errors.The method therefore uses the same model for recovery and relabeling.
  • Cross-Architecture Generalization: The condensed data generalize across ResNet-18, ResNet-50, ResNet-101, and ViT-T, although ViT performance is suboptimal.Visual comparisons also show clearer class semantics and contours than MTT’s more blurred synthetic images.

4 Related Work

Data condensation methods aim to create compact synthetic datasets while retaining essential information and comparable performance. Prior approaches include four main categories and have primarily succeeded on smaller or lower-resolution datasets.

  • Data condensation creates compact synthetic datasets that preserve essential information while reducing training time and storage needs.
  • Prior methods are mainly categorized as Meta-Model Matching, Gradient Matching, Distribution Matching, and Trajectory Matching.
  • Existing solutions primarily excel on MNIST, CIFAR, Tiny-ImageNet, downscaled ImageNet, or ImageNet subsets.

5 Conclusion

SRe2L introduces a three-step dataset condensation process—squeezing, recovering, and relabeling—for large-scale ImageNet-1K data. The method outperforms existing condensation approaches, while full replacement of the original dataset remains infeasible.

  • SRe2L uses sequential squeezing, recovering, and relabeling steps to condense large-scale ImageNet-1K data.The process is presented as a more efficient and effective approach to harnessing large-scale datasets.
  • The proposed method outperforms existing state-of-the-art dataset condensation approaches by a significant margin.
  • Complete substitution of the full dataset with condensed data is not yet feasible because a performance disparity persists.
  • Future work targets ImageNet-21K condensation and additional modalities, including language and speech.The authors also identify extra storage for soft labels as a limitation.

A.1 Dataset Statistics

The appendix describes dataset variants used in the experiments, including Tiny-ImageNet and ImageNette/ImageWoof configurations. These datasets differ in class counts and image resolutions.

  • Tiny-ImageNet contains 200 classes derived from ImageNet-1K.
  • Each Tiny-ImageNet class comprises 500 images with 64×64 resolution.
  • ImageNette/ImageWoof contain 10 classes, with each image having 112×112 resolution.

A.2 Squeezing Details

The appendix details recovery settings, model configurations, and ablations across Tiny-ImageNet and ImageNet-1K. It emphasizes that longer training and stronger augmentation can hinder recovery, while SRe2L reduces state storage relative to MTT.

  • Data Augmentation: More data augmentation during squeezing decreases the final accuracy of recovered data on Tiny-ImageNet.
  • Data Augmentation: Longer training budgets and more squeezing augmentations make compressed-model data recovery more intricate.The appendix characterizes this increased recovery difficulty as undesirable.
  • Recovering Settings: ResNet-50 models with V2 weights pose a greater data-extraction challenge than models with V1 weights.
  • Recovering Settings: The V2 recovery challenge is attributed to enhanced training recipes involving more training and data augmentation.
  • Regularization Terms: The experiments omit L2 and total-variation image-prior regularizers because the focus is information recovery rather than image smoothness.
  • Memory Consumption and Computational Cost: SRe2L stores statistical data from each BN layer for synthetic-image optimization instead of all model states across training epochs.This contrasts with MTT's trajectory-alignment requirement.

A.4 Relabeling & Validation Details

Validation uses condensed data and retargeted labels, with a separate teacher-based soft-labeling procedure for synthesized images.

  • Relabeling: A recovery-model architecture supplies soft labels as a teacher for synthesized images through fast knowledge distillation.Post-validation training uses 300 epochs and temperature τ = 20.
  • Validation: Tiny-ImageNet validation trains for 100 epochs using condensed data and retargeted labels.Other training parameters follow the condensing configuration in Table 7a.
  • Validation: ImageNet-1K validation uses CutMix with p = 1.0 and β = 1.0.The validation model follows the parameter configuration in Table 7b.

B Low-Resolution Data

SRe2L is evaluated beyond its primary high-resolution setting, showing stronger accuracy as ImageNet-1K resolution increases and competitive results on CIFAR-100 but a gap on CIFAR-10.

  • ImageNet-1K: Validation accuracy increases with ImageNet-1K image resolution, indicating better effectiveness for high-resolution datasets.The comparison includes down-sampled resolutions of 112×112 and 64×64.
  • CIFAR: CIFAR-100 results are on par with leading methods including DM, FrePo, MTT, and TESLA.The experiment uses an adapted ResNet-18 backbone across SRe2L’s three phases.
  • CIFAR: A clear performance gap remains on the smaller CIFAR-10 dataset.The authors report this contrast alongside the CIFAR-100 results.
  • CIFAR: The CIFAR experiments suggest benefits for lower-resolution datasets while retaining computational efficiency and processing speed.These experiments use 32×32 images and fewer classes than the larger benchmark settings.

D Theoretical Analysis

The theoretical analysis estimates condensed-data generalization through mutual-information bounds and PAC-style generalization-error bounds, accounting for the smaller synthetic training set.

  • Generalization framework: The analysis evaluates generalization by comparing models trained on condensed data with performance on the original validation data.Generalization error is used to assess the condensed dataset’s ability to preserve performance.
  • Mutual-information analysis: Mutual information between original or condensed inputs and final-layer representations is used to analyze generalization.The analysis uses the same network architecture to bound mutual information.
  • Mutual-information analysis: A leave-one-out upper bound conservatively bounds mutual information between input samples and model representations.Here, X denotes the input sample and D denotes the model’s output representation.
  • Generalization-error bound: PAC generalization bounds depend on hypothesis-class cardinality and the number of training examples.For synthetic data, Ntrn = |Csyn|; for full data, Ntrn = Nori, with confidence parameter δ between 0 and 1.
  • Generalization-error bound: The hypothesis-space estimate is linked to the number of class-homogeneous clusters distinguished by the backbone network.The analysis estimates clusters through mutual information between inputs and representations.

E More Visualization of Synthetic Data

Additional visualizations compare SRe2L with MTT on Tiny-ImageNet and show SRe2L synthetic samples from ImageNet-1K, emphasizing semantic detail and object structure.

  • Visualization comparison: SRe2L synthetic data shows stronger semantic information than MTT, with more object textures, shapes, and details.The comparison covers Tiny-ImageNet and additional ImageNet-1K visualizations.
Loading 2306.13092v3…