Source-linked AI summary
CAFE: Learning to Condense Dataset by Aligning Features
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, Yang You
TL;DR
Dataset condensation seeks compact synthetic training sets, but gradient matching can overfit dominant-gradient samples and miss the broader data distribution. CAFE aligns real and synthetic features across layers, adds discrimination supervision, and uses dynamic bi-level optimization. Across datasets it generally outperforms prior methods, with an up-to-11% gain on SVHN and stronger generalization to unseen architectures.
Problem
Gradient-based condensation can overfit hard or noisy samples with dominant gradients, overlooking representative samples and weakening generalization to unseen architectures.
Method
CAFE aligns layer-wise real and synthetic features, uses real-sample classification to inject discriminative power, and dynamically adjusts bi-level SGD updates.
Results
CAFE generally outperforms the state of the art across datasets, including an up-to-11% gain on SVHN and stronger generalization to unseen architectures.
Takeaways & Limitations
Preserving feature distributions and discriminative power enables synthetic sets that characterize original data and generalize across learning settings.
Abstract
from arXiv · showhide
Dataset condensation aims at reducing the network training effort through condensing a cumbersome training set into a compact synthetic one. State-of-the-art approaches largely rely on learning the synthetic data by matching the gradients between the real and synthetic data batches. Despite the intuitive motivation and promising results, such gradient-based methods, by nature, easily overfit to a biased set of samples that produce dominant gradients, and thus lack global supervision of data distribution. In this paper, we propose a novel scheme to Condense dataset by Aligning FEatures (CAFE), which explicitly attempts to preserve the real-feature distribution as well as the discriminant power of the resulting synthetic set, lending itself to strong generalization capability to various architectures. At the heart of our approach is an effective strategy to align features from the real and synthetic data across various scales, while accounting for the classification of real samples. Our scheme is further backed up by a novel dynamic bi-level optimization, which adaptively adjusts parameter updates to prevent over-/under-fitting. We validate the proposed CAFE across various datasets, and demonstrate that it generally outperforms the state of the art: on the SVHN dataset, for example, the performance gain is up to 11%. Extensive experiments and analyses verify the effectiveness and necessity of proposed designs.
1. Introduction
CAFE addresses dataset-condensation methods that can overfit to hard, high-gradient samples by aligning real and synthetic features across layers while preserving discriminative power. Its dynamic bi-level optimization supports adaptive updates, and experiments report stronger generalization and state-of-the-art performance, including an 11% SVHN gain.
- Background: Dataset condensation reduces large training sets to compact synthetic sets intended to preserve model performance while lowering training cost.Earlier approaches include coreset selection and meta-learning, but coreset selection is typically NP-hard and meta-learning can be difficult to scale.
- Motivation: CAFE targets the bias and weak cross-architecture generalization of gradient matching, which can overlook representative easy samples while overfitting hard examples or noise.These high-gradient samples may vary across architectures, limiting transfer to unseen networks.
- Method: CAFE aligns layer-wise features between real and synthetic samples to supervise distribution consistency across the dataset rather than only matching dominant gradients.This broader feature matching attends to all samples and aims to avoid overfitting hard or noisy examples.
- Method: CAFE adds a discrimination loss that classifies real samples using affinities to synthetic class clusters, injecting discriminative capability into the synthetic data.The loss complements distribution alignment by encouraging each synthetic class cluster to represent its corresponding real cluster.
- Optimization: CAFE uses dynamic bi-level optimization to adapt the number of SGD updates for the network and synthetic data, alleviating under- and over-fitting.The method alternates synthetic-image and network updates through automatically adjusted outer- and inner-loop steps.
- Results: 11%: CAFE outperforms the runner-up on SVHN when learning a 1 image/class synthetic set, while experiments also show stronger generalization to unseen architectures.The reported results support state-of-the-art performance with strong generalization and robustness across learning settings.
2. Related Work
Related work spans dataset-condensation methods, coreset selection, and generative models. CAFE differs from prior feature-distribution matching by aligning layer-wise rather than final-layer features, adding explicit discrimination, and retaining dynamic bi-level optimization.
- Dataset Condensation: Dataset-condensation research includes meta-learning, ridge-regression simplifications, generative networks, and differentiable Siamese augmentation to improve efficiency or scalability.These methods target reduced optimization cost or improved training of synthetic sets.
- Dataset Condensation: CAFE differs from feature-distribution matching [52] by aligning layer-wise features, explicitly enforcing discriminative synthetic images, and using dynamic bi-level optimization.Method [52] uses only final-layer features and removes bi-level optimization to reduce training cost.
- Coreset Selection: Coreset methods select real points using heuristics such as cluster proximity, gradient-space diversity, or forgetfulness, but cannot ensure an optimal training subset for deep networks.Greedy selection also lacks guarantees that the selected subset satisfies its criterion optimally.
- Generative Models: Unlike generative models that synthesize human-plausible images, dataset condensation seeks informative training samples; CAFE can outperform selected real images for model training.The paper distinguishes visual realism from training utility as the primary objective.
3. Method
CAFE condenses datasets by aligning real and synthetic features across network layers, enforcing discrimination, and dynamically alternating synthetic-data and network updates. These components aim to preserve data distribution, improve synthetic-set discriminability, and reduce over- or under-fitting.
- 3. Method: CAFE aligns layer-wise real and synthetic features, adds discrimination loss, and uses dynamic bi-level optimization to condense datasets.The method comprises three modules: layer-wise feature alignment, discrimination loss, and dynamic bi-level optimization.
- 3.1. Overview: Synthetic images are optimized in an outer loop while network parameters are trained on synthetic images in an inner loop.The alternating updates train the synthetic set across different network parameters, supporting generalization to those networks.
- 3.2. Layer-wise Features Alignment: Feature alignment matches real and synthetic representations at each convolutional layer using category-wise feature averaging and mean squared error.Real and synthetic batches with the same label are embedded through multiple layers, averaged, and compared to preserve feature distributions.
- 3.3. Discrimination Loss: The discrimination loss classifies real samples by their inner-product affinity to synthetic class centers, injecting discriminant capability into the synthetic images.The total synthetic-image objective combines feature-alignment loss with weighted discrimination loss, Ltotal = Lf + βLd.
- 3.4. Dynamic Bi-level Optimization: Dynamic bi-level optimization uses query-set performance to adapt inner-loop sampling and stop when performance converges, reducing over- and under-fitting.A queue tracks query-set performance, and new inner-loop networks are sampled when notable improvement occurs.
4. Experiments
CAFE is evaluated across five datasets, component ablations, hyper-parameter studies, unseen architectures, and visual analyses. It achieves strong condensation performance while its feature-alignment and dynamic optimization components improve distribution capture and generalization.
- 4.2. Comparison to the State-of-the-art Methods: CAFE achieves the best results on all five datasets at 1 image per class, improving over other methods by 11% on SVHN and 6.5% on FashionMNIST.At 10 and 50 images per class, CAFE exceeds DC by 0.7%–2.6% on most datasets; CAFE+DSA exceeds DSA by 1.7% on CIFAR10 at 50 images per class.
- 4.3. Ablation Studies: CAFE jointly uses discrimination loss, layer-wise feature alignment, and dynamic bi-level optimization, whose combination achieves the highest ablation performance.Discrimination loss alone reaches 49.78%, while adding feature alignment and dynamic optimization progressively improves performance; the latter reduces over- and under-fitting by adapting loop termination.
- 4.3. Ablation Studies: Feature alignment across more layers improves testing performance, with an approximately 4% gain without discrimination loss and 3% gain with it compared with using only the first layer.The average boost from adding each layer is about 1% without discrimination loss and 0.7% with it, indicating complementary layer-wise supervision.
- 4.3. Ablation Studies: The default λ1 = 0.05 and λ2 = 0.05 settings outperform DC by 1.6%, while equal weighting of feature and discrimination losses performs best.Performance is relatively insensitive to λ1 and λ2, but excessive loss weight on discrimination loss degrades performance; γ = 10 gives the best result with less training time than DC.
- 4.3. Ablation Studies: When trained with ConvNet-generated CIFAR10 synthetic data, CAFE surpasses DC on AlexNet, VGG11, ResNet18, and three-layer MLP by 5.25%, 1.79%, 4.42%, and 7.96%, respectively.The synthetic set is learned on one architecture and evaluated on unseen architectures, demonstrating stronger cross-architecture generalization than DC.
- 4.4. Visualizations: Compared with DC, CAFE synthetic images are more visually similar to original CIFAR10 images and contain more semantic information, while its feature distributions cover the real dataset more broadly.t-SNE visualizations show DC features concentrated near a small decision-boundary region, whereas DL, LFA, and CAFE capture information across the real dataset.
5. Conclusion
CAFE combines feature alignment, discrimination modeling, and dynamic bi-level optimization to condense datasets while preserving distributional and discriminative properties. Across datasets, it outperforms prior methods with lower computation, though ImageNet remains future work.
- CAFE preserves real-feature distributions and discriminant power through layer-wise feature alignment, discrimination loss, and dynamic bi-level optimization.These modules target distribution consistency, discriminative synthetic data, and adaptive SGD steps to avoid over- and under-fitting.
- CAFE consistently outperforms state-of-the-art methods across various datasets with less computation cost.
- The authors identify more challenging datasets such as ImageNet as future directions for dataset condensation.