Dataset Condensation with Contrastive Signals

Motivation

• DC can underperform random selection on fine-grained tasks because class-wise gradient matching overlooks contrastive signals between classes.On the fine-grained Automobile dataset, DC reaches 11% test accuracy versus 12.2% for random selection.Source: §1 · pp. 1–2
• Fine-grained classification requires preserving discriminative features such as logos, police signs, and trailers while suppressing common features such as wheels, headlights, roads, and trees.Source: §1 · pp. 1–2

Contribution

• DCC modifies the gradient-matching loss to capture contrastive signals by matching summed gradients across all classes rather than only same-class gradients.Source: §1 · p. 2
• DCC also tracks kernel velocity to analyze training dynamics and uses a bi-level warm-up strategy to stabilize optimization.Source: §1 · p. 2
• Experiments report that DCC outperforms DC on fine-grained tasks, general benchmarks including SVHN, CIFAR-10, and CIFAR-100, and downstream continual-learning tasks.The method is presented as generating informative small synthetic datasets that reduce total data storage in continual learning.Source: §1 · p. 2

Selection-based methods

• Selection-based methods find a K-sample coreset by reducing the difference between training-set and coreset loss gradients.Source: §2 · p. 2
• C-score, GradNd, and EL2N-based approaches rank samples using consistency, expected gradient-norm, or error-vector criteria to prune training data.Source: §2 · p. 2

Limitations

• Selection methods are ineffective when task-conditional information is evenly distributed among training samples.Source: §2 · p. 2
• Under the stated equal-information condition, any selection method retains only a small amount of conditional mutual information, and empirical gaps over random selection are often insignificant.Source: §2 · p. 2

Synthesis-based methods

• Synthesis-based methods generate a small dataset intended to achieve performance similar to the full training dataset, rather than selecting existing samples.Source: §2 · p. 2

3.2. A Motivating Example

• Class-wise gradient matching can fail when class-common features dominate, producing ineffective synthetic datasets for fine-grained classification.The analysis states that when α ≪ β, the discriminative-feature ratio R(S) approaches zero; this matches failures on tasks with shared appearance and fine-grained distinctions.Source: §3.2 · pp. 3–4
• The resulting synthetic dataset contains class-discriminative features independently of the proportion of class-common features in the original training data.The motivating analysis characterizes the collective-gradient solution as emphasizing differences between classes rather than shared features.Source: §3.2 · p. 4
• In the MNIST motivating example, DC generates near-prototype images, whereas collective gradient matching emphasizes class differences such as the left half of class “8” images.The repeated pattern across settings is attributed to leveraging contrastive signals rather than chance or dataset bias.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Dataset Condensation with Contrastive Signals

• DCC alternately trains the classifier and synthetic dataset with periodic classifier initialization, while its bi-level warm-up applies class-wise matching during early optimization.The warm-up addresses early training peaks associated with periodically induced chaotic transience and is designed to stabilize DCC optimization.Source: §3.3 · pp. 4–5
• DCC captures contrastive signals by summing loss gradients across classes before measuring the distance between training and synthetic datasets.Unlike DC, which computes class-specific gradient distances and then sums them, DCC performs the summation first, preserving interclass differences.Source: §3.2 · p. 4, §3.3 · p. 5

Experimental Setup

• The evaluation covers SVHN, CIFAR-10, CIFAR-100, and fine-grained Automobile, Terrier, Fish, Truck, Insect, and Lizard datasets using 1, 10, or 50 images per class.Synthesis methods are evaluated over five learned synthetic datasets and 100 classifiers trained from scratch.Source: §4.1 · p. 6

Results on Fine-Grained Datasets

• DCC always outperforms Random on the fine-grained datasets, whereas DC performs worse than Random on Automobile, Terrier, and Fish.These results indicate that DCC captures class-discriminative features more effectively.Source: §4.2 · p. 6, Table 1 · p. 6
• DCC-generated Automobile images are sharper and emphasize distinctive class patterns, whereas DC images have ambiguous interclass differences.Examples include the limousine’s long body and multiple windows and the Model T’s distinctive body frame.Source: §4.2 · pp. 6–7, Figure 5 · p. 6

Results on Benchmark Datasets

• DCC achieves the best performance on SVHN, CIFAR-10, CIFAR-100, and fine-grained tasks, while DC and DSA outperform Random and selection methods on benchmark datasets.The benchmark comparison reports a gap of +15% in SVHN for DC and DSA over competing methods, while KIP is unstable across datasets and images per class.Source: §4.2 · p. 7, Table 2 · p. 7

Cross-Architecture Generalization

• DCC generalizes across ConvNet, LeNet, AlexNet, VGG-11, and ResNet-18 target architectures when synthetic data are learned using ConvNet.The comparison uses CIFAR-10 with 50 images per class and reports mean test accuracy over 100 runs.Source: §4.2 · p. 7, Table 3 · p. 8

Ablation Study

• The bi-level warm-up strategy improves DCC effectiveness on CIFAR datasets, although its benefit is negligible when the synthetic dataset capacity is relatively large.The ablation compares the proposed warm-up settings with and without warm-up.Source: Table 4 · p. 8, §4.2 · p. 8

Conclusion

• Existing condensation methods perform poorly on fine-grained tasks because they are biased toward reconstructing class prototypes.DCC instead captures subtle interclass differences through class-collective gradient matching.Source: §5 · p. 9
• DCC significantly outperforms baselines on fine-grained tasks and general vision classification benchmarks.The conclusion presents this as the paper’s main experimental finding.Source: §5 · p. 9

Future Directions

• For datasets with many classes, bi-level warm-up may be insufficient to control the large number of contrastive signals involved in DCC.The paper identifies class subgrouping as a possible direction for future work.Source: §5 · p. 9

An issue with class-wise gradient matching

• Class-wise matching optimizes each class separately toward the average gradient of its training samples, overlooking interclass contrastive information.The analysis formulates the resulting synthetic set through the class-wise objective.Source: A. A Motivating Example · p. 11
• When class-common features dominate class-discriminative features, class-wise gradient matching can produce synthetic datasets ineffective for classification.This limitation is especially relevant to fine-grained tasks with shared appearance across classes.Source: A. A Motivating Example · p. 11

Leveraging contrastive signals

• The modified collective objective sums loss gradients over all classes, incorporating contrastive terms such as (µ+ − µ−) and (s1 − s2).These terms encode differences between class means and class-specific components.Source: A. A Motivating Example · p. 11
• The resulting synthetic dataset contains only class-discriminative features and is independent of the proportion of class-common features in the original training data.This is the stated implication of the modified objective.Source: A. A Motivating Example · p. 12

Benchmark continual learning

• DSAC and DSA use condensed datasets as rehearsal examples in a continual-learning sequence of CIFAR10, SVHN, and TrafficSigns.The comparison evaluates performance across the second and third tasks.Source: B. Application: Continual Learning · p. 12
• DSAC dominates the other baselines on the T2 and T3 tasks in the CIFAR10-SVHN-TrafficSigns sequence.The baselines include DSAC, DSA, and RB.Source: B. Application: Continual Learning · p. 12

• SVHN and CIFAR-10 contain 10 classes, while CIFAR-100 contains 100 classes; all use 32 × 32-pixel images.Source: C. Datasets · p. 12
• ImageNet contains 1,281,167 training images, 100,000 test images, and 1,000 classes.Source: C. Datasets · p. 12
• Table 6 lists the ImageNet classes subsampled for each fine-grained image classification dataset.Source: Table 6 · p. 13