Source-linked AI summary
Fixed Anchors Are Not Enough: Dynamic Retrieval and Persistent Homology for Dataset Distillation
Muquan Li, Hang Gou, Yingyi Ma, Rongzheng Wang, Ke Qin, Tao He
TL;DR
Decoupled dataset distillation is limited by static residual anchors that create fit–complexity gaps and pull-to-anchor effects, weakening diversity and generalization. RETA combines dynamic feature-space patch retrieval with persistent topological alignment to address these failures. Across benchmarks, it reports consistent state-of-the-art performance, including 64.3% accuracy on ImageNet-1K with ResNet-18 at IPC=50, 3.1% above the best prior.
Problem
Static residual anchors in decoupled dataset distillation create fit–complexity gaps and pull-to-anchor effects that erode intra-class diversity and weaken generalization.
Method
RETA retrieves feature-aligned real patches using a fit–complexity score and applies persistent-homology alignment to preserve class topology.
Results
RETA consistently achieves state-of-the-art accuracy across multiple datasets and architectures, reaching 64.3% on ImageNet-1K with ResNet-18 at IPC=50, 3.1% above the best prior.
Takeaways & Limitations
Dynamic retrieval and topology alignment jointly improve synthetic-data fidelity and preserve class geometry across standard image-distillation settings.
Takeaways & Limitations
RETA relies on a frozen teacher, per-class retrieval pools, topology-design choices, and a hand-crafted complexity proxy, while PTA adds modest distillation-time overhead.
Abstract
from arXiv · showhide
Decoupled dataset distillation (DD) compresses large corpora into a few synthetic images by matching a frozen teacher's statistics. However, current residual-matching pipelines rely on static real patches, creating a fit-complexity gap and a pull-to-anchor effect that reduce intra-class diversity and hurt generalization. To address these issues, we introduce RETA -- a Retrieval and Topology Alignment framework for decoupled DD. First, Dynamic Retrieval Connection (DRC) selects a real patch from a prebuilt pool by minimizing a fit-complexity score in teacher feature space; the chosen patch is injected via a residual connection to tighten feature fit while controlling injected complexity. Second, Persistent Topology Alignment (PTA) regularizes synthesis with persistent homology: we build a mutual k-NN feature graph, compute persistence images of components and loops, and penalize topology discrepancies between real and synthetic sets, mitigating pull-to-anchor effect. Across CIFAR-100, Tiny-ImageNet, ImageNet-1K, and multiple ImageNet subsets, RETA consistently outperforms various baselines under comparable time and memory, especially reaching 64.3% top-1 accuracy on ImageNet-1K with ResNet-18 at 50 images per class, +3.1% over the best prior.
1. Introduction
Decoupled dataset distillation separates supervision from distributional alignment, but fixed residual anchors can create fit–complexity and pull-to-anchor failures. RETA addresses these issues with dynamic retrieval and persistent topology alignment, achieving strong results across datasets and architectures.
- 1. Introduction: Decoupled dataset distillation stabilizes optimization by separating supervised learning from distributional alignment and enabling independent control of both objectives.This design also supports architecture-agnostic supervised learning and stronger pretrained representations for alignment.
- 1. Introduction: Residual matching preserves fine-scale structure by injecting a small real patch through a residual connection, but fixed patches may misalign with synthetic features and contract intra-class diversity.These effects correspond to the fit–complexity gap and pull-to-anchor effect described for prevailing pipelines.
- 1. Introduction: RETA combines Dynamic Retrieval Connection and Persistent Topology Alignment to address fit–complexity gaps and pull-to-anchor effects in decoupled dataset distillation.DRC retrieves feature-aligned, low-complexity real patches, while PTA matches persistent topological signatures to preserve class geometry.
- 1. Introduction: 64.3% top-1 accuracy on ImageNet-1K with ResNet-18 at IPC=50 surpasses the best prior by 3.1%.The reported result is part of broader state-of-the-art performance across multiple architectures and datasets.
2. Related work
Dataset distillation synthesizes a compact dataset that preserves the training behavior of a much larger corpus. Prior work includes bilevel optimization, gradient matching, and trajectory matching approaches.
- 2. Related work: Dataset distillation synthesizes a small set of examples intended to preserve the training behavior of a much larger corpus.The goal is to replace repeated training on the full dataset with a compact surrogate.
- 2. Related work: Most prior work formulates dataset distillation as bilevel optimization in which synthetic data are updated to mimic training on real data.This formulation incurs expensive nested optimization and large memory footprints.
- 2. Related work: Major dataset-distillation families include gradient matching, which aligns real and synthetic per-batch gradients, and trajectory matching.
3. Preliminary
Decoupled dataset distillation separates supervised and alignment objectives, but direct pixel optimization can erase fine structures. Residual matching counters this erosion by periodically injecting real patches, while fixed connections still risk geometric collapse.
- 3. Preliminary: Decoupled dataset distillation learns a compact synthetic dataset by separating supervised learning from distributional alignment.The formulation uses a supervised objective on synthetic data and an alignment objective matching real-data statistics.
- 3. Preliminary: Direct pixel updates under repeated resampling can wash out fine structures that the alignment term alone cannot recover.This information-vanishing problem arises because decoupled DD does not simulate long training trajectories on real data.
- 3. Preliminary: Residual matching periodically anchors synthetic images to the real-data manifold by injecting a randomly selected real patch through a residual connection.The connection forms a convex combination of the optimized synthetic image and a real sample, with α controlling the connection ratio.
- 3. Preliminary: A fixed residual connection can simultaneously induce fit–complexity mismatch and pull-to-anchor effects that collapse class geometry.
4. Method
RETA addresses residual-matching failures by dynamically selecting anchors that balance feature fit and complexity, then aligning real and synthetic feature topology with persistent homology.
- Motivation: RETA targets fit–complexity gaps and pull-to-anchor contraction, which can enlarge generalization error and prematurely merge intra-class clusters.The analysis links static or nearby anchors to both excessive injected complexity and shrinking representation diversity.
- Joint Framework: Together, DRC chooses which real patches to connect at each stage, while PTA constrains the global topology those connections should preserve.The frozen teacher remains in the computation graph so topology-loss gradients can reach synthetic inputs, with cached features reducing overhead.
- Dynamic Retrieval Connection: DRC derives a stage-wise anchor from a class-specific real-image pool by minimizing a teacher-space fit–complexity score.A frozen encoder embeds the current synthetic query and candidate patches before retrieval.
- Dynamic Retrieval Connection: The selected patch is resampled and mixed through a residual update, while the score penalizes irregular high-frequency content to regulate injected complexity.This design jointly shrinks the fit gap and controls the complexity gap in the post-connection bound.
- Persistent Topology Alignment: PTA models class-wise teacher features with a mutual k-NN Vietoris–Rips-like filtration and persistent homology over connected components and loops.The filtration thresholds mutual-neighbor edges and completes cliques to form a flag complex.
- Persistent Topology Alignment: PTA converts persistence diagrams into persistence images and aligns real and synthetic summaries, with the topology loss added to the training objective.The method uses persistence images as a stable differentiable signal for preserving multi-scale structure.
5. Experiment
RETA is evaluated across benchmarks, architectures, ablations, efficiency, corruption robustness, and continual learning. It consistently improves accuracy while preserving a favorable computational trade-off and broader synthetic diversity.
- 5.2. Comparison with the State-of-the-art: RETA achieves the best reported accuracy across five benchmarks, IPC values, and backbone depths, including gains of 3.5 points on ImageNette and 3.1 points on ImageNet-1K.The comparison covers ResNet-18/50/101 and IPC ∈ 1, 10, 50, with RETA consistently surpassing FADRM+.
- 5.2. Comparison with the State-of-the-art: RETA also leads across CNN and transformer architectures on ImageNet-1K, with gains persisting from lightweight mobile models to deeper and wider networks.The evaluated architectures include MobileNetV2, ShuffleNetV2-0.5x, EfficientNet-B0, Wide-ResNet50-2, DenseNet variants, and Swin-Tiny.
- 5.3. Ablation Study: Enabling both DRC and PTA produces the largest ablation gains, exceeding the sum of individual improvements and reaching a +3.4-point boost on Tiny-ImageNet.DRC improves local feature fidelity, while PTA preserves global class geometry, yielding complementary local- and global-scale refinement.
- 5.3. Ablation Study: Hyperparameter sweeps show interior optima: DRC peaks at 52.9% with λ=0.1, while topology alignment peaks at 52.5% with λtopo=0.5.Accuracy declines when either complexity control or topology regularization is weighted too weakly or too strongly.
- 5.4. Computational Efficiency: RETA adds modest overhead over FADRM+—1.31s versus 1.09s and 13.4GB versus 11.0GB—while remaining lighter than heavier decoupled baselines.The reported ImageNet-1K comparison measures single-image generation time on one RTX 4090 GPU and peak GPU memory.
- 5.5. Robustness to Corruptions: RETA improves robustness under corruption and continual learning, achieving about +3.2 points over FADRM+ on ImageNet-Subset-C and maintaining an advantage as replay memory grows.Its distilled sets perform best across all listed corrupted subsets and continue improving in the class-incremental setting.
6. Conclusion
RETA addresses two failure modes of static residual anchors by combining adaptive patch retrieval with persistent topology alignment. It achieves consistent state-of-the-art accuracy across datasets and transfers across CNN and ViT students, while relying on several fixed design choices and adding modest distillation overhead.
- 6. Conclusion: RETA combines Dynamic Retrieval Connection and Persistent Topology Alignment to improve diversity and preserve class geometry in decoupled dataset distillation.DRC adaptively injects real patches using a fit–complexity score, while PTA uses persistent-homology loss to preserve multi-scale class topology.
- 6. Conclusion: RETA delivers consistent state-of-the-art accuracy across multiple datasets under standard images-per-class budgets and transfers robustly across CNN and ViT students.
- 6. Conclusion: RETA relies on a frozen teacher, per-class retrieval pools, and several topology and hypergraph choices, including k-NN construction, persistence-image grids, and λtopo.Its DRC complexity proxy is hand-crafted, and PTA adds modest overhead during distillation.