Source-linked AI summary
DeCO: Discriminative Evidence Composition for Fine-Grained Dataset Distillation
Chuixuan Fan, Guang Li, Shijie Wang, Dongzhan Zhou, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui Wang
TL;DR
Fine-grained distillation can miss localized evidence when it primarily matches global image statistics. DeCO preserves spatially diverse discriminative regions through attention-guided composition, and it consistently outperforms representative baselines across three benchmarks and IPC budgets.
Problem
Existing distillation objectives primarily capture global statistics and may overlook localized evidence such as object parts, textures, and region-specific structures required for FGVC.
Method
DeCO uses a pretrained TransFG teacher to score patches, suppresses spatial redundancy, aggregates regions by class, and grid-composes same-class regions for hard-label training.
Results
DeCO consistently outperforms representative coreset and dataset-distillation baselines across CUB-200-2011, FGVC-Aircraft, and Stanford Cars under different IPC budgets.
Takeaways & Limitations
Localized, spatially diverse evidence can be preserved in compact distilled images while downstream students train with standard hard labels without teacher logits.
Abstract
from arXiv · showhide
Dataset distillation compresses a large training set into a compact synthetic set while preserving its downstream utility. However, existing methods primarily preserve global image statistics and may overlook the localized evidence essential for fine-grained visual classification (FGVC), such as object parts, subtle textures, and region-specific structures. We formulate fine-grained dataset distillation as budgeted discriminative-evidence preservation and propose Discriminative Evidence Composition (DeCO). DeCO uses attention rollout from a pretrained TransFG teacher to identify informative patches, applies spatial diversification to reduce redundant coverage, and organizes the resulting regions into class-wise evidence banks. Multiple same-class regions are then packed into compact grid-composed images. The teacher is used only for dataset construction, whereas downstream students are trained with standard hard-label supervision without teacher logits. Experiments on CUB-200-2011, FGVC-Aircraft, and Stanford Cars show that DeCO consistently outperforms representative coreset and dataset-distillation baselines under different IPC budgets.
1 Introduction
Fine-grained dataset distillation must preserve localized, class-specific evidence that global-statistics objectives may overlook. DeCO addresses this as budgeted discriminative-evidence preservation using spatially diverse regions composed into compact images.
- Global-statistics objectives may overlook object parts, textures, and region-specific structures needed for fine-grained classification.
- DeCO formulates distillation as preserving dense and spatially diverse class-specific evidence under a fixed image budget.
- DeCO scores patches with attention rollout, diversifies selected regions, aggregates them by class, and packs them into grid-composed images.
- The teacher guides dataset construction, while students use standard hard-label supervision without teacher logits.
- Across three FGVC benchmarks and different IPC budgets, DeCO consistently outperforms representative coreset and dataset-distillation baselines.
2 Method
DeCO mines discriminative regions with a frozen TransFG teacher, suppresses spatial redundancy, and organizes selected crops into class-wise evidence banks. Same-class regions are grid-composed into distilled images for hard-label student training.
- Discriminative Evidence Mining: A TransFG teacher is trained on the original set and used with attention rollout to score candidate patches by accumulated class-token attention.
- Discriminative Evidence Mining: Greedy distance-based spatial suppression retains high-scoring candidates while reducing redundant neighboring coverage and promoting complementary evidence.
- Evidence Bank and Grid Composition: Selected locations become fixed-size crops organized into class-specific evidence banks.
- Evidence Bank and Grid Composition: For each distilled image, M = g^2 same-class regions are arranged with a grid-composition operator and assigned the class label.
- Evidence Bank and Grid Composition: Grid composition allocates the fixed pixel budget to multiple discriminative regions, especially when IPC = 1 requires one image to represent an entire class.
- Student Training: After construction, the teacher is discarded; students train on the distilled set using standard hard labels without logits, soft labels, or auxiliary distillation losses.
3 Experiments
DeCO is evaluated across three fine-grained benchmarks and IPC budgets using fixed distilled images and hard-label student training. It achieves the strongest reported accuracy pattern and yields more localized student attention than RDED.
- Experimental Settings: Experiments cover CUB-200-2011, FGVC-Aircraft, and Stanford Cars at IPC budgets of 1, 3, and 5.
- Experimental Settings: DeCO uses four regions per distilled image, a 28% region-area ratio, and images generated once before student training.
- Experimental Settings: Comparisons include Uniform, RDED, SRe2L++, FADRM+, and DeCOrand, with RDED evaluated under a unified hard-label protocol.
- Student Attention Visualization: DeCO-trained students generally concentrate Grad-CAM responses on discriminative object regions more than RDED-trained students.
- Main Results: At IPC = 1, DeCO reaches 65.53% on CUB-200-2011, 66.04% on FGVC-Aircraft, and 63.94% on Stanford Cars.
- Main Results: DeCO achieves the highest accuracy on all three datasets at every evaluated IPC budget and consistently outperforms DeCOrand.
4 Conclusion
DeCO preserves localized discriminative evidence through attention-guided selection, spatial diversification, class-wise aggregation, and grid composition. Across three fine-grained benchmarks, it improves compact distilled-image training under varying budgets.
- DeCO combines attention-guided region selection, spatial diversification, class-wise evidence aggregation, and grid composition for compact hard-label student training.
- Dataset Distillation: Dataset distillation methods are categorized into gradient, distribution, trajectory, decoupled, and generative distillation approaches.
- Patch-based and Fine-grained Dataset Distillation: Patch-based methods improve pixel utilization by composing multiple local regions, but confidence-based selection may repeatedly retain similar parts or background content.
- Fine-grained Visual Classification: Fine-grained classification benefits from localized object parts, textures, and region-specific structures because categories can differ subtly despite visual similarity.
B An Evidence-Preservation Analysis
The analysis explains why grid-composed images can support hard-label training when selected regions retain positive evidence and avoid excessive redundancy. It is an informal rationale, not a formal downstream-accuracy guarantee.
- The analysis is explicitly informal and does not guarantee downstream accuracy; it clarifies how evidence strength, spatial diversity, and composition affect supervision reliability.
- Evidence preservation: A composed image retains positive expected class evidence when the expected regional evidence exceeds composition distortion, µc > ϵA.
- Evidence concentration under spatial diversity: Proposition 1 bounds the probability that a composed image fails to retain positive class evidence under the stated assumptions.
- Evidence concentration under spatial diversity: Spatial redundancy controls evidence concentration: composing more regions helps when their correlations remain controlled.
- Effective evidence regions: The failure bound decreases proportionally to 1/Meff, so highly correlated regions limit the benefit of increasing region count.
- Effective evidence regions: DeCO uses M = 4 regions per image, with spatial suppression encouraging complementary rather than repetitive evidence.
- Intermediate crop sizes balance evidence strength and contextual information, whereas very small or large crops increase composition distortion or redundant content.
C Additional Experimental Results
Additional experiments examine sensitivity to region area ratio, teacher patch size, and attention rollout depth using IPC=1 and the standard hard-label training protocol. Figure 3 reports mean top-1 accuracy across cropped-region area ratios on two benchmarks.
- Sensitivity experiments use IPC=1, a randomly initialized TransFG student, and the same hard-label training protocol as the main experiments.
- Figure 3 compares mean top-1 accuracy under different cropped-region area ratios on CUB-200-2011 and FGVC-Aircraft.
C.1 Impact of Region Area Ratio
Region area ratio trades off localized evidence against contextual information. Intermediate region sizes provide the most favorable and stable balance, motivating a common 28% ratio across benchmarks.
- Very small regions preserve local parts or textures but may discard structural information, while excessively large regions add background and reduce discriminative-evidence density.
- Intermediate region sizes balance local evidence and contextual information, with performance remaining stable across an intermediate range.
- The main experiments use a common region area ratio of 28% across all benchmarks.
C.2 Impact of Initial Patch Size
Teacher patch size has limited effect on DeCO, while rollout depth is varied independently and stabilizes at later Transformer blocks.
- Teacher patch sizes of 16 × 16 and 32 × 32 are compared for their effect on DeCO’s initial patch granularity.
- Accuracy changes by at most 0.2 percentage points when changing the teacher patch size.The selected patch mainly determines the center of a larger evidence crop, which is later cropped and resized.
- Attention rollout depth is varied by accumulating head-averaged, residual-normalized attention through selected Transformer blocks.Only rollout depth changes; region extraction, evidence-bank construction, composition, and student training remain fixed.
- Accuracy changes by only 0.15 points on CUB-200-2011 and 0.08 points on FGVC-Aircraft between rollout depths 5 and 11.Performance improves through early and middle blocks before becoming stable at later depths.
C.4 Qualitative Comparison with RDED
Under the same four-region budget, DeCO retains more localized, complementary class-specific evidence than RDED’s larger or weakly localized regions.
- DeCO and RDED are compared using the same four-region composition budget.
- DeCO retains more localized class-specific evidence, whereas RDED may preserve larger background or weakly localized regions.The figure also shows top-20 predicted class probabilities for a representative CUB-200-2011 category.
- Teacher attention and spatial suppression select complementary class-specific regions for DeCO.This changes the information retained within the fixed pixel budget rather than increasing the number of composed regions.
D Visualization of Distilled Samples
The paper visualizes DeCO’s IPC=1 distilled samples across the evaluated fine-grained datasets and their class subsets.
- At IPC=1, CUB-200-2011 samples are visualized for the first 100 classes and the last 100 classes.
- At IPC=1, Stanford Cars samples are visualized for the first 100 classes and the remaining 96 classes.
- At IPC=1, distilled samples from all 100 FGVC-Aircraft classes are visualized.
NeurIPS Paper Checklist
The checklist finds that the paper’s claims match its evaluated contributions and scope, while identifying omissions in limitations discussion, open-access code, statistical uncertainty reporting, compute disclosure, and societal-impact discussion.
- The abstract and introduction accurately describe DeCO’s focus, contributions, and evaluation on three fine-grained benchmarks without claiming broader generalization.
- The manuscript does not report full hardware configuration, per-experiment execution time, total compute, or a dedicated discussion of limitations and both positive and negative societal impacts.
- Theory assumptions and proofs: Appendix B states Proposition 1’s assumptions and provides a complete proof of its evidence-failure bound using Chebyshev’s inequality.
- Experimental result reproducibility: The paper provides methodological and experimental details sufficient to reproduce the reported results, with additional implementation information in the appendix and supplement.
- Open access to data and code: The experiments use public benchmark datasets, but no open-access code repository or public implementation release was available at submission time.