PaperlayerView exact revision on arXiv

Source-linked AI summary

DeCO: Discriminative Evidence Composition for Fine-Grained Dataset Distillation

Chuixuan Fan, Guang Li, Shijie Wang, Dongzhan Zhou, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui Wang

arXiv:2608.25480v1cs.CVcs.AI

TL;DR

Fine-grained distillation can miss localized evidence when it primarily matches global image statistics. DeCO preserves spatially diverse discriminative regions through attention-guided composition, and it consistently outperforms representative baselines across three benchmarks and IPC budgets.

  • Problem

    Existing distillation objectives primarily capture global statistics and may overlook localized evidence such as object parts, textures, and region-specific structures required for FGVC.

    Source: §1 · p. 1

  • Method

    DeCO uses a pretrained TransFG teacher to score patches, suppresses spatial redundancy, aggregates regions by class, and grid-composes same-class regions for hard-label training.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    DeCO consistently outperforms representative coreset and dataset-distillation baselines across CUB-200-2011, FGVC-Aircraft, and Stanford Cars under different IPC budgets.

    Source: §1 · p. 2, §3.2 · p. 4

  • Takeaways & Limitations

    Localized, spatially diverse evidence can be preserved in compact distilled images while downstream students train with standard hard labels without teacher logits.

    Source: §1 · p. 2, §2.2 · p. 3

Abstract

from arXiv · show

Dataset distillation compresses a large training set into a compact synthetic set while preserving its downstream utility. However, existing methods primarily preserve global image statistics and may overlook the localized evidence essential for fine-grained visual classification (FGVC), such as object parts, subtle textures, and region-specific structures. We formulate fine-grained dataset distillation as budgeted discriminative-evidence preservation and propose Discriminative Evidence Composition (DeCO). DeCO uses attention rollout from a pretrained TransFG teacher to identify informative patches, applies spatial diversification to reduce redundant coverage, and organizes the resulting regions into class-wise evidence banks. Multiple same-class regions are then packed into compact grid-composed images. The teacher is used only for dataset construction, whereas downstream students are trained with standard hard-label supervision without teacher logits. Experiments on CUB-200-2011, FGVC-Aircraft, and Stanford Cars show that DeCO consistently outperforms representative coreset and dataset-distillation baselines under different IPC budgets.

1 Introduction

Fine-grained dataset distillation must preserve localized, class-specific evidence that global-statistics objectives may overlook. DeCO addresses this as budgeted discriminative-evidence preservation using spatially diverse regions composed into compact images.

  • Global-statistics objectives may overlook object parts, textures, and region-specific structures needed for fine-grained classification.Source: §1 · p. 1
  • DeCO formulates distillation as preserving dense and spatially diverse class-specific evidence under a fixed image budget.Source: §1 · p. 1
  • DeCO scores patches with attention rollout, diversifies selected regions, aggregates them by class, and packs them into grid-composed images.Source: §1 · p. 2, Figure 1 · p. 2
  • The teacher guides dataset construction, while students use standard hard-label supervision without teacher logits.Source: §1 · p. 2
  • Across three FGVC benchmarks and different IPC budgets, DeCO consistently outperforms representative coreset and dataset-distillation baselines.Source: §1 · p. 2

2 Method

DeCO mines discriminative regions with a frozen TransFG teacher, suppresses spatial redundancy, and organizes selected crops into class-wise evidence banks. Same-class regions are grid-composed into distilled images for hard-label student training.

Discriminative Evidence Mining

  • A TransFG teacher is trained on the original set and used with attention rollout to score candidate patches by accumulated class-token attention.Source: §2.1 · pp. 2–3
  • Greedy distance-based spatial suppression retains high-scoring candidates while reducing redundant neighboring coverage and promoting complementary evidence.Source: §2.1 · p. 3

Evidence Bank and Grid Composition

  • Selected locations become fixed-size crops organized into class-specific evidence banks.Source: §2.2 · p. 3
  • For each distilled image, M = g^2 same-class regions are arranged with a grid-composition operator and assigned the class label.Source: §2.2 · p. 3
  • Grid composition allocates the fixed pixel budget to multiple discriminative regions, especially when IPC = 1 requires one image to represent an entire class.Source: §2.2 · p. 3

Student Training

  • After construction, the teacher is discarded; students train on the distilled set using standard hard labels without logits, soft labels, or auxiliary distillation losses.Source: §2.3 · p. 3

3 Experiments

DeCO is evaluated across three fine-grained benchmarks and IPC budgets using fixed distilled images and hard-label student training. It achieves the strongest reported accuracy pattern and yields more localized student attention than RDED.

Experimental Settings

  • Experiments cover CUB-200-2011, FGVC-Aircraft, and Stanford Cars at IPC budgets of 1, 3, and 5.Source: §3.1 · p. 3
  • DeCO uses four regions per distilled image, a 28% region-area ratio, and images generated once before student training.Source: §3.1 · p. 3
  • Comparisons include Uniform, RDED, SRe2L++, FADRM+, and DeCOrand, with RDED evaluated under a unified hard-label protocol.Source: §3.1 · p. 3

Student Attention Visualization

  • DeCO-trained students generally concentrate Grad-CAM responses on discriminative object regions more than RDED-trained students.Source: Figure 2 · p. 4, §3.3 · p. 4

Main Results

  • At IPC = 1, DeCO reaches 65.53% on CUB-200-2011, 66.04% on FGVC-Aircraft, and 63.94% on Stanford Cars.Source: §3.2 · p. 4
  • DeCO achieves the highest accuracy on all three datasets at every evaluated IPC budget and consistently outperforms DeCOrand.Source: §3.2 · p. 4

4 Conclusion

DeCO preserves localized discriminative evidence through attention-guided selection, spatial diversification, class-wise aggregation, and grid composition. Across three fine-grained benchmarks, it improves compact distilled-image training under varying budgets.

  • DeCO combines attention-guided region selection, spatial diversification, class-wise evidence aggregation, and grid composition for compact hard-label student training.Source: §4 · p. 5

Dataset Distillation

  • Dataset distillation methods are categorized into gradient, distribution, trajectory, decoupled, and generative distillation approaches.Source: A Related Work · p. 9

Patch-based and Fine-grained Dataset Distillation

  • Patch-based methods improve pixel utilization by composing multiple local regions, but confidence-based selection may repeatedly retain similar parts or background content.Source: A Related Work · p. 9

Fine-grained Visual Classification

  • Fine-grained classification benefits from localized object parts, textures, and region-specific structures because categories can differ subtly despite visual similarity.Source: A Related Work · p. 9

B An Evidence-Preservation Analysis

The analysis explains why grid-composed images can support hard-label training when selected regions retain positive evidence and avoid excessive redundancy. It is an informal rationale, not a formal downstream-accuracy guarantee.

  • The analysis is explicitly informal and does not guarantee downstream accuracy; it clarifies how evidence strength, spatial diversity, and composition affect supervision reliability.Source: B An Evidence-Preservation Analysis · p. 9

Evidence preservation

Evidence concentration under spatial diversity

Effective evidence regions

C Additional Experimental Results

Additional experiments examine sensitivity to region area ratio, teacher patch size, and attention rollout depth using IPC=1 and the standard hard-label training protocol. Figure 3 reports mean top-1 accuracy across cropped-region area ratios on two benchmarks.

  • Sensitivity experiments use IPC=1, a randomly initialized TransFG student, and the same hard-label training protocol as the main experiments.Source: C Additional Experimental Results · p. 10
  • Figure 3 compares mean top-1 accuracy under different cropped-region area ratios on CUB-200-2011 and FGVC-Aircraft.Source: Figure 3 · p. 11

C.1 Impact of Region Area Ratio

Region area ratio trades off localized evidence against contextual information. Intermediate region sizes provide the most favorable and stable balance, motivating a common 28% ratio across benchmarks.

  • Very small regions preserve local parts or textures but may discard structural information, while excessively large regions add background and reduce discriminative-evidence density.Source: §C.1 · p. 11
  • Intermediate region sizes balance local evidence and contextual information, with performance remaining stable across an intermediate range.Source: §C.1 · p. 11
  • The main experiments use a common region area ratio of 28% across all benchmarks.Source: §C.1 · p. 11

C.2 Impact of Initial Patch Size

Teacher patch size has limited effect on DeCO, while rollout depth is varied independently and stabilizes at later Transformer blocks.

  • Teacher patch sizes of 16 × 16 and 32 × 32 are compared for their effect on DeCO’s initial patch granularity.Source: §C.2 · p. 11
  • Accuracy changes by at most 0.2 percentage points when changing the teacher patch size.The selected patch mainly determines the center of a larger evidence crop, which is later cropped and resized.Source: §C.2 · p. 11
  • Attention rollout depth is varied by accumulating head-averaged, residual-normalized attention through selected Transformer blocks.Only rollout depth changes; region extraction, evidence-bank construction, composition, and student training remain fixed.Source: §C.3 · p. 11
  • Accuracy changes by only 0.15 points on CUB-200-2011 and 0.08 points on FGVC-Aircraft between rollout depths 5 and 11.Performance improves through early and middle blocks before becoming stable at later depths.Source: §C.3 · p. 11

C.4 Qualitative Comparison with RDED

Under the same four-region budget, DeCO retains more localized, complementary class-specific evidence than RDED’s larger or weakly localized regions.

  • DeCO and RDED are compared using the same four-region composition budget.Source: §C.4 · p. 11, Figure 4 · p. 12
  • DeCO retains more localized class-specific evidence, whereas RDED may preserve larger background or weakly localized regions.The figure also shows top-20 predicted class probabilities for a representative CUB-200-2011 category.Source: Figure 4 · p. 12
  • Teacher attention and spatial suppression select complementary class-specific regions for DeCO.This changes the information retained within the fixed pixel budget rather than increasing the number of composed regions.Source: §C.4 · p. 12

D Visualization of Distilled Samples

The paper visualizes DeCO’s IPC=1 distilled samples across the evaluated fine-grained datasets and their class subsets.

NeurIPS Paper Checklist

The checklist finds that the paper’s claims match its evaluated contributions and scope, while identifying omissions in limitations discussion, open-access code, statistical uncertainty reporting, compute disclosure, and societal-impact discussion.

  • The abstract and introduction accurately describe DeCO’s focus, contributions, and evaluation on three fine-grained benchmarks without claiming broader generalization.Source: NeurIPS Paper Checklist · p. 18
  • The manuscript does not report full hardware configuration, per-experiment execution time, total compute, or a dedicated discussion of limitations and both positive and negative societal impacts.Source: NeurIPS Paper Checklist · pp. 18, 22

Theory assumptions and proofs

  • Appendix B states Proposition 1’s assumptions and provides a complete proof of its evidence-failure bound using Chebyshev’s inequality.Source: NeurIPS Paper Checklist · p. 19

Experimental result reproducibility

  • The paper provides methodological and experimental details sufficient to reproduce the reported results, with additional implementation information in the appendix and supplement.Source: NeurIPS Paper Checklist · pp. 19–20

Open access to data and code

  • The experiments use public benchmark datasets, but no open-access code repository or public implementation release was available at submission time.Source: NeurIPS Paper Checklist · p. 20

Related papers

Loading 2608.25480v1…