PaperlayerView exact revision on arXiv

Source-linked AI summary

DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers

Xianing Chen, Qiong Cao, Yujie Zhong, Jing Zhang, Shenghua Gao, Dacheng Tao

arXiv:2204.12997v2cs.CV

TL;DR

Vision transformers’ strong performance depends on large training datasets, motivating more data-efficient training. DearKD distills CNN inductive biases early and then trains without distillation, with reported gains across full, partial, and data-free settings.

  • Problem

    Vision transformers lack certain inductive biases and their strong performance depends on enormous training datasets, motivating data-efficient training.

    Source: §1 · p. 1

  • Method

    DearKD distills CNN classification logits and intermediate-layer inductive biases early, then trains the transformer without distillation; for data-free training it adds a DeepInversion-based boundary-preserving intra-divergence loss.

    Source: §1 · p. 2

  • Results

    DearKD reports superior performance over baselines and state-of-the-art methods across ImageNet, partial ImageNet, data-free settings, and downstream tasks; its data-free DeiT-Ti reaches 71.2% on ImageNet, 1.0% below its full-ImageNet counterpart.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The framework applies early CNN-bias distillation followed by undistilled transformer training across full-data, partial-data, and data-free settings.

    Source: §1 · p. 2, §6 · p. 8

    DF-DearKD has difficulty handling human-related classes because feature statistics store limited information, and generating many samples costs time and computation while leaving a gap versus real-image training.

    Source: E. Limitation and Future works · p. 12

Abstract

from arXiv · show

Transformers are successfully applied to computer vision due to their powerful modeling capacity with self-attention. However, the excellent performance of transformers heavily depends on enormous training images. Thus, a data-efficient transformer solution is urgently needed. In this work, we propose an early knowledge distillation framework, which is termed as DearKD, to improve the data efficiency required by transformers. Our DearKD is a two-stage framework that first distills the inductive biases from the early intermediate layers of a CNN and then gives the transformer full play by training without distillation. Further, our DearKD can be readily applied to the extreme data-free case where no real images are available. In this case, we propose a boundary-preserving intra-divergence loss based on DeepInversion to further close the performance gap against the full-data counterpart. Extensive experiments on ImageNet, partial ImageNet, data-free setting and other downstream tasks prove the superiority of DearKD over its baselines and state-of-the-art methods.

1. Introduction

DearKD addresses vision transformers’ reliance on large training datasets by transferring CNN inductive biases early, then letting transformers train without distillation. It evaluates full-data, partial-data, and data-free settings, with reported gains in each setting.

1. Introduction

  • Transformers’ self-attention models long-range dependencies, but their lack of certain inductive biases means strong performance relies heavily on large training datasets.CNN locality and weight sharing provide inductive biases associated with sample efficiency.Source: §1 · p. 1
  • DeiT distills CNN knowledge from classification logits, which can leave shallow transformer layers without explicit guidance to capture inductive biases; distillation throughout training may also hinder transformers’ own biases and representations.DearKD confines distillation to its first stage to address these drawbacks.Source: §1 · pp. 1–2
  • DearKD distills CNN classification logits and intermediate-layer features during early training, then trains the transformer without distillation in a second stage.A Multi-Head Convolutional-Attention layer mimics convolution without constraining self-attention, and an aligner addresses feature misalignment.Source: §1 · p. 2
  • For data-free training, DearKD adds a DeepInversion-based boundary-preserving intra-divergence loss that diversifies generated images by separating positive samples in latent space while maintaining class boundaries.The framework is evaluated with full ImageNet, partial ImageNet, and no real training images.Source: §1 · p. 2
  • 71.2% on ImageNet is the data-free DearKD result, only 1.0% below its full-ImageNet counterpart; with 50% of ImageNet, DearKD outperforms the baseline transformer trained on all data.With full ImageNet, it achieves state-of-the-art image-classification performance with similar or less computation.Source: §1 · p. 2

2. Related work

Related work covers knowledge distillation, vision transformers, and data-free distillation. The cited approaches motivate DearKD’s two-stage approach and its use of a boundary-preserving loss for data-free training.

Knowledge Distillation

  • Knowledge distillation transfers information from a teacher to a student, including output statistics, attention, activation boundaries, Jacobians, and structured knowledge; the cited methods do not address distillation between different architectures.The paper notes that its teacher has lower capacity than its student.Source: §2 · p. 2

Vision Transformers

  • Vision transformers can perform computer-vision tasks, but their lack of inductive bias is associated with lagging behind CNNs in low-data regimes; approaches that insert convolutions can alter their intrinsic properties.Prior CNN-to-transformer distillation does not account for differences in their representations or transformer-native inductive biases.Source: §2 · p. 2

Data-Free KD

  • Data-free distillation methods include GAN-based synthesis and prior-based data generation, but the cited approaches face mode collapse; DearKD uses a boundary-preserving intra-divergence loss with DeepInversion to generate diverse samples.The pipeline illustration separates convolutional-inductive-bias distillation from transformer-intrinsic-bias learning.Source: §2 · p. 3, Figure 2 · p. 3

3. Data-efficient Early Knowledge Distillation

DearKD first uses convolutional inductive-bias guidance in early transformer layers, then removes distillation so the transformer can learn its own inductive biases. The reported attention-distance pattern shifts from locality in Stage I toward non-local representations in Stage II.

3. Data-efficient Early Knowledge Distillation

  • DearKD uses two stages: early convolutional guidance supports locality, while later training without distillation lets transformers explore their expressive capacity.The method is motivated by the possibility that guidance can restrict transformers in later training.Source: §3 · p. 3, §3.2 · p. 4

3.1. DearKD: Stage I

  • Stage I uses relative-position Multi-Head Convolutional-Attention and distills intermediate CNN features to provide spatial and local-pattern guidance to transformer layers.An aligner matches transformer-token size to CNN-feature size before the intermediate-layer distillation loss is applied.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2. DearKD: Stage II

  • Stage II keeps the relative position encoding unchanged, and average attention distance indicates a shift from locality-focused attention toward non-local representations.The passage attributes the initial locality focus to convolutional inductive-bias distillation and describes the later shift after Stage II training.Source: §3.2 · p. 4

4. DF-DearKD: Training without Real Images

DF-DearKD adapts DearKD to training without real images by adding image generation and a boundary-preserving intra-divergence loss. DeepInversion supplies synthetic training images, while the new loss increases their diversity without moving class boundaries.

4. DF-DearKD: Training without Real Images

  • DF-DearKD adds an image-generation component to DearKD for training without access to real images.The framework introduces a boundary-preserving intra-divergence loss to increase the diversity of generated samples.Source: §4 · pp. 4–5
  • DeepInversion synthesizes images by optimizing a randomly initialized input for a target label, using classification, image-regularization, and diversity terms.Image regularization combines image-prior penalties with batch-normalization feature-statistic matching.Source: §4 · p. 5
  • The boundary-preserving intra-divergence loss separates easiest positive samples in latent space while leaving class boundaries unaffected.It is proposed to address over-clustering associated with hard, ambiguous samples from Adaptive DeepInversion.Source: §4 · p. 5, Figure 6 · p. 5
  • A triplet-style margin keeps anchor-positive pairs closer than anchor-negative pairs, countering the risk that diversity optimization pushes samples outside decision boundaries.The distances use the hardest positive and hardest negative samples in latent space.Source: §4 · p. 5

5. Experiments

The experiments evaluate DearKD’s training setup, data efficiency, ImageNet performance, downstream transfer, and data-free variant. DearKD improves on DeiT across data amounts, achieves strong ImageNet comparisons, and DF-DearKD outperforms other data-free methods.

5. Experiments

  • The experiments assess DearKD through ablations, data-efficiency comparisons, state-of-the-art comparisons, downstream generalization, and DF-DearKD results.The setup uses three model variants based on DeiT and a pre-trained RegNetY-16GF teacher.Source: §5 · p. 6, §5.1 · p. 6

5.1. Implementation Details

  • The DeiT-based DearKD variants increase attention heads while keeping vector dimension unchanged, and the method can extend to other vision transformers.Models use 224 × 224 images and combine MHCA and MHSA blocks; training uses AdamW for 300 epochs, with the first 250 in stage one.Source: §5.1 · p. 6

5.2. Ablation Study

  • The two-stage learning framework achieves the best 74.8% Top-1 accuracy among the ablation settings.MHCA reaches 72.5% Top-1; adding hidden-stage distillation to MHCA brings +2.3%, while hidden-stage distillation alone decreases performance.Source: §5.2 · p. 6, Table 1 · p. 6

5.3. Analysis of Data Efficiency

  • DearKD with 50% of ImageNet training data outperforms the DeiT baseline trained with all data, and its advantage grows as data volume decreases.With all training data, DearKD exceeds the DeiT baseline by about 2.6% absolute accuracy.Source: §5.3 · pp. 6–7, Table 2 · p. 6

5.4. Comparison with Full ImageNet

  • DearKD-Ti reaches 74.8% Top-1 accuracy and DearKD-S reaches 81.5% Top-1 accuracy; DearKD-S beats Swin-T with 7M fewer parameters.DearKD achieves state-of-the-art performance on most evaluated downstream datasets, including CIFAR-10/100, Flowers, and Cars.Source: §5.4 · p. 7

5.5. Performance of DF-DearKD

  • DF-DearKD significantly outperforms other data-free methods, and its DeiT-Ti result is only 1.0% below training from scratch; its intra-divergence loss yields an 8.6% accuracy increase.Its generated images have the highest LPIPS score among the compared methods, though they remain less diverse than real images.Source: §5.5 · p. 8

6. Conclusion

DearKD uses a two-stage framework to distill CNN inductive biases into a transformer before training it without distillation. For the data-free setting, it adds a boundary-preserving intra-divergence loss, and experiments report superior performance over baselines and state-of-the-art methods.

6. Conclusion

  • DearKD first distills inductive biases from early CNN intermediate layers, then trains the transformer without distillation.The second stage is intended to let the transformer fully use its capacity.Source: §6 · p. 8
  • For data-free DearKD, a boundary-preserving intra-divergence loss based on DeepInversion generates diverse training samples.Source: §6 · p. 8
  • Experiments on ImageNet, partial ImageNet, data-free settings, and downstream tasks report that DearKD surpasses baselines and state-of-the-art methods.Source: §6 · p. 8

Appendix

The image regularization term combines an image-prior penalty with batch-normalization regularization of feature-map distributions.

Appendix

B. Generated samples from DF-DearKD

The method generates high-fidelity, high-resolution samples from an ImageNet-pretrained RegNetY-16GF model using only the pretrained teacher.

B. Generated samples from DF-DearKD

C. Analysis of the number of epochs for the first stage of DearKD

The first-stage epoch ablation reports its best ImageNet Top-1 accuracy at 250 epochs; the text also describes choices for 300- and 1000-epoch training schedules.

C. Analysis of the number of epochs for the first stage of DearKD

D. More implement details of DF-DearKD

DF-DearKD filters ambiguous samples and synthesizes 600k images, then trains the student from scratch with a two-stage schedule and data augmentation.

D. More implement details of DF-DearKD

  • After filtering samples with ResNet-101 logits below 0.1, 600k images are synthesized; RegNetY-16GF supplies target labels, while ResNet-101 is the teacher.RegNetY-16GF reaches 100% accuracy on the generated samples; the ResNet-101 teacher has 77.37% ImageNet top-1 accuracy.Source: D. More implement details of DF-DearKD · p. 12
  • The student trains from scratch for 1000 epochs, with 800 epochs in stage one, using AdamW at learning rate 0.0005, cosine scheduling, and batch size 1024.The experiments use four NVIDIA TESLA V100 GPUs.Source: D. More implement details of DF-DearKD · p. 12
  • Training uses Mixup, Cutmix, Random Erasing, and Random Augmentation.Source: D. More implement details of DF-DearKD · p. 12

E. Limitation and Future works

DF-DearKD’s generated-data setting has class-specific and computational constraints, while the authors identify further research directions for inductive biases and data privacy.

E. Limitation and Future works

Related papers

Loading 2204.12997v2…