Contrastive Model Inversion for Data-Free Knowledge Distillation

1 Introduction

• Data-free KD is needed when teacher training data are unavailable, and model inversion recovers data for student training.Such data should match the original distribution and be diverse enough to support comprehensive student learning.Source: §1 · p. 1
• Existing inversion methods can collapse to highly similar instances, while harder or adversarial examples may lack natural image statistics and appear unrealistic.The cited methods fit priors or mine harder samples without explicitly constraining diversity.Source: §1 · pp. 1–2
• CMI produces more visually plausible instances and significantly superior knowledge-distillation performance on CIFAR-10, CIFAR-100, and Tiny-ImageNet than prior methods.The paper also reports that its approach makes generated data more diverse and realistic.Source: §1 · p. 2
• CMI contrasts a crop with its source image as a positive pair and images from different instances as a negative pair.Its stated contributions include addressing mode collapse while keeping generated data closer to the real distribution.Source: §1 · p. 2
• CMI defines data diversity through instance discrimination and incorporates it explicitly into the optimization objective.The paper observes that, at the same data volume, higher diversity indicates stronger instance discrimination.Source: §1 · p. 2

2 Related Works

• Model inversion reconstructs inputs from a pre-trained model's parameters and has been used for security, interpretability, and knowledge transfer.The standard formulation seeks an input whose mapped representation is close to that of the target input.Source: §2 · p. 2
• Data-free KD learns a student from a cumbersome teacher without real-world data, with prior approaches broadly using adversarial training or data priors.The cited priors include total variation and batch-normalization statistics.Source: §2 · p. 2
• CMI uses contrastive learning's instance-discrimination capability to model data diversity in model inversion.The related-work discussion describes contrastive learning as treating each sample as a distinct category and learning to distinguish samples.Source: §2 · p. 2

3.1 Preliminary

• BN-statistic regularization, one-hot class-prior assumptions, and adversarial teacher–student disagreement are three inversion techniques whose combination yields a unified inversion framework for data-free knowledge distillation.Source: §3.1 · p. 2
• Because inversion criteria admit non-unique solutions, conventional methods can repeatedly synthesize duplicates; CMI adds an explicit diversity constraint.The proposed method aims to keep generated data closer to the real data distribution while alleviating mode collapse.Source: §3.1 · pp. 2–3

3.2 Contrastive Model Inversion

• CMI treats data diversity as sample distinguishability, but teacher-embedding Euclidean distance can be unsuitable because its meaning is unclear, structured information may be missed, and it is unbounded.The paper instead learns a distinguishability metric through contrastive learning.Source: §3.2 · p. 3
• Contrastive learning pulls augmented views of the same instance together and separates different instances, providing a learned metric that can be optimized for diversity.The discriminator operates on teacher features, and the paper relates minimizing contrastive loss to maximizing the defined diversity objective.Source: §3.2 · pp. 3–4
• CMI comprises a generator, teacher network, instance discriminator, and memory bank.The discriminator takes penultimate-layer representations and global pooling of intermediate features as input.Source: §3.2 · p. 4
• CMI synthesizes one batch per timestamp with a freshly initialized generator, comparing new samples with historical samples stored in a memory bank.The generator and latent codes are optimized, while augmentation supplies local and global views; the combined objective adds contrastive loss to the inversion criterion.Source: §3.2 · p. 4

15 end

• The contrastive objective uses the current synthetic batch and history data in memory bank B, stopping gradients on global views while backpropagating through local views.The authors report that this provides clearer gradients for local pattern synthesis.Source: §15 · p. 4
• Synthetic images stored in memory bank B are used for downstream distillation.Source: §15 · p. 4

3.3 Decision Adversarial Distillation

• The decision adversarial loss focuses on boundary samples because large teacher–student disagreement can select outliers, and it moves inputs toward the decision boundary for more teacher information.Adversarial learning is enabled only when teacher and student predictions match.Source: §3.3 · p. 4

4.1 Settings

• Experiments evaluate ResNet, VGG, and wide ResNet architectures on CIFAR-10, CIFAR-100, and Tiny-ImageNet, using synthetic data to train students.Teachers are trained on labeled data; the reported optimization uses Adam for the generator and SGD for student training.Source: §4.1 · pp. 4–5

4.2 Benchmarks on Knowledge Distillation

• CMI achieves superior or comparable performance to most baselines, including with a WRN-40-2 teacher that has about 10× fewer parameters than ResNet-34.The benchmark covers CIFAR-10, CIFAR-100, and Tiny-ImageNet.Source: §4.2 · p. 5, Table 1 · p. 5
• CMI produces more plausible and diverse images than DeepInversion and DFQ, with better local details such as eyes and textures.DeepInversion uses pixel updates, while DFQ uses a generator; DFQ can have more severe mode collapse.Source: §4.2 · p. 5, Figure 2 · p. 5

4.3 Ablation Study

• Cutting both the contrastive loss and generator leads to poor performance, while removing the generator appears to cause more severe degradation.The authors attribute imbalanced optimization to pixel updates affecting only pixels inside a crop.Source: §4.3 · p. 6
• Increasing αcr benefits student learning and improves synthetic-data FID, with particularly significant FID improvements in shallow layers.Shallow layers contain more low-level features; the results examine different data amounts and contrast settings.Source: §4.3 · p. 6, Figure 3 · p. 6, Table 3 · p. 6
• A stronger instance discriminator is reported to help student learning across the tested discriminator designs.The designs include no discriminator, a linear discriminator, and the nonlinear discriminator used in CMI.Source: §4.3 · p. 6, Table 4 · p. 6

• The proposed model inversion approach aims to guarantee synthetic-data diversity and bring benefits to downstream distillation tasks.Source: §5 · p. 6
• The paper reports extensive experiments to demonstrate the approach’s effectiveness and explore how it works.Source: §5 · p. 6
• Model inversion in knowledge distillation may offer insight into learned knowledge and how it transfers between networks.Source: §5 · p. 6