Source-linked AI summary
IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves
Feyza Yavuz, Mert Bülent Sarıyıldız, Diane Larlus
TL;DR
Multi-teacher distillation usually requires real images, but those data may be unavailable or restricted. The paper optimizes procedural Dead Leaves samples with teacher-guided decorrelation losses, and IDeaL produces competitive students, matching or surpassing a 1K-image ImageNet subset while narrowing the real-data gap.
Problem
Multi-teacher distillation typically relies on large real-image datasets, which may be unavailable because of licensing, privacy, or proprietary restrictions.
Method
The method optimizes procedural Dead Leaves images through frozen teacher encoders using patch- and image-level decorrelation losses, then distills all teachers into a student.
Results
IDeaL closes much of the performance gap with real-image distillation; with 1K images, it matches or surpasses a 1K-image ImageNet subset and improves over Dead Leaves across tasks.
Takeaways & Limitations
Teacher information can tailor procedural synthetic data for multi-teacher distillation without real images, producing students that capture complementary teacher knowledge.
Takeaways & Limitations
Procedural synthetic data, including IDeaL, scales less well than ImageNet, particularly for dense tasks, at larger subset sizes.
Abstract
from arXiv · showhide
Multi-teacher distillation has emerged as a way to combine complementary teacher models into a single student model that exhibits the strengths of all its teachers. The student is trained to mimic the output of the teachers on a set of images, typically the union of the individual teacher's training sets, assuming this data is available. In this paper, we question that assumption and explore alternative options. We first study how far one can go when distilling from teachers fed with different types of noise. Then, we show that information contained in the teachers can be leveraged to tailor the noise for multi-teacher distillation: we propose a method that, thanks to decorrelation losses at both patch and image levels, generates teacher-specific, improved samples optimized for data-free distillation. Experiments show that our most effective samples, IDeaL, lead to strong students that successfully capture complementary information from the teachers, yielding surprisingly competitive results that substantially narrow the gap with students distilled from real images. Moreover, given a limited budget of 1K images for distillation, students distilled using our IDeaL samples match or surpass the performance of those distilled using a 1K-image subset of ImageNet.
1 Introduction
The paper asks whether multi-teacher distillation can work without real images and develops teacher-tailored synthetic data to narrow the gap with real-image distillation.
- Multi-teacher distillation typically depends on large real-image datasets, which may be unavailable, costly, or restricted by privacy, licensing, or proprietary policies.
- The study replaces real images with procedural surrogates, first comparing noise options and finding that Dead Leaves consistently outperforms other alternatives.
- IDeaL optimizes Dead Leaves using teacher attention information and patch- and image-level decorrelation so samples are informative for multiple teachers.
- The controlled UNIC-based evaluation measures distilled students across diverse classification and dense-prediction tasks while isolating the effect of synthetic data.
- 1K IDeaL images match or surpass a 1K-image ImageNet subset, while 1M IDeaL samples produce competitive students across downstream tasks.
2 Related Work
Prior work addresses data-free learning through teacher statistics, model inversion, or procedural data, whereas this paper targets multi-teacher distillation with jointly optimized synthetic images.
- Knowledge distillation trains a student from teacher outputs or intermediate features, and multi-teacher distillation aligns one student with several complementary teachers.
- This approach uses procedural synthetic data optimized through teacher attention features to be informative across all teachers simultaneously.
- Existing data-free methods often synthesize data using teacher activations, internal statistics, or class information, introducing assumptions about teacher-side knowledge.
- ViT inversion methods use patch-similarity regularization but target single-model quantization and small sample sets rather than multi-teacher distillation.
- Procedural datasets support data-free pretraining and single-teacher distillation, but prior methods generally use them as-is and do not address heterogeneous teachers.
3 Background
The framework distills multiple ViT teachers into one ViT student by matching patch and global features, while replacing the usual real-image training set with synthetic alternatives.
- The setup trains a ViT student from multiple pretrained ViT teachers, each producing patch features and a global CLS feature.
- The distillation objective sums teacher-specific losses comparing student and teacher CLS and patch features using similarity measures such as cosine similarity or smooth-ℓ1 distance.
- The combined objective encourages a unified student representation that captures complementary knowledge encoded by the teachers.
- The method uses a two-stage pipeline: improve synthetic inputs, then train the student with the resulting data as the distillation set.
- Unlike standard multi-teacher methods using large real-image collections, this study assumes no real distillation data and evaluates Gaussian noise and procedural datasets.
4 Optimizing images for multi-teacher distillation
The method optimizes structured-noise images through frozen teacher encoders, encouraging diverse patch representations within images and diverse global representations across images before distillation.
- Image optimization: Synthetic images are initialized from structured noise, optimized through frozen teacher encoders, and stored for subsequent student distillation.
- Patch decorrelation loss: Patch decorrelation addresses homogeneous noise by encouraging distinct patch representations across all teachers’ attention features.
- Patch decorrelation loss: Minimizing the patch loss drives off-diagonal patch similarities toward zero, making patches dissimilar within each image.
- Image decorrelation loss: Image decorrelation prevents mode collapse by encouraging distinct global representations across generated samples.
- Total image generation loss: The total generation loss combines total-variation smoothness with patch- and image-decorrelation terms summed across teachers.
- Training the student: The decoupled two-stage design pays generation cost once, allowing the synthetic dataset to be reused across student architectures or distillation configurations.
5 Results
The experiments replace ImageNet with synthetic data in a controlled UNIC multi-teacher distillation setting, then evaluate IDeaL across classification, transfer, segmentation, and depth tasks. IDeaL consistently improves over Dead Leaves and performs competitively with real-image subsets in data-scarce settings, while scaling less effectively at larger sizes.
- 5 Results: IDeaL encodes teacher-representation information through pixel optimization with patch- and image-level decorrelation losses, using frozen teachers to generate distillation samples.The samples are initialized from Dead Leaves and then used to train the student through multi-teacher distillation.
- 5 Results: The controlled evaluation uses UNIC with four ImageNet-trained ViT-B/16 teachers and tests ImageNet classification, transfer learning, semantic segmentation, and depth estimation.Synthetic sources are compared at 1K, 10K, 100K, and 1M samples, with three random subsets for each applicable source and size.
- 5.2 From real images to data-free distillation: Dead Leaves is the strongest structured-noise baseline: 100K samples achieve 65.6 transfer accuracy, matching the 65.3 score from a 1K ImageNet subset.Gaussian noise performs poorly at every size, while FractalDB remains below Dead Leaves.
- 5.3 Tailoring Dead Leaves for multi-teacher distillation: IDeaL improves over original Dead Leaves in all settings and tasks, including 14% and 17% relative gains on ImageNet classification with 1K and 1M samples.The largest reported gain is an 18% relative improvement for 1M-sample semantic segmentation; transfer learning improves by 7% overall.
- 5.3 Tailoring Dead Leaves for multi-teacher distillation: With only 1K samples, IDeaL students surpass the minimum teacher performance in transfer learning, semantic segmentation, and depth estimation.In the 1M setting, transfer learning gains +2.3 Top-1 accuracy and segmentation gains +4.1 mIoU, while ImageNet classification reaches 78.2 versus 78.4 for the minimum teacher.
- 5.3 Tailoring Dead Leaves for multi-teacher distillation: In data-scarce classification, IDeaL outperforms ImageNet subsets by +3.3 and +3.4 transfer Top-1 accuracy at 1K and 10K images, respectively.The 1K IDeaL student also exceeds the 1K ImageNet student by +1.5 ImageNet accuracy, while 10K IDeaL matches 10K ImageNet.
- 5.3 Tailoring Dead Leaves for multi-teacher distillation: At larger subset sizes, procedural synthetic data including IDeaL scales less effectively than ImageNet, especially for dense prediction tasks.A performance gap with ImageNet remains at 1M samples, despite IDeaL outperforming Dead Leaves by 20% on ImageNet classification and 27% on semantic segmentation.
- 5.3 Tailoring Dead Leaves for multi-teacher distillation: Using more teachers during optimization improves performance, with joint optimization over all four teachers producing the strongest results.The ablation also finds LPD is the main source of gains, LID provides a complementary boost, and combining both losses gives the best results.
6 Conclusion
The paper develops data-free multi-teacher distillation using optimized structured noise that captures information from all teachers. IDeaL improves students across tasks without teacher- or dataset-specific information.
- IDeaL uses pixel-level optimization to improve structured noise and capture information from all teachers.The objective is designed to remain truly data-free.
- IDeaL significantly boosts student performance across all tasks compared with non-optimized synthetic distillation sets.
- The optimization objectives require no teacher- or dataset-specific information and can potentially extend to other synthetic data, teachers, and downstream tasks.
Supplementary Material for IDeaL: Data-Free Multi-Teacher Distillation
The supplementary material details the UNIC-based distillation framework, its teacher and student configurations, losses, and synthetic-sample generation procedure.
- The experiments adopt UNIC as the multi-teacher distillation framework and use it consistently across experiments.
- The distillation loss combines cosine-similarity and smooth-ℓ1 terms applied to both [CLS] and patch features.Cosine similarity aligns feature directions, while smooth-ℓ1 penalizes magnitude differences.
- The setup uses four ViT-B/16 ImageNet teachers: DINO, iBOT, DeiT-3, and dBOT-ft.All teachers use 16-pixel patches and 224 × 224 inputs.
- The student is a ViT-B/16 model matching the teachers’ architecture and uses teacher-specific projectors, token-specific projectors, and ladder-of-projectors connections.
- Sample-generation hyperparameters receive no detailed search because moderate changes to batch size, iterations, and subset size only slightly affect downstream performance.The ablation also includes samples initialized from Gaussian noise.
B Detailed transfer learning results from Tab. 1
Detailed transfer results show that IDeaL improves over Dead Leaves across concept-generalization, fine-grained, and long-tail classification, with the largest gain on long-tail tasks.
- IDeaL improves transfer performance across all three sub-tasks, achieving an overall 7% gain over Dead Leaves.
- 12% is the largest reported IDeaL gain, occurring on long-tail classification at subset sizes of 100K and 1M.Smaller subsets remain aligned with the overall 7% trend.
- At 1K samples, unoptimized Dead Leaves outperforms a 1K-image ImageNet subset by 4% on long-tail classification.Its fine-grained classification gap is only 1%.
C Extended results with improved Gaussian noise
Extended results show that optimization also strengthens Gaussian-noise distillation, while IDeaL remains superior to Dead Leaves across tasks and competitive with real-data students in scarce-data settings.
- IDeaL outperforms original Dead Leaves across all tasks and exceeds minimum teacher performance across three tasks, even with 1K samples.
- In data-scarce classification settings, IDeaL matches or outperforms students distilled on ImageNet subsets.
- Improved Gaussian noise also produces stronger students than pure Gaussian noise after applying the optimization objective.
- The optimization gain is larger for Gaussian noise than for Dead Leaves because Gaussian noise is a much weaker starting point.
- The extended table reports ImageNet, concept-generalization, fine-grained, long-tail, segmentation, and depth metrics across teachers and synthetic-data students.
- More than a 2× improvement is reported for the optimization claim.
- The Gaussian-noise ablation generates 10K samples per configuration and evaluates optimization losses including LPD, LID, and LTV.
D Ablation on losses with improved Gaussian noise
The ablation finds that combining LID and LPD produces the best results for improved Gaussian noise. LID alone also substantially improves performance across all tasks.
- Using both LID and LPD produces the best results for improved Gaussian noise.
- LID alone yields a substantial performance gain across all tasks.LID leverages information from the [CLS] tokens.
- The results highlight the effectiveness of the proposed optimization method even without both loss components.
E Scaling performance analysis
The scaling analysis compares procedural synthetic datasets, Dead Leaves, IDeaL, and ImageNet across six tasks and four distillation-set sizes. IDeaL scales slightly better than Dead Leaves on transfer tasks but remains limited on dense prediction tasks.
- Procedural synthetic data does not scale well as the distillation set grows.
- IDeaL shows slightly better and more stable scaling than Dead Leaves, especially on concept generalization and long-tail classification.
- IDeaL struggles to scale on semantic segmentation and depth estimation, particularly relative to ImageNet.
- The comparison covers ImageNet classification, concept generalization, fine-grained and long-tail classification, semantic segmentation, and depth estimation at 1K to 1M samples.
F Additional qualitative results
The paper provides qualitative examples of IDeaL and improved Gaussian noise generated with the same optimization setup. The initial image type is the only varied component.
- The qualitative results include examples from both Improved Dead Leaves and improved Gaussian noise.
- Both example sets use exactly the same generation setup; only the initial images vary.
- Figure C shows IDeaL samples initialized from Dead Leaves and produced by the optimization method.
- Figure D shows improved Gaussian noise samples initialized from Gaussian noise and produced by the same optimization method.