A Comprehensive Survey on Knowledge Distillation

• Large models, including LLMs and foundational vision models, impose substantial runtime and memory costs, especially when deployed on edge devices.Their complex architectures and over-parameterization make training and inference challenging, despite high performance.Source: §1 · p. 2
• Knowledge distillation trains a lightweight student under a more complex teacher’s supervision, often by mimicking the teacher’s output distribution as soft labels.Unlike pruning, quantization, and low-rank factorization, KD does not modify the network’s layers or parameters.Source: §1 · p. 2
• KD also supports knowledge transfer to data-scarce target tasks and data-free distillation through synthetic data when privacy concerns prevent storing sensitive data.Key challenges include deciding what knowledge to transfer, selecting algorithms, and designing teacher and student architectures.Source: §1 · pp. 2–3
• Existing surveys overlook recent feature-based methods, adaptive and contrastive distillation, foundation models and LLMs, and KD for 3D inputs such as point clouds.These gaps motivate broader coverage of rapidly developing distillation settings.Source: §1 · p. 3
• This survey classifies KD by sources, algorithms, schemes, modalities, and applications, covering feature-based methods; adaptive and contrastive algorithms; offline, online, and self-distillation; and video, speech, text, multi-view, and 3D data.It additionally examines self-supervised learning, foundational models, transformers, diffusion models, visual recognition, and LLMs, provides quantitative comparisons, and discusses challenges and future directions.Source: §1 · pp. 3–4

2 Sources

• Knowledge distillation sources are classified into logits, intermediate features, and similarities, with Tables 2–3 summarizing representative methods.Logits are final-layer outputs, features are intermediate outputs, and similarities capture relationships among features, channels, samples, and related structures.Source: §2 · p. 5, Figure 3 · p. 5

2.1 Logit-based Distillation

• Logit-based distillation makes students mimic teacher predictions or soft-label distributions, but depends on predefined classes and offers limited intermediate-layer guidance.Methods refine logits through normalization, temperature adaptation, smoothing, or loss decoupling; soft-label distillation uses temperature-controlled softmax and KL divergence.Source: §2.1 · pp. 6–7

2.2 Feature-based Distillation

• Feature-based distillation aligns teacher and student intermediate representations, providing richer and more generalizable knowledge but requiring careful layer selection, feature alignment, and transformation, especially across architectures.Transformations may match spatial dimensions or channels and include attention, cross-layer matching, masking, MLPs, diffusion-based denoising, and other feature refinements.Source: §2.2 · pp. 7–9

2.3 Similarity-based Distillation

• Similarity-based distillation transfers structural knowledge by matching higher-order relations at instance, feature/channel, class, and spatial levels rather than exact predictions or feature values.Examples include sample correlations and relation graphs, channel correlation maps, inter- and intra-class similarities, and pixel-to-pixel or pixel-to-region relations.Source: §2.3 · pp. 9–10

3 Schemes

• The three schemes suit different conditions: offline distillation uses an available pretrained teacher, online distillation avoids that requirement, and self-distillation supports settings without labeled data.Figure 4 contrasts a pretrained frozen teacher in offline distillation, simultaneous teacher-student training online, and identical teacher-student networks in self-distillation.Source: §3 · p. 10, Figure 4 · p. 11

3.1 Offline Distillation

• Offline distillation uses a larger pretrained, frozen teacher in a two-stage process and remains the most widely used scheme despite resource and teacher-bias limitations.Adaptive techniques can improve student performance (Cho & Hariharan, 2019; Shu et al., 2021), while offline distillation is the only practical approach for proprietary LLM teachers accessible through APIs.Source: §3.1 · p. 11

3.2 Online Distillation

• Online distillation trains teacher and student simultaneously, often using multiple students, branch ensembles, previous predictions, or mutual contrastive learning instead of a large pretrained teacher.This single-stage scheme includes virtual teachers formed from groups of students (Wang et al., 2024a), multi-branch logits, and mutual contrastive embeddings (Yang et al., 2023b).Source: §3.2 · p. 11

3.3 Self-distillation

• Self-distillation is an online scheme in which identical networks exchange knowledge across training stages, layers, or classifiers rather than relying on a larger pretrained teacher.It was introduced by Furlanello et al. (2018), with later methods using previous epochs, deeper layers, or multiple classifiers; theoretical analyses connect it to self-distillation mechanisms and label smoothing (Mobahi et al., 2020; Zhang & Sabuncu, 2020).Source: §3.3 · p. 12

4 Algorithms

• The survey organizes KD algorithms into attention-based, adversarial, multi-teacher, cross-modal, graph-based, adaptive, and contrastive approaches.These families differ in the information transferred and the mechanisms used to guide the student.Source: §4 · p. 12

4.1 Attention-based Distillation

• Attention distillation transfers teacher attention maps or refined features so students prioritize task-relevant regions rather than exactly matching predictions.Methods use channel or spatial attention, CAM or GradCAM, self-attention, diffusion-based feature denoising, and CBAM refinement; AttnFD particularly targets small-object segmentation.Source: §4.1 · pp. 12–13

4.2 Adversarial Distillation

• Adversarial distillation uses generated data, discriminators, or perturbed samples to improve transfer when data are unavailable, outputs or features misalign, or robustness is required.Data-free methods synthesize samples under privacy, transmission, or legal constraints; discriminator-based methods align logits or intermediate features and extend to GAN and diffusion-model compression, while ARD transfers teacher predictions under adversarial perturbations and later methods refine teacher reliability, perturbation adaptation, and distribution balance.Source: §4.2.1 · p. 13, §4.2.2 · p. 14, §4.2.3 · p. 14

4.3 Multi-teacher Distillation

• Multi-teacher distillation combines diverse teachers through averaged or perturbed logits, voting, adaptive weighting, auxiliary branches, progressive transfer, or knowledge amalgamation to broaden student knowledge and generalization.Its central challenges are aggregating teachers with different capacities, assigning their contributions, and controlling computational complexity.Source: §4.3 · p. 15

4.4 Cross-modal Distillation

• Cross-modal distillation uses a richly annotated modality during training to improve a less-annotated modality while allowing inference without the auxiliary modality.Applications transfer information among RGB, depth, events, frames, LiDAR, radar, thermal, optical flow, text, images, audio, and video through parallel streams, aligned features, masking, or reconstruction.Source: §4.4 · p. 16, Figure 5 · p. 16, Table 4 · p. 17

4.5 Graph-based Distillation

• Graph-based distillation transfers higher-order relational knowledge by representing features or instances as vertices and dependencies as weighted edges.Graph structures capture local and global relationships, regulate multi-teacher information flow, and support either GNN performance improvement or compression into compact GNNs and MLPs using logits, features, and graph connectivity.Source: §4.5 · pp. 16–17

4.6 Adaptive Distillation

• Adaptive distillation dynamically changes losses, weights, teacher structure, or sample and region emphasis to avoid transferring teacher errors and address teacher–student capacity gaps.Examples replace incorrect predictions with labels, decay teacher influence during training, use teaching assistants or pruning, and weight samples, masks, channels, layers, or classes according to uncertainty, difficulty, discrepancy, or energy.Source: §4.6 · pp. 17–18, Table 5 · p. 18

4.7 Contrastive Distillation

• Contrastive distillation aligns student and teacher feature spaces by treating corresponding features as positives and unrelated features as negatives, with extensions addressing noisy negatives, capacity mismatch, and structured relations.Applications include semantic segmentation, super-resolution, CLIP, sentence embeddings, medical imaging, visual grounding, and relation or class-wise semantic transfer; WCoRD combines global mutual-information transfer with local mini-batch alignment.Source: §4.7 · pp. 18–19, Figure 6 · p. 19
• Contrastive distillation aligns features, preserves relational knowledge, integrates local and global information, and addresses multimodal alignment, dense feature mapping, efficient compression, imbalanced data, noisy labels, and computational constraints.Its flexibility underlies this versatility across feature learning, model compression, and challenging data conditions.Source: §4.7 · p. 20

Modalities

• Knowledge distillation has been effectively applied beyond images to 3D/multi-view data, text, speech, and video.Source: §5 · p. 20
• The reviewed distillation methods are categorized according to their modality-specific tasks.Source: §5 · p. 20

5.1.1 3D Object Detection

• KD improves 3D object detection by transferring spatial, depth, geometric, and cross-modal knowledge to lightweight models across LiDAR, camera-only, and sensor-fusion settings.Methods address sparsity, occlusion, density variation, weather-induced domain gaps, missing depth, and cross-sensor inconsistencies using feature, logit, geometric, and BEV distillation.Source: §5.1.1 · pp. 20–21

5.1.2 3D Shape Classification

• For 3D shape classification, KD combines logit, feature, graph-entropy, offline negative-weight, and self-distillation methods for complete or incomplete point clouds.JGEKD targets non-independent and identically distributed point-cloud data, while offline and self-distillation frameworks improve efficiency and incomplete-cloud classification.Source: §5.1.2 · p. 22

5.1.3 3D Semantic Segmentation

• KD supports 3D semantic segmentation by transferring high-capacity teacher knowledge while addressing sparsity, varying density, hard classes, and limited point-wise annotations.Point-to-voxel, multilevel, self-distillation, self-supervised, and vision-language-model distillation provide fine-grained, coarse, graph-consistent, or externally sourced supervision.Source: §5.1.3 · pp. 22–23

5.1.4 3D Domain Adaptation

• KD addresses 3D domain adaptation and depth estimation through self-distillation, cross-modal fusion, self-ensembling, and lightweight multi-view or RGB-radar models.These methods target noisy or incomplete target domains, improved point-cloud upsampling, monocular depth estimation, and dense depth prediction from sparse measurements.Source: §5.1.4 · p. 23, §5.1.5 · p. 23

5.1.6 3D Representation Learning

• 3D representation learning uses KD to transfer semantic, geometric, visual, and textual knowledge from 2D data, foundation models, images, text, or point-cloud sequences.Approaches include contrastive 2D-to-3D transfer, superpixel-driven self-supervision for image-LiDAR alignment, single- or cross-modal learning, and self-supervised 4D representation learning.Source: §5.1.6 · p. 23

5.1.7 3D Recognition

• Cross-modal KD supports visual place recognition by transferring knowledge between camera images and LiDAR point clouds through frameworks such as PointMCD and DistilVPR.PointMCD uses a multi-view framework, while both methods target recognizing previously visited locations from visual input.Source: §5.1.7 · p. 23

5.1.8 3D Completion

• KD improves 3D completion by transferring priors or learned knowledge for shape, scene, and depth reconstruction from incomplete or sparse observations.Methods include semi-supervised prior distillation, loss distillation, hierarchical self-distillation, and knowledge transfer for missing geometry or depth information.Source: §5.1.8 · p. 24

5.1.9 3D Registration

• Beyond these applications, KD compresses models for point-cloud registration and supports 3D generation, stylization, sampling, and accordance detection.Registration methods transfer knowledge from large models to efficient ones while preserving performance, whereas other tasks employ varied or all types of distillation.Source: §5.1.9 · p. 24, §5.1.10 · p. 24

5.2 Multi-view Input

• Multi-view distillation spans 3D object detection, 3D shape recognition, and specialized applications such as robotic manipulation and medical diagnostics, adapting knowledge transfer across diverse modalities and challenges.Existing work is broadly divided into 3D object detection and 3D shape recognition, with other tasks represented by individual studies.Source: §5.2 · p. 24, §5.2.3 · p. 26

5.2.1 3D Object Detection

• 3D object detection methods transfer LiDAR, camera, temporal, and task-specific knowledge to multi-view students, including BEV alignment, geometry compensation, cross-modal outputs, and response distillation.SimDistill generates multimodal features from multi-view images; other methods use LiDAR-guided BEV or feature distillation, cross-task and adversarial training, structural-temporal regularization, or response matching.Source: §5.2.1 · pp. 24–25

5.2.2 3D Shape Recognition

• For 3D shape recognition, PointMCD bridges 2D visual and 3D geometric domains, while GMViT uses a large model to improve smaller students for memory-limited devices.These approaches target the representation gap between 2D and 3D inputs and the large model sizes that constrain deployment.Source: §5.2.2 · p. 25

5.2.3 Other Tasks

• Other-task applications include multi-view classification, robotic manipulation, BEV detection, stereo depth, semantic segmentation, action recognition, and myocardial infarction diagnosis, using tailored teacher-student or multi-teacher schemes.Examples include teacher-policy transfer from multiple cameras to one, self-supervised probabilistic depth transfer, multi-view co-tuning with feature and output losses, hierarchical-transformer distillation, and multi-task ECG representation learning.Source: §5.2.3 · p. 26

• KD transfers knowledge from large BERT- and GPT-like models to lightweight students, supporting high accuracy across diverse NLP tasks while reducing computational costs.Applications include translation, question answering, generation, event detection, retrieval, recognition, NER, summarization, NLU, sentiment analysis, and classification.Source: §5.3 · p. 26

Neural Machine Translation

• NMT distills teacher outputs to students, achieving comparable translation quality with lower computational costs and enabling more efficient, scalable systems (Kim & Rush, 2016; Tan et al., 2019; Wang et al., 2021a; Liang et al., 2022a; Jin et al., 2024).The approach addresses the computational intensity of large transformer-based translation models.Source: §5.3 · pp. 26–27

Question Answering

• In QA, KD transfers answer predictions and intermediate representations such as attention distributions, preserving accuracy while making systems more efficient (Hu et al., 2018; Izacard & Grave, 2020; Yang et al., 2020; Liu et al., 2021b; Yin et al., 2024a).This transfer supports deployment in real-world applications with reduced computational costs.Source: §5.3 · p. 27

Text Generation

• For text generation, KD teaches students to mimic teacher vocabulary distributions and language patterns, with BERT-based contextual transfer improving efficiency and GAN-based distillation enhancing fluency and diversity (Chen et al., 2019c; Haidar & Rezagholizadeh, 2019).The two approaches target efficient generation and improved output quality through different forms of teacher knowledge.Source: §5.3 · p. 27

Other NLP Applications

• KD supports event detection, document retrieval, text recognition, NER, and summarization by transferring task-specific capabilities such as cross-lingual knowledge, relevance scores, visual-contextual features, contextual classification, and output distributions (Ren et al., 2021; Yu et al., 2021; Liu et al., 2019a; Shakeri et al., 2019; Chen et al., 2021c; Bhunia et al., 2021; Wang & Du, 2021; Liang et al., 2021; Ge et al., 2024; Chen et al., 2017; Liu et al., 2020b; Nguyen & Luu, 2022).These transfers produce smaller, more efficient models while retaining detection, ranking, recognition, contextual understanding, or summarization behavior.Source: §5.3 · p. 27

Understanding and Classification

• In NLU, sentiment analysis, and text classification, students learn teacher output probabilities, internal or feature representations, and decision patterns to preserve linguistic or classification performance at lower computational cost (Liu et al., 2019c; Tang et al., 2019; FitzGerald et al., 2022; Lehečka et al., 2020; Malki, 2023; Xu & Yang, 2017; Li & Li, 2021).These applications transfer linguistic understanding, sentiment classification, or general classification capabilities from teacher to student.Source: §5.3 · p. 27

Overview

• KD addresses diverse speech-processing tasks by reducing model complexity for real-time or embedded deployment while retaining task-relevant representations and performance.The survey motivates KD through the computational demands of deep acoustic models and identifies speech recognition, enhancement, speaker verification, translation, synthesis, separation, language understanding, spoofing detection, audio classification, conversational AI, and audio retrieval as application areas.Source: §5.4 · p. 27

Speech Recognition (ASR)

• KD enables rapid, resource-efficient speech recognition while preserving accuracy and improving adaptation to poor acoustic environments and expert domains.Applications include challenging settings such as medical transcription.Source: §5.4 · p. 28

Speech Enhancement and Speaker Recognition

• KD enables speech enhancement through high-fidelity representations for denoising and model compression, while speaker recognition and verification retain identity features with faster, more secure, and more robust authentication.Speaker systems are described as more robust to adversarial attacks, cross-domain variation, noise, and multilingual conditions.Source: §5.4 · p. 28

Speech Translation

• KD improves multilingual speech translation by sharing knowledge between high- and low-resource languages, supporting fluent, low-latency translation for real-time applications.This application combines multilingual knowledge transfer with efficiency improvements.Source: §5.4 · p. 28

Speech Synthesis and Speech Separation

• KD compresses generative speech-synthesis and speech-separation models while retaining naturalness or speaker distinction, enabling expressive speech and real-time separation on low-power devices.For synthesis, distillation also preserves expressive features; for separation, it supports cocktail-party-effect reduction.Source: §5.4 · p. 28

Spoken Language Identification and Understanding

• KD transfers multilingual teacher knowledge to lighter models for low-resource language identification and spoken dialogue understanding, maintaining high performance with faster inference.The mechanism supports language adaptation through information transfer from high-resource multilingual teachers.Source: §5.4 · p. 28

Deepfake Speech and Spoofing Detection

• KD strengthens deepfake and spoofing detection by transferring knowledge from models trained across diverse synthetic-speech datasets into lightweight detection architectures.This targets reliable detection of synthetic voice fraud while reducing deployment cost.Source: §5.4 · p. 28

Audio Analysis, Conversational AI, and Retrieval

• KD improves sound-event feature learning for embedded audio classification, compresses contextual knowledge for fast spoken question answering, and reduces audio-text embedding size for efficient multimedia retrieval.These applications target IoT and edge platforms, quick-answer-producing conversational systems, and generalized multimedia search architectures.Source: §5.4 · p. 28

Video Input

• Video KD has been applied to action recognition, classification, object and instance segmentation, partially relevant video retrieval, trajectory forecasting, and masked video modeling.These applications are summarized in Table 8 by distillation source.Source: §5.5 · pp. 28–29
• KD enables smaller video models to learn from larger teachers, reducing computational and memory requirements with minimal performance reduction for edge deployment.MobileVOS transfers structural information through pixel-wise representation distillation while maintaining feature consistency across frames; Kim et al. (2024a) distills sequence-level teacher similarity information to an online frame-based model, and Mullapudi et al. (2019) applies feature distillation to live-video semantic segmentation.Source: §5.5 · p. 29
• KD supports cross-modal transfer across video-to-image, video-to-text, text-to-video, audio-to-video, and video-to-audio tasks.Pan et al. (2020) uses KD in a spatio-temporal graph model for video captioning to improve robustness against spurious correlations.Source: §5.5 · p. 29
• V2I-DETR distills temporal knowledge from a video teacher to an image student for medical video lesion detection, achieving real-time inference with high accuracy.The approach transfers knowledge across modalities while targeting efficient medical-video processing.Source: §5.5 · p. 29

6.1 Distillation in Large Language Models

• LLM distillation divides into white-box methods, which transfer logits or internal features, and black-box methods, which use predictions through ICL, instruction following, or chain-of-thought.White-box methods can transfer more internal knowledge but require greater computational cost and model access; black-box methods apply to closed-source models but may generalize less broadly.Source: §6.1 · pp. 30–31
• White-box LLM distillation includes BERT compression methods such as DistilBERT, MiniLM, MobileBERT, TinyBERT, and PKD, alongside newer decoder-focused methods including MetaDistill, GKD, PC-LoRA, and MiniLLM.These methods distill probabilities, attention, hidden states, features, or self-generated sequences, and may use feedback, progressive compression, or reverse Kullback-Leibler divergence.Source: §6.1 · pp. 30–31
• Black-box LLM distillation uses in-context examples, generated instructions, or teacher rationales to transfer task knowledge, zero-shot capabilities, and reasoning to smaller language models.ICL combines context-learning and language-modeling objectives; instruction following generates filtered task data; chain-of-thought transfers intermediate reasoning steps, including positive and negative examples.Source: §6.1 · p. 31

6.2 Foundation Model Distillation

• Foundation-model distillation transfers knowledge from large architectures such as CLIP, Chat-GPT, DALL-E, SAM, DINO, ViT, DETR, Swin Transformer, and DeIT into smaller task-specific models.This addresses computational and deployment constraints while supporting downstream applications across language, vision, multimodal, segmentation, detection, and generation tasks.Source: §6.2 · p. 32

6.2.1 Vision-language Models and Knowledge Distillation

• CLIP-based VLM distillation transfers aligned visual-textual representations to open-vocabulary detection and segmentation, weakly supervised segmentation, prompt learning, image generation and editing, and specialized multimodal tasks.KD adapts CLIP for dense pixel-level understanding, minimal-label mask generation, learnable prompts, and applications including retrieval, depth estimation, human-object interaction, and domain generalization.Source: §6.2.1 · pp. 33–34

6.2.2 Segment Anything (SAM)

• SAM distillation transfers its promptable segmentation capabilities into lighter task-specific models and uses pseudo-labels from weak supervision to reduce annotation costs while maintaining competitive performance.Pseudo-labels can be generated from image-level tags, points, or bounding boxes, enabling models to learn object boundaries with limited manual annotation.Source: §6.2.2 · pp. 34–35

6.2.4 Vision Transformers

• Knowledge distillation has been explored for Vision Transformers and DETR to reduce computational demands while supporting specialized or real-time, resource-limited vision applications.DINO pretrained features support unsupervised object discovery, open-world counting, and zero-shot semantic segmentation. ViT distillation methods are summarized in Table 9.Source: §6.2.4 · pp. 35–36, Table 9 · p. 35

6.3 Distillation in Self-supervised learning

• Self-supervised distillation commonly uses identical networks with different training roles: the student learns by backpropagation while the teacher is an EMA of the student, as in MOCO (He et al., 2020; Chen et al., 2020d) and BYOL.SimCLR (Chen et al., 2020c) instead uses two augmented views and minimizes their extracted embeddings.Source: §6.3 · p. 36
• DINO (Caron et al., 2021) and DINOv2 (Oquab et al., 2023) distill global and local image patches through EMA-updated teacher networks, making local-to-global representations suitable for segmentation.Teacher and student representations are explicitly forced to be similar.Source: §6.3 · p. 37
• Beyond standard SSL, BINGO (Xu et al., 2021), multi-teacher masked modeling (Wang et al., 2023l), parallel consistency learning (Song et al., 2023a), and pre-trained-teacher integration introduce more complex distillation designs.Other work compresses pre-trained SSL networks using similarity losses, queues, auxiliary pretraining, or student MLP layers, with similarity-based distillation sometimes outperforming supervised training (Abbasi Koohpayegani et al., 2020).Source: §6.3 · p. 37

6.4 Diffusion Distillation

• Diffusion distillation transfers teacher outputs and internal representations to smaller students, achieving comparable quality with far fewer sampling steps and lower inference cost.This addresses the hundreds or thousands of denoising steps required by diffusion models.Source: §6.4 · p. 37
• Diffusion methods accelerate generation through feature alignment, sampling-process reduction, adversarial or data-free output matching, and consistency-based direct transformations.Feature methods align internal representations; sampling methods merge denoising steps; adversarial and data-free methods use GAN losses or bootstrapping; consistency models often use one step.Source: §6.4 · pp. 37–38
• Progressive distillation (Salimans & Ho, 2022) repeatedly merges two denoising steps, whereas distribution matching distillation (Yin et al., 2024c) matches a one-step student directly to the multi-step teacher’s output distribution.Both approaches shorten the iterative sampling procedure, but the latter targets a single-step model directly.Source: §6.4 · pp. 37–38
• Consistency models reformulate denoising as a direct noisy-to-clean transformation; latent variants and stability improvements provide inference speedups, but training requires careful tuning to avoid quality collapse.This includes methods by Song et al. (2023c), Song & Dhariwal (2023), Geng et al. (2024), Luo et al. (2023a), Xie et al. (2024a), and Lu & Song (2024).Source: §6.4 · p. 38
• Diffusion distillation offers distinct trade-offs: sampling-process methods accelerate generation but complicate training, adversarial and data-free methods improve flexibility but may reduce stability, and consistency models enable low-latency generation when calibrated.Cross-modal and universal approaches extend applicability across diverse generation tasks.Source: §6.4 · p. 38

6.5 Visual Recognition Distillation

• Visual recognition distillation trains compact students to replicate larger, resource-intensive teacher models.The survey presents knowledge distillation as broadly applied to computer vision tasks.Source: §6.5 · p. 38
• The surveyed applications include depth estimation, face recognition, segmentation, medical imaging, detection, tracking, pose estimation, super resolution, action recognition, and image retrieval.These tasks define the scope of visual recognition coverage summarized in Table 11.Source: §6.5 · p. 38, Table 11 · p. 38
• Table 11 categorizes methods by feature-based, logit-based, similarity-based, or combined knowledge, alongside offline, online, and self-distillation schemes.The table summarizes visual recognition methods according to both their knowledge sources and distillation schemes.Source: Table 11 · p. 38, §6.5 · p. 38

Comparison considerations

• Fair cross-method comparison is difficult because teacher models, epochs, batch sizes, hardware resources, architectures, and reporting conventions can materially affect distillation results.For segmentation, reported results are taken directly from original papers whenever possible; otherwise, the survey reports results from its own implementations.Source: §7 · pp. 41, 44

Classification

• Comparisons cover classification on CIFAR-100 and ImageNet, using widely adopted datasets to evaluate diverse distillation methods.The classification tables report knowledge source, teacher and student architectures, and post-distillation accuracy, including similar- and different-architecture settings.Source: §7 · pp. 41, 44, Table 12 · p. 42, Table 13 · p. 43

Semantic segmentation

• Semantic-segmentation comparisons use PascalVOC and Cityscapes, combining results from original papers with own implementations when specified results were unavailable.These datasets are widely used in existing distillation methods, and unavailable results are marked with an asterisk.Source: §7 · pp. 41, 44, Table 14 · p. 44, Table 15 · p. 45

LLM distillation

• LLM comparisons include black-box and white-box distillation on commonly used datasets and models to improve fairness despite substantial variation across studies.The compared datasets include GLUE, Multilingual NER, DollyEval, IMDB, GSM8K, CrossFit, CommonsenseQA, SVAMP, Universal NER Benchmark, OpenBookQA, BBH, StrategyQA, and MATH.Source: §7 · p. 43
• Table 16 compares LLM distillation methods by compression rate and improvement over the teacher, using student-before-and-after results when teacher performance is unavailable.This reporting convention is marked with an asterisk, and the comparison emphasizes maintaining comparable performance while compressing large models.Source: §7 · p. 44, Table 16 · p. 46

8.1 Principles of Method Classification

• The survey classifies methods by distillation sources, schemes, algorithms, modalities, and applications according to each method’s primary contribution.Methods spanning multiple categories are assigned by their central innovation, with cross-references preserving their multidimensional nature.Source: §8.1 · p. 46

8.2 Guidelines for Selecting Knowledge Distillation Strategies

• Task constraints guide selection of distillation sources, schemes, and algorithms, while the optimal strategy depends on each use case’s goals and context.Table 17 provides a decision-oriented mapping from task scenarios to suitable KD components.Source: §8.2 · p. 46, Table 17 · p. 47

8.3 Quantitative Comparison

• Directly ranking distillation methods is difficult because results depend on hyperparameters, teacher weights, metrics, dataset sizes, and teacher-student architectural differences.Tables 12–16 report cross-domain results and architecture settings, but the survey avoids definitive superiority claims.Source: §8.3 · pp. 46–47

8.4 Practical and Ethical Aspects

• KD extends beyond compression to privacy, security, fairness, energy efficiency, and intellectual-property concerns, but privacy gains are not guaranteed and commercial teachers may raise copyright issues.Studies report reduced memorization and sensitive-information generation, while other work identifies vulnerabilities in data-free distillation.Source: §8.4 · pp. 47–48

8.5 Challenges

• KD challenges include selecting informative sources, choosing teacher-student schemes, bridging capacity gaps, and transferring knowledge across different architectures.Logit distillation may fail without labels or inaccessible intermediate outputs; feature, similarity, offline, online, self-, multi-teacher, and CoT approaches address different conditions but retain resource or alignment difficulties.Source: §8.5 · pp. 48–49

8.6 Future Directions

• Future work emphasizes feature and adaptive distillation, including embedding transfer, selective knowledge delivery, teacher assistants, adaptive losses, and reduced late-training distillation.These approaches aim to improve student performance, avoid transferring incorrect or unused knowledge, and narrow teacher-student gaps.Source: §8.6 · p. 49
• Foundation models and LLMs offer major distillation opportunities through rich embeddings, open-vocabulary transfer, black-box and CoT reasoning distillation, compact pre-trained models, and integration with quantization or pruning.Distilled students can approach teacher performance with lower resource demands, while applications also include attacks, backdoor mitigation, anomaly detection, dataset distillation, and compression.Source: §8.6 · pp. 49–50

9 Conclusion

• The survey organizes recent knowledge-distillation methods by sources, schemes, algorithms, modalities, applications, and performance comparisons, extending beyond existing surveys.It particularly emphasizes adaptive and contrastive distillation as increasingly important algorithms.Source: §9 · p. 50
• It connects knowledge distillation to 3D modalities such as point clouds and applications in self-supervised learning, diffusion models, foundation models, and LLMs.Self-supervised learning primarily uses self-distillation, diffusion-model distillation reduces generation steps, and LLMs are distilled into smaller versions.Source: §9 · p. 50
• Distilling rich knowledge from foundation models such as CLIP into other models has become increasingly widespread.Source: §9 · p. 50