PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions

Luyang Fang, Xiaowei Yu, Jiazhang Cai, Yongkai Chen, Shushan Wu, Zhengliang Liu, Zhenyuan Yang, Haoran Lu, Xilin Gong, Yufang Liu, Terry Ma, Wei Ruan, Ali Abbasi, Jing Zhang, Tao Wang, Ehsan Latif, Weihang You, Hanqi Jiang, Wei Liu, Wei Zhang, Soheil Kolouri, Xiaoming Zhai, Dajiang Zhu, Wenxuan Zhong, Tianming Liu, Ping Ma

arXiv:2504.14772v2cs.CLcs.LGstat.ML

TL;DR

LLMs require costly computation and massive datasets, while preserving their reasoning and linguistic diversity during compression remains difficult. This survey analyzes KD and DD methodologies and their integration, covering model transfer, compact-data synthesis, applications, theory, and evaluation. It concludes that their complementary use supports resource-efficient LLM compression, while preserving advanced capabilities and adapting deployment across domains remains an open challenge.

  • Problem

    LLMs impose substantial computational and data demands, while existing compression approaches must preserve emergent reasoning, linguistic diversity, and complex knowledge transfer.

  • Method

    The survey synthesizes Knowledge Distillation and Dataset Distillation methods, their integration, theoretical foundations, evaluation protocols, and healthcare and education applications.

  • Results

    The survey concludes that complementary KD and DD strategies can compress model size and training data while preserving contextual reasoning, cross-domain generalization, and linguistic diversity.

  • Takeaways & Limitations

    Integrating KD and DD offers a framework for more computationally efficient, data-scalable, resource-aware LLM deployment across domains such as healthcare and education.

  • Takeaways & Limitations

    Open challenges include evaluating extracted reasoning traces, eliciting sequential reasoning from black-box teachers, reducing uncertainty-quantification costs, and limiting computational overhead in data-selection methods.

Abstract

from arXiv · show

The exponential growth of Large Language Models (LLMs) continues to highlight the need for efficient strategies to meet ever-expanding computational and data demands. This survey provides a comprehensive analysis of two complementary paradigms: Knowledge Distillation (KD) and Dataset Distillation (DD), both aimed at compressing LLMs while preserving their advanced reasoning capabilities and linguistic diversity. We first examine key methodologies in KD, such as task-specific alignment, rationale-based training, and multi-teacher frameworks, alongside DD techniques that synthesize compact, high-impact datasets through optimization-based gradient matching, latent space regularization, and generative synthesis. Building on these foundations, we explore how integrating KD and DD can produce more effective and scalable compression strategies. Together, these approaches address persistent challenges in model scalability, architectural heterogeneity, and the preservation of emergent LLM abilities. We further highlight applications across domains such as healthcare and education, where distillation enables efficient deployment without sacrificing performance. Despite substantial progress, open challenges remain in preserving emergent reasoning and linguistic diversity, enabling efficient adaptation to continually evolving teacher models and datasets, and establishing comprehensive evaluation protocols. By synthesizing methodological innovations, theoretical foundations, and practical insights, our survey charts a path toward sustainable, resource-efficient LLMs through the tighter integration of KD and DD principles.

1 Introduction

The survey frames Knowledge Distillation (KD) and Dataset Distillation (DD) as complementary strategies for compressing LLMs and improving data efficiency while preserving advanced capabilities. It unifies their methodologies, integration, theoretical foundations, and evaluation challenges for resource-aware deployment.

  • Motivation: LLMs create computational, energy, environmental, and data-efficiency challenges because training and inference rely on immense resources and massive datasets.Public corpora also raise concerns about data quality, sustainability, and maintaining diversity.
  • Motivation: KD transfers knowledge from large teacher models to smaller students, but LLM scale complicates preserving distributed knowledge, reasoning, and problem-solving abilities.Traditional KD primarily aligns outputs or intermediate representations, whereas LLMs encode higher-order capabilities.
  • Motivation: DD complements KD by synthesizing compact, high-fidelity datasets that preserve essential knowledge, rare reasoning patterns, and linguistic diversity.The survey cites LIMA, where 1,000 examples achieved teacher-level performance, as evidence of compact data’s potential.
  • Survey scope: The survey examines standalone KD and DD advances before exploring their joint potential for model compression, training efficiency, and resource-aware deployment.Covered KD methods include rationale-based, uncertainty-aware, multi-teacher, dynamic, adaptive, and task-specific approaches; DD includes optimization, synthesis, and data selection.
  • Survey scope: Its distinctive contribution is a unified KD–DD framework that addresses scalability, data scarcity, computational barriers, and reasoning retention through methods and evaluation protocols.The survey also reviews theoretical guarantees and evaluation dimensions including calibration, robustness, memory usage, and compression level.

2 Fundamentals of Distillation

KD compresses models by transferring teacher behavior or representations to students, while DD synthesizes compact datasets that retain useful learning signals. In LLMs, these paradigms must address scale, architectural heterogeneity, reasoning preservation, data diversity, and computational cost.

  • Comparison and challenges: KD is mature and widely adopted, whereas DD remains promising for data efficiency, privacy, and decentralized training but faces unresolved LLM scalability and diversity challenges.Reported benchmark patterns describe KD students retaining over 95% of teacher performance, while DD often reaches 80–90% of full-real-data performance.
  • Knowledge Distillation: KD transfers teacher knowledge to compact students through soft outputs, intermediate representations, attention, or relational knowledge.Its objective combines hard-label cross-entropy with a distillation loss, while later methods match hidden states and other representations.
  • Knowledge Distillation: LLM distillation differs from traditional KD because autoregressive token distributions, heterogeneous architectures, cross-layer dependencies, and evolving teachers complicate knowledge transfer.LLM methods must preserve linguistic coherence, logical inference, and reasoning trajectories rather than only localized features.
  • Dataset Distillation: Dataset distillation synthesizes new, compact samples rather than merely selecting representative real examples, aiming to preserve effective training with far less data.The distilled dataset can be orders of magnitude smaller while achieving comparable or improved performance.
  • Dataset Distillation: DD methods align synthetic and original data through meta-learning, gradient matching, training dynamics, final performance, or distribution matching.Distribution Matching uses metrics such as Maximum Mean Discrepancy, while optimization methods refine synthetic samples against real-data behavior.

3 Methodologies and Techniques for Knowledge Distillation in LLMs

Knowledge-distillation methods for LLMs increasingly target reasoning, uncertainty, and adaptive knowledge transfer rather than only final predictions. The section also identifies unresolved challenges in evaluating extracted reasoning and controlling the computational cost of uncertainty-aware approaches.

  • Rationale-Based KD: Rationale-based KD trains students to predict teacher rationales alongside final answers, encouraging internalized reasoning rather than answer shortcuts.This can improve interpretability and reduce reliance on large labeled datasets.
  • Rationale-Based KD: Rationale-based distillation can balance accuracy, interpretability, and resource efficiency in applications such as healthcare and law.The approach supports smaller models while making decision processes more transparent.
  • Rationale-Based KD: Difficulty-aware KPOD progressively teaches reasoning steps from simpler to more complex ones using weighted token-generation loss.The design is reported to strengthen generalization and interoperability.
  • Challenges and Future Directions: Open questions include standardized evaluation of reasoning traces, black-box prompting for intermediate steps, and extraction of sequential reasoning chains.These issues are especially relevant when ground-truth reasoning paths or teacher internal representations are unavailable.
  • Uncertainty-Aware KD: Uncertainty-aware KD distills predictive distributions or quantifies uncertainty in students, reducing the risk of propagating incorrect knowledge and overconfident predictions.Bayesian approaches interpret teacher knowledge as a prior and derive a student posterior for uncertainty quantification.
  • Uncertainty-Aware KD: Bayesian KD uses teacher-informed priors over student parameters, with posterior sampling providing uncertainty estimates for the distilled model.The framework connects KD objectives with Bayesian posterior optimization and supports tools such as SGLD for sampling.
  • Challenges and Future Directions: Uncertainty quantification can impose substantially higher computational costs than standard prediction, motivating efficient alternatives and more flexible teacher-informed priors.The survey highlights variational inference and scenario-specific adaptations as future directions.

3.3 Multi-Teacher KD

Multi-teacher KD combines heterogeneous teacher expertise through aggregation, weighting, feature alignment, and specialized teacher settings. It can broaden reasoning, multilingual, and domain capabilities, but introduces access, scaling, heterogeneity, and conflict-resolution challenges.

  • Aggregation and Weighting: Multi-teacher KD consolidates heterogeneous teacher expertise into one student, using a distillation loss such as KL divergence against aggregated teacher outputs.Uniform averaging is a baseline, while later methods address unequal teacher quality and conflicting signals.
  • Aggregation and Weighting: Teacher weighting can prioritize predictions matching ground truth or adaptively merge teacher signals in gradient space.Feature-embedding alignment offers another way to capture complementary knowledge beyond simple output losses.
  • Specialized Teacher Settings: TinyLLM transfers reasoning by combining in-context example generation with teacher-forcing CoT, while MERGEDISTILL combines multilingual and monolingual models through evaluation and vocabulary mapping.These settings target reasoning diversity, multilinguality, and specialized language knowledge.
  • Challenges: Multi-teacher KD can broaden domain coverage and generalization, but access to all teachers may be infeasible because of licensing, privacy, or asynchronous deployment constraints.The approach also requires careful alignment across heterogeneous architectures and tokenization schemes.
  • Challenges: Pairwise interactions among K teachers scale roughly as O(K^2), increasing computational overhead and complicating synchronization of distinct objectives and representations.Domain-specific evaluation is proposed to test whether expertise is retained or diluted.
  • Challenges: Conflicting or contradictory teacher outputs can cause students to homogenize errors or fail to benefit from teachers’ distinct perspectives.Confidence-based weighting and context-dependent teacher selection are identified as possible conflict-resolution strategies.

3.4 Dynamic and Adaptive KD

Dynamic and adaptive KD replaces fixed teacher-student hierarchies with co-evolutionary training or self-distillation. These approaches improve adaptability but introduce convergence, calibration, bias, and computational challenges.

  • Simultaneous Training: Dynamic KD jointly refines teacher and student models through bidirectional interaction rather than relying on a static pretrained teacher.The student adapts to real-time teacher feedback, while the teacher adjusts guidance according to student progress.
  • Simultaneous Training: BiLD aligns pairwise differences among top-k logits, combining teacher-led knowledge capture with student-led feedback to support joint learning.Its two components are the teacher-led logits difference loss and the student-led logits difference loss.
  • Simultaneous Training: Competitive multi-modal distillation cycles through instruction tuning, evaluation, and augmentation with assessor-guided feedback.New instructions generated during augmentation replace or supplement the original instruction pool.
  • Self-distillation: Self-distillation partitions a model into sequential sections whose shallow classifiers learn from the deepest classifier acting as teacher.The framework uses intermediate outputs and shallow classifiers while the final classifier uses all blocks.
  • Self-distillation: Transitive and ensemble teacher distillation modify self-distillation by using successive classifiers or ensemble predictions as guidance targets.Theory also guarantees improved test accuracy for ensemble self-distillation when data has a multi-view structure.
  • Challenges and Future Directions: Dynamic KD remains constrained by unstable joint convergence, uncalibrated self-distillation errors, bias amplification, and higher computational overhead.Proposed directions include bias detection, consistency checks, uncertainty-aware supervision, and lightweight feedback protocols.

3.5 Vision KD for Autonomous Driving

Vision KD for autonomous driving transfers semantic, causal, and spatial-temporal reasoning from vision-language models to lightweight planners. The surveyed methods align features, train surrogate tasks, or use contrastive objectives for safer generalization under strict latency constraints.

  • Motivation and Taxonomy: Autonomous-driving KD must transfer spatial-temporal awareness and causal logic, not only logits, because safety-critical deployment requires real-time decisions.The section contrasts rationale alignment, surrogate-task distillation, and contrastive planning as three strategy families.
  • Rationale-Based Feature Alignment: VERDI aligns student feature maps from perception, prediction, and planning with semantic embeddings of teacher-generated rationales.This organizes the student’s latent space around linguistic reasoning and targets zero-shot robustness.
  • Surrogate Task Distillation: DiMA uses a shared scene encoder and surrogate tasks such as masked-token reconstruction and future prediction to transfer MLLM knowledge to a lightweight planner.Reasoning-task gradients enrich shared representations while the planner remains independent at inference.
  • Contrastive Planning: VLP aligns agent-centered bird’s-eye-view features with linguistic prototypes through contrastive learning, while Vi-LAD additionally distills attention maps.The stated goal is semantically distinct representations for long-tail objects and socially compliant navigation.

3.6 Task-Specific KD

Task-specific KD combines instruction tuning with domain-focused datasets to transfer specialized capabilities into smaller models. This is motivated by the mismatch between simple predefined instructions and complex real-world tasks.

  • Task-Specific Instruction Tuning: Task-specific distillation pairs instruction tuning with teacher-generated task knowledge while preserving instruction-following capabilities.The approach transfers specialized behavior through instruction-response data.
  • Task-Specific Instruction Tuning: Alpaca fine-tuned LLaMA-7B on 52,000 instruction-response pairs generated by text-davinci-003, while FLAN trained a 137B model on over 60 NLP datasets.Both approaches phrase tasks as natural-language instructions to support generalization to unseen tasks.
  • Domain-Specific Distillation: Predefined instructions can be simpler than real-world cases, motivating domain-specific distillation for fields such as medicine and programming.These solutions generally design specialized datasets for model fine-tuning.

3.7 Theoretical Studies

Theoretical KD studies explain teacher-student transfer through soft labels, convergence, architecture, capacity, and data geometry. The survey presents these factors as guidance for selecting methods according to model capacity, computational budget, and application domain.

  • Teacher-Student Theory: KD theory interprets soft labels as dark knowledge about teacher uncertainty and inter-class relationships, or as label-smoothing regularization.These interpretations explain why softened outputs can provide information beyond hard labels.
  • Convergence and Generalization: Convergence analyses commonly study linear models, deep linear models, and Gaussian processes, while broader work provides criteria for evaluating teacher quality.The surveyed theory therefore spans simplified architectures and more general teacher-performance conditions.
  • Architectural Factors: The capacity gap between teacher and student influences transfer effectiveness: excessive gaps can exceed student representational power, whereas small gaps may reduce the benefit of distillation.Teacher and student capacity must therefore be considered when designing KD systems.
  • Data Factors: Larger distillation datasets do not necessarily improve teacher fidelity and can negatively affect student generalization.Data geometry, including angular alignment with the teacher’s weight vector, also enters theoretical generalization bounds.
  • Comparative Synthesis: The survey’s comparative tables organize KD methods by core idea, strengths, limitations, use cases, data requirements, robustness, cost, and flexibility.These summaries are intended to guide method selection across model and application settings.

4 Methodologies and Techniques for Dataset Distillation in LLMs

Dataset distillation compresses large training corpora into compact datasets through selection, optimization, and generative synthesis. These methods improve efficiency but face trade-offs involving generalization, linguistic diversity, computational cost, and evaluation breadth.

  • Overview: Dataset distillation synthesizes compact datasets that retain essential information from large corpora for efficient LLM training.Approaches include optimization-based synthesis and generative data production.
  • Optimization-Based Distillation: Optimization-based methods learn synthetic samples by matching real-data training behavior, including gradients, embeddings, or training trajectories.Gradient matching aligns updates from real and synthetic data, while embedding- and trajectory-based methods preserve latent distributions or optimization paths.
  • Optimization-Based Distillation: LLM-specific methods reduce distillation overhead through random-feature approximation, model pooling, and representative rather than random text selection.RFAD, FRePo, and DREAM target computational efficiency and reduced overfitting or sampling redundancy.
  • Generative-Model-Based Distillation: Generative methods learn linguistic distributions in latent space and generate synthetic sequences through learned generators rather than directly optimizing individual data points.GAN-based and latent-code methods aim to preserve expressive structure and diversity.
  • Generative-Model-Based Distillation: Generated datasets can be made computationally efficient and representationally faithful, but generative distillation risks distribution drift and requires extensive inference.Monitoring, computational amortization, and hybrid optimization-generation pipelines are proposed responses.
  • Data Selection: Data selection methods balance efficiency, accuracy, and adaptability, but semantic weakness, computational expense, and changing data importance limit practical reliability.Jaccard similarity may miss nuanced meaning, while LESS, STAFF, and Data Shapley can require costly model evaluation and may struggle with dynamic training.

5 Integration of Knowledge Distillation and Dataset Distillation

Integrating knowledge distillation with dataset distillation transfers teacher capabilities through compact synthetic data. The survey describes unified, prompt-based, and soft-prompt approaches for reducing data and computation while retaining useful model functionality.

  • Integrated KD and DD: Combining KD and DD transfers reasoning capabilities through minimal representative datasets, reducing dependence on large-scale data while maintaining advanced functionalities.The integration balances model compression with sustainable data utilization.
  • Integrated KD and DD: Optimization-based distillation can overfit distillation-phase training dynamics and scale poorly as dataset size or architectural complexity increases.These issues arise from unrolling and storing computation graphs across multiple gradient steps.
  • Integrated KD and DD: SRe2L reframes dataset distillation as data-frugal knowledge distillation through teacher pretraining, model-inversion coreset synthesis, and soft-label transfer.The method decouples dataset optimization from neural-network training and uses synthesized samples with teacher soft labels for student distillation.
  • Integrated KD and DD: Subsequent approaches improve cross-architecture generalizability, informative-patch selection, and synthetic-data quality through diverse architectures, pruning, and advanced cropping.These methods extend the integrated knowledge-transfer framework beyond a single distillation configuration.
  • Prompt-Based Synthetic Data Generation: Prompt-based generation uses large teacher LLMs to create compact training sets for low-resource adaptation, task-specific augmentation, and self-distillation.The survey distinguishes static, automatically optimized, and soft prompt-based generation.
  • Prompt-Based Synthetic Data Generation: Soft prompts steer frozen LLMs with learnable embeddings, enabling controllable generation of structured, domain-specific, or targeted synthetic text.This avoids altering internal model parameters and reduces reliance on manual prompt engineering.
  • Prompt-Based Synthetic Data Generation: Prompt-based methods generate datasets tailored to specific needs through static prompts, automatic optimization, or soft-prompt frameworks.The survey presents these approaches as versatile mechanisms for improving model training and performance.

6 Evaluation and Metrics for LLM Distillation Techniques

Evaluation of LLM distillation must jointly assess retained performance, efficiency, robustness, calibration, and teacher–student knowledge transfer. The survey combines standardized benchmarks with metrics for text quality, computational footprint, security, uncertainty, and identity preservation.

  • Evaluation Framework: A rigorous evaluation framework measures performance, efficiency, robustness, and knowledge-transfer efficacy in distilled LLMs.The goal is to assess task retention while balancing computational and data efficiency.
  • Performance Evaluation: GLUE, GSM8K, MATH, and MMLU provide standardized reference points for natural-language understanding, mathematical reasoning, and broad knowledge assessment.These benchmarks support comparisons across distillation approaches.
  • Performance Evaluation: 90–95% of teacher performance with 5–10x model-size reduction is a typical reported trade-off for distilled students on referenced benchmarks.The figures summarize representative performance retention and compression patterns described by the survey.
  • Text and Task Metrics: Perplexity, exact match, F1, BLEU, and ROUGE quantify predictive performance and generalization across language modeling, question answering, and generation.These metrics assess how effectively students preserve teacher capabilities while limiting degradation.
  • Text and Task Metrics: MAUVE measures divergence between human and model text distributions, while BERTScore compares candidate and reference texts through contextual-embedding similarity.Self-BLEU and distinct-n additionally address repetition and diversity.
  • Higher-Order Capabilities: Reasoning evaluation uses mathematical problem-solving and logical deduction, while generation evaluation includes summarization, question answering, and retrieval-augmented performance.These tests target higher-order capabilities beyond surface-level accuracy.
  • Efficiency Evaluation: Inference speed, memory consumption, model size, FLOPs, latency, and peak memory quantify the computational footprint of distillation.The survey describes latency comparisons from hundreds of milliseconds to tens of milliseconds per query.
  • Robustness Evaluation: Robustness evaluation stress-tests distilled models under domain shifts, adversarial perturbations, and noisy inputs using ASR and prompt-focused PDR.Higher PDR indicates greater vulnerability to prompt attacks.

7 Applications and Use Cases of Distillation

Distillation supports healthcare and education applications by transferring LLM capabilities into compact models that reduce computational demands while retaining useful performance. Reported uses include clinical prediction, summarization, patient interaction, information extraction, multimodal question answering, and large-scale educational deployment.

  • Clinical Decision Support: CKLE improved heart-failure and hypertension prediction by up to 4.48% over state-of-the-art models while using smaller local models.The framework transfers knowledge from a text-based teacher into multimodal EHR students, addressing privacy and deployment constraints.
  • Clinical Decision Support: OptimCLM achieved 22.8× model-size compression and 28.7× speedup with under 5% AUROC loss on hospital outcome tasks.The framework combines ensemble learning, distillation, pruning, and quantization for clinical BERT models.
  • Patient Interaction and Information Extraction: Distilled clinical models can preserve practical utility across patient interaction and information extraction, including competitive trial matching and a 6-point F1 advantage over GPT-3.5 for adverse-drug-event extraction.Trial-LLAMA matched or surpassed GPT-3.5 on aggregate patient–trial matching metrics, while PubMedBERT used 1,000× fewer parameters than its teacher.
  • Patient Interaction: On medical MMLU questions, a distilled DeBERTav3 student reached 39.3% accuracy versus 28.9% for a few-shot baseline.The teacher generated synthetic multiple-choice questions with answer-likelihood scores for student training.
  • Multimodal Applications: ClinKD achieved state-of-the-art Med-GRIT 270k accuracy by distilling image-text alignment and medical concepts into a vision-language student.The approach improves grounding for radiology-image answers and addresses visual hallucination concerns.
  • Education: EduChat reduced its teacher model by about 80%, lowering memory use from over 40 GB to under 8 GB and delivering responses in under 1.5 seconds during school deployment.The system served over 200 schools and more than 50,000 students, supporting thousands of simultaneous sessions on standard servers and tablets.

8 Open Challenges and Future Directions

The survey identifies unresolved challenges in preserving emergent reasoning, linguistic diversity, fairness, and privacy while distilling rapidly evolving models and datasets. It proposes architecture-aware methods, fairness-conscious objectives, incremental adaptation, and holistic evaluation as future directions.

  • Preserving Emergent Capabilities: Traditional logit or layer matching can lose chain-of-thought reasoning, in-context learning, long-range dependencies, narrative coherence, and coreference resolution.Future methods should explicitly preserve deeper contextual relationships and reasoning mechanisms while retaining linguistic diversity.
  • Dynamic Models and Data: Trillion-parameter teachers and continually expanding corpora make repeated KD cycles computationally expensive, while compressing datasets without losing rare linguistic patterns remains difficult.The challenge is amplified when teachers undergo fine-tuning, domain adaptation, or reinforcement learning and require incremental updates.
  • Trustworthiness and Fairness: Distillation can amplify teacher and dataset biases, discard contextual safeguards, and underrepresent minority instances needed for equitable behavior.The survey points toward fairness-aware objectives and multi-objective optimization balancing compression, predictive performance, and group-wise error gaps.
  • Training Stability: Joint teacher-student training creates stability, scalability, and knowledge-coherence challenges because conflicting objectives can destabilize convergence.The survey suggests tailored loss functions or staged curricula to promote synergy in multi-model settings.
  • Architectural Mismatch: Architecture-aware distillation must align transferred reasoning across heterogeneous models, adapt loss weighting to student capacity, and preserve informative dataset structure.These questions target the performance gap between heterogeneous teachers and students without abandoning efficiency gains.
  • Agent Systems: Agent distillation must selectively preserve planning pathways, reasoning chains, and tool-usage patterns required for complex tasks.Multi-agent systems may also require targeted integration of specialized knowledge from multiple teacher agents.
  • Evaluation: Current evaluations often omit deeper reasoning, emergent behavior, adaptability, bias, fairness, and ethical alignment.The survey calls for holistic protocols combining nuanced capability measures with fairness-specific metrics and traditional performance assessments.

9 Conclusion

The survey presents KD and DD as complementary strategies for compressing LLMs while retaining advanced capabilities, but emphasizes unresolved challenges in preservation, adaptation, trustworthiness, and evaluation.

  • KD and DD jointly address computational efficiency, data scalability, and retention of contextual reasoning, cross-domain generalization, and linguistic diversity.The survey combines methodological innovations including multi-teacher architectures, rationale-based guidance, adaptive teacher-student co-evolution, and data filtering or synthesis.
  • Preserving emergent abilities in smaller models requires methods that capture nuanced teacher distributions and structural dependencies.
  • Distilled data must balance coverage of rare phenomena with efficiency while continuously aligning with evolving teachers and tasks.
  • Bias mitigation and prevention of inherited harmful hallucinations require carefully calibrated, uncertainty-aware distillation protocols.
  • Future evaluation frameworks should move beyond standard accuracy to assess language understanding, compositional reasoning, and alignment with societal values.The survey identifies robust safety and domain-appropriateness checks as complementary requirements for deployment in diverse and critical applications.

Declarations

The declarations report partial support from NSF, NIH, and IES, and state that the authors have no conflict of interest.

  • The work was partially supported by the U.S. National Science Foundation, National Institutes of Health, and Institute of Education Sciences.
  • The authors declare that there is no conflict of interest.
Loading 2504.14772v2…