PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Distillation for Large Language Models

Alejandro Paredes La Torre, Barbara Flores, Diego Rodriguez

arXiv:2603.13765v1cs.CLcs.AI

TL;DR

Large language models require substantial computational resources, motivating methods that preserve capability in smaller models. This work combines knowledge distillation, coding-focused chain-of-thought reinforcement learning, and post-training quantization. The resulting students retain substantial teacher capability across language and coding tasks while reducing deployment costs.

  • Problem

    Large language models demand substantial computation, memory, and time, limiting their use in resource-constrained and real-time settings.

  • Method

    The study distills Qwen 3B into Qwen 0.5B across language and coding datasets, adds GRPO-guided chain-of-thought training for coding, and applies post-training quantization.

  • Results

    Knowledge distillation compresses large LLMs into lightweight students without sacrificing core performance, while CoT prompting with reinforcement learning produces richer reasoning on coding tasks.

  • Takeaways & Limitations

    The combined approach supports compact, efficient models for deployment in resource-constrained settings, including specialized coding tasks.

Abstract

from arXiv · show

We propose a resource-efficient framework for compressing large language models through knowledge distillation, combined with guided chain-of-thought reinforcement learning. Using Qwen 3B as the teacher and Qwen 0.5B as the student, we apply knowledge distillation across English Dolly-15k, Spanish Dolly-15k, and code BugNet and PyTorrent datasets, with hyperparameters tuned in the English setting to optimize student performance. Across tasks, the distilled student retains a substantial portion of the teacher's capability while remaining significantly smaller: 70% to 91% in English, up to 95% in Spanish, and up to 93.5% Rouge-L in code. For coding tasks, integrating chain-of-thought prompting with Group Relative Policy Optimization using CoT-annotated Codeforces data improves reasoning coherence and solution correctness compared to knowledge distillation alone. Post-training 4-bit weight quantization further reduces memory footprint and inference latency. These results show that knowledge distillation combined with chain-of-thought guided reinforcement learning can produce compact, efficient models suitable for deployment in resource-constrained settings.

1 Introduction

The study targets resource-intensive LLM deployment by compressing models through knowledge distillation and evaluating compact students across language and coding domains. It additionally combines coding-focused chain-of-thought reinforcement learning with quantization-oriented deployment goals.

  • 1.1 Problem: Knowledge distillation is used to create smaller language models with comparable performance and lower computational requirements.The motivation is deployment in settings with limited compute, memory, or latency budgets.
  • 1.2 Objectives: Post-training quantization is added to reduce distilled-model memory footprint and inference latency for resource-constrained deployment.This is presented as a secondary contribution complementing distillation and reasoning objectives.
  • 1.3 Contributions: Guided reinforcement learning with chain-of-thought traces is applied specifically to coding tasks to refine student outputs.The procedure uses GRPO and CoT-annotated Codeforces problem-solution pairs.

2 Related Work

Prior work established teacher-student distillation for neural networks and extended it to language models. Recent decoder-only LLM research further combines distillation with chain-of-thought prompting and reinforcement learning for complex tasks.

  • 2.1 Context: Hinton et al. introduced teacher-student knowledge distillation, while later image-classification work explored teacher-free and self-distillation alternatives.Teacher-free methods remove the external teacher, whereas self-distillation uses a model’s own predictions.
  • 2.1 Context: DistilBERT demonstrated NLP distillation by training a student on teacher logits as soft labels.The cited work reduced model size while retaining much of the teacher’s language understanding.
  • 2.1 Context: Gu et al. applied knowledge distillation to decoder-only LLMs using large open-source teachers and compact student counterparts in English.This work provides the LLM distillation foundation extended here across additional domains.
  • 2.1 Context: DeepSeek-related work added chain-of-thought prompting and reinforcement learning to guide smaller models through complex tasks.The present study adapts these techniques specifically to coding rather than foundational LLM training.

3 Methodology

The methodology combines transformer-based autoregressive modeling, teacher-student distillation, coding-focused reinforcement learning, and post-training quantization. Together, these components define how compact students learn task distributions, generate reasoning traces, and reduce deployment cost.

  • 3 Methodology: The Transformer provides the autoregressive architecture, using scaled dot-product attention as its core mechanism.The methodology frames subsequent language-model objectives within this architecture.
  • 3 Methodology: Decoder-only models assign probabilities to each token conditioned on its preceding context.The target function operates over token sequences and model parameters.
  • 3.1 Knowledge Distillation: Knowledge distillation trains the student to align its predicted distribution with the teacher’s distribution over the vocabulary.The objective uses teacher and student distributions in a KL-divergence-based soft-target formulation.
  • 3.2 Chain of Thought: Group Relative Policy Optimization is used in coding generation to encourage long reasoning traces through reinforcement learning.The objective includes a clipped policy term, group-based advantages, and a KL-divergence regularizer.
  • 3.2 Chain of Thought: The distillation process uses teacher-generated sequences while the student adjusts weights using a joint distillation loss.The teacher supplies task-distribution information through generated sequences.
  • 3.3 Post-Training Quantization: Post-training quantization compresses the distilled model without additional training or fine-tuning.The paper applies this technique to reduce memory footprint and inference latency for on-device deployment.
  • 3.3 Post-Training Quantization: Uniform affine quantization maps full-precision weights to low-bit integer representations using a scale factor, zero-point, rounding, and an integer range.The mapping is followed by dequantization to recover an approximation of the original weights.
  • 3.3 Post-Training Quantization: Four-bit weight quantization is applied to linear layers with GPTQ to reduce model size while maintaining task performance with negligible degradation.GPTQ minimizes layer-wise reconstruction error using approximate second-order information.

4 Experiments

The experiments evaluate knowledge distillation across English, Spanish, and coding tasks, then extend coding distillation with reinforcement learning guided by chain-of-thought objectives.

  • English knowledge distillation: 5 × 10^-4 produced the greatest loss reduction after epoch 3 and the best Rouge-L results in English Dolly distillation.This learning rate also matched the recommendation from the original paper.
  • Spanish knowledge distillation: The Spanish student trained with knowledge distillation achieved lower training loss and higher Rouge-L than the supervised fine-tuned counterpart.The advantage was especially pronounced during early training stages.
  • Spanish knowledge distillation: The Spanish distilled student closely approached the teacher despite using 10% of the total available parameters.The experiment used a Qwen 3B teacher and trained the student with supervised fine-tuning and knowledge distillation.
  • Coding knowledge distillation: BugNet evaluates code correction using incorrect code, console error output, and corresponding correct code, with Qwen2 3B as teacher and Qwen2 0.5B as student.The student learns through knowledge distillation after teacher supervised fine-tuning.
  • Coding knowledge distillation: PyTorrent evaluates code generation from natural-language and programming-language pairs, using the same teacher-student distillation procedure.A further PyTorrent experiment uses Qwen2.5 1.5B and Qwen2.5 0.5B models before adding reinforcement learning.
  • Chain-of-thought reinforcement learning: The coding reinforcement-learning setup uses rewards for reasoning keywords, correctness, output length, and strict procedural formatting.The training sequence shifts generation toward longer outputs containing targeted keywords while tracking KL divergence and reward increases.

5 Conclusions

The conclusions report that performance depends on the model's prior knowledge and training technique, while knowledge distillation compresses LLMs and chain-of-thought reinforcement learning enriches coding reasoning.

  • Conclusions: Performance varies with the implicit knowledge already available in the language model and with the technique employed.The conclusion contrasts faster, stronger performance with full knowledge against training under partial knowledge across language datasets.
  • Conclusions: Adding chain-of-thought to coding training enabled the model to handle a more complex task.The coding experiment is presented as confirmation of the broader dependence on training technique.
  • Conclusions: Knowledge distillation compressed large LLMs into lightweight students without sacrificing core performance.The paper also reports richer reasoning when chain-of-thought prompting is integrated with reinforcement learning, including on coding tasks.

Token-wise Knowledge Distillation for Autoregressive Language Models

Token-wise knowledge distillation trains a student autoregressive language model to match the teacher's token distributions at each timestep. The KL-divergence objective is equivalent to cross-entropy with soft teacher targets, preserving information beyond the ground-truth token.

  • Autoregressive formulation: An autoregressive language model factorizes sequence probability into conditional next-token probabilities given preceding context.The sequence is represented as tokens x1:T, with x<t denoting the prefix before timestep t.
  • Distillation objective: At each timestep, knowledge distillation compares teacher and student probability distributions over the vocabulary using token-wise KL divergence.The teacher distribution guides the student's distribution for the same preceding context.
  • Distillation objective: Expanding the KL divergence separates the teacher entropy term from the cross-entropy term involving student probabilities.The displayed expansion contains teacher-log-teacher and teacher-log-student components.
  • Soft-target training: Because teacher entropy does not depend on student parameters, minimizing KL divergence is equivalent to minimizing token-level cross-entropy with soft targets.The soft targets preserve relative probabilities assigned to non-maximum tokens, described as dark knowledge that improves generalization.

MiniLLM Framework

MiniLLM formulates knowledge distillation for language models with reverse KL divergence at the sequence level. Student-generated outputs receive teacher likelihood evaluations, turning distillation into a policy-gradient reinforcement learning problem.

  • Reverse KL divergence replaces forward KL divergence as MiniLLM's distillation objective.
  • At sequence level, qθ(o | q) represents the student distribution over complete outputs, while pT(o | q) represents the teacher distribution.
  • On-policy student sampling supplies outputs whose teacher log-probabilities guide optimization through a policy-gradient estimator.
  • Reverse KL is mode-seeking, concentrating student probability mass on high-probability teacher outputs.
  • MiniLLM empirically improves generation fidelity relative to conventional token-level forward KL distillation.

Group Relative Policy Optimization (GRPO) for Chain-of-Thought

GRPO trains reasoning-focused policies by sampling output groups, normalizing their rewards into relative advantages, and optimizing clipped policy updates with KL regularization. Unlike standard PPO, it removes the learned value-function baseline and explicit critic network.

  • GRPO samples a group of G outputs from the old policy for each prompt and assigns each output a scalar reward.
  • Within each group, rewards are normalized to compute relative advantages for policy optimization.
  • The GRPO objective uses a clipping parameter ϵ and KL regularization strength β toward a reference policy πref.
  • GRPO replaces standard PPO's learned value-function baseline with group-relative reward normalization, eliminating the need for an explicit critic network.
  • For chain-of-thought training, rewards reflect correctness and reasoning quality, shifting policies toward higher-reward reasoning trajectories while controlling reference-model deviation.
Loading 2603.13765v1…