PaperlayerView exact revision on arXiv

Source-linked AI summary

Masking Teacher and Reinforcing Student for Distilling Vision-Language Models

Byung-Kwan Lee, Yu-Chiang Frank Wang, Ryo Hachiuma

arXiv:2512.22238v1cs.LGcs.AIcs.CV

TL;DR

Large VLMs are impractical on mobile and edge devices, and the teacher–student size gap makes distillation difficult. Masters masks and progressively restores the teacher, then applies offline RL with accuracy and transferability rewards. It reports outperforming existing compact VLMs and exceeding large ones.

  • Problem

    Large VLMs are impractical for mobile or edge deployment, while the substantial teacher–student size gap makes distillation unstable and can degrade performance.

    Source: §1 · p. 2

  • Method

    Masters masks non-dominant teacher weights and progressively restores teacher capacity, using offline RL with pre-generated responses and accuracy and distillation rewards.

    Source: §1 · pp. 2–4

  • Results

    Masters consistently outperforms existing compact VLMs and exceeds large ones.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The framework provides stable, scalable knowledge transfer from large to compact models.

    Source: §5 · p. 11

    Offline training limits adaptability to real-time feedback.

    Source: §5 · p. 11

Abstract

from arXiv · show

Large-scale vision-language models (VLMs) have recently achieved remarkable multimodal understanding, but their massive size makes them impractical for deployment on mobile or edge devices. This raises the need for compact yet capable VLMs that can efficiently learn from powerful large teachers. However, distilling knowledge from a large teacher to a small student remains challenging due to their large size gap: the student often fails to reproduce the teacher's complex, high-dimensional representations, leading to unstable learning and degraded performance. To address this, we propose Masters (Masking Teacher and Reinforcing Student), a mask-progressive reinforcement learning (RL) distillation framework. Masters first masks non-dominant weights of the teacher to reduce unnecessary complexity, then progressively restores the teacher by gradually increasing its capacity during training. This strategy allows the student to learn richer representations from the teacher in a smooth and stable manner. To further refine knowledge transfer, Masters integrates an offline RL stage with two complementary rewards: an accuracy reward that measures the correctness of the generated responses, and a distillation reward that quantifies the ease of transferring responses from teacher to student. Unlike online think-answer RL paradigms that are computationally expensive and generate lengthy responses, our offline RL leverages pre-generated responses from masked teachers. These provide rich yet efficient guidance, enabling students to achieve strong performance without requiring the think-answer process.

1. Introduction

Large VLMs are impractical for mobile and edge deployment, while the size gap between teacher and student makes distillation unstable. Masters progressively restores a masked teacher and uses offline reinforcement learning with accuracy and transferability rewards; the paper reports gains over existing compact VLMs and large ones.

1. Introduction

  • Large VLMs impose computational costs that make them impractical on mobile or edge devices, motivating compact yet capable models.Distillation is a common route, but the large teacher–student size gap can destabilize learning and degrade performance.Source: §1 · p. 2
  • Masters masks the teacher’s non-dominant weights, then progressively restores its capacity so the student learns from coarse patterns toward richer representations.The passage describes this as enabling smoother and more stable knowledge transfer.Source: §1 · p. 2, Figure 3 · p. 3
  • Dual RL rewards assess response correctness and ease of transfer, addressing the risk that inaccurate or linguistically complex responses hinder distillation.The accuracy reward uses LLM-as-a-Judge; the distillation reward measures transferability between teacher and student.Source: §1 · pp. 3–4, Figure 4 · p. 4
  • Masters trains on responses pre-generated by masked teachers and students, avoiding explicit think-answer generation during RL and reducing training time and computation.The design also uses teacher responses suited to student capacity and the student’s own responses to maintain alignment as it learns.Source: §1 · p. 3
  • Masters combines teacher weight masking, progressive distillation, multi-response learning, and offline RL in one framework, reporting gains over compact VLMs and large ones.The framework is presented as a path toward efficient and deployable VLMs.Source: §1 · pp. 3–4

2. Related Work

Prior efficient-VLM work has used architectural changes and knowledge distillation, while distillation research has developed multiple ways to align teacher and student knowledge. The paper identifies the size gap as an unresolved challenge and directly addresses it by masking and progressively restoring the teacher.

2. Related Work

  • Efficient-VLM approaches have modified model architecture or propagation, while knowledge distillation transfers a large teacher’s knowledge to a smaller student.Distillation can provide rich logits or intermediate representations to compensate for the smaller model’s limited representational capacity.Source: §2 · p. 4
  • Distillation methods have progressed from logit and feature alignment to probabilistic, relational, contrastive, multi-teacher, reasoning-trace, and RL-based approaches.Other explored directions include alternative objectives, dynamic intermediate-layer and vision-attention distillation, multi-step methods, and cross-token general distillation.Source: §2 · p. 4
  • The teacher–student parameter gap remains a challenge: students struggle with rich, high-dimensional teacher representations, contributing to unstable training and performance degradation.The authors characterize direct efforts to narrow this gap as limited.Source: §2 · p. 5
  • Masters directly targets the gap by masking non-dominant teacher weights and progressively restoring the teacher as the student learns more complex representations.The paper presents this as capacity-aligned training intended to smooth optimization and mitigate instability in direct distillation.Source: §2 · p. 5

3. Masters

Masters combines magnitude-based teacher masking and progressive restoration with multi-response distillation and offline reinforcement learning. The framework pairs response accuracy with transferability rewards to refine student outputs.

3.1. Magnitude-based Teacher Masking

  • Masters masks low-magnitude teacher weights, then progressively restores teacher capacity so the student moves from coarse-grained knowledge toward refined representations.Masking is temporary; per-layer thresholds avoid the excessive pruning that can make the model non-functional at inference.Source: §3.1 · pp. 5–6, §3.2 · pp. 6–7

3.2. Mask-Progressive Distillation

  • Performance improves as the number of generated responses increases, converging around eight; the authors choose eight to balance training efficiency and performance.Figure 5 compares performance by the number of teacher and student responses.Source: Figure 5 · p. 6
  • The progressive strategy pairs teacher responses from changing masking ratios with student-generated responses, adapting guidance to the student’s evolving capacity.The paper frames this as addressing representation mismatch and the single-answer constraint of conventional SFT datasets.Source: §3.2 · pp. 7–8

3.3. Reinforcing Student with Dual Rewards

  • Masters uses offline RL with multiple pre-generated teacher and student responses, reducing the computational overhead of generating long responses during online training.The pre-generated responses cover all training questions and support scaled-up training with diverse responses.Source: §3.3 · p. 8
  • The RL stage combines an LLM-judge accuracy reward with a distillation reward based on teacher–student logit divergence, where lower divergence yields a higher reward.The total reward is the sum of the accuracy and distillation rewards.Source: §3.3 · pp. 8–9

4. Experiments

Experiments report that Masters reaches competitive results across model scales, while step-wise studies and ablations examine the effects of progressive distillation, response generation, and reward design.

4.1. Implementation Details

  • Experiments cover Qwen and InternVL student-teacher series and are conducted primarily on NVIDIA A100 80GB GPUs.The training setup generates eight responses per question from masked teacher checkpoints on a 1.5M-sample dataset.Source: §4.1 · p. 9

4.2. Step-wise Dissection of Masters

  • Mask-progressive distillation improves on naive JSD distillation, and adding RL feedback yields a cumulative performance boost.The step-wise comparison attributes improvements to capacity alignment and feedback on correctness and transferability.Source: §4.2 · p. 10
  • The paper reports that gradual teacher-size scaling consistently yields better performance than one-shot distillation from a large teacher, and associates it with smoother convergence and richer representations.One example increases teacher size from 14B to 38B; after warm-up with a mid teacher, performance accelerates significantly when distilling from the large teacher.Source: §4.2 · p. 10
  • With mask-progressive distillation and RL feedback, Masters matches or surpasses many recent open- and closed-source VLMs across model scales.Figure 6 compares inference hours on one GPU across models described in Tab. 3.Source: §4.2 · p. 10, Figure 6 · p. 10

4.3. Ablation Studies with Configurations

  • Performance steadily improves as the number of pre-generated responses increases, converging around eight responses, which the authors select to balance efficiency and performance.The response-count comparison is reported in Fig. 5.Source: §4.3 · p. 10
  • The optimal maximum masking ratio is 0.2 for most teachers, while Qwen3-VL-32B and InternVL3.5-8B/-14B perform best at 0.4.For InternVL3 distillation to an 8B student, gradual teacher scaling with 14B, 38B, and 78B teachers performs best.Source: §4.3 · p. 11
  • A balanced 1:1 mix of teacher- and student-generated responses addresses the limits of using either source alone, and removing any tested component degrades performance.The component comparison includes mask-progressive distillation, mid-teacher scaling, multi-response generation, and reward feedback.Source: §4.3 · p. 11
  • Removing the accuracy reward leaves performance below the large teacher, while omitting distillation-reward normalization makes performance comparable to using only accuracy reward.The authors report that both rewards are essential for identifying high-quality responses; Table 5 summarizes the configuration ablations.Source: Table 5 · p. 10, §4.3 · p. 11

5. Discussion and Conclusion

Masters avoids the inference-speed sacrifice associated with think-answer approaches, but its offline training limits real-time feedback. The paper reports strong, scalable knowledge transfer and identifies efficient online distillation as future work.

5. Discussion and Conclusion

  • Masters is trained offline, which limits its adaptability to real-time feedback.The authors report that online sampling for 1.5M samples would take over 30 days on 256 A100 GPUs for a single model, compared with two days for Masters.Source: §5 · p. 11
  • Think-answer methods increase latency and computational cost, while Masters attains strong performance without sacrificing inference speed.The paper identifies think-answer approaches as impractical for real-world or on-device use because of their computational cost.Source: §5 · p. 11
  • The conclusion presents Masters as a scalable approach to knowledge transfer and suggests data-driven masking and RL-based replay buffers for handling hard examples.Source: §5 · p. 11

A. Related Work of Efficient VLMs

Efficient VLM research responds to the computational demands of large models in real-world and edge settings. Related work explores both architectural and training strategies for lightweight models.

A. Related Work of Efficient VLMs

  • Visual instruction tuning and the scaling of LLMs accompanied the emergence of large-scale open-source and closed-source VLMs.Source: A. Related Work of Efficient VLMs · p. 23
  • Large VLMs impose substantial computational demands in real-world and on-device settings, motivating interest in lightweight models with fast inference.Source: A. Related Work of Efficient VLMs · p. 23
  • Related approaches include additional visual encoders, multiple vision backbones, rational embeddings, shared or repetitive FFNs, expanded hidden dimensions, aligned training, Mamba, and mixture-of-experts designs.Source: A. Related Work of Efficient VLMs · p. 23

B. The Objective of GRPO

The GRPO objective derives the student’s update from combined accuracy and distillation rewards, while KL regularization is used to stabilize training.

B. The Objective of GRPO

  • The GRPO objective uses a policy ratio and normalized-reward advantage, with a clipped update term and KL divergence from a reference policy.The clipped term limits the policy ratio to [1 − ε, 1 + ε], while the KL term regularizes deviation from the reference policy.Source: B. The Objective of GRPO · p. 23
  • The student’s total reward sums the accuracy and distillation rewards, and the resulting advantage guides its update.The student policy corresponds to its logit-softmax output.Source: B. The Objective of GRPO · p. 23
  • Because the student is updated only once per iteration, its old and new policies are identical and the ratio is one, making clipping redundant.The ratio is retained for gradient flow, and β is set to 0.1 for stable regularization.Source: B. The Objective of GRPO · p. 23

C. Visual Instruction Tuning Data

The visual instruction-tuning dataset combines real-world and synthetic sources in a 1.5M-sample collection spanning varied reasoning and perception tasks.

C. Visual Instruction Tuning Data

  • The visual instruction-tuning dataset contains 1.5M samples drawn from both real-world and synthetic sources.The listed sources include datasets and filtered synthetic data.Source: C. Visual Instruction Tuning Data · p. 24
  • Its listed materials include a wide range of visual instruction and question-answering datasets, including COCO-ReM, VQA-v2, AI2D, ChartQA, and DocVQA.The source list also includes filtered synthetic knowledge and perception data.Source: C. Visual Instruction Tuning Data · p. 24
  • Filtered synthetic data cover coarse- and fine-grained perception, relations, attributes, and logic reasoning.Source: C. Visual Instruction Tuning Data · p. 24

D. Additional Parsing Prompts for Accuracy Reward

The additional parsing prompts specify binary accuracy evaluation for generated responses and provide criteria and output formats for the evaluators.

D. Additional Parsing Prompts for Accuracy Reward

  • One prompt evaluates generated text against a ground-truth summary in the context of a question.It instructs the evaluator to compare the generated response with the ground truth.Source: D. Additional Parsing Prompts for… · p. 24
  • A response receives 1 when it correctly captures the core point, and 0 when it is incorrect, irrelevant, repetitive, or empty.Source: D. Additional Parsing Prompts for… · p. 24
  • The prompts prescribe constrained score outputs, including an integer-only response and an answer tag with a brief explanation.The integer-only prompt scores an overall summary; the tagged format is used for the question, ground-truth, and generated-text evaluation.Source: D. Additional Parsing Prompts for… · pp. 24–25
Loading 2512.22238v1…