Source-linked AI summary
On-Policy Distillation for Vision-Language Model Adaptation, an Effective Paradigm on Low-Quality Multimodal Data
Hongyuan Zhang, Xianda Guo, Yanlun Peng, Qianlong Yang, Yubin Guo, Pinhan Fu, Mulin Chen, Xiaozhen Qiao, Ping Luo
TL;DR
Vision-language distillation commonly applies a fixed teacher-derived target despite changing reliability under class and domain shifts. OnPoKD learns a validation-guided policy to construct adaptive targets from teacher, zero-shot prior, and hard-label supervision, improving Base-to-novel and Cross-dataset transfer without test-time overhead.
Problem
Fixed teacher-centered targets can be unreliable when teacher predictions vary in quality across class or domain shifts, while zero-shot and label supervision may remain useful.
Method
OnPoKD uses a lightweight on-policy controller to adaptively mix teacher, frozen zero-shot prior, and hard-label supervision using reliability cues and validation feedback.
Results
OnPoKD consistently improves Base-to-novel generalization and Cross-dataset transfer, raising average harmonic mean over PromptKD from 83.73 to 84.62 across eleven benchmarks.
Takeaways & Limitations
Adaptive target construction improves the robustness and transferability of vision-language distillation while preserving the student's original inference architecture and test-time cost.
Takeaways & Limitations
Policy learning uses held-out source-domain validation feedback and excludes novel-class images and target-domain test samples from policy updates.
Abstract
from arXiv · showhide
Knowledge distillation offers an efficient route to transfer a task-adapted vision-language teacher to a compact student. The training target in current vision-language distillation methods is typically constructed from the teacher prediction and applied uniformly to all training samples, making it unreliable under class and domain shifts. In this paper, we argue that distillation target construction should be treated as a dynamic training decision rather than a fixed recipe. To this end, we propose OnPoKD, an on-policy distillation framework for vision-language model adaptation. To the best of our knowledge, OnPoKD is the first framework that applies on-policy distillation to vision-language model adaptation by learning target construction as a policy decision. OnPoKD learns a lightweight controller that constructs sample-wise adaptive targets using reliability and disagreement cues from the teacher model, student model, and zero-shot prior. Instead of relying on a fixed teacher prediction, the controller dynamically balances teacher supervision, zero-shot prior guidance, and hard-label anchoring through bounded policy actions, allowing the distillation target to adapt to varying sample reliability and training stages. The policy controller is updated with validation feedback, encouraging target construction to optimize transferability rather than merely fitting the training distribution. Since the controller is only used during training, OnPoKD can be seamlessly integrated into existing vision-language distillation pipelines while preserving the original inference architecture and test-time cost. Extensive experiments on Base-to-novel generalization and Cross-dataset transfer benchmarks show that OnPoKD consistently improves over strong vision-language distillation baselines.
1. Introduction
Vision-language adaptation must improve target-task discrimination without losing the zero-shot prior that supports novel classes and shifted domains. OnPoKD addresses this tension by learning adaptive, validation-guided distillation targets and improves transfer across class and dataset shifts.
- Fixed teacher-centered distillation targets are unreliable because teacher reliability varies across samples under class and domain shifts.
- Across eleven Base-to-novel benchmarks, ONPOKD raises average harmonic mean over PromptKD from 83.73 to 84.62.
- OnPoKD treats target construction as an on-policy decision that provides sample-wise and stage-wise adaptive supervision.
- A lightweight controller balances adapted-teacher, zero-shot-prior, and hard-label supervision using reliability and disagreement cues, with validation-guided updates.
- Experiments show improved robustness and transferability on both Base-to-novel generalization and Cross-dataset transfer benchmarks.
2. Related Work
Vision-language distillation transfers adapted recognition into compact students, but fixed teacher targets can fail when teacher reliability changes across classes or domains. ONPOKD instead adapts targets using the teacher, zero-shot prior, and hard-label guidance.
- Knowledge distillation transfers information from a stronger teacher to a smaller or more deployable student by matching softened predictive distributions.
- Standard distillation assumes the teacher distribution is appropriate for every sample, an assumption that breaks under class and domain shifts.
- A single teacher-centered target cannot express reliability changes or determine when hard labels should override uncertain soft targets.
- ONPOKD keeps teacher-dominant supervision when reliable and uses the frozen zero-shot prior or hard-label guidance when soft targets are unreliable.
3. Method
OnPoKD replaces fixed teacher-centered distillation targets with bounded, sample-wise actions that adapt supervision using reliability, disagreement, and training progress. A lightweight controller mixes teacher, zero-shot prior, and hard-label guidance during training, with validation feedback steering target construction while preserving a compact inference path.
- Problem formulation: Conventional distillation applies the same teacher-centered target to every image, despite reliability changes under class shift and optimization.The zero-shot prior and hard labels can provide complementary supervision when teacher predictions are uncertain or conflicting.
- Training and inference: At inference, the controller and prior branch are removed, leaving the original student architecture and test-time cost unchanged.The adaptive machinery is therefore used only to construct training targets and update the student.
- On-policy controller: OnPoKD learns a lightweight controller that constructs sample-wise targets from teacher, student, and zero-shot-prior reliability and disagreement cues.The controller also conditions its actions on normalized training progress rather than relying on a fixed supervision recipe.
- Policy-guided target construction: The controller emits bounded mixture, sample-weight, and temperature actions, combining teacher supervision with prior guidance and hard-label anchoring.Caps and projection keep prior and hard-label contributions auxiliary, while bounded scalar actions limit sample reweighting and target softness.
- Training procedure: A teacher-dominant initialization keeps early training conservative, while zero-initialized controller outputs and caps prevent random auxiliary targets from replacing useful teacher supervision.Validation feedback is applied periodically, and the controller can adapt sample importance and target softness within bounded ranges.
4.1. Experimental Setup
The evaluation tests open-vocabulary preservation and robustness under dataset shift using Base-to-novel and Cross-dataset settings. Experiments use a fixed ViT-B/16 CLIP backbone and compare OnPoKD with zero-shot, prompt-learning, and distillation baselines.
- Base-to-Novel Generalization: Base-to-novel evaluation adapts on base-class data and tests both base and novel classes.Reported metrics are base accuracy, novel accuracy, and their harmonic mean (HM).
- Cross-Dataset Evaluation: Cross-dataset evaluation adapts on ImageNet and transfers to other datasets without using their training images.The benchmark reports top-1 accuracy for each target dataset and average accuracy across targets.
- Implementation Details: Training-time and memory overhead are compared between OnPoKD and PromptKD on two representative ablation datasets.The supplied caption identifies this comparison but does not provide the overhead values.
- Datasets: The Base-to-novel benchmark contains eleven datasets spanning objects, fine-grained categories, textures, scenes, actions, and remote sensing.The listed datasets include ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, and UCF101.
- Implementation Details: All comparisons use ViT-B/16 CLIP and include zero-shot CLIP, four prompt-learning methods, and PromptKD.OnPoKD keeps the teacher, student, and test-time inference architecture unchanged; only target construction differs from the underlying distillation pipeline.
4.2. Base-to-Novel Generalization
Across eleven Base-to-novel benchmarks, ONPOKD improves PromptKD’s harmonic mean, with larger gains on novel than base classes and improvements on every dataset. The pattern is consistent with adaptive balancing of teacher, prior, and label guidance.
- Base-to-Novel Generalization: 83.73 to 84.62 HM: ONPOKD improves PromptKD on average across eleven Base-to-novel datasets.The average gain is accompanied by larger improvement on novel classes (+1.40) than base classes (+0.30).
- Base-to-Novel Generalization: ONPOKD improves HM on all eleven benchmarks, including +2.88 HM on EuroSAT and +2.49 HM on FGVCAircraft.Other reported gains include +1.64 HM on DTD, +0.83 HM on StanfordCars, and +0.33 HM on ImageNet.
- Base-to-Novel Generalization: The policy retains teacher-centered supervision for reliable base knowledge while adding zero-shot prior and hard-label guidance when teacher transfer reliability is lower.This adaptive balance is presented as the mechanism underlying the broader novel-class gains.
4.3. Cross-Dataset Transfer
On Cross-dataset transfer from ImageNet to ten unseen target datasets, ONPOKD achieves the best average accuracy and wins on most targets. Gains are largest on visually or semantically different datasets, with small decreases on three datasets where PromptKD is already strong.
- Cross-Dataset Evaluation: 72.66 average accuracy: ONPOKD improves PromptKD by 1.33% when transferring from ImageNet to ten target datasets.The target datasets are not seen during adaptation.
- Cross-Dataset Evaluation: ONPOKD achieves the strongest accuracy on seven of ten target datasets.Largest gains are +5.69 on EuroSAT, +3.11 on DTD, +2.11 on Cars, +1.79 on Aircraft, and +1.54 on Flowers102.
- Cross-Dataset Evaluation: ONPOKD decreases accuracy on Caltech101 (-0.57), Food101 (-0.32), and UCF101 (-1.07) relative to PromptKD.The passage characterizes these as datasets where PromptKD is already strong.
4.4. Ablation Study
The ablations show that OnPoKD's gains come from adaptive control of complementary supervision sources, policy actions, and reliability cues rather than simply adding supervision. Their roles vary across datasets, making sample-wise target construction and validation feedback important.
- Supervision sources: OnPoKD outperforms teacher-only distillation and fixed mixtures because the policy decides when to trust the teacher, prior, and label.Relative to T-KD, it improves FGVCAircraft HM by 3.65%, EuroSAT HM by 1.22%, DTD accuracy by 5.00%, and EuroSAT Cross-dataset accuracy by 6.88%.
- Policy actions: The full policy controls target mixing, sample weighting, and temperature because these actions determine what to trust, how much to trust it, and how sharply to match it.Adaptive mixing alone helps EuroSAT but harms FGVCAircraft, while weighting and temperature recover performance on the more fine-grained dataset.
- Reliability cues: Uncertainty cues are the most important policy-state signal, with their removal reducing FGVCAircraft HM by 9.69% and EuroSAT HM by 3.30%.Teacher-student disagreement, teacher-prior conflict, and feature alignment add complementary relational information.
- Reliability cues: The learned policy increases prior mass under uncertainty on EuroSAT but keeps the prior dormant on FGVC-Aircraft with a weak teacher.This state-to-action behavior is consistent with dataset-dependent supervision roles.
4.5. Hyperparameter Analysis
Hyperparameter sweeps support bounded, feedback-stabilized intervention: moderate auxiliary capacity, regularization, and action strength provide the strongest Base-to-novel results, while Cross-dataset transfer is less monotonic.
- Cross-dataset transfer: Cross-dataset transfer is less monotonic and remains sensitive to the exact intervention scale and policy setting.The best Cross-dataset transfer occurs at different settings from the best Base-to-novel balance, supporting bounded rather than uniformly stronger intervention.
- Auxiliary cap: Raising the auxiliary cap from 0.10 to 0.40 improves EuroSAT HM from 90.32 to 92.02, but increasing it to 0.50 lowers HM to 90.95.The cap improves the balance while base accuracy stays high, but excessive auxiliary mass weakens Base-to-novel generalization.
- Policy regularization: Policy regularization of α_r=0.05 gives the best Base-to-novel result, reaching 97.64 Base, 87.01 Novel, and 92.02 HM.Without regularization, EuroSAT HM falls to 87.71, indicating overly aggressive target updates.
- Action strength: Action strength s=0.50 yields the best EuroSAT Base-to-novel HM of 92.02, whereas s=0.25 gives higher Base accuracy but lower Novel accuracy and HM.The results show that stronger intervention can improve the overall balance even when it does not maximize Base accuracy alone.
- Design implication: The sweeps support adaptive target construction that corrects unreliable teacher targets while keeping each policy intervention bounded and feedback-stabilized.The auxiliary cap controls available non-teacher supervision, while regularization and action scale control how safely it is applied.
5. Conclusion
OnPoKD treats distillation target construction as a policy decision that adaptively combines teacher, zero-shot prior, and hard-label supervision using reliability cues and validation feedback. Experiments show improved robustness and transferability without changing the student's inference architecture or adding test-time computation.
- Framework: OnPoKD adaptively mixes adapted-teacher, zero-shot-prior, and hard-label supervision instead of imposing a fixed teacher-centered target.The controller uses reliability cues and validation feedback to shape target construction during training.
- Deployment: The student keeps the same inference architecture and adds no test-time computation because the policy only shapes the training objective.This preserves the original deployment path while changing how the student is trained.
- Results: Experiments on Base-to-novel generalization and Cross-dataset transfer show improved robustness and transferability, including gains on FGVCAircraft, DTD, and EuroSAT.Ablations attribute the gains to the combined contributions of supervision sources, validation feedback, and reliability-aware policy cues.