Source-linked AI summary
PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation
Heng Li, Yong Zhang, Ning Cheng, Zhigen Li, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao
TL;DR
Vertical-domain few-shot classification challenges small language models because limited supervision leaves them without sufficient domain knowledge, while fixed KD-to-RL schedules do not account for sample-level differences in acquisition needs. PIVOT routes samples between OPD and GRPO using teacher-evaluated sequence perplexity. On Banking77 and HWU64, it outperforms continued OPD and globally synchronized OPD→GRPO under matched post-warm-up student optimization steps, with more stable training dynamics.
Problem
Globally fixed KD-to-RL schedules do not model sample-level differences in how much teacher-guided acquisition vertical-domain few-shot classification requires.
Source: §1 · pp. 1–2Method
PIVOT progressively routes samples between OPD and GRPO according to teacher-evaluated sequence perplexity.
Source: §1 · p. 2Results
Takeaways & Limitations
Adaptive sample-level transitions provide an effective alternative to fixed two-stage distillation and refinement schedules in the evaluated vertical-domain few-shot classification setting.
Source: §6 · p. 8The experiments cover vertical-domain few-shot classification with relatively structured outputs and reward signals; extension to more complex generation tasks remains open.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Vertical-domain few-shot classification remains challenging for small language models, as limited supervision makes it difficult to acquire domain-specific decision knowledge. On-Policy Distillation (OPD) can improve teacher-guided adaptation by supervising student-generated rollouts, while GRPO-based reinforcement learning can further refine downstream predictions. However, existing KD-to-RL pipelines typically rely on globally fixed transition schedules, ignoring that different samples may require different amounts of teacher-guided acquisition before reward-driven refinement. We propose PIVOT (Perplexity-Informed Transition Optimization), a dynamic transition framework that routes samples between OPD and GRPO according to teacher-evaluated sequence perplexity. PIVOT moves low-perplexity samples to GRPO for reward-driven refinement while keeping high-perplexity samples under OPD for continued domain knowledge acquisition. Experiments on Banking77 and HWU64 show that PIVOT consistently outperforms continued OPD and globally synchronized OPD$\rightarrow$GRPO baselines under the same number of post-warm-up student optimization steps, achieving stronger downstream performance and more stable training dynamics.
1 Introduction
The introduction frames vertical-domain few-shot adaptation as a domain-knowledge challenge and motivates moving from teacher-guided acquisition to reward-driven refinement at different times for different samples. It presents PIVOT’s perplexity-informed routing and reports stronger results than fixed alternatives on Banking77 and HWU64 under matched student optimization steps.
1 Introduction
- Limited supervision leaves small language models short of domain-specific knowledge, while OPD aligns teacher supervision with student-generated trajectories during adaptation.The introduction highlights professional domains such as finance, where prediction depends on specialized terminology and semantics.Source: §1 · p. 1
- In vertical-domain few-shot adaptation, OPD first supports domain-knowledge acquisition, while its marginal benefit diminishes as reward-driven refinement becomes more useful.This motivates shifting the dominant learning signal from teacher-guided acquisition to refinement as training progresses.Source: §1 · p. 2
- PIVOT routes samples progressively from OPD to GRPO using teacher-evaluated sequence perplexity, keeping harder samples in acquisition and sending more stable rollouts to refinement earlier.The framework is designed to avoid premature reinforcement learning on under-acquired samples and over-distillation on samples ready for refinement.Source: §1 · p. 2
- PIVOT outperforms continued OPD and globally synchronized OPD→GRPO on Banking77 and HWU64 under the same post-warm-up optimization steps, with more stable training dynamics.The authors also report stronger downstream refinement effectiveness than fixed two-stage schedules.Source: §1 · p. 2
- Prolonged teacher-guided distillation does not necessarily improve downstream GRPO refinement in vertical-domain few-shot adaptation.Source: §1 · p. 2
2 Method
PIVOT pairs OPD-based domain-knowledge acquisition with GRPO refinement and estimates sample acquisition using teacher-side perplexity. It uses acquisition scores to transition samples asynchronously between the two objectives.
2.1 Domain Knowledge Acquisition and Reward-Driven Refinement
- OPD aligns teacher supervision with evolving student rollouts for domain acquisition, while GRPO refines grouped responses through relative reward optimization.The two objectives serve complementary roles: acquisition under OPD precedes reward-driven refinement under GRPO.Source: §2.1 · pp. 2–3
2.2 Heterogeneous Acquisition Dynamics and Acquisition-State Estimation
- Teacher-evaluated, length-normalized sequence perplexity is aggregated into prompt-level acquisition scores and normalized within each mini-batch for routing.The candidate-level perplexities are combined into a prompt-level score; absolute perplexities are not directly comparable across prompts.Source: §2.2 · pp. 3–4
- Lower acquisition scores indicate samples closer to rollout-coverage saturation, while higher scores indicate continued benefit from teacher-supervised acquisition.Acquisition and transition timing vary across samples.Source: §2.2 · pp. 3–4
2.3 Asynchronous KD-to-RL Transition Scheduling
- A cosine-increased transition ratio progressively routes the lowest-scoring samples to GRPO while remaining samples continue OPD.Samples with lower acquisition-state scores are treated as closer to rollout-coverage saturation and transition earlier.Source: §2.3 · p. 4
- A binary routing indicator selects each sample’s GRPO or OPD loss within the step’s training objective.The indicator is 1 for GRPO refinement and 0 for continued OPD acquisition.Source: §2.3 · p. 4
3 Experiments
Under matched post-warm-up optimization budgets, PIVOT outperforms continued OPD and globally synchronized OPD→GRPO on both datasets, with smoother training dynamics. The experiments also examine scaling, sample-routing rules, and transition schedules.
3.1 Experimental Setup
- The experiments use class-balanced few-shot training sets of 385 Banking77 examples and 640 HWU64 examples, with official test splits and fixed sampling seed 42.The default models are Qwen2.5-0.5B-Instruct student and Qwen2.5-7B-Instruct teacher; scaling and stronger-teacher settings are also evaluated.Source: §3.1 · p. 4
- Trainable student methods share a two-epoch off-policy SFT warm-up followed by 480 post-warm-up update steps; PIVOT routes samples progressively using teacher-evaluated sequence perplexity.Lower normalized perplexity samples transition earlier to GRPO, while higher-perplexity samples continue OPD.Source: §3.1 · pp. 5–6
- GRPO rewards intent-label matches, with auxiliary formatting and consistency rewards; evaluation uses greedy decoding and test-set micro accuracy.Invalid or unparsable responses count as incorrect.Source: §3.1 · p. 5
3.2 Main Results
- PIVOT improves over continued OPD and globally synchronized OPD→GRPO on Banking77 and HWU64 under the same post-warm-up steps, and also surpasses teacher inference on both datasets.Table 1 reports test-set classification accuracy under 480 post-warm-up student optimization steps.Source: Table 1 · p. 5, §3.2 · p. 6
3.3 Optimization Dynamics
- Progressive sample-level transition achieves the highest final test accuracy with smoother entropy and completion-length dynamics than globally synchronized switching.The synchronized method shows abrupt completion-length increases and entropy spikes near its transition; PIVOT surpasses it before the global transition point.Source: Figure 2 · p. 5, §3.3 · p. 6
- High-perplexity samples continue improving more during OPD, while low-perplexity samples saturate earlier, indicating that transition timing can differ across samples.The figure groups Pass@32 performance by teacher-side perplexity buckets.Source: Figure 3 · p. 6
3.4 Scaling Analysis
- The scaling analysis evaluates PIVOT across student and teacher configurations, with Table 2 defining Δ as PIVOT's improvement over OPD→GRPO.The supplied passage identifies the scaling analysis but gives no configuration-specific results.Source: §3.4 · p. 6, Table 2 · p. 7
3.5 Ablation and Routing Analysis
- Across routing and schedule ablations, progressive sample-level routing outperforms globally synchronized switching, cosine routing performs best, and the strongest routing retains high-perplexity samples in OPD while moving low-perplexity samples to GRPO earlier.The table caption defines routing columns by which samples are preferentially sent to GRPO earlier; Figure 3 supports the different saturation patterns.Source: §3.5 · pp. 6–7, Figure 3 · p. 6, Table 3 · p. 7
4 Discussion
The discussion distinguishes teacher alignment from readiness for downstream refinement and argues that transition timing should vary across samples. It limits the study's demonstrated setting to OPD and GRPO for vertical-domain few-shot classification.
4.1 Distillation Quality and Refinement Readiness Are Not Necessarily Aligned
- The strongest downstream GRPO performance frequently occurs before full OPD convergence, so stronger distillation quality does not necessarily mean greater refinement readiness.The authors suggest prolonged imitation may narrow student rollouts around teacher-preferred responses, potentially reducing candidate diversity and observability for GRPO.Source: §4.1 · p. 7
- The authors conclude that maximizing teacher alignment alone may be insufficient because distillation and downstream refinement pursue partially different optimization objectives.This follows their observation that stronger imitation does not always translate into more effective downstream optimization.Source: §4.1 · p. 7
4.2 Implications for Adaptive Distillation-to-Refinement Transition
- Different samples show different OPD dynamics, supporting sample-specific rather than globally synchronized transition timing.Low teacher-side perplexity correlates with more stable rollouts and stronger Pass@32 performance, while high-perplexity samples often benefit from prolonged OPD before GRPO.Source: §4.2 · p. 7
- The study focuses on OPD and GRPO for vertical-domain few-shot classification; broader relevance to other iterative imitation and preference-optimization settings is presented as a possibility.The authors point to optimization dynamics as relevant alongside teacher-alignment quality when considering downstream refinement.Source: §4.2 · pp. 7–8
5 Related Work
Related work covers adaptive supervision within OPD and methods coordinating distillation with reinforcement learning. The paper distinguishes its focus on sample-level transition dynamics from prior globally shared objectives or fixed schedules.
5.1 On-Policy Distillation and Adaptive Optimization
- On-policy distillation aligns teacher supervision with the student's evolving rollout distribution to improve exposure alignment during autoregressive generation.The cited studies include Agarwal et al. (2024b) and Gu et al. (2024).Source: §5.1 · p. 8
- Adaptive OPD studies address heterogeneous rollout conditions through entropy-aware objectives, uncertainty-calibrated weighting, rollout coordination, competence-based schedules, and analyses of unreliable teacher guidance on unfamiliar prefixes.This paper instead studies how acquisition dynamics should govern the transition from distillation to downstream RL refinement.Source: §5.1 · p. 8
5.2 KD-RL Coordination and Adaptive Transition
- Prior KD-RL work explores complementary reinforcement learning, supervised signals, theoretical links between OPD and KL-regularized RL, and joint distillation-RL optimization.The cited examples include RLKD, KDRL, SuperRL, G-OPD, and CoDistill-GRPO.Source: §5.2 · p. 8
- The paper identifies a gap in existing coordination methods: they primarily use globally shared objectives or fixed schedules rather than explicitly modeling sample-level transitions.Related work also describes GRPO and reward-based optimization for downstream behavior beyond supervised imitation.Source: §5.2 · p. 8
6 Conclusion
The paper concludes that PIVOT adaptively schedules OPD and GRPO by teacher-evaluated sequence perplexity. On Banking77 and HWU64, it outperforms continued OPD and globally synchronized OPD→GRPO under matched post-warm-up steps, with more stable training dynamics.
6 Conclusion
- PIVOT uses teacher-evaluated sequence perplexity for progressive sample-level OPD-to-GRPO scheduling and outperforms continued OPD and globally synchronized OPD→GRPO on both datasets under equal post-warm-up steps.The authors present adaptive sample-level transition as an alternative to fixed two-stage schedules.Source: §6 · p. 8
7 Limitations
The study identifies limitations in PIVOT’s routing rationale, evaluation scope, and compute comparisons.
7 Limitations
- PIVOT uses teacher-evaluated sequence perplexity as a heuristic, without a theoretical account of why it reflects refinement readiness across optimization settings.The authors leave broader theoretical analysis for future work.Source: §7 · p. 9
- Experiments cover vertical-domain few-shot classification with relatively structured outputs and reward signals, leaving transfer to more complex generation tasks open.The authors identify evaluation on more diverse tasks as future work.Source: §7 · p. 9
- Comparisons match student optimization steps rather than total compute, and PIVOT adds teacher-inference overhead after transition relative to the globally synchronized baseline.The authors therefore characterize results as step-matched, not strictly compute-matched.Source: §7 · p. 9
AI Assistance Statement
The authors report using generative AI tools to assist with language editing and say they reviewed the generated content.
AI Assistance Statement
- ChatGPT and Gemini assisted with language editing during manuscript preparation.The statement names both tools.Source: AI Assistance Statement · p. 9
- The authors say they reviewed and verified all generated content.This is their stated review process.Source: AI Assistance Statement · p. 9
- The authors take responsibility for the final manuscript’s accuracy, integrity, and originality.The statement assigns responsibility to the authors.Source: AI Assistance Statement · p. 9
A.1 Dataset Construction
The experiments use class-balanced few-shot training sets drawn from the original training splits, with official test splits used for evaluation.
A.1 Dataset Construction
- Banking77 uses 5 examples per intent for 385 training examples across 77 intents, while HWU64 uses 10 per intent for 640 examples across 64 intents.Both few-shot training sets are class-balanced and sampled from the original training splits.Source: §A.1 · p. 11
- The sampling seed is fixed at 42, and evaluation uses the official test splits with micro accuracy reported.Unless otherwise stated, experiments use the same sampled few-shot split.Source: §A.1 · p. 11
- Main results use a single sampled split because OPD and GRPO training are computationally expensive.The stated rationale applies to the main few-shot results.Source: §A.1 · p. 11
A.2 Prompt Templates
The appendix specifies prompt and prediction handling, baseline training procedures, a Banking77 data-augmentation comparison, and the GRPO reward components.
A.2 Prompt Templates
- Prompts contain the full task label set; Banking77 adds natural-language intent definitions, while HWU64 uses only the intent labels.The prompt-template tables are identified as Banking77 and HWU64 templates.Source: §A.2 · p. 11, Table 5 · p. 12, Table 6 · p. 12
A.3 Output Parsing
- Predictions are parsed from the first valid label after “Intent Category:”; unmatched outputs count as incorrect, with no manual correction.If multiple valid labels follow the field, the first match is used.Source: §A.3 · p. 11
A.4 Baseline Implementations
- GRPO-based methods and OPD use the same two-epoch off-policy SFT warm-up; OPD→GRPO then switches globally at a selected checkpoint within 480 post-warm-up steps.OPD runs for all 480 steps, while GRPO follows warm-up directly; the global transition checkpoint is selected within the same step budget.Source: §A.4 · p. 11
- The listed baselines also include teacher-only inference, label-only SFT-ZeroShot, teacher-response SeqKD, and teacher-filtered DataAug-SFT on Banking77.SFT-ZeroShot uses gold labels, while SeqKD fine-tunes on teacher-generated responses.Source: §A.4 · p. 11
A.5 Data Augmentation Baseline
- On the same 5-shot Banking77 split, the augmentation pipeline groups semantically similar intents, prompts a teacher to generate diverse, label-discriminative queries using intent information and anchor examples, then retains samples reclassified as the target intent.Source: §A.5 · p. 11
- 71.75 test accuracy on Banking77 is reported for DataAug-SFT, below continued OPD and OPD→GRPO; the comparison table covers teacher-generated augmentation and OPD/GRPO strategies.DataAug-SFT uses 5,119 teacher-filtered synthetic examples from the same 5-shot split; the authors say the transition yields an additional gain not explained solely by more synthetic supervision or extended optimization.Source: §A.5 · p. 11, Table 7 · p. 12
A.6 Training Hyperparameters
- Table 9 summarizes the main training hyperparameters used across all experiments; those experiments were conducted on a server with four NVIDIA A100 GPUs.Prompt-template tables cover Banking77, HWU64, and teacher-based data augmentation.Source: §A.6 · pp. 11–12, Table 5 · p. 12, Table 6 · p. 12
A.7 Reward Function
- GRPO reward combines correctness (+1/−1), candidate alignment (+0.5/−0.5), and valid-format (+0.1) components, with no explicit length penalty.The method also uses a KL penalty coefficient of 0.001 and normalizes advantages within each response group.Source: §A.7 · p. 12