PaperlayerView exact revision on arXiv

Source-linked AI summary

OVD: On-policy Verbal Distillation

Jing Xiong, Hui Shen, Shansan Gong, Yuxin Cheng, Jianghan Shen, Chaofan Tao, Haochen Tan, Haoli Bai, Lifeng Shang, Ngai Wong

arXiv:2601.21968v1cs.CL

TL;DR

Existing reinforcement-learning distillation methods are limited by memory efficiency and their use of verbal feedback. OVD matches student trajectories using discrete teacher scores, outperforming existing methods across Web Q&A and mathematical reasoning benchmarks, including a 25.7% math improvement.

  • Problem

    Existing reinforcement-learning distillation approaches remain limited in memory efficiency and their ability to leverage verbal feedback.

  • Method

    OVD replaces token-level probability matching with trajectory matching supervised by discrete verbal scores from teacher models.

  • Results

    Across Web Q&A and mathematical reasoning benchmarks, OVD substantially outperforms existing methods, achieving a 25.7% improvement on math benchmarks from one randomly selected instance.

  • Takeaways & Limitations

    OVD enables memory-efficient black-box distillation while providing unbiased gradient estimates with reduced variance and convergence guarantees under mixture training.

Abstract

from arXiv · show

Knowledge distillation offers a promising path to transfer reasoning capabilities from large teacher models to efficient student models; however, existing token-level on-policy distillation methods require token-level alignment between the student and teacher models, which restricts the student model's exploration ability, prevent effective use of interactive environment feedback, and suffer from severe memory bottlenecks in reinforcement learning. We introduce On-policy Verbal Distillation (OVD), a memory-efficient framework that replaces token-level probability matching with trajectory matching using discrete verbal scores (0--9) from teacher models. OVD dramatically reduces memory consumption while enabling on-policy distillation from teacher models with verbal feedback, and avoids token-level alignment, allowing the student model to freely explore the output space. Extensive experiments on Web question answering and mathematical reasoning tasks show that OVD substantially outperforms existing methods, delivering up to +12.9% absolute improvement in average EM on Web Q&A tasks and a up to +25.7% gain on math benchmarks (when trained with only one random samples), while also exhibiting superior training efficiency. Our project page is available at https://OVD.github.io

1. Introduction

OVD addresses the memory, alignment, and exploration limitations of token-level reinforcement-learning distillation by matching student trajectories with discrete verbal scores from teacher models. Experiments and theoretical analysis show improved performance and efficiency across Web Q&A and mathematical reasoning tasks.

  • Complex multi-step reasoning remains challenging, while the computational and resource costs of training large reasoning models hinder deployment.
  • Token-level distillation creates severe reinforcement-learning memory bottlenecks by storing teacher probability distributions over the full vocabulary at every decoding step.It also constrains exploration through strict student–teacher distribution matching and does not capture hierarchical reasoning structure.
  • OVD reformulates distillation as trajectory matching, using discrete verbal feedback from teacher models to supervise student reasoning trajectories during on-policy reinforcement learning.Teacher agents score reasoning correctness, enabling the student to learn reasoning patterns and interaction behaviors.
  • OVD replaces full-vocabulary logit supervision with verbal trajectory evaluation, greatly reducing memory cost and promoting on-policy exploration.
  • 25.7% performance improvement is achieved on math benchmarks by scaling trajectory sampling on a single randomly selected instance.The experiments cover Web Q&A and mathematical reasoning benchmarks and report superior sample efficiency relative to existing RL-based web search methods.
  • OVD’s rejection sampling scheme yields unbiased gradient estimates, and an interactive environment agent is key to success on Web Q&A tasks.

2. Why Verbal Distillation?

OVD adopts verbal scoring instead of token-level distillation to address token-level supervision’s memory bottleneck and lack of semantic, vocabulary-aligned feedback. It evaluates reasoning at step or trajectory level, enabling more interpretable supervision without token-level alignment.

  • Memory Bottleneck: Token-level distillation treats all tokens equally, requires teacher token probabilities, and incurs prohibitive memory overhead for long reasoning rollouts.These limitations hinder hierarchical reasoning supervision and may exclude black-box teachers whose token-level distributions are unavailable.
  • Memory Bottleneck: O(N · L · V ) memory arises from storing vocabulary logits for every token, with V ≈32K–128K and long-horizon trajectories.In GRPO, maintaining logits for all N samples simultaneously creates this overhead.
  • Verbal Feedback: OVD replaces token-level logits with 0–9 verbal scores, reducing memory to O(N · K · v) because K ≪L for step-level scoring.The teacher assesses correctness, relevance, and coherence while providing interpretable explanations of errors and improvements.
  • Memory Bottleneck: O(N = 32) rollouts require 240 GB for logits alone, making token-level distillation impractical for reinforcement learning.At fixed sequence length L = 8192, logits require ∼16× more memory than the KV cache.

3. On-policy Verbal Distillation

OVD transfers reasoning capabilities through on-policy trajectory optimization guided by discrete verbal teacher feedback. Verbal rejection sampling combines student exploration with teacher demonstrations while supporting stable, theoretically justified training.

  • Framework: OVD trains a student model to produce high-quality multi-step reasoning trajectories through teacher-guided trajectory optimization with verbal feedback.The student samples reasoning steps, while the teacher evaluates trajectories toward correct answers.
  • Verbal Rejection Sampling: On-policy sampling uses the student’s own distribution, while low-scoring trajectories are rejected and replaced through teacher-guided resampling.The resulting training samples contain complete trajectories and combine student explorations with teacher acceptance signals.
  • Verbal Rejection Sampling: 10 discrete scores, {0, 1, . . . , 9}, replace full-vocabulary logits for step- and trajectory-level quality assessment.This compact score space supports black-box teachers and preserves judgment uncertainty through score sampling.
  • Mixture Training: The training batch mixes accepted student trajectories with teacher trajectories, allowing learning from both exploration and expert demonstrations.As student quality improves, the mixture shifts toward accepted student trajectories, mitigating distribution shift while maintaining on-policy learning.
  • Theoretical Analysis: Theoretical results establish unbiased gradient estimation, variance reduction, convergence under mixture training, and approximation error decreasing as O(1/v).The variance reduction comes from replacing rejected trajectories with lower-variance teacher demonstrations.

4. Experiments

Experiments evaluate OVD on Web Q&A and mathematical reasoning benchmarks, including eight Web Q&A datasets and diverse math benchmarks. OVD consistently outperforms search-based baselines, while ablations show that SFT-based agents and moderate rejection thresholds improve performance and that verbal feedback enables sample-efficient mathematical RL training.

  • Evaluation Tasks: OVD is evaluated on Web Q&A and mathematical reasoning tasks.The Web Q&A evaluation spans eight single-hop and multi-hop question-answering benchmarks, while the mathematical evaluation uses diverse reasoning benchmarks.
  • Web Q&A Results: OVD consistently outperforms Search-o1, Search-R1, and ZeroSearch across eight Web Q&A test datasets and both model backbones.Table 3 reports exact match (EM) scores; OVD† uses score sampling during testing rather than deterministic threshold-based selection.
  • Ablation Studies: A moderate testing rejection threshold of 5 outperforms thresholds of 0 and 10, balancing student exploration and teacher guidance.Threshold 0 accepts all trajectories, whereas threshold 10 causes teacher intervention in nearly all cases.
  • Ablation Studies: SFT-based environment agents achieve an average score of 0.486 versus 0.241 for prompt-based agents, a 101.7% relative improvement.The best SFT configuration is T10 with QR test T10, while the best prompt-based configuration is T5 with QR test T10.
  • Mathematical Reasoning Results: 55.3% average accuracy at 800 training steps is achieved by OVD with 1 random training example per problem, versus 52.7% for one-example RLVR.OVD improves the baseline performance (23.9%) by over 30 percentage points and exceeds same-budget one-example RLVR by 2.6 percentage points.
  • Mathematical Reasoning Results: 55.2% average accuracy is reached by OVD at 300 training steps with 2 training examples per problem, approaching 55.3% at 800 steps.This demonstrates accelerated convergence when OVD leverages additional training examples.

5. Conclusion · A. Appendix

OVD transfers reasoning capabilities from large teachers to smaller students through verbal feedback while replacing token-level probability matching with trajectory-level optimization. The framework reduces memory consumption, enables black-box distillation, and offers unbiased, lower-variance gradient estimates with convergence guarantees.

  • 5. Conclusion: OVD is a memory-efficient framework for transferring reasoning capabilities from large teacher models to smaller student models via verbal feedback.
  • 5. Conclusion: OVD replaces token-level probability matching with trajectory-level optimization.
  • 5. Conclusion: This trajectory-level formulation significantly reduces memory consumption during distillation.
  • 5. Conclusion: OVD enables black-box distillation from teacher models.
  • 5. Conclusion: Theoretical analysis shows that OVD provides unbiased gradient estimates.
  • 5. Conclusion: The analysis also establishes reduced gradient-estimate variance and convergence under the mixture training distribution.

A.1. Theoretical Proofs

This section provides detailed proofs for the theoretical results presented in Section 3.5.

  • A.1. Theoretical Proofs: The section contains detailed proofs.
  • A.1. Theoretical Proofs: These proofs address theoretical results.
  • A.1. Theoretical Proofs: The theoretical results were presented in Section 3.5.

A.1.1. PROOF OF PROPOSITION 3.3 (CONVERGENCE UNDER MIXTURE TRAINING) · A.1.2. PROOF OF THEOREM 3.1 (UNBIASED GRADIENT ESTIMATION)

The proofs establish that verbal rejection sampling converges toward teacher-level performance under mixture training and yields an unbiased gradient estimator for the resulting mixture-distribution objective. Together, they characterize convergence through acceptance and quality-gap dynamics and explain how accepted student trajectories and teacher demonstrations jointly shape learning.

  • A.1.1. PROOF OF PROPOSITION 3.3 (CONVERGENCE UNDER MIXTURE TRAINING): OVD’s convergence proof decomposes expected reward under the mixture distribution, bounds accepted student trajectories’ performance gap, and combines these steps to show convergence toward teacher-level performance.The proof explicitly proceeds through mixture decomposition, accepted-trajectory quality, and convergence analysis.
  • A.1.1. PROOF OF PROPOSITION 3.3 (CONVERGENCE UNDER MIXTURE TRAINING): As training progresses, the student generates higher-quality trajectories, increasing the acceptance rate while the performance gap δt between accepted student and teacher trajectories decreases monotonically.These are the two stated convergence properties: increasing acceptance rate and decreasing quality gap.
  • A.1.1. PROOF OF PROPOSITION 3.3 (CONVERGENCE UNDER MIXTURE TRAINING): αt →1 shifts training from teacher demonstrations toward the student policy, forming a curriculum that progresses from stable teacher-guided learning to autonomous exploration and refinement.Initially αt ≈0, while later αt →1.
  • A.1.1. PROOF OF PROPOSITION 3.3 (CONVERGENCE UNDER MIXTURE TRAINING): The bound Ey∼ptrain[R(y)] ≥ [1 −αtδt] · J(πT ) guarantees asymptotic convergence as αt →1 and δt →0, while β controls how aggressively low-quality trajectories are rejected.The convergence rate depends on the product αtδt and, in practice, on the temperature parameter β.
  • A.1.2. PROOF OF THEOREM 3.1 (UNBIASED GRADIENT ESTIMATION): The unbiased-gradient proof uses threshold acceptance a(y) = 1[S(y) ≥θ], where verbal scores S(y) and thresholds θ range over {0, 1, . . . , 9}.A trajectory is accepted when S(y) ≥θ and rejected otherwise.
  • A.1.2. PROOF OF THEOREM 3.1 (UNBIASED GRADIENT ESTIMATION): Rejected student trajectories are replaced by teacher demonstrations, producing pRS(y) = πS(y) · 1[S(y) ≥θ] + (1 −αt) · πT (y).Here αt = Ey∼πS[1[S(y) ≥θ]] is the expected acceptance rate.
  • A.1.2. PROOF OF THEOREM 3.1 (UNBIASED GRADIENT ESTIMATION): The derived estimator matches the policy-gradient expression for the mixture objective, confirming that it is unbiased for that objective.Its two terms combine accepted student trajectories with teacher demonstrations weighted by 1 −αt.
  • A.1.2. PROOF OF THEOREM 3.1 (UNBIASED GRADIENT ESTIMATION): Unlike importance sampling requiring teacher-to-student density ratios and teacher logits, verbal rejection sampling uses binary quality decisions and can distill from black-box teachers.The gradient’s student term learns from successful exploration, while its teacher term supplies demonstrations when student samples are insufficient.

A.1.3. PROOF OF PROPOSITION 3.2 (VARIANCE REDUCTION)

The proof shows that verbal rejection sampling reduces gradient-estimator variance by replacing rejected student trajectories with teacher demonstrations. A threshold separates accepted trajectories used as-is from rejected trajectories replaced for greater training stability.

  • Rejection Sampling Estimator: The estimator uses a student trajectory when S(y) ≥ θ and a teacher trajectory when S(y) < θ.Here y ∼ πS, y′ ∼ πT, and αt = Pr(S(y) ≥ θ).
  • Variance Comparison: Rejected student trajectories are replaced with teacher demonstrations, which typically have lower variance because of their consistent high quality.This replacement is the source of the variance reduction relative to vanilla on-policy learning.
  • Variance Comparison: Under bounded teacher variance and higher variance for rejected than accepted trajectories, replacing rejected trajectories yields a variance-reduction lower bound.The comparison assumes E_y′∼πT[∥R(y′)∇θ log πS(y′)∥2] ≤ VT.
  • Practical Implications: The threshold mechanism stabilizes training by filtering low-reward, high-variance trajectories, especially when the student policy is weak and exploration is noisy.The supplied practical implication also describes more stable gradient estimates and faster convergence during early training.

A.1.4. PROOF OF PROPOSITION 3.4 (SCORE GRANULARITY AND APPROXIMATION QUALITY) · A.1.5. ADDITIONAL THEORETICAL REMARKS

The proof models verbal scoring as uniform quantization of normalized trajectory quality, establishing O(1/v) approximation error and improved rejection-sampling alignment as score granularity increases. Additional remarks relate verbal scoring to importance sampling and state an O(1/T) convergence rate, while noting practical limits on vocabulary size.

  • A.1.4. PROOF OF PROPOSITION 3.4 (SCORE GRANULARITY AND APPROXIMATION QUALITY): The discretization approximation error decreases as O(1/v) under uniform scalar quantization.The proof bounds the quantization error pointwise and then averages it over trajectories.
  • A.1.4. PROOF OF PROPOSITION 3.4 (SCORE GRANULARITY AND APPROXIMATION QUALITY): Trajectory quality Q(y) is normalized to [0, 1] and uniformly quantized into v integer scores.The setup defines the discrete score as S_v(y) = floor((v − 1) · Q(y)).
  • A.1.4. PROOF OF PROPOSITION 3.4 (SCORE GRANULARITY AND APPROXIMATION QUALITY): The normalized score reconstructs quality on the grid {0, 1/(v − 1), ..., 1}.The reconstruction is ˜S_v(y) = S_v(y)/(v − 1).
  • A.1.4. PROOF OF PROPOSITION 3.4 (SCORE GRANULARITY AND APPROXIMATION QUALITY): Larger v enables finer rejection thresholds, improving alignment between verbal-score acceptance and ideal true-quality decisions.The approximation error in quality assessment decreases as 1/v, supporting more precise trajectory selection.
  • A.1.4. PROOF OF PROPOSITION 3.4 (SCORE GRANULARITY AND APPROXIMATION QUALITY): Very large vocabularies can exceed teacher calibration ability and reduce sample efficiency because each score level needs sufficient samples.The practical discussion gives v = 100 as an example of potentially excessive granularity.
  • A.1.5. ADDITIONAL THEORETICAL REMARKS: OVD verbal scores act as a proxy for the importance ratio π_T(y)/π_S(y) when black-box teachers expose no teacher probability π_T(y).The acceptance probability a(y) is interpreted as a soft importance weight.
  • A.1.5. ADDITIONAL THEORETICAL REMARKS: Under bounded rewards and Lipschitz policy gradients, OVD converges at O(1/T), with a better constant factor at higher acceptance rates.Here T denotes the number of iterations, and higher acceptance rates retain more on-policy samples.

A.2. RL Scaling Random Sample Example

The section illustrates RL scaling by repeatedly sampling and training on one randomly selected Omni-MATH problem instance, using example problems in combinatorics and analytic geometry.

  • RL Scaling Random Sample Example: RL scaling repeatedly samples and trains on a single randomly selected Omni-MATH problem instance, without requiring an extensive dataset.The example is intended to demonstrate the effectiveness of the approach under this single-instance training setup.
  • RL Scaling Random Sample Example: One sampled problem asks how many attempts preceded the successful guess of a three-digit lock code.The code is guessed correctly only on the last attempt after randomly selecting digits from a set of five.
  • RL Scaling Random Sample Example: Another sampled problem asks for the distance between the foci of a specified ellipse.The ellipse is given by 25x2 −100x + 4y2 + 8y + 16 = 0.

A.3. Reward Convergence Across Training Steps … B. Related Work

The appendices show how OVD’s reward convergence, verbal scoring, trajectory-based algorithm, implementation, hyperparameters, and related-work positioning support its training framework. Moderate rejection thresholds improve convergence, while discrete scores and task-specific trajectories enable efficient optimization across mathematical reasoning and Web Q&A.

  • A.3. Reward Convergence Across Training Steps: Moderate rejection thresholds (5-7) typically provide the best balance between faster convergence and higher final rewards across Qwen-2.5-7B and LLaMA-3.2-3B.Rejection thresholds significantly affect learning dynamics.
  • A.4. Verbal Scoring Prompt: OVD’s verbal-scoring prompt rates each reasoning step from 1-10 and outputs a single score token, avoiding token-level probability distributions.The rubric maps 1-3 to incorrect or irrelevant, 4-6 to partially correct, 7-9 to correct and well-reasoned, and 10 to optimal.
  • Verbal Scoring Prompt: The scoring rubric distinguishes incorrect, partially correct, well-reasoned, and optimal steps using ranges 1-3, 4-6, 7-9, and 10.The prompt instructs the teacher to output only one number from 1 to 10.
  • A.5. Complete Algorithm: OVD adapts trajectory generation by using sequential reasoning steps for mathematics and interleaved search queries, retrieved documents, and reasoning steps for Web Q&A.Algorithm 1 is presented as a unified framework for both task types.
  • A.5. Complete Algorithm: Each training iteration samples student trajectories, queries teacher scores, computes rewards and advantages, and updates the student with clipped policy gradients and importance ratios.The algorithm includes teacher-trajectory reward recomputation when applicable.
  • A.6.1. IMPLEMENTATION DETAILS: Experiments use 8 AMD Instinct MI210 accelerators with 64 GB HBM2e each, while simulated search environments run 7B models on one A100 and 14B models across two A100s.Distributed training uses 4 GPUs per training instance.
  • B. Related Work: The implementation details cover vanilla prompting, chain-of-thought, retrieval-augmented generation, advanced search baselines, and reinforcement-learning methods for preference optimization, reasoning, and agent training.The related-work section frames these methods as approaches for improving language models.
  • A.6.2. HYPERPARAMETERS: For mathematics, QwQ-32B scores trajectories on a 0-9 scale and Qwen2.5-Math-1.5B is the untuned student; Web Q&A uses simulated search agents with step-level verbal scores.Web students include Qwen2.5-3B-Base and LLaMA-3.2-3B-Base.

B.1. Post-training Methods for Language Models · B.2. Knowledge Distillation for Large Language Models

The paper situates OVD within post-training methods spanning preference optimization, reinforcement learning with environment interaction, and knowledge distillation. It contrasts token-level, sequence-level, and on-policy distillation according to their supervision signals and data-generation strategies.

  • B.1. Post-training Methods for Language Models: RLHF aligns language models with human preferences by training reward models from human feedback, later scaling from open-ended generation to instruction following.The passage identifies RLHF as a standard alignment approach and cites its progression from reward-model training to instruction-following settings.
  • B.1. Post-training Methods for Language Models: DPO directly optimizes the policy without explicit reward modeling, simplifying RLHF, while IPO and ORPO address overfitting and reference-model requirements.IPO provides a theoretical framework for DPO overfitting issues, whereas ORPO removes the need for a reference model through odds-ratio optimization.
  • B.1. Post-training Methods for Language Models: RL with environment interaction enhances reasoning and agentic capabilities, but effectively leveraging environment-derived supervision remains an open problem.Existing approaches include self-training from model-generated rationales and environment-based supervision, according to the provided passages.
  • B.2. Knowledge Distillation for Large Language Models: Knowledge distillation compresses large models into smaller ones and can be categorized by training paradigm and supervision signal.This framing motivates distinctions between token-level and sequence-level distillation methods.
  • B.2. Knowledge Distillation for Large Language Models: Token-level distillation supplies dense supervision at every decoding step, yielding richer gradients than sequence-level methods.Its objectives include forward KL for mode covering, reverse KL for mode seeking, and adaptive schemes based on token importance.
  • B.2. Knowledge Distillation for Large Language Models: Sequence-level distillation trains students from teacher-generated trajectories, providing global and structured signals through output imitation and related supervision forms.The passage lists imitation learning, symbolic supervision, and unified f-divergence formulations as extensions or alternatives.
  • B.2. Knowledge Distillation for Large Language Models: Off-policy distillation pre-generates teacher data but suffers distribution mismatch, whereas on-policy methods let students generate samples; GKD interpolates between both strategies.Black-box on-policy distillation is identified as a more recent extension of this direction.
Loading 2601.21968v1…