PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Language Feedback via Variational Policy Distillation

Yang Li, Erik Nijkamp, Semih Yavuz, Shafiq Joty

arXiv:2605.15113v2cs.LG

TL;DR

Language-feedback distillation is limited by passive teachers whose diagnostic abilities can plateau. VPD co-evolves teacher and student policies through variational EM, and consistently outperforms standard RLVR and self-distillation baselines across code-generation and scientific-reasoning benchmarks.

  • Problem

    Language-feedback methods rely on passive teachers whose unrefined diagnostic interpretations can impose a ceiling on the supervision available to improving students.

  • Method

    VPD jointly refines a feedback-aware teacher and student through EM, using adaptive trust-region teacher updates followed by token-level on-policy distillation.

  • Results

    VPD consistently outperforms standard RLVR and self-distillation baselines across competitive-programming and scientific-reasoning benchmarks using diverse diagnostic feedback sources.

  • Takeaways & Limitations

    Co-evolving the teacher and student extracts more value from diagnostic language feedback than passive distillation within the evaluated tasks.

  • Takeaways & Limitations

    VPD still lags pure sparse RL in mathematical reasoning and base-model cold-starts, while shared weights constrain the teacher’s representational capacity.

Abstract

from arXiv · show

Reinforcement learning from verifiable rewards (RLVR) suffers from sparse outcome signals, creating severe exploration bottlenecks on complex reasoning tasks. Recent on-policy self-distillation methods attempt to address this by utilizing language feedback to generate dense, token-level supervision. However, these approaches rely on a fixed, passive teacher to interpret the feedback. As the student policy improves, the teacher's zero-shot assessment capabilities plateau, ultimately halting further learning. To overcome this, we propose Variational Policy Distillation (VPD), a framework that formalizes learning from language feedback as a Variational Expectation-Maximization (EM) problem. VPD co-evolves both policies: in the E-step, the teacher is actively refined on trajectory outcomes via an adaptive trust-region update, translating textual feedback into a dynamically improved target token distribution. In the M-step, the student internalizes this dense distributional guidance on its own on-policy rollouts. By continuously improving the teacher's ability to extract actionable signals from textual critique, VPD overcomes the limitations of passive distillation. Evaluated across diverse sources of diagnostic feedback on scientific reasoning and code generation tasks, VPD consistently outperforms both standard RLVR and existing self-distillation baselines. Finally, by stress-testing our framework on rigid mathematical reasoning and cold-start regimes, we illuminate the fundamental bounds of feedback-driven self-distillation compared to pure environment-driven RL.

1 Introduction

The introduction identifies sparse RLVR outcomes and passive self-teachers as barriers to learning from language feedback. It presents VPD as a variational EM framework that co-optimizes teacher interpretation and student policy distillation.

  • Introduction: RLVR relies largely on sparse, binary outcome signals, creating a severe credit-assignment bottleneck for complex reasoning.
  • Introduction: Language feedback offers dense, localized supervision by explaining not only that an attempt failed, but also why and how to fix it.
  • Introduction: Passive self-teachers can become counterproductive when feedback is noisy or difficult to map into structural token adjustments, and their zero-shot advantage diminishes as students learn basic corrections.
  • Introduction: VPD frames language-feedback learning as variational inference, treating the feedback-conditioned teacher as an approximate posterior optimized alongside the student.
  • Introduction: The E-step refines the teacher using successful and failed trajectories, while the M-step distills its improved token-level guidance onto the student’s on-policy rollouts.
  • Introduction: VPD formalizes on-policy feedback learning as variational EM with unpaired preference optimization and an adaptive trust region for stable on-policy KL distillation.

2 Preliminaries

This section formalizes RLVR as contextual-bandit language generation optimized for verifier rewards while constrained by a reference policy, then introduces on-policy self-distillation as a dense-feedback alternative. It also identifies passive, zero-shot teacher interpretation as a fundamental limitation of existing self-distillation.

  • Reinforcement Learning from Verifiable Rewards: RLVR models language generation as a contextual bandit in which policy πθ produces responses and a verifier assigns scalar outcome reward r(x, y).The context is the user prompt x, and responses are generated autoregressively.
  • Reinforcement Learning from Verifiable Rewards: The standard RLVR objective maximizes expected reward while penalizing divergence from the supervised fine-tuned reference policy πref.The KL penalty coefficient β controls the strength of this constraint, and modern pipelines use GRPO and related variants.
  • Reinforcement Learning from Verifiable Rewards: Sparse outcome rewards create an exploration bottleneck when a model samples no correct answer for a prompt.The objective’s gradients depend on verifier-assigned scalar outcomes, which may provide little learning signal for complex reasoning tasks.
  • On-Policy Self-Distillation: On-policy self-distillation uses textual feedback C, such as compiler errors or LLM critique, to construct dense token-level supervision.SDPO conditions the model itself on feedback and treats the resulting policy as an on-policy self-teacher.
  • On-Policy Self-Distillation: SDPO aligns the unconditioned student policy with a feedback-conditioned copy by minimizing token-level KL divergence on the student’s own rollouts.Forward KL or JS divergence can be substituted depending on the desired training dynamics.
  • On-Policy Self-Distillation: Because stop-gradient prevents explicit teacher optimization, the self-teacher interprets feedback zero-shot using only its pre-existing diagnostic capacity.This leaves teacher refinement as a fundamental limitation of the approach.

3 Variational Policy Distillation

Variational Policy Distillation frames learning from language feedback as variational inference, enabling teacher–student co-evolution through alternating teacher refinement and student distillation. A dynamic student-anchored trust region keeps the teacher’s targets locally reachable for stable projection into the student’s unconditioned policy.

  • Variational Policy Distillation: VPD treats language-feedback learning as variational inference, allowing the teacher to co-evolve with the student instead of remaining a fixed heuristic.The teacher actively learns to extract deeper insights from textual feedback.
  • EM Optimization: The E-step refines the feedback-conditioned teacher toward the reward-tilted optimal policy, while the M-step distills its token-level distribution into the unconditioned student on on-policy rollouts.The teacher uses diagnostic feedback C, whereas the student must internalize the corrections without C at inference time.
  • E-Step Teacher Refinement: The E-step trains the teacher off-policy on the student’s diverse exploration trajectories, avoiding the sparse-reward bottleneck that standard on-policy reinforcement learning would reintroduce.The teacher objective has the mathematical form of a KL-regularized RL objective, but reuses trajectories already generated by the student.
  • Dynamic Reference Prior: The reference prior is dynamically anchored to the current student policy, and student likelihoods are frozen during each E-step to create an iterative trust-region update.This student-relative posterior is intended to prevent teacher–student distribution shift and stabilize optimization.
  • M-Step Student Distillation: Constraining the teacher within the student’s local trust region keeps its target distribution reachable, making M-step distillation stable and avoiding extreme gradient variance and mode collapse.The student minimizes token-level KL divergence to the updated teacher while the teacher parameters remain fixed.

4 Experiments

The experiments compare VPD with reinforcement-learning, distillation, and hybrid baselines across multiple model scales and reasoning domains. They also test VPD with environment execution, successful sibling rollouts, and LLM self-critique as diagnostic feedback sources.

  • Experimental Questions: The experiments investigate VPD’s performance relative to standard RL and self-distillation, its use of diagnostic feedback sources, and its limits in challenging regimes.These are the three primary experimental questions stated by the paper.
  • Models and Benchmarks: The evaluation uses Qwen3-1.7B, Qwen3-8B, and OLMo3-7B-Instruct across scientific, code-generation, and mathematical reasoning benchmarks.Scientific reasoning uses SciKnowEval; code generation uses LiveCodeBench; mathematical reasoning uses DAPO-Math, Math500, AIME24/25, and AMC23.
  • Baselines: VPD is benchmarked against pure RL with GRPO, pure distillation with SDPO, and single-phase hybrid mechanisms.The hybrids include Joint Loss, Advantage Reshaping, and Advantage ... as listed in the supplied passage.
  • Feedback Sources: VPD is evaluated with diagnostic feedback from verifiable environment execution, successful sibling rollouts, and self-critique by an LLM judge.The feedback sources are intended to test the versatility of the on-policy distillation framework.
  • Environment Feedback (LiveCodeBench): On LiveCodeBench, Qwen3-8B is evaluated without reasoning mode on the v6 subset under public and private unit-test settings.The environment provides runtime errors and failed unit-test assertions as deterministic, rich feedback.

2. Contrastive Sibling Rollouts (Sci-

VPD extends feedback-driven self-distillation with autonomous self-critique when sibling rollouts lack correct positive pairs, while exposing limits in cold-start and mathematical-reasoning settings. Its shared-weight design also improves computational efficiency, though performance depends on appropriate E-step scheduling and a dynamic reference prior.

  • Self-Critique via LLM Judge: Autonomous self-critique lets the model diagnose failed trajectories when all sibling rollouts are incorrect, enabling the E-step to refine the teacher without positive pairs.This setup mimics a self-reflection loop in which the model generates critiques for its own failed trajectories.
  • Limits of Self-Distillation: Self-distillation methods, including SDPO and VPD, struggle relative to pure RL in cold-start base-model and challenging mathematical-reasoning regimes.These findings are presented as empirical limits of language-driven self-distillation.
  • Cold Start Problem: SDPO performance on base models rapidly collapses to near zero, suggesting self-distillation requires rudimentary instruction-following competence to process diagnostic feedback.The limitation is attributed to the base model’s inability to properly digest feedback in its prompt.
  • Mathematical Reasoning: On DAPO-Math, SDPO suffers severe training collapse, while VPD delays collapse but pure GRPO remains dominant, achieving 83.8% on Math500 with Qwen3-8B.The passage hypothesizes that strict, non-forgiving mathematical derivations drive this vulnerability.
  • Computational Efficiency: VPD eliminates separate-teacher VRAM overhead through shared weights and adds zero sampling or environment-verification cost by reusing the same on-policy rollouts.The explicit E-step still introduces additional gradient computation.
  • E-Step Update Frequency: Both overly frequent and overly infrequent E-step updates degrade performance, motivating the main-experiment schedule of one E-step per five M-step updates.The ablation uses Qwen3-1.7B on SciKnowEval and denotes this schedule as F5.
  • Dynamic Reference Prior: Replacing VPD’s dynamic student-anchored reference prior with a fixed reference model causes severe performance degradation on Qwen3-1.7B.The dynamic prior constrains the teacher target distribution within a sliding trust region reachable by the student.

5 Related Work

Prior work spans verifiable-reward reinforcement learning, iterative self-improvement, on-policy self-distillation, and language-feedback supervision. VPD distinguishes itself by jointly optimizing a feedback-aware teacher and student within a trust-region-constrained EM framework using shared weights.

  • Reinforcement Learning with Verifiable Rewards: RLVR optimizes language models with deterministic, rule-based environments but suffers from sparse rewards.Examples include Python interpreters, formal theorem provers, and exact-match graders.
  • Iterative Self-Improvement and Preference Optimization: Self-improvement methods use model-generated rollouts through self-play, self-rewarding, or iterative preference optimization.Examples include SPIN, Self-Rewarding LLMs, and iterative DPO-based approaches.
  • Privileged Information and On-Policy Self-Distillation: On-policy self-distillation uses privileged information such as ground-truth answers, historical solution traces, and system-prompt behaviors to generate training targets.OPSD conditions a teacher on ground-truth answers, while OPCD distills historical traces and system-prompt behaviors into model weights.
  • VPD's Distinction: VPD embeds self-distillation in EM, actively optimizes the teacher to distinguish successful from failed trajectories, and constrains updates within a dynamic trust region.This differs from static prompts and ad-hoc off-policy sampling by producing mathematically bounded teacher targets.
  • Learning from Language Feedback: Language-feedback research treats natural language as dense supervision through feedback-information theory, value estimation, verbal reflections, and textual gradients.These mechanisms support feedback-based credit assignment and iterative response improvement.
  • VPD's Distinction: VPD uses a single shared-weight network (θ = ϕ) to combine feedback-aware teacher and student, embedding critiques into a mathematically bounded EM cycle without separate critique models.This design addresses the computational cost and heuristic fine-tuning used by some language-feedback approaches.

6 Conclusion · A Theoretical Derivations

The paper presents VPD as a co-evolutionary EM framework that jointly improves a feedback-aware teacher and student, with formal derivations supporting its alternating optimization. Despite delaying collapse and outperforming standard self-distillation, VPD remains behind sparse RL in mathematical reasoning and base-model cold starts.

  • 6 Conclusion: VPD reframes self-distillation as a co-evolutionary Expectation-Maximization algorithm.It jointly trains a feedback-aware teacher and an unconditioned student instead of relying on frozen heuristic oracles or static reference models.
  • 6 Conclusion: The E-step actively updates the teacher through unpaired preference learning to extract sharper insights from environmental critiques.The passage identifies BCO as the teacher-update method.
  • 6 Conclusion: VPD significantly improves upon standard self-distillation and delays training collapse in challenging scenarios.The conclusion nevertheless characterizes these gains as insufficient to surpass pure sparse reinforcement learning in some settings.
  • 6 Conclusion: VPD ultimately lags behind pure sparse RL, such as GRPO, in mathematical reasoning and base-model cold-start regimes.This limitation is attributed to reliance on unstructured textual feedback, which can be noisy or imprecise for strict logical reasoning.
  • A Theoretical Derivations: The theoretical appendix derives the closed-form optimal policy for the KL-regularized RLVR objective.It also demonstrates equivalence between this objective and minimizing reverse KL divergence.
  • A Theoretical Derivations: The appendix establishes the Expectation-Maximization lower bound underlying alternating optimization of the teacher q_ϕ and student π_θ.These derivations formally support the variational framework introduced in Section 3.

A.1 Derivation of the Optimal Target Distribution … A.4 Theoretical Implications of the Dynamic Reference Model

The appendix derives the exponentially reward-tilted optimal target for KL-regularized RLVR, proves reverse-KL minimization is equivalent to the original objective, and formulates VPD as variational teacher refinement followed by student distillation. With a dynamic student reference, the teacher gains an adaptive alignment bonus and operates within a sliding trust region while remaining stationary within each E-step.

  • A.1 Derivation of the Optimal Target Distribution: The KL-regularized RLVR objective combines expected reward with a KL penalty against the reference policy.The objective is defined for context x as expected reward minus β times D_KL(π(·|x) ∥ π_ref(·|x)).
  • A.1 Derivation of the Optimal Target Distribution: The constrained optimization yields an exponentially reward-tilted policy as the optimal target distribution.The probability normalization introduces a partition function Z(x), producing the normalized target.
  • A.2 Equivalence of Reverse KL and the RLVR Objective: Minimizing reverse KL divergence between the student and optimal policy exactly maximizes the original RLVR objective.The β log Z(x) term is constant with respect to the student parameters θ.
  • A.3 Variational Expectation-Maximization Lower Bound: Because the partition function is analytically intractable, VPD introduces a feedback-conditioned teacher distribution as an approximate posterior.The teacher distribution q_ϕ(y|x,C) conditions on dense language feedback C.
  • A.3 Variational Expectation-Maximization Lower Bound: The ELBO lower-bounds the intractable objective, and maximizing it is equivalent to minimizing the teacher’s divergence from the optimal policy.The maximum objective value is β log Z(x), while the E-step optimizes teacher parameters using Binary Classifier Optimization on implicit rewards.
  • A.3 Variational Expectation-Maximization Lower Bound: In the M-step, the student minimizes divergence to the refined teacher on on-policy rollouts, avoiding privileged feedback at inference.Finite teacher capacity leaves residual error, so the optimized teacher serves as a tractable surrogate for the optimal policy.
  • A.4 Theoretical Implications of the Dynamic Reference Model: Using the current student as reference transforms the teacher objective by adding an adaptive alignment bonus favoring trajectories the student rates above the base model.The extra term is −E_qϕ[log(πθ/πref)].
  • A.4 Theoretical Implications of the Dynamic Reference Model: The dynamic reference acts as a sliding trust region: teacher updates pursue diagnostic feedback but are penalized for departing from the student’s active distribution.Student likelihoods are frozen before each E-step, making the target stationary within that step while it shifts across training.

B Method … C.3 Self-Critique via LLM Judge (SciKnowEval)

VPD formulates language-feedback learning as a co-evolutionary EM procedure that refines a teacher and distills its guidance into an on-policy student. The paper establishes monotonic progress under idealized updates, compares hybrid alternatives, and evaluates feedback mechanisms across code and scientific reasoning tasks.

  • B.1 VPD Algorithm: VPD alternates on-policy rollouts, verifier-based success/failure partitioning, teacher refinement in the E-step, and stop-gradient student distillation in the M-step.The E-step refines the teacher using trajectory outcomes and diagnostic feedback, while the M-step updates the student against the refined teacher distribution.
  • B.2 Convergence Properties of the VPD EM Procedure: Under globally optimal E- and M-steps, the RLVR objective is non-decreasing across iterations: J(θk+1) ≥ J(θk).The proof attributes improvement to successive reductions in the teacher- and student-side divergences.
  • B.2 Convergence Properties of the VPD EM Procedure: The resulting objective sequence converges because it is monotonically non-decreasing and upper-bounded by β log Z(x).For approximate updates, generalized EM preserves monotonic improvement when each step does not decrease its sub-objective, such as with sufficiently small learning rates.
  • B.3 Hybrid Distillation and RL: Hybrid baselines combine dense SDPO distillation with verifier-driven GRPO either through weighted losses, fused advantages, or distillation-guided advantage reweighting.The reweighting variant uses clipped exponential token weights and a decaying α to transition from teacher guidance toward pure verifier-driven GRPO.
  • C.1 Environment Feedback (LiveCodeBench): LiveCodeBench v6 evaluates generalization by returning compiler errors or assertion failures as rollout feedback while reserving final performance evaluation for the benchmark protocol.Experiments use Qwen3-8B with “thinking mode” disabled and follow the SDPO evaluation protocol.
  • C.2 Contrastive Sibling Rollouts as Feedback (SciKnowEval): SciKnowEval tests Biology, Chemistry, Materials Science, and Physics using a sparse exact-match verifier that returns only binary final-answer correctness.Because the environment lacks textual critiques, successful sibling rollouts from the same prompt provide teacher-conditioning feedback when available.
  • C.3 Self-Critique via LLM Judge (SciKnowEval): The self-critique setting addresses prompts whose N rollouts all fail by having the model generate diagnostic critiques for failed trajectories.The critique compares the failed attempt with a reference containing only the correct multiple-choice letter, requiring the model to infer the reasoning gap.
  • C.3 Self-Critique via LLM Judge (SciKnowEval): Generated critiques become E-step feedback, after which teacher conditioning and the subsequent E- and M-step optimizations proceed as in the contrastive setting.The self-critique configuration retains N = 8 rollouts and F = 5 update frequency.

C.4 Base Model “Cold Start”

The cold-start experiment directly optimizes Qwen3-4B-Base on SciKnowEval using on-policy generation, exposing severe limitations in processing textual critique. Pure RL elicits reasoning, while SDPO collapses rapidly and VPD preserves a functional learning trajectory by actively optimizing the teacher and constraining its trust region.

  • Motivation and Setup: The experiment directly optimizes Qwen3-4B-Base with on-policy generation on the SciKnowEval benchmark.The setup uses a pure “cold start” scenario to examine reliance on pre-existing instruction-following capabilities.
  • Prompting and Formatting: Because the base model lacks instruction-following alignment, it struggles to parse and integrate textual critique into subsequent reasoning.This limitation causes the standard self-distillation teacher’s target distribution to rapidly degrade.
  • Hyperparameters and Stability: 0% pass rate: standard SDPO immediately collapses within the first few steps, whereas pure RL successfully elicits reasoning through sparse reward maximization.All hyperparameters match those detailed in Table C.2.
  • Hyperparameters and Stability: VPD’s E-step prevents rapid collapse by actively optimizing the teacher policy and constraining the trust region, maintaining a functional learning trajectory substantially longer.Additional curves show VPD delaying training degradation across Biology, Chemistry, and Physics, while SDPO collapses almost immediately.

C.5 Mathematical Reasoning Experimental Setup

The mathematical reasoning experiments train on DAPO-Math and evaluate on Math500 with binary symbolic exact-match rewards. They use contrastive sibling feedback and test VPD under optimization conditions where flawed intermediate targets can destabilize self-distillation.

  • Dataset and Evaluation Protocol: Training uses DAPO-Math, while evaluation uses Math500 with a symbolic exact-match grader assigning binary reward to boxed final answers.The environment parses the model’s final answer enclosed in a boxed tag.
  • Feedback Mechanism for Mathematics: The contrastive sibling mechanism gives the teacher a successful sibling trajectory to diagnose algebraic or conceptual errors in incorrect derivations.This mechanism generates diagnostic feedback C for mathematical reasoning.
  • Optimization Dynamics: 83.8% accuracy is achieved by pure sparse RL (GRPO) when fully correct derivations are strictly rewarded.Mathematical reasoning is described as uniquely unforgiving because subtle arithmetic hallucinations in intermediate targets can poison the student policy.
  • Optimization Dynamics: Standard SDPO suffers severe training collapse when it distills flawed linguistic explanations, whereas VPD uses N = 8 trajectories per prompt and asymmetric update frequency F = 5.Detailed DAPO-Math training hyperparameters are provided in Table C.4.
  • Additional Training Curves: Additional training curves cover AIME24, AIME25, and AMC23, where VPD delays SDPO’s severe collapse but GRPO remains the most robust optimization method.These curves are reported as consistent with the Math500 findings.

C.6 Ablations

Ablations show that VPD benefits from an asymmetric E-step schedule and a dynamic reference prior. F = 5 balances teacher adaptation with student convergence, while a fixed prior causes severe instability from distribution shift.

  • E-Step Update Frequency: F = 1 prevents adequate student convergence because the teacher distribution shifts rapidly during synchronous updates.The synchronous schedule also increases temporal runtime overhead.
  • E-Step Update Frequency: F = 10 yields stale guidance because the teacher’s diagnostic capabilities stagnate relative to the student’s learning pace.This guidance fails to appropriately address the student’s newly emerging logic.
  • Dynamic vs. Fixed Reference Prior: The Fixed Prior ablation anchors the teacher’s E-step BCO objective to the frozen Qwen3-1.7B initialization, unlike the Dynamic Prior.The Dynamic Prior continually updates its reference to match the active student policy at each expectation-maximization cycle.
  • Dynamic vs. Fixed Reference Prior: Fixed Prior training exhibits severe instability because student distribution drift pulls the teacher’s optimized targets backward toward outdated base behavior.The student evolves through M-steps and discovers novel successful reasoning paths, increasing its divergence from the initial base model.
Loading 2605.15113v2…