PaperlayerView exact revision on arXiv

Source-linked AI summary

Inductive Feedback for Mixed-Policy Distillation

Amir Moeini, Huaijiang Zhu, Daniel Havir, Shangtong Zhang

arXiv:2609.35390v1cs.LG

TL;DR

Verbal feedback can provide step-specific supervision, but feedback-conditioned on-policy distillation may transfer unrelated teacher preferences and leave corrective guidance unused. The paper constructs a student-anchored target from feedback-induced token evidence and a symmetric shared-rollout objective; it reports gains over standard and contrastive baselines across knowledge and agentic benchmarks.

  • Problem

    Feedback-conditioned on-policy distillation can transfer teacher preferences unrelated to feedback and leave prescriptive guidance unused when student rollouts diverge from suggested corrections.

    Source: §1 · pp. 2–3

  • Method

    The method scores feedback-induced changes in teacher predictions as evidence about next-token hypotheses, reweights the student distribution within a trust region, and trains with a symmetric shared-rollout estimator.

    Source: §1 · p. 3, §3 · p. 4

  • Results

    The method improves over SDPO and W2S-OPD across trivia, embodied, scientific, and tool-use benchmarks, with the highest mean score across all four reported benchmarks.

    Source: §1 · p. 3, §5.2 · p. 8

  • Takeaways & Limitations

    The results show that verbal feedback can support learning across knowledge and agentic benchmarks through student-anchored targets and shared supervision along student and corrected-policy rollouts.

    Source: §1 · p. 3

    Gains are less clearly separated on biology, materials science, and tool use because reported confidence intervals overlap in those comparisons.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Verbal feedback can identify errors and prescribe corrections, providing rich supervision for language-model post-training even when reliable programmatic verifiers are unavailable. Such feedback, often generated by a capable model, can be used to condition the teacher in on-policy distillation, which trains the student to match the teacher's predictions on student-generated rollouts. However, this approach can transfer teacher preferences that the feedback did not motivate, while leaving much of the feedback's guidance unused. We find that both problems come from the standard on-policy distillation objective, specifically the divergence it minimizes and the distribution it uses as its target. Our proposed method addresses both limitations. First, to isolate the information conveyed by the feedback from the teacher's inherent preferences, we treat verbal feedback as evidence for or against the hypothesis that a particular token comes next at a given prefix. We then adopt a probabilistic confirmation framework which uniquely determines an ordering over the vocabulary based on the teacher's predictions before and after it receives feedback. Using a confirmation score consistent with this ordering, we construct a target distribution within a trust region of the student. Second, to learn from guidance that student rollouts can leave unused, we derive a simple shared-rollout estimator of a symmetric divergence between the student and target distributions over rollouts, reusing student and feedback-conditioned teacher rollouts in both directions through importance weighting. Empirical evaluations show that our method outperforms the common on-policy distillation recipe and a recent contrastive variant on knowledge-based and agentic benchmarks.

1 INTRODUCTION

Verbal feedback can provide step-specific, explanatory supervision, including when reliable programmatic verifiers are unavailable. The paper modifies feedback-conditioned on-policy distillation to isolate feedback-induced guidance and learn from corrections that student rollouts may miss.

1 INTRODUCTION

  • Verbal feedback links parts of a rollout to task criteria, explaining where credit belongs and why, beyond what a single scalar reward provides.It is presented as a practical supervision source when reliable programmatic verifiers are unavailable.Source: §1 · p. 2
  • Feedback-conditioned OPD can transfer teacher preferences unrelated to the guidance and leave prescriptive feedback unused when student rollouts diverge from suggested corrections.The teacher interprets the judge’s feedback to provide token-level supervision; the judge can spend additional test-time compute reasoning about the rollout and criteria.Source: §1 · p. 2

A distillation target anchored to the student

  • The corrected target scores feedback-induced prediction changes as evidence about next-token hypotheses, then reweights the student’s distribution within a trust region.It equals the student’s distribution when feedback leaves the teacher’s predictions unchanged and bounds token reweighting.Source: §1 · p. 3

A symmetric objective with shared rollouts

  • A symmetric shared-rollout estimator reuses student and feedback-conditioned teacher rollouts in both directions to supervise student behavior and suggested corrections.Under the analyzed student–target mixture, pooling prevents arbitrarily large importance weights and bounds gradient variance.Source: §1 · p. 3

Empirical gains on knowledge and agentic benchmarks

  • The method improves over SDPO and W2S-OPD on trivia, embodied, scientific, and tool-use benchmarks.The authors evaluate the method against SDPO, which follows standard feedback-conditioned OPD, and the recent contrastive variant W2S-OPD.Source: §1 · p. 3

2 PRELIMINARIES

The preliminaries set up token-level credit assignment and explain how teacher predictions can supervise student-generated rollouts. They contrast that signal with scalar-reward updates and describe how privileged feedback can condition the teacher.

Policy gradients and token-level credit

  • Policy-gradient methods optimize expected rollout reward, using an advantage to compare a chosen token with alternatives at its prefix.The advantage estimates the return after the token relative to a baseline independent of that token.Source: §2 · p. 3
  • With one scalar reward per rollout, GRPO applies a group-normalized reward to every token, without distinguishing which choices caused the outcome.Such updates identify better-performing rollouts but not the responsible token choices.Source: §2 · p. 3

On-policy distillation

  • On-policy distillation samples student rollouts and pulls the student’s next-token distribution toward the teacher’s at visited prefixes.The reverse-KL update gives a token-level advantage from teacher–student log-probability differences, whose magnitude has no uniform bound.Source: §2 · p. 4

Distillation from verbal feedback

  • A teacher conditioned on a reference solution, execution feedback, or verbal feedback can provide token-level supervision using context unavailable to the student at inference.In the verbal-feedback setting, a judge evaluates a student rollout and returns text; the teacher uses it to produce a next-token target, without requiring judge token probabilities or a shared vocabulary.Source: §2 · p. 4

3 FEEDBACK AS EVIDENCE: A CORRECTED DISTILLATION TARGET

This section treats feedback as evidence about whether a token comes next and uses the teacher’s predictions before and after feedback to construct a student-anchored target. A relative-distance confirmation score determines the token ordering, while a trust region controls how the student distribution changes.

Feedback as evidence

  • The teacher’s predictions with and without feedback serve as posterior and prior probabilities for the hypothesis that token v comes next; the confirmation score measures support for or against it.Holding the prompt, student rollout, and prefix fixed isolates the change attributable to feedback.Source: §3 · p. 4

Choosing the confirmation score

  • The partial-entailment framework requires probability dependence, monotonicity, ratio dependence under disconfirmation, and complementarity; the theorem fixes the score’s ordering up to a strictly increasing transformation.Complementarity requires tokens and their complements to rank in reverse order.Source: §3 · pp. 4–5
  • The selected relative-distance score measures probability movement toward zero or one against the remaining distance, yielding a signed score in [−1, 1] that is zero when feedback changes nothing.For increases it divides the gain by 1 − Pprior(v); for decreases it divides the signed change by Pprior(v).Source: §3 · p. 5
  • Relative distance ranks confirmation consistently with token complements, whereas log-probability difference can reverse the student’s preference for the correct token.In Figure 1’s synthetic task, feedback raises the teacher’s probability of correct token c from 95% to 98% and wrong token n from 1% to 1.2%; the log-probability difference scores n higher because its proportional increase is larger.Source: §3 · p. 5, Figure 1 · p. 5

Constructing the target

  • The corrected target maximizes expected confirmation score while penalizing KL divergence from the current student distribution, producing a reweighted student distribution.Positive scores upweight tokens, negative scores downweight them, zero leaves their probabilities unchanged, and full confirmation or refutation gives weights e^β or e^−β before normalization.Source: §3 · pp. 5–6

Properties of the target

  • When feedback leaves the teacher’s distribution unchanged, the target equals the student distribution and contributes no local distillation gradient; bounded scores also constrain each target probability’s multiplicative change.Under reverse-KL OPD, the corrected target yields a bounded token-level advantage.Source: §3 · p. 6

4 A SHARED OBJECTIVE FOR STUDENT AND TEACHER ROLLOUTS

This section combines supervision at student-visited prefixes with supervision along corrected-policy rollouts. A shared mixture of student and corrected-policy rollouts estimates both directions of a symmetric divergence using importance weighting.

A rollout-level objective

  • The Jeffreys divergence combines reverse-KL supervision at student-visited prefixes with forward-KL supervision at corrected-policy prefixes.The forward term shapes student predictions along rollouts it cannot yet reliably generate.Source: §4 · pp. 6–7

One pool for both directions

  • Each rollout is sampled wholly from the student with probability λ or the corrected policy with probability 1 − λ, forming a mixture used to estimate both terms.Importance weights recover the prefix distribution required by each term.Source: §4 · p. 7
  • Because the sampling mixture contains both policies, its importance ratios are bounded, avoiding arbitrarily large ratios from direct policy-to-policy importance weighting.Training uses a semi-gradient, holding the target, sampled prefixes, and importance ratios fixed while differentiating token losses.Source: §4 · p. 7
  • Shared rollouts give each term strictly larger asymptotic effective sample size than separate pools with the same allocation, unless prefix distributions have disjoint support.Bounded importance ratios also yield a variance bound for the estimated semi-gradient.Source: §4 · p. 7

5 RESULTS

The experiments compare feedback-distillation methods on knowledge and agentic benchmarks, then examine rollout allocation and target reweighting. The method leads the benchmark comparisons, while ablations show that transfer outcomes vary with the training choices.

5.1 EXPERIMENTAL SETUP

  • Training evaluates trivia-fact acquisition and held-out embodied, tool-use, and scientific-reasoning tasks, using shared judge feedback and matched training settings across methods.The setup reports avg@16 for ToolAlpaca and SciKnowEval, with three checkpoint evaluations and t-based 95% confidence intervals.Source: §5.1 · pp. 7–8

5.2 MAIN RESULTS

  • 98.3% accuracy on Trivia Fantasy, versus 61.0% for SDPO and 41.7% for W2S-OPD, accompanies the method’s highest mean score across all benchmarks and SciKnowEval subjects.Source: Table 1 · p. 8, §5.2 · p. 8
  • On held-out tasks, EmbodiedEval pass rate reaches 82.1% versus SDPO’s 73.7%, while biology, materials science, and tool-use gains over the stronger baseline are less clearly separated.The reported gains are 0.7, 0.5, and 2.2 percentage points, respectively; confidence intervals overlap in these comparisons.Source: §5.2 · p. 8

5.3 ABLATIONS

  • With one quarter of the rollout pool from the teacher, ToolAlpaca reaches 55.9%, while every SciKnowEval subject stays within 2.1 points of the untrained student.More teacher rollouts lower all four transfer-subject scores without further ToolAlpaca gain; across the sweep, task scores vary less than biology and chemistry transfer scores.Source: §5.3 · p. 9, Figure 2 · p. 9
  • At the same 100-step budget, increasing β from 1 to 10 raises ToolAlpaca accuracy from 51.1% to 57.7%, while transfer effects differ by subject.Chemistry improves by 3.7 points, physics changes little, and biology and materials decline at the upper end; transfer stays below the untrained student throughout.Source: Figure 3 · p. 9, §5.3 · p. 10

A RELATED WORK

  • Related work distinguishes this approach by changing only feedback while holding question and response fixed, reweighting student predictions with confirmation scores, and sharing rollouts across both KL directions.Source: A RELATED WORK · p. 13

B PROOFS AND DERIVATIONS

The appendix formalizes the confirmation-score conditions and derives properties of the corrected target. It shows how the target stays anchored to the student when feedback does not change the teacher’s predictions, and relates its token-level losses to score-based updates.

B.1 CONFIRMATION AXIOMS AND THE RELATIVE-DISTANCE SCORE

  • Under A0–A3, any confirmation score is a strictly increasing transformation of relative distance, fixing the score ordering.The appendix also shows that ratio dependence follows from A0 and A2.Source: §B.1 · pp. 13–14
  • Relative distance reverses rankings for complementary token hypotheses, whereas the log-ratio can rank both a token and its complement in the same order.For the example, log(0.02/0.01) > log(0.60/0.40), and the complements retain that ordering; relative distance reverses it.Source: §B.1 · p. 14

B.2 PROPERTIES OF THE CORRECTED TARGET

  • When feedback leaves every teacher prediction unchanged, the corrected target equals the held-fixed student distribution; relative-distance scores also bound token reweighting by e^-β and e^β.At that unchanged target, both token KL terms, including their skew versions, have zero local gradient when the target is held fixed.Source: §B.2 · p. 14
  • At the evaluation point, the reverse token-KL update follows the expected confirmation score, while the forward-term advantage increases with the score and remains bounded.The reverse-term per-token advantage is β(S(k) − E¯q[S]) ∈ [−2β, 2β].Source: §B.2 · p. 14

B.3 POOLED ESTIMATOR AND GRADIENT VARIANCE

The appendix analyzes shared student–target rollout sampling for estimating both divergence directions, including its effective-sample-size and gradient-variance properties. It also bounds token-level skew divergences and relates the skew objective to the symmetric divergence as α approaches zero.

B.3 POOLED ESTIMATOR AND GRADIENT VARIANCE

  • Sampling rollouts from the student–target mixture and importance-weighting each direction recovers the prefix distributions needed for both divergence terms.The analysis assumes a fixed context and finite horizon; the KL and gradient results additionally require common rollout support and finite token gradients.Source: §B.3 · pp. 14–15
  • The pooled gradient’s second moment is bounded by a mixture-weighted sum of the component-gradient second moments, with factors depending only on mixture proportions.At λ = 1/2, G/2 is a convex combination of the two component gradients.Source: §B.3 · p. 15
  • The rollout-update variance bound allows arbitrary within-rollout position correlations, and fixed source counts remove the mixture estimator’s nonnegative between-source variance.The same bound holds for independent rollouts with source counts proportional to λ and 1 − λ.Source: §B.3 · p. 15
  • Whenever student and target prefix distributions overlap, shared sampling strictly improves both asymptotic ESS fractions over separate pools.For λ = 1/2, each shared-pool fraction is 1/(2 − κ).Source: §B.3 · pp. 15–16

B.4 TOKEN-LEVEL SKEW DIVERGENCE

  • The two rollout-level skew-divergence terms have a sum bounded by 2 log(1/α), independently of horizon; their token-level objective approaches the symmetric divergence as α → 0.For finite vocabulary and common support, 0 ≤ Lα ≤ (1 − α) J(q, p) and lim α→0 Lα = J(q, p).Source: §B.4 · p. 16

C SYNTHETIC PREFERENCE-REVERSAL EXPERIMENT

The synthetic experiment tests whether confirmation scores preserve the correct token’s preference when teacher probabilities change. Log-probability differences can reverse that preference, while relative-distance scores keep the correct token ranked higher.

C SYNTHETIC PREFERENCE-REVERSAL EXPERIMENT

  • On high-prior examples, the log-probability difference shifts relative preference toward the wrong token.Examples prescribe teacher priors and posteriors for correct, wrong, and third tokens; log-probability difference yields 0.031 < 0.182, while relative distance yields 0.600 > 0.002.Source: C SYNTHETIC PREFERENCE-REVERSAL… · pp. 16–17
  • At η ≈0.596, exact updates with log-probability differences reverse the correct token’s preference, while relative-distance scores keep its log-odds positive.The shared token distribution starts uniform and undergoes K = 20 updates.Source: C SYNTHETIC PREFERENCE-REVERSAL… · p. 17
  • The neural-policy test uses 1,000 fixed inputs, a 32–128–3 MLP, five paired seeds, and reports mean log-odds and three-token argmax accuracy.High-prior sets are nested across η; curves show means ± standard errors on the fixed training set.Source: C SYNTHETIC PREFERENCE-REVERSAL… · p. 17

D PRACTICAL IMPLEMENTATION

The implementation bounds token-level divergence, reuses student and feedback-conditioned teacher rollouts, and approximates teacher-derived scores over a top-k vocabulary union. Forward-weight clipping was limited in the reported EmbodiedEval runs.

D PRACTICAL IMPLEMENTATION

  • At α = 0.01 per prefix, each skewed token divergence is capped at log(1/α), and the ideal loss is bounded above by (1 −α)J(q, pcorr).As α →0, the ideal loss recovers Jeffreys divergence; fixing α per token avoids a full-rollout loss cap and diminishing conditional losses.Source: D PRACTICAL IMPLEMENTATION · p. 17
  • The estimator reuses student rollouts and samples feedback-conditioned teacher rollouts, using the latter as proposals for the forward term targeting pcorr.This permits batched scoring over the pool rather than scoring at every decoding step; the feedback was written about the student rollout.Source: D PRACTICAL IMPLEMENTATION · p. 17
  • The implemented reverse weight q(u)/mraw(u) is bounded by 1/λ, while the forward weight is clipped at cIS because πF is only a proposal for pcorr.With λ = 1/2 and cIS = 2, clipping affected about 1% of forward weights on average in three runs and at most 4.2% in an update.Source: D PRACTICAL IMPLEMENTATION · p. 17
  • Teacher-derived scores are computed on the union of both branches’ top-k tokens, while probabilities outside that union retain normalized anchor mass.Missing branch probabilities receive a common positive floor, and scores are zero outside the union; the remaining vocabulary contribution is evaluated analytically.Source: D PRACTICAL IMPLEMENTATION · pp. 17–18
  • Gradients flow through explicit student-policy terms, while the corrected target, normalizer, sampled prefixes, and importance weights remain fixed.Training therefore uses a semi-gradient of the token losses.Source: D PRACTICAL IMPLEMENTATION · p. 18
  • Table 2 compares skewed and unskewed objectives for perturbed neural autoregressive policies using relative value gaps at α = 0.01.Entries are Monte Carlo estimates with 4,096 rollouts per policy, shared prefixes, and exact vocabulary sums.Source: Table 2 · p. 18, D PRACTICAL IMPLEMENTATION · p. 18

E FORWARD-WEIGHT CLIPPING ON EMBODIEDEVAL

  • Forward importance-weight clipping averaged 1.11%, 0.96%, and 1.02% across three EmbodiedEval runs, with a maximum recorded rate of 4.19%.Runs used raw-teacher rollouts with λ = 1/2 and cIS = 2.Source: E FORWARD-WEIGHT CLIPPING ON… · p. 18

F.1 BENCHMARKS

The benchmarks cover fictional-fact recall, embodied robot planning, tool-call prediction, and scientific reasoning. On EmbodiedEval, most of the teacher-rollout score increase with feedback comes from enabling judge reasoning.

F.1 BENCHMARKS

  • Trivia Fantasy tests recall of 20 fictional facts through 10-option questions, with all task information supplied through judge feedback and evaluation against an answer key.The student may assign near-zero probability to the correct option and commit incorrectly before the teacher can correct it.Source: §F.1 · p. 18
  • EmbodiedEval tests a vision-language robot planner on three-turn tool-call conversations, using 12 held-out suites of 20 tasks and suite-level pass rates.Its behavioral criteria have no programmatic verifiers; training scenarios come equally from pools of 1,751 and 2,014 scenarios.Source: §F.1 · p. 18
  • ToolAlpaca evaluates API-call prediction on 68 held-out requests involving tools absent from training, while SciKnowEval covers Level-3 reasoning across four scientific domains.SciKnowEval uses multiple-choice questions and scores the extracted final option.Source: §F.1 · pp. 18–19
  • Most of the EmbodiedEval teacher-rollout score increase with feedback comes from enabling judge reasoning, with smaller gains at higher effort.Figure A.1 reports mean score differences over 10 repetitions with 95% confidence intervals.Source: §F.1 · p. 18, Figure A.1 · p. 19

F.2 TRAINING AND IMPLEMENTATION

The training setup standardizes shared optimization and sampling settings while retaining method-specific configurations. Baseline implementations differ in how feedback enters teacher context and how targets are constructed.

F.2 TRAINING AND IMPLEMENTATION

  • Tables 3 and 4 distinguish settings shared across methods and settings specific to each method.Training uses verl with Megatron-LM, and vLLM for generation and teacher scoring; rollout generation runs asynchronously with training.Source: §F.2 · p. 19, Table 4 · p. 20
  • On SciKnowEval and ToolAlpaca, prompts are limited to 2,048 tokens and model thinking mode is disabled, following SDPO.When no correction is needed, the judge’s fixed acknowledgement is omitted from the teacher context.Source: §F.2 · p. 19
  • SDPO gives the teacher feedback in addition to task context and sampled prefix, whereas W2S-OPD reweights frozen initial-student probabilities on a top-20 intersection.For W2S-OPD, the intersection includes both teacher branches and the anchor’s top-20 lists, each augmented with the sampled token.Source: §F.2 · p. 19
  • All methods use Adam with weight decay 0.01, gradient-norm clipping at 1, temperature 1 and top-p 0.95 sampling, and Gemini 3.7 Flash as feedback judge.Rollouts become stale when their generating policy is more than four weight synchronizations old.Source: Table 3 · p. 20

Related papers

Loading 2609.35390v1…