PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning

Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao, Yuezhi Zhou

arXiv:2608.19181v1cs.LGcs.AIcs.CL

TL;DR

Long-context OPD can favor locally plausible responses when teacher support disagrees with response-level task verification. GC-OPD calibrates these signals using group-normalized residuals and token allocation, achieving the highest average performance across five benchmarks, including 39.31 to 40.47 for Qwen3-4B over vanilla OPD.

  • Problem

    In long-context tasks, trajectory-level teacher support becomes progressively less aligned with verifier rewards as inputs lengthen.

  • Method

    GC-OPD forms a signed residual between group-normalized verifier rewards and OPD scores, then distributes it across tokens using relative OPD advantages while retaining dense guidance.

  • Results

    Across five benchmarks, GC-OPD achieves the highest average, raising Qwen3-4B from 39.31 to 40.47 over vanilla OPD.

  • Takeaways & Limitations

    GC-OPD integrates response-level verification with dense token-level guidance, with residual calibration and RACA contributing to its aggregate gains.

  • Takeaways & Limitations

    When either signal has near-zero within-group variance, GC-OPD sets the residual to zero and reduces to vanilla OPD for that group.

Abstract

from arXiv · show

On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Vanilla OPD reaches 39.31 and 43.56 under the same setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance. Code is available at https://github.com/SolereZhang/GC-OPD.

1 Introduction

Long-context tasks expose a mismatch between dense token-level OPD guidance and response-level verifier outcomes, with alignment worsening as inputs grow longer. GC-OPD addresses this mismatch by forming a signed group-normalized residual and distributing it across tokens while retaining the original OPD advantage.

  • Motivation: On-policy distillation gives dense token-level teacher guidance on student-generated responses through teacher–student log-probability advantages.This dense signal supports recent OPD methods.
  • Motivation: Long-context responses must integrate distant evidence and satisfy global constraints, yet may remain locally plausible while omitting required information.Students can struggle to locate and use relevant information as input position or lexical form changes.
  • Method: GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group, then uses their difference as a signed teacher–verifier disagreement residual.Subtracting the group-normalized OPD assessment focuses calibration on the discrepancy between verifier and teacher assessments.
  • Motivation: Trajectory-level OPD scores become progressively less aligned with verifier rewards over longer inputs in two distributed-evidence tasks.The pattern is measured by pairwise disagreement rate and OPD preference gap in Multi-Table Extraction and High-Recall Retrieval.
  • Method: RACA distributes the residual across response tokens according to relative OPD advantages while retaining the original OPD advantage.This combines response-level verifier information with token-dependent calibration.

2 Related Work

Related work spans long-context reasoning and verifiable post-training, teacher-centric on-policy distillation, verifier-aware distillation, and methods for trajectory-level feedback with token-level credit. GC-OPD is positioned as requiring neither step labels nor auxiliary continuations for finer-grained token allocation.

  • Long-context reasoning and verifiable post-training: Long-context benchmarks reveal that nominal context capacity does not ensure reliable retrieval, evidence integration, or reasoning across long inputs.Post-training approaches address these capabilities through long-instruction data, length-aware optimization, reinforcement-learning curricula, and task-specific feedback.
  • Teacher-centric on-policy distillation: MiniLLM and GKD distill student-generated sequences, while ExOPD, FiRe-OPD, and PowerOPD modify teacher-direction extrapolation, trajectory filtering, token reweighting, or sampled-token rewards.Related analyses also study teacher–student compatibility and whether large token-level disagreement is learnable.
  • Verifier-aware on-policy distillation: Verifier-aware distillation methods connect verifier outcomes to teacher supervision through response routing, peer-response conditioning, outcome-class margins, or token-update gating.SCOPE, MOPD, Uni-OPD, and SG-OPD exemplify these interfaces.
  • Trajectory-level feedback and token-level credit: A terminal verifier provides response-level reward without specifying tokenwise influence, motivating additional labels, auxiliary continuations, or alternative credit-assignment mechanisms.GC-OPD requires neither step labels nor auxiliary continuations and uses RACA to allocate trajectory-level feedback across tokens.

3 Background and Motivation

Vanilla OPD provides dense token-level teacher-preference guidance, whereas verifiers measure response-level task outcomes, so their rankings can disagree. On two long-context evidence-aggregation tasks, this disagreement increases with prompt length and OPD preferences shift against verifier orderings.

  • OPD’s dense token signal reflects teacher preference rather than verified task outcome.Positive values indicate greater teacher support than student support for sampled tokens; negative values indicate the converse.
  • Trajectory-level OPD scores summarize mean teacher support, while verifier rewards encode binary correctness or graded partial success.Averaging token advantages avoids direct scaling with response length.
  • Teacher–verifier disagreement occurs when OPD and verifier rankings differ, such that a higher-OPD response can receive a lower verifier reward.The two signals assess different properties of the same response.
  • 40.6%→64.0% and +0.35→−0.37: Multi-Table Extraction shows rising disagreement and declining OPD preference across prompt-length ranges.The disagreement rate rises from 40.6% below 8K to 64.0% at 32–64K, while the preference gap declines from +0.35 to −0.37.
  • 35.2%→60.2% and +0.65→−0.35: High-Recall Retrieval exhibits the same longer-context deterioration in alignment.The corresponding values change from 35.2% to 60.2% and from +0.65 to −0.35.

4 Method

GC-OPD calibrates dense token-level OPD guidance with group-normalized verifier feedback by forming a signed disagreement residual. RACA distributes this residual across tokens while preserving the original OPD signal.

  • Group calibration: GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group, then uses their difference as a signed teacher–verifier disagreement residual.Separate normalization avoids comparing raw scales and preserves graded outcome spacing within groups.
  • Residual construction: The residual vanishes when group-normalized assessments coincide and grows with their discrepancy, focusing calibration on disagreement rather than reinforcing OPD’s existing preferences.Its sign indicates whether verifier assessment exceeds OPD assessment, while its magnitude measures their difference.
  • Residual construction: When verifier and OPD rankings conflict, the residual assigns a larger value to the response preferred by the verifier, directing correction toward higher-reward responses.If either signal has near-zero within-group standard deviation, GC-OPD sets the residual to zero and reduces to vanilla OPD for that group.
  • Relative-advantage-based credit assignment: RACA allocates each trajectory-level residual across tokens using relative OPD advantages, giving larger positive credits to tokens whose advantages exceed the response mean.The positive bounded credit preserves the residual’s sign while retaining within-response OPD ordering.
  • Advantage construction: The original token-level OPD advantage remains the base term, while βc_t^(i)ρ^(i) supplies the residual correction; β = 0 recovers vanilla OPD without extra forward passes.GC-OPD changes advantage construction but leaves the clipped actor objective unchanged.

5 Experiments

GC-OPD outperforms vanilla OPD across both model scales, with gains concentrated on structured reasoning and evidence-aggregation benchmarks. Ablations show that signed residualization and RACA each contribute to the improvement, while teacher–verifier disagreement extends across task families.

  • Overall performance: GC-OPD raises the five-task average from 39.31 to 40.47 for Qwen3-4B and from 43.56 to 44.65 for Qwen3-8B.It achieves the highest five-task average among evaluated shared-setup implementations at both model scales.
  • Overall performance: GC-OPD improves DocMath, MRCR, and CorpusQA for both students, with the largest gains on CorpusQA.Frames and LBv1QA show smaller or model-dependent changes, so benefits are not uniform across tasks.
  • Token allocation ablation: RACA raises the average from 44.28 to 44.65 over uniform allocation, whereas Absolute OPD reaches only 43.93.Absolute OPD discards the sign of the OPD advantage, potentially assigning large credits to strongly negative OPD values.
  • Disagreement analysis: Teacher–verifier disagreement extends beyond the two initially analyzed tasks to additional task families and a prompt-macro aggregate over all nine families.Figure 4 reports pairwise disagreement and top-1 mismatch for four families with at least 500 prompts.

6 Conclusion

GC-OPD addresses teacher–verifier disagreement in long-context OPD by calibrating group-normalized verifier rewards against trajectory-level OPD scores and distributing the signed residual across tokens with RACA. It improves vanilla OPD across five benchmarks while preserving dense token-level guidance.

  • Method: GC-OPD forms a signed residual between group-normalized verifier rewards and trajectory-level OPD scores, then distributes it across tokens with RACA.This preserves dense OPD guidance while supporting binary and graded rewards without cross-task calibration.
  • Results: 39.31 to 40.47: GC-OPD improves vanilla OPD for Qwen3-4B across five benchmarks.The corresponding Qwen3-8B average improves from 43.56 to 44.65.
  • Results: 43.56 to 44.65: GC-OPD improves vanilla OPD for Qwen3-8B across five benchmarks.The corresponding Qwen3-4B average improves from 39.31 to 40.47.
  • Analysis: Ablations identify residual calibration as the main contributor, with RACA providing an additional gain over uniform token allocation.Diagnostics also reveal stronger disagreement over longer inputs and task-dependent prevalence across the training mixture.

A Task-Conditioned Disagreement Analysis

This section complements length-conditioned analysis with a task-conditioned view of teacher–verifier disagreement across four GoLongRL task families. It measures disagreement through pairwise ordering conflicts and top-1 mismatches within informative rollout groups.

  • Task-conditioned analysis: The analysis covers four GoLongRL task families with at least 500 prompts, using frozen responses and fixed Qwen3-8B rollouts.Multi-Table Extraction and High-Recall Retrieval use responses from prompts shorter than 32K; the other task rows use eight fixed Qwen3-8B rollouts.
  • Disagreement measures: Pairwise disagreement is the fraction of response pairs with distinct verifier rewards whose OPD and verifier orderings disagree, averaged across prompts.The measure is computed within informative rollout groups.
  • Disagreement measures: Top-1 mismatch is the fraction of informative groups where the highest-scoring trajectory-level OPD response does not achieve the group’s highest verifier reward.Both disagreement measures are evaluated over groups containing at least two distinct verifier rewards.
  • Task-conditioned results: Figure 4 reports prompt-macro pairwise disagreement and top-1 mismatch rates for four GoLongRL task families, with an overall pool spanning all nine families.Bars summarize rollout groups with at least two distinct verifier rewards.

B GC-OPD Implementation Details · C Experimental Details · C.1 Models, Data, and Training Configuration

GC-OPD implements a guarded, group-calibrated residual update that preserves token-level OPD guidance while incorporating verifier rewards. Experiments use official Qwen3 student checkpoints, a Qwen3-30B-A3B-Thinking-2507 teacher, and a 32K-filtered GoLongRL training file.

  • B GC-OPD Implementation Details: GC-OPD forms a trajectory-level residual from group-normalized verifier rewards and trajectory-level OPD scores.The residual is defined using group means and standard deviations over G responses.
  • B GC-OPD Implementation Details: If either group-level signal has negligible variation, the residual is zero and the update reduces to vanilla OPD.The group-level guard compares both signal standard deviations against τG.
  • B GC-OPD Implementation Details: RACA assigns residual credit using each token’s OPD advantage relative to its response mean and within-response standard deviation.Negligible token variation gives unit credit, and the credit reflects relative OPD support rather than token correctness.
  • B GC-OPD Implementation Details: The resulting token advantages are clipped to [−amax, amax] before updating the policy with the same surrogate as vanilla OPD.Given OPD advantages and verifier rewards, the procedure requires no additional teacher or student forward pass.
  • B GC-OPD Implementation Details: The algorithm freezes the old student for rollout generation, samples G responses per prompt, verifies them, and scores their tokens under the teacher and old student.It then computes group statistics before applying the guarded residual and token-credit updates.
  • C.1 Models, Data, and Training Configuration: Students are initialized from official Qwen3-4B and Qwen3-8B checkpoints, while Qwen3-30B-A3B-Thinking-2507 serves as teacher.Students generate in no-thinking mode, and the teacher scores sampled response tokens without generating separate responses.
  • C. Experimental Details: The 32K-filtered GoLongRL training file contains 9,527 prompts, with median, 90th-percentile, and maximum lengths of 9,923, 26,940, and 32,766 tokens.Raw rows receive no training; trained methods use the common configuration and are evaluated at their final checkpoint.
  • C.1 Models, Data, and Training Configuration: Each training run used eight 80-GB NVIDIA H800 or H100 GPUs, with rollout tensor parallelism and actor/reference sequence parallelism both set to 8.These settings define the reported compute configuration for the training runs.

C.2 Ablation Configurations

The ablations use shared Qwen3-8B training and evaluation settings to isolate the effects of added trajectory-level signals and token-credit allocation. Comparisons hold core components fixed while varying verifier feedback, OPD-derived terms, or residual distribution across tokens.

  • Shared configuration: All ablations use independent Qwen3-8B runs with identical initialization, teacher, filtered data, optimizer, rollout budget, 100-step horizon, and step-100 five-benchmark evaluation.This shared configuration isolates the ablation variables from training and evaluation differences.
  • Added-signal ablation: Added-signal variants retain vanilla OPD, RACA credit, and coefficient 0.10 while changing the signal added to the OPD advantage.Vanilla OPD serves as the common anchor for this comparison.
  • Added-signal ablation: Additional OPD adds another OPD-derived term, Direct reward adds group-normalized verifier reward, and GC-OPD uses their signed difference as the residual.This separates verifier feedback from an additional OPD-derived signal and tests whether relative OPD preference improves the correction.
  • Token-allocation ablation: Token-allocation variants hold the signed residual and β = 0.10 fixed while comparing RACA, uniform allocation, and absolute-OPD credit assignment.Absolute OPD allocates credit by tokenwise OPD-advantage magnitude, whereas RACA uses Equation 16.

C.3 Baselines under the Shared Setup · C.4 Evaluation Implementation

The study compares training-signal mechanisms under a shared controlled setup, then evaluates five long-context benchmarks with explicitly defined scorers, serving limits, and aggregation rules.

  • C.3 Baselines under the Shared Setup: All trained baselines share initialization, teacher, filtered data, rollout budget, optimizer, horizon, and clipped surrogate, isolating each method’s principal training-signal mechanism.Each method generates its own trajectories on-policy; comparisons are controlled mechanism comparisons rather than exact reproductions of original recipes.
  • C.3 Baselines under the Shared Setup: Vanilla OPD uses the detached token advantage t = log pT,t − log pS,t, while the verifier’s terminal score is not added; equivalently, β = 0 recovers vanilla OPD.The native verifier still runs in the shared pipeline.
  • C.3 Baselines under the Shared Setup: ExOPD uses t = λ log pT,t + (1 − λ) log p0,t − log pS,t with λ = 1.25, using the frozen Qwen3 initialization as p0 rather than a KL regularizer.The reference affects only the detached token target and is distinct from GC-OPD’s verifier reward.
  • C.3 Baselines under the Shared Setup: Uni-OPD adds a group-level bidirectional margin shift to original tokenwise OPD advantages, using verifier-defined correct and incorrect sets, δ = 0.4, and no shift when either set is absent.The controlled comparison omits Uni-OPD’s offline and online data-balancing components while retaining group scope and mean statistics.
  • C.3 Baselines under the Shared Setup: FiRe-OPD applies a 20th-percentile trajectory cutoff, confidence-and-confusion token weights clipped to [0, 1], and αF = βF = 1 recomputed within actor micro-batches.Filtered trajectories contribute no policy loss, and weight normalization gives valid-token weights mean one.
  • C.4 Evaluation Implementation: DocMath scores each example by the maximum of rule-based numerical/equation checking and semantic-equivalence judgments, aggregating four simple/complex and short/long splits.The evaluation units and aggregation rules are recorded in Table 7.
  • C.4 Evaluation Implementation: FRAMES uses cover exact match or semantic equivalence; MRCR requires an exact random prefix before character-level SequenceMatcher scoring across 2-, 4-, and 8-needle variants.CorpusQA reports overall accuracy across four domains, while LongBench v1 QA reports the macro mean across five datasets.
  • C.4 Evaluation Implementation: Evaluation serves at most 120,000 input tokens with 131,072-token model length, caps generation at 8,192 tokens, middle-truncates overlong inputs, and reports the naive average of five benchmark scores.Students use no-thinking mode; semantic judging uses Qwen3-30B-A3B-Instruct-2507 with a 32,768-token context and 2,048-token output cap.

D Auditable Long-Context Case Study

The paper audits GC-OPD on a 17,265-token GoLongRL example whose native answer is B, then independently rerolls eight responses to illustrate offline signal recomputation. The audit uses official Qwen3 checkpoints and reports representative RACA token signals under the final GC-OPD setting.

  • Case setup: 17,265 tokens: the GoLongRL prompt asks which matrix receives the largest relative improvement from parallel NUMA optimization, with native label B.The options are A rajat31, B HV15R, C cage15, and D ldoor; contextual observations support B but do not directly measure speedup.
  • Audit protocol: Eight responses were independently rerolled from the official Qwen3-8B base checkpoint using Qwen3-30B-A3B-Thinking-2507 as teacher.Sampling used temperature 1, top-p = 1, top-k = −1, seed 42, and a 10,240-token response cap.
  • Signal audit: Table 8 reports representative token-level RACA recalculations for the correct-B and incorrect-D responses under the final GC-OPD setting.Values were recomputed offline with β = 0.10 and amax = 10, rounded for display, while ranking checks used unrounded values.
  • Signal audit: Figure 5 visualizes the selected context evidence and all eight independently rerolled student responses for the 17,265-token case.The visualization identifies B as the native label.
Loading 2608.19181v1…