PaperlayerView exact revision on arXiv

Source-linked AI summary

CA-OPD: Confidence-Aware On-Policy Distillation for Structured Visual Prediction

Menghao Li, Linjie Mu, Yin Wang, Haotian Hu, Yannian Gu, Lujiayi Xue, Fanyi Wang

arXiv:2609.02401v1cs.CV

TL;DR

Autoregressive visual prediction is vulnerable to compounding errors, while existing interleaved distillation methods use rigid ranking and underuse intervention signals. CA-OPD uses teacher confidence for selective correction, progressively shifts rollout control to the student, and aligns supervision with intervention outcomes; it improves across GUI grounding and OCR benchmarks, with gains depending on intervention placement, rollout scheduling, and adaptive supervision.

  • Problem

    Autoregressive visual prediction can compound token errors, while existing interleaved distillation methods rely on rigid ranking and underuse intervention decisions as supervision signals.

  • Method

    CA-OPD uses teacher confidence to selectively correct unreliable student transitions, gradually transfers rollout control through a strict-to-relaxed schedule, and applies intervention-aligned token supervision.

  • Results

    CA-OPD consistently improves across GUI grounding and OCR benchmarks over standard OPD and competing distillation methods, with gains linked to strategic intervention placement, progressive rollout control, and adaptive supervision.

  • Takeaways & Limitations

    Effective on-policy distillation depends jointly on constructing reliable trajectories and aligning knowledge transfer with intervention decisions, rather than increasing intervention frequency alone.

  • Takeaways & Limitations

    CA-OPD assumes sufficiently calibrated teacher probabilities and validates only shared teacher–student tokenizers, leaving tokenizer mismatch and open-ended generation untested.

Abstract

from arXiv · show

Autoregressive vision language models unify heterogeneous perception tasks but are highly susceptible to compounding errors. On-policy distillation (OPD) bridges the training-inference mismatch by training students on their own rollouts. However, unreliable student predictions, especially early in training, can derail the trajectory and degrade the quality of teacher supervision. While recent interleaved distillation methods allow the teacher to verify and replace student tokens, they primarily rely on rigid ranking metrics rather than exact teacher confidence, and they overlook how intervention decisions can inform token-level supervision. To address this, we introduce Confidence-Aware On-Policy Distillation (CA-OPD), a framework that couples reliable rollout construction with adaptive supervision. CA-OPD utilizes teacher confidence to selectively correct unreliable student transitions, gradually transferring rollout control to the student via a strict-to-relaxed schedule. Crucially, CA-OPD aligns knowledge transfer with these intervention decisions: corrected positions receive direct cross-entropy supervision from the teacher's prediction, while retained positions benefit from the teacher's full predictive distribution. Evaluated in a multi-teacher setting for GUI grounding and optical character recognition, CA-OPD substantially improves the Qwen3.5-0.8B baseline across all six target benchmarks, including gains of $9.50$ points on ScreenSpot-Pro and $6.72$ points on OCRBench-v2 English. Controlled studies further show that the gains depend on intervention placement, progressive rollout control, and intervention-aligned supervision, rather than intervention frequency alone.

1 INTRODUCTION

Autoregressive visual prediction exposes subsequent generation to compounded errors, motivating confidence-aware intervention during on-policy distillation. CA-OPD couples selective rollout repair with intervention-aligned supervision and improves performance across GUI grounding and OCR benchmarks.

  • Autoregressive structured prediction makes each generated token part of the next input, so local errors can corrupt subsequent states and degrade the full trajectory.
  • Existing interleaved distillation methods use teacher ranking to verify proposals, but ranking does not directly measure the teacher’s confidence or exploit intervention decisions for supervision.
  • CA-OPD uses teacher confidence to selectively replace unreliable student transitions while progressively shifting rollout control from stronger early intervention toward greater student control.
  • Corrected positions receive direct supervision from the teacher’s prediction, whereas retained positions receive the teacher’s full predictive distribution.
  • CA-OPD improves all six target benchmarks relative to Qwen3.5-0.8B, including 9.50 points on ScreenSpot-Pro and 6.72 points on OCRBench-v2 English.
  • Controlled studies attribute gains to confidence-based intervention placement, a strict-to-relaxed rollout schedule, and adaptive supervision rather than intervention frequency alone.

2 RELATED WORK

Related work frames structured visual prediction as sequential token generation and distinguishes offline, on-policy, and interleaved distillation. CA-OPD replaces rank-based intervention with teacher confidence and adapts control as the student improves.

  • Structured visual outputs such as text, bounding boxes, coordinates, and actions are serialized into token sequences through shared autoregressive interfaces.
  • Because committed tokens update later contexts, errors in OCR symbols, coordinates, or actions can compound and alter the semantic interpretation of the output.
  • Traditional knowledge distillation transfers teacher predictive distributions, while sequence-level distillation uses complete teacher-generated trajectories.
  • On-policy and interleaved distillation address training–inference state mismatch by querying teachers on student-generated states and replacing poorly ranked proposals.
  • CA-OPD refines interleaved distillation by using teacher confidence instead of relative rank and dynamically relaxing intervention as the student improves.

3 PROBLEM FORMULATION

The formulation represents image–instruction inputs and structured outputs as autoregressive token sequences whose prefixes define visited states. OPD trains on student-generated states, while CA-OPD adaptively controls which proposals enter the rollout.

  • 3.1 STRUCTURED AUTOREGRESSIVE PREDICTION: An input pair x consists of an image I and textual instruction q, and the model generates a length-L sequence over vocabulary V.
  • 3.1 STRUCTURED AUTOREGRESSIVE PREDICTION: The autoregressive state at step t is st = (x, y<t), and student and frozen teacher networks define next-token distributions conditioned on that state.
  • 3.1 STRUCTURED AUTOREGRESSIVE PREDICTION: Structured output tokens may encode text, spatial coordinates, or discrete actions, with each committed token updating the future context.
  • 3.2 ON-POLICY DISTILLATION AND STATE VISITATION: OPD constructs training trajectories by sampling directly from the student policy and aligns the student with the teacher over actively visited states.
  • 3.2 ON-POLICY DISTILLATION AND STATE VISITATION: The OPD objective uses a loss mask mt to determine which visited positions contribute to the distillation loss.
  • 3.2 ON-POLICY DISTILLATION AND STATE VISITATION: The supervision objective is decoupled from the behavior policy used to construct rollouts, allowing training targets and trajectory control to differ.
  • 3.2 ON-POLICY DISTILLATION AND STATE VISITATION: Standard OPD uses µ = πS, whereas interleaved methods permit teacher intervention; CA-OPD adaptively gates retained proposals to balance reliable trajectories with student exposure.

4 CONFIDENCE-AWARE ON-POLICY DISTILLATION

CA-OPD constructs rollouts by testing each student proposal against teacher confidence, replacing weak proposals and aligning supervision with that decision. A progressive threshold schedule increasingly returns rollout control to the student as training advances.

  • Confidence-Aware Rollout Control: At each decoding step, the teacher evaluates the student’s proposed token, retaining highly supported proposals and replacing unreliable ones with the teacher’s prediction.Retained positions use the teacher’s predictive distribution, while replaced positions use direct cross-entropy supervision.
  • Confidence-Aware Rollout Control: The student proposal is rejected when its teacher-assigned negative log-likelihood exceeds the step-dependent threshold.A lower teacher NLL indicates stronger support for the exact proposal.
  • Confidence-Aware Rollout Control: Rejected proposals are replaced by the teacher’s deterministic prediction, changing the autoregressive state for subsequent generation without propagating gradients through the discrete rollout operation.The final committed token is the student proposal when retained and the teacher prediction when rejected.
  • Intervention-Aligned Supervision: For retained positions, CA-OPD applies forward KL supervision from the teacher distribution and can restrict transfer to the teacher’s top-k predictions to reduce computational cost.The truncated KL value is lower-bounded at zero because unnormalized retained probabilities can make it negative.
  • Progressive Transfer of Rollout Control: A cosine schedule increases the intervention threshold from a strict initial criterion to a relaxed later one, allowing more student proposals to remain as training progresses.The schedule controls the confidence criterion rather than imposing a fixed replacement rate, facilitating a natural transfer of rollout control.

5 EXPERIMENTS

CA-OPD is evaluated against controlled distillation baselines across GUI grounding, OCR, retention benchmarks, and targeted ablations. Results indicate that selective intervention, prefix repair, progressive rollout control, and confidence-aware placement—not intervention frequency alone—drive improvements.

  • Main results: CA-OPD surpasses Offline KD, standard OPD, and SKD across six target benchmarks spanning GUI grounding and OCR.All controlled methods use the same student initialization, training data, and optimization budget, with results averaged over three runs.
  • Main results: ScreenSpot-Pro rises from 40.82 to 45.92 (+5.10 points), while CC-OCR increases from 63.93 to 67.32 (+3.39 points).
  • Main results: CA-OPD’s annealed variant improves five of six target metrics over a fixed confidence threshold while maintaining held-out RefCOCO and MMBench performance near competing baselines.The held-out evaluation indicates no measured compromise in retained general capabilities.
  • Ablation study: Only the combination of confidence-based intervention placement and prefix writeback produces the full improvement; random interventions or gating without writeback remain close to standard OPD.The ablation holds student initialization, training data, and optimization budgets constant.
  • Ablation study: A strict-to-relaxed rollout schedule outperforms fixed and reversed schedules, while deterministic teacher corrections outperform sampled corrections.These results support progressively transferring rollout control from teacher to student and aligning supervision with intervention decisions.
  • Gating mechanism: Increasing the SKD replacement rate from 0.16% to 3.41% improves SS-Pro from 41.20 to 44.00 but still trails CA-OPD by 1.92 points on SS-Pro and 0.86 points on OCRv2-EN.Random interventions at a comparable budget yield only marginal gains, showing that placement matters beyond frequency.
  • Gating mechanism: CA-OPD-triggered proposals have median teacher rank 3, with 100% of grounding and 88% of OCR proposals within the teacher’s top-25.Thus, rank-based gating would retain many proposals rejected for insufficient absolute teacher support.
  • Data-regime analysis: CA-OPD improves over OPD across all output-length, initial-competence, and teacher-rejection-density strata.The nearly constant margin across rejection-density terciles indicates gains are not confined to examples requiring frequent intervention.

6 CONCLUSION

The paper concludes that CA-OPD jointly constructs reliable student-conditioned trajectories and adapts supervision to teacher interventions. Its formulation and implementation distinguish teacher-written corrections from retained student positions while progressively relaxing rollout control.

  • Conclusion: CA-OPD uses teacher confidence to selectively repair student-generated trajectories and couples rollout intervention with token-level supervision.
  • Method: Teacher-written positions receive cross-entropy supervision, whereas retained student positions receive distributional distillation.
  • Rollout schedule: The strictest acceptance criterion applies during the warm-up fraction, followed by cosine relaxation that permits more student proposals to remain.The main configuration uses ω = 0.1, τstart = 0.693, and τend = 1.386, annealing the teacher-probability floor from α = 0.50 to α = 0.25 over 471 optimizer steps.
  • Implementation: Algorithm 1 constructs the rollout token by token, replacing rejected student proposals with the teacher argmax and restarting generation from the first replaced position.Retained positions use a top-64 teacher distribution for the distillation objective.
  • Configuration: The experimental configuration uses an SFT-initialized Qwen3.5-0.8B student and separate grounding and OCR Qwen3.5-9B teachers.The optimizer settings are shared across controlled methods, while gate-specific settings apply to CA-OPD variants.

C BENCHMARK AND EVALUATION DETAILS

The evaluation uses standardized benchmark-specific protocols for GUI grounding, OCR, and capability retention. Results are reported with consistent harnesses and run-level averages across independent training runs.

  • Common protocol: Every benchmark column uses an identical evaluation harness, prompt, preprocessing pipeline, and decoding configuration across methods.Reported means and standard deviations are computed across three independent training runs.
  • GUI grounding: ScreenSpot-v2 and ScreenSpot-Pro use the official Qwen3.5 tool-call prompt, eight sampled decodings, and a score based on points falling inside target boxes.Sampling uses temperature 0.7, top-p 0.8, top-k 20, and presence penalty 1.5.
  • OCR: OCRBench-v2 is scored separately in English and Chinese, CC-OCR is averaged across four tracks, and OmniDocBench reports full-page 1−NED.All OCR benchmarks use greedy decoding with thinking disabled.
  • Retention: RefCOCO, RefCOCO+, and RefCOCOg use eight official splits with ACC@0.5, while MMBench uses EN-dev-v1.1 accuracy.Table 1 reports the mean across the eight RefCOCO-family splits.

D.1 FINE-GRAINED TARGET-BENCHMARK RESULTS

CA-OPD improves fine-grained performance across ScreenSpot-Pro and OCRBench-v2 categories while preserving held-out capabilities. The analyses indicate that intervention placement, schedule design, and selective gating matter beyond replacement frequency.

  • CA-OPD is best in all six ScreenSpot-Pro application categories.
  • CA-OPD exceeds OPD in all eight OCRBench-v2 capability categories and has the best listed score in five.SFT initialization remains strongest in Recognition, Calculation, and Reasoning.
  • Held-out capability gains do not come at the expense of measured retention, with controlled methods differing by at most 0.43 points in any RefCOCO/+/g or MMBench column.
  • Annealed gating lowers cross-run deviations on SS-Pro, CC-OCR, and OmniDoc from 1.46, 0.96, and 1.13 to 0.72, 0.51, and 0.33.The deviations reflect run-to-run variability across three independent runs.
  • Accuracy does not increase monotonically with replacement rate: random placement offers little benefit, and high-frequency fixed floors do not consistently beat annealing.
  • Both schedule direction and terminal floor matter: relaxing to α = 0.10 loses part of the gain, while reversing the schedule loses more.Linear relaxation is weaker than cosine relaxation at the same endpoints.

E.3 GATING DYNAMICS OVER TRAINING

The gating analyses examine how confidence-based interventions vary with training and across predefined difficulty and competence strata. Annealed CA-OPD intervenes more early, improves over OPD across strata, and retains more mastered behavior than OPD.

  • Under annealing, replacement is more frequent early and decreases as the confidence criterion relaxes; fixed-threshold replacement also declines as student reliability improves.
  • Within-type OCR difficulty tertiles are required because global bucketing would confound difficulty with metric type and output length.
  • Teacher-rejection density ρ is the fraction of output tokens with teacher probability below 0.25; low ρ indicates strong endorsement and high ρ frequent rejection.Its correlation with initial competence is r = −0.51, while its correlation with output length is r = −0.10.
  • CA-OPD improves over OPD in every initial-competence group and retains 93.83 in the all-correct group versus OPD’s 90.18 and initialization’s 96.03.
  • The annealed method improves over OPD in every OCR stratum across both languages, whereas the fixed floor falls below OPD only in the hardest Chinese tertile.
  • Annealed CA-OPD’s margin over OPD is nearly constant across rejection-density tertiles: +4.95, +5.31, and +5.05.In the low-ρ group, OPD falls below initialization while both gated variants improve on it.
  • In a single-teacher, raw-pretrained setting with fixed α = 0.25, CA-OPD improves SS-v2 by 1.74 points and SS-Pro by 5.48 points.Only the within-table OPD–CA-OPD comparison is meaningful because initialization and training regime differ from the main experiments.
  • A policy-gradient variant reaches 88.61 on SS-v2, 45.56 on SS-Pro, 51.58 on OCRv2-EN, and 53.04 on OCRv2-ZH in one run.It was not repeated across runs and is kept outside the controlled main table.

H TRAINING COST AND COMPUTE MATCHING

CA-OPD incurs higher training cost than ungated OPD because interleaved teacher queries occur during generation. The reported comparisons use matched hardware, data, and optimizer-step conditions.

  • CA-OPD and SKD cost about 1.5× ungated OPD in median wall-clock seconds per optimizer step.The gate arithmetic itself is not the dominant overhead.
  • All methods use the same 8×A800 node, identical data, batch size, and 471 optimizer steps.
  • The fixed-α throughput depends on the final chunked gate implementation; earlier per-token runs took approximately 109 seconds per step without changing final scores.The SS-Pro cross-run deviation for the final configuration is 0.53.

I LIMITATIONS AND SCOPE

CA-OPD’s scope depends on calibrated teacher probabilities and shared teacher–student tokenizers. Its sequential mixed-prefix rollout also increases cost and does not fully prevent erosion of mastered behavior.

  • CA-OPD assumes teacher probabilities are sufficiently calibrated to identify unreliable student transitions.Systematic teacher–student mismatch, such as domain shift, may require retuning the confidence floor.
  • The evaluation uses shared teacher–student tokenizers and leaves mismatched tokenizers and open-ended generation untested.
  • Mixed-prefix rollout queries the teacher at every visited state and costs about 1.5× ungated OPD.
  • Annealed CA-OPD reduces but does not eliminate erosion on consistently solved SS-Pro examples, reaching 93.83 versus the initialization’s 96.03.
Loading 2609.02401v1…