PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillation

Siyuan Liu, Kanghui Tian, Yue Duan, Yutao He, Shangdong Yang, Jian Zhang, Yinghuan Shi

arXiv:2609.39120v1cs.CV

TL;DR

Limited student visual perception remains a bottleneck in multimodal on-policy distillation, even as prior work improves teacher-side guidance. S-OPD adds teacher-calibrated contrast and policy agreement objectives to student training. Experiments show consistent gains across benchmarks and settings, including up to 4.25 points on LogicVista.

  • Problem

    Teacher-side improvements provide text-token guidance that students with limited visual perception may struggle to ground in visual inputs.

    Source: §1 · p. 2

  • Method

    S-OPD uses teacher-calibrated contrast between original- and masked-image student policies and agreement between original- and noise-perturbed-image policies.

    Source: §1 · p. 3

  • Results

    S-OPD yields consistent improvements across student scales, distillation paradigms, and benchmarks, with gains reaching 4.25 points on LogicVista.

    Source: §1 · p. 3

  • Takeaways & Limitations

    S-OPD strengthens student perceptual learning within token-level distillation and can add gains when combined with teacher-side supervision methods.

    Source: §1 · p. 3, §5 · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) improves reasoning by providing token-level supervision from a teacher on a student's own trajectories. Existing methods primarily focus on enhancing this teacher-side guidance (e.g., by enriching teacher inputs and refining teacher feedback), yet we find that limited student perception is another critical bottleneck in multimodal OPD. By providing oracle visual facts, the performance of OPD-trained students can still be substantially improved for both weak and strong teachers. To address this bottleneck, we propose S-OPD, a simple multimodal on-policy distillation framework that explicitly strengthens student perceptual learning through two objectives. Specifically, Teacher-calibrated Policy Contrast separates student policies under original and masked images with teacher-based token-level gating, strengthening the student's reliance on visual evidence during reasoning. Policy Agreement aligns student policies under original and noise-perturbed images, further improving perceptual robustness to visual noise. Notably, our method can be seamlessly plugged into existing OPD frameworks, requiring no additional data annotations, model parameters or inference operations. Extensive experiments on eight benchmarks across student scales and distillation paradigms demonstrate consistent performance improvements, with gains of up to 4.25 points on LogicVista. When combined with existing teacher-side supervision methods, our method can yield further gains. Code is available at https://github.com/Sirilaw/S-OPD.

1 INTRODUCTION

The introduction argues that student visual perception remains a bottleneck in multimodal on-policy distillation despite advances in teacher-side supervision. It proposes S-OPD to strengthen student perceptual learning, reporting gains across tasks and settings.

1 INTRODUCTION

  • Existing OPD improvements primarily enrich teacher inputs or refine teacher feedback, while limited student perception may hinder grounding of text-token guidance in visual inputs.OPD provides dense token-level supervision on student-generated trajectories.Source: §1 · p. 2
  • Oracle visual facts substantially improve OPD-trained students’ Geometry3K accuracy, including with a stronger GRPO-trained teacher.The facts describe diagram content without providing answers or reasoning steps.Source: §1 · p. 2
  • Across four benchmarks and both student scales, higher visual-sensitivity quartiles generally achieve higher accuracy.Visual sensitivity is estimated from the KL divergence between student policies under original and masked images.Source: §1 · p. 3, Figure 3 · p. 3
  • S-OPD combines teacher-calibrated policy contrast under original and masked images with policy agreement under original and noise-perturbed images.Teacher responses gate contrast at tokens affected by visual evidence removal; agreement encourages stable predictions under noise.Source: §1 · p. 3
  • S-OPD delivers consistent gains across student scales, distillation paradigms, and benchmarks, reaching 4.25 points on LogicVista.Teacher-side supervision methods gain further when combined with S-OPD.Source: §1 · p. 3

2 RELATED WORK

Prior OPD research chiefly improves teacher-side supervision, while S-OPD focuses on student-side learning. It distinguishes itself by using teacher responses to select tokens for visual policy contrast.

2 RELATED WORK

  • OPD addresses the mismatch between fixed teacher-generated sequences and student-generated inference trajectories through dense token-level teacher feedback.Recent methods enrich teacher inputs or refine feedback; S-OPD instead targets student-side learning.Source: §2 · pp. 3–4
  • Multimodal OPD work enriches supervision sources, constructs teacher–student asymmetry, and refines distillation signals; S-OPD shapes student perception with cross-view KL objectives.Its objectives promote sensitivity to evidence removal and stability under content-preserving perturbations.Source: §2 · p. 4
  • Unlike prior visual contrast and consistency applications, S-OPD uses changes in teacher-assigned token probabilities to select tokens for student policy contrast.Teacher feedback supplies both distillation targets and token-level calibration for student perceptual learning.Source: §2 · p. 4

3 METHOD

S-OPD augments token-level teacher–student distillation with objectives that shape student responses to visual evidence. Its masked-image contrast is teacher-calibrated, while its noise-view agreement promotes stable predictions.

3 METHOD

  • S-OPD uses two KL objectives to strengthen student sensitivity to visual evidence and robustness to content-preserving perturbations.Teacher-Calibrated Policy Contrast (TPC) contrasts original and masked-image policies using teacher-selected tokens; Policy Agreement (PA) aligns original and mildly perturbed-image policies.Source: §3 · p. 4

3.1 PROBLEM FORMULATION

  • Oracle diagram facts substantially improve Geometry3K accuracy even after OPD with a stronger GRPO-trained teacher, and higher masked-view sensitivity generally tracks higher accuracy across benchmarks.The method motivates shaping responses to evidence removal and mild perturbations; masking removes visual evidence, while mild noise is intended to preserve task content.Source: §3.1 · p. 5

3.2 TEACHER-CALIBRATED POLICY CONTRAST

  • TPC increases the student’s original-versus-masked policy divergence where the teacher indicates that removed visual evidence matters.The teacher gate selects positions where its probability is higher under the original image; the selected-token contrast is weighted by the teacher’s log-probability drop.Source: §3.2 · p. 6

3.3 POLICY AGREEMENT

  • PA minimizes divergence between original- and noisy-image student policies to encourage consistent predictions under mild perturbations.The perturbations are intended to preserve task-relevant content, and the noisy-image policy provides a fixed target for each update.Source: §3.3 · p. 6

3.4 OVERALL OBJECTIVE

  • S-OPD combines teacher–student distillation with weighted policy-contrast and policy-agreement objectives, without extra model parameters or inference operations.On Geometry3K, the method achieves higher performance and reduces the teacher–student log-probability gap during training; additional computation is confined to training.Source: Figure 4 · p. 5, §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate S-OPD across student sizes, teacher strengths, and reasoning domains, then ablate its objectives and training choices. Results show consistent average gains, with benefits depending on the student–teacher setting and the objective used.

4.1 MAIN RESULTS

  • S-OPD improves average performance in all four student–teacher configurations, with gains of +1.37 versus +0.61 points for 2B and 4B students using the 8B teacher, and +0.92 and +0.82 points with the GRPO-trained teacher.Table 1 reports the main results across eight benchmarks; the gain depends on both student capacity and teacher strength.Source: §4.1 · p. 7, Table 1 · p. 8
  • Across every configuration, S-OPD improves average scores in mathematical, logical, and general reasoning; the largest reported task gain is 4.25 points on LogicVista for the 4B student with the GRPO-trained teacher.With the 8B teacher, the 2B student improves on all eight benchmarks, including MathVista (+1.80) and ZeroBench (+2.70).Source: §4.1 · p. 7
  • Adding S-OPD to VA-OPD raises average performance by 1.03 points for the 2B student and 0.72 points for the 4B student.Reported task gains include MathVision (+2.70) for 2B and MMMU-val (+1.67) for 4B.Source: §4.1 · p. 7

4.2 ABLATION STUDIES

  • Combining TPC and PA improves over OPD on Geometry3K and the other three categories at both scales, with average gains of 1.60 and 1.04 points; TPC favors in-domain learning while PA supports transfer.TPC raises Geometry3K accuracy by 1.63 and 2.26 points for the 2B and 4B students, versus 1.00 and 1.27 points from PA; PA achieves the highest general-reasoning scores at both scales.Source: §4.2 · pp. 7–8, Table 2 · p. 8
  • Teacher-calibrated gating exceeds student-based and no gating by 1.15 and 1.02 average points, while random gating with the same token count trails it by 0.69 points.Applying policy contrast to all tokens reduces the average score from 38.60 to 38.06, underscoring the value of selective teacher calibration.Source: §4.2 · p. 8, Table 3 · p. 9
  • S-OPD improves both standard OPD and OPSD across 2B and 4B students, with average gains of 1.60 and 1.04 points under standard OPD and 0.71 and 0.66 under OPSD.Figure 5 summarizes gains across student scales and distillation paradigms.Source: Figure 5 · p. 8, §4.2 · p. 9
  • Average performance varies by only 0.74 points across TPC coefficients and 0.50 across PA coefficients; increasing the masking ratio from 0.2 to 0.6 raises the average from 38.49 to 39.08, while stronger masking lowers it to 38.74.Gaussian noise performs best at σ = 0.2; the selected main-experiment settings are λTPC = λPA = 0.02, ρ = 0.6, and σ = 0.2.Source: §4.2 · p. 9, Table 4 · p. 9, Table 5 · p. 9
  • S-OPD adds no trainable parameters, autoregressive rollouts, or inference overhead, but increases training time per step by 28.6% for the 2B student on ViRL39K and 16.0% for the 4B student on Geometry3K.Its additional computation is confined to training and evaluates perturbed image views using the same generated responses.Source: §4.2 · p. 9

5 CONCLUSION

The paper concludes that student visual perception remains a bottleneck in multimodal OPD even with stronger teacher supervision. S-OPD complements teacher-side improvements with student perceptual learning, yielding consistent gains across eight benchmarks without deployment-time overhead.

5 CONCLUSION

  • S-OPD combines teacher-calibrated policy contrast for sensitivity to removed visual evidence with policy agreement for stability under visual noise.Experiments report consistent average improvements across student scales and teacher strengths, with further gains when combined with methods that enhance teacher supervision.Source: §5 · p. 10

A.1 TRAINING DETAILS

The appendix describes the training setup and image perturbations used for S-OPD. Masking and Gaussian noise are applied to pixel values before model-specific visual preprocessing.

A.1 TRAINING DETAILS

  • Experiments use the verl framework on NVIDIA A100 80GB GPUs, with PyTorch FSDP for student training and vLLM for rollouts and frozen-teacher forward passes.Table 6 summarizes key configurations for the main ViRL39K experiments and Geometry3K ablations.Source: §A.1 · p. 14, Table 6 · p. 14
  • Masking independently replaces 16 × 16 pixel blocks with black pixels at an expected ratio of 0.6 while preserving image dimensions and the visual-token sequence structure.Figure 6 illustrates masked images at different ratios.Source: §A.1 · p. 14, Figure 6 · p. 14
  • Gaussian perturbation adds independent zero-mean noise to each RGB channel in the [0, 1] pixel range, using σ = 0.2 before clipping and conversion back to 8-bit pixels.At σ = 0.2, the noise corresponds to approximately 51 intensity levels on the [0, 255] scale; Figure 7 visualizes perturbation strengths.Source: §A.1 · p. 14

A.2 EVALUATION DETAILS

Evaluation spans eight benchmarks and specifies dataset splits, scoring procedures, and prompt conventions. A separate Geometry3K intervention compares image-only inputs with inputs augmented by oracle visual facts, finding improvements at both student scales but a qualified result for 4B.

A.2 EVALUATION DETAILS

  • Checkpoints are evaluated on eight named benchmarks with greedy decoding and shared Qwen3-VL prompts; scoring uses official exact-match or multiple-choice rules where available and answer extraction or semantic grading otherwise.The protocol uses MathVista’s 1,000-example mini split, WeMath Core (Strict), and the MMMU validation split; MathVerse excludes its text-only subset.Source: §A.2 · pp. 14–15, Table 7 · p. 15

A.3 ORACLE VISUAL-FACT INTERVENTION DETAILS

  • For the Geometry3K oracle-fact intervention, diagram annotations are rendered as geometric statements without answers or solution steps, and paired inputs differ only by adding those statements.Evaluation uses the official 601-example test split; Accuracy Gap is the paired difference between oracle-fact and image-only accuracy.Source: §A.3 · pp. 15–16, Figure 8 · p. 16
  • For the 2B student, S-OPD raises image-only accuracy from 34.4% to 44.1% and narrows the Accuracy Gap from 7.8 to 2.3 points; for 4B, image-only accuracy rises from 50.7% to 52.1% and the gap falls from 15.1 to 12.1 points.For 4B, oracle-fact accuracy decreases from 65.9% to 64.2%, and OPD with the GRPO teacher retains the smallest gap.Source: §A.3 · p. 16, Figure 8 · p. 16

B.1 VISUAL PERCEPTION BEYOND REASONING BENCHMARKS

S-OPD is evaluated on fine-grained visual-understanding and hallucination-oriented benchmarks, extending assessment beyond in-domain reasoning. It improves or matches OPD in most model–data configurations, with benefits across the three benchmarks.

B.1 VISUAL PERCEPTION BEYOND REASONING BENCHMARKS

  • 10 of 12 model–data configurations improve or match OPD across three out-of-domain visual benchmarks.The evaluations cover fine-grained visual understanding on V∗Bench and ZoomBench and visual-illusion and language-hallucination diagnosis on HallusionBench.Source: §B.1 · pp. 16–17, Table 8 · p. 17
  • V∗Bench improves by 1.05 points for the 2B student on either dataset and by 0.63 points for the 4B student trained on ViRL39K.On ZoomBench, gains are 0.95 and 0.71 points for Geometry3K-trained 2B and 4B students, respectively, and 0.47 points for the ViRL39K-trained 4B student.Source: §B.1 · p. 17
  • HallusionBench performance improves or is preserved across all four settings, with gains of up to 0.53 points.The reported trend spans model sizes, training datasets, and diagnostic tasks.Source: §B.1 · pp. 16–17

B.2 MAIN COMPONENT ANALYSIS

TPC improves mathematical benchmark performance, while PA yields task- and scale-dependent gains; combining them gives the strongest overall average. The teacher gate retains fewer than half of valid response tokens on average, and tested coefficient settings outperform OPD.

B.2 MAIN COMPONENT ANALYSIS

  • TPC improves in-domain accuracy and all four mathematical reasoning benchmarks for both student scales, while PA’s gains vary by task and scale.PA notably improves LogicVista and ZeroBench for the 2B student and MMMU for the 4B student.Source: §B.2 · p. 17
  • 1.60 and 1.04 points are the combined objectives’ gains over vanilla OPD in overall average for the 2B and 4B students, respectively.The combination also improves in-domain accuracy and out-of-domain average performance, though it does not preserve every component’s best individual score.Source: §B.2 · p. 17

B.3 EMPIRICAL SPARSITY OF THE TEACHER GATE

  • 46.87% of valid response tokens are retained by the teacher gate; rollout-weighted mean activation is 48.29%, with a 48.39% median.Across 3,840 rollouts and 8,649,621 valid response tokens, more than half of tokens are filtered on average even with τ = 0.Source: §B.3 · p. 18

B.4 IMPACT OF THE COEFFICIENTS OF TPC AND PA

  • 0.92 and 1.02 points are the ranges of average performance across tested TPC and PA coefficients on ViRL39K; all settings beat OPD’s 36.09 baseline.Both coefficients reach their highest average score at 0.02; λPA = 0.04 gives 37.39 versus 37.46 at 0.02, and both defaults are set to 0.02 for ViRL39K.Source: §B.4 · p. 18, Table 10 · p. 18

B.5 TOKEN-LEVEL LOG-PROBABILITY ANALYSIS

S-OPD maintains comparable teacher–student log-probability gaps over complete rollouts while reducing gaps in selected middle response segments. The segment-level pattern is consistent with improved learning during intermediate reasoning rather than only final-answer prediction.

B.5 TOKEN-LEVEL LOG-PROBABILITY ANALYSIS

  • S-OPD reaches comparable final teacher–student log-probability gaps to OPD over complete rollouts at both student scales.The full-rollout average includes planning, intermediate reasoning, and final-answer generation, and can obscure changes within individual stages.Source: §B.5 · p. 18, Figure 9 · p. 19
  • S-OPD shows smaller Q2 gaps for the 4B student on Geometry3K and reduced Q3 gaps for the 2B student on ViRL39K during later training.Both 4B settings also show smaller Q3 gaps during portions of training; Q1–Q4 divide each rollout into four equal-length segments.Source: §B.5 · p. 18, Figure 9 · p. 19

B.6 COMPUTATIONAL OVERHEAD ANALYSIS

S-OPD adds auxiliary training evaluations but reuses the same student-generated response, with no additional inference overhead. Sample-level cases illustrate differences in how OPD and S-OPD use visual evidence across mathematical, spatial, and fine-grained visual reasoning.

B.6 COMPUTATIONAL OVERHEAD ANALYSIS

  • Training time per step rises by 28.6% for the 2B student on ViRL39K and 16.0% for the 4B student on Geometry3K, with response lengths nearly unchanged.Masked-image teacher evaluation and masked/noisy-image student evaluation reuse the same response; inference has no additional computational overhead.Source: §B.6 · p. 19

C.1 SAMPLE-LEVEL ROLLOUT ANALYSIS

  • Three selected cases compare OPD errors with S-OPD correct answers on problems requiring image-only information, illustrating visual grounding rather than adding quantitative evidence.The cases are from MathVerse, LogicVista, and MMMU-val and cover geometric dimensions, spatial motion relations, and fine-grained visual patterns.Source: §C.1 · p. 20, Figure 10 · p. 20

C.1.1 MATHVERSE: TRIANGULAR PRISM VOLUME CALCULATION

  • S-OPD assigns 4 cm and 2 cm to the triangular face and 8 cm to prism length, yielding the correct volume of 32 cm3.The OPD baseline reads the same visible measurements but confuses the base dimensions with prism length, producing an incorrect answer of 8 cm3.Source: §C.1.1 · p. 20

C.1.2 LOGICVISTA: GEAR ROTATION DIRECTION REASONING

  • The OPD baseline correctly reads the initial counterclockwise arrow and first gear reversal but misses the second reversal; S-OPD applies both and answers counterclockwise.The contrast concerns relational composition across two gear contacts, not initial arrow recognition.Source: §C.1.2 · p. 21

C.1.3 MMMU-VAL: MUSCLE TISSUE IMAGE IDENTIFICATION

  • S-OPD distinguishes skeletal from cardiac muscle by comparing long, parallel fibers with the branching arrangement associated with cardiac muscle.Both models detect striation, but the OPD baseline relies on that shared, non-discriminative attribute.Source: §C.1.3 · p. 21

C.2 TOKEN-LEVEL PROBABILITY VISUALIZATION AND ANALYSIS

The visualization compares student token probabilities for original and masked images, then applies teacher responses to selectively retain student discrepancies. Across three visual-reasoning domains, the gated contrast concentrates on visually grounded, decision-relevant steps.

C.2 TOKEN-LEVEL PROBABILITY VISUALIZATION AND ANALYSIS

  • For each example, the student generates a response from the original image, then student and teacher score that same response under original and patch-masked images.This controlled comparison isolates how visual evidence removal changes token probabilities.Source: §C.2 · p. 22
  • The signed student contrast compares token log-probabilities under original and masked images; gated yellow is strongest when student policy changes and teacher gives positive visual support.Orange and blue mark positive and negative student changes, while darker colors indicate larger magnitudes; darker purple marks stronger teacher support.Source: §C.2 · p. 22
  • Examples show gating emphasizes unit-circle quantities in MathVerse, structural-pattern tokens in LogicVista, and branch relations and the ancestor answer in MMMU-val.In MathVerse, the gated contrast focuses on quantities needed to infer sec θ; in LogicVista, tokens distinguish two sets.Source: §C.2 · p. 22
  • Across three domains, teacher gating preserves student discrepancies only when supported by teacher responses, concentrating contrast on visually grounded, decision-relevant steps.Teacher and student emphasize different tokens; the gated contrast is not a uniform rescaling of student sensitivity.Source: §C.2 · p. 22, Figure 11 · p. 23
Loading 2609.39120v1…