PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual-Advantage On-Policy Distillation for Vision-Language Models

Ruiqi Liu, Xiaolei Lv, Gengsheng Li, Ximo Zhu, Zhiheng Wang, Zhengbo Zhang, Junkai Chen, Zhiheng Li, Bo Li, Jun Gao, Shu Wu

arXiv:2605.21924v1cs.CV

TL;DR

Standard on-policy distillation can improve VLM outputs without strengthening reliance on fine-grained visual detail. The paper introduces VA-OPD, which uses teacher-relative visual advantage to prioritize visual supervision, and reports improvements over standard OPD on every benchmark. Its evidence supports stronger visual reliance, while VA remains a proxy rather than a direct measure of internal perceptual processes.

  • Problem

    Standard on-policy distillation can match teacher outputs without ensuring that VLM students rely on fine-grained visual detail.

  • Method

    VA-OPD uses visual advantage for rollout-level reweighting and separate token-level KL averaging across high-VA and low-VA groups.

  • Results

    VA-OPD improves over standard on-policy distillation on every benchmark across three teacher sizes, two training corpora, and eight benchmarks.

  • Takeaways & Limitations

    The gains are accompanied by stronger student reliance on fine-grained visual detail rather than only closer output mimicry.

  • Takeaways & Limitations

    VA is a proxy for visual dependency, not a direct measurement of internal perceptual processes such as attention allocation or feature grounding.

Abstract

from arXiv · show

On-policy knowledge distillation has proven effective for language models, yet its application to vision-language models (VLMs) remains underexplored. We observe that standard on-policy distillation can improve a student's output quality while failing to strengthen its reliance on visual input: on vision-critical tokens, the student's predictions remain largely unchanged whether or not fine-grained visual detail is present, even though the teacher's predictions depend heavily on it.To make this difference observable, we introduce visual advantage (VA), the token-level log-probability difference when the teacher scores a student-generated rollout with versus without access to fine-grained visual detail. VA is concentrated in a small minority of tokens, and these high-VA tokens are the ones that actually carry the visual supervision signal. This motivates a distillation objective that treats them differently from language scaffolding, so their contribution is not diluted by the abundant surrounding language tokens.We propose Visual-Advantage On-Policy Distillation (VA-OPD), which uses VA at two granularities: rollout-level reweighting by trajectory-averaged VA, and token-level KL averaged within high-VA and low-VA groups separately. We train on two math datasets (Geometry3K and ViRL39K) and evaluate on eight benchmarks covering both mathematical reasoning and visual understanding, across three teacher sizes (4B, 8B, and 32B) on the Qwen3-VL family. VA-OPD improves over standard on-policy distillation on every benchmark, with the gain growing monotonically along both the teacher-size and data-scale axes, suggesting that these factors compound consistently.

1 Introduction

Standard on-policy distillation can match teacher outputs without strengthening a VLM’s reliance on fine-grained visual detail. VA-OPD exposes and prioritizes the sparse visual signal, improving benchmark performance while increasing visual reliance.

  • Visual advantage identifies how much each student-generated token depends on fine-grained visual detail according to the teacher.
  • Standard OPD improves task accuracy while rollout visual advantage remains nearly flat, so output matching does not strengthen visual reliance.
  • The top-VA minority carries the visual supervision signal, whereas uniform KL dilutes it across language scaffolding.
  • VA-OPD reweights rollouts by trajectory-averaged VA and separately averages KL for high-VA and low-VA token groups.
  • VA-OPD improves over standard on-policy distillation across three teacher sizes, two training corpora, and eight benchmarks.Reported gains include HallusionBench +2.5, AI2D +2.4, and, with ViRL39K, Math Avg +3.8 and Visual Avg +2.5.
  • During VA-OPD training, rising rollout VA accompanies accuracy gains, indicating stronger visual reliance rather than only closer output mimicry.

2 Background

Standard OPD trains a student to match teacher token distributions but does not directly ensure dependence on fine-grained visual detail. VA measures this dependency through a degraded-image counterfactual, motivating grouped treatment of the sparse high-VA tokens.

  • Standard OPD minimizes reverse KL on student rollouts, constraining predicted tokens without ensuring attention to fine-grained visual detail.
  • 2.1 Visual Advantage: Visual advantage is the non-negative teacher log-probability difference for a token with versus without fine-grained visual detail.High VA indicates dependence on details such as precise diagram numbers; near-zero VA marks scaffolding or information preserved by degradation.
  • 2.1 Visual Advantage: The degraded image removes fine-grained detail while preserving global layout and color and keeping visual token counts aligned.
  • 2.2 Motivating Observations: The top 10% of tokens carry approximately 93% of total VA mass, showing that visual dependency is sparse within rollouts.
  • 2.2 Motivating Observations: Masking the highest-VA 10% drops MathVersemini by approximately 2.9 points, while low-VA masking changes performance within ±0.1 and random masking drops it by approximately 0.3 points.
  • Because uniform KL scales every token by 1/T, grouped normalization is used to prevent high-VA supervision from being diluted by scaffolding.

3 Visual-Advantage On-Policy Distillation

VA-OPD uses visual advantage at rollout and token granularities to direct distillation toward visually grounded behavior. It reweights sibling rollouts by relative trajectory-averaged VA and separately averages KL over high- and low-VA token groups.

  • 3.1 Rollout-Level Reweighting: VA-OPD reweights sibling rollouts by trajectory-averaged VA, giving more gradient mass to rollouts with stronger visual reliance.The trajectory average is normalized within each K-rollout group and converted into softmax weights with τ = 1.0.
  • 3 Visual-Advantage On-Policy Distillation: The teacher computes visual advantage by comparing each token’s log-probability with and without fine-grained visual detail.VA supplies the signal used for both rollout-level reweighting and token-level grouped KL.
  • 3.2 Token-Level Grouped KL: Tokens are sorted by VA into high- and low-VA groups, whose per-token KL losses are averaged separately before being combined.The high-VA group contains the top p_v fraction of tokens, with p_v fixed at 0.2 in the experiments.
  • 3.2 Token-Level Grouped KL: The grouped loss weights the high-VA and low-VA KL averages using λ, with λ = 0.5 as the default.Size normalization makes the high-VA contribution depend on its group size and λ rather than the low-VA group or total rollout length.
  • 3.3 Overall Objective: Combining rollout weights with grouped token KL yields the per-prompt VA-OPD loss without requiring extra annotations, reward models, or student rollout changes.The teacher performs one additional forward pass per rollout without fine-grained visual detail for VA computation, while inference remains unchanged.

4 Experiments

The experiments evaluate VA-OPD across benchmarks, teacher sizes, data scales, component ablations, training trajectories, and efficiency. VA-OPD consistently improves over Standard OPD, with gains linked to stronger visual reliance and complementary contributions from both components.

  • 4.2 Main Results: VA-OPD improves over Standard OPD on all eight benchmarks, including mathematical reasoning and visual-understanding tasks.The primary 8B→2B Geometry3K comparison reports gains from +0.2 on MMStar to +3.3 on WeMath, with +2.9 Math Avg and +1.5 Visual Avg.
  • 4.2 Main Results: The largest visual-understanding gains occur on HallusionBench (+2.5) and AI2D (+2.4), where performance depends on fine-grained visual detail.Benchmarks admitting text-only shortcuts show smaller gains, including MMMU (+0.6) and MMStar (+0.2).
  • 4.3 Scaling Across Data and Teacher Size: VA-OPD’s gains grow with both training-data scale and teacher size: Math Avg rises from +2.9 to +3.8 with ViRL39K and reaches +3.7 with a 32B teacher.With teacher size varied, Math/Visual Avg gains are +2.1/+0.8 at 4B, +2.9/+1.5 at 8B, and +3.7/+2.0 at 32B.
  • 4.4 Ablation Study and Controls: Both rollout-level reweighting and token-level grouped KL contribute to the full VA-OPD gain, while removing either component measurably degrades performance.Full VA-OPD reaches +2.8 on MathVerse, +2.5 on HallusionBench, and +1.7 on OCRBench.
  • 4.4 Ablation Study and Controls: VA-OPD increases accuracy and all-token mean VA together, whereas Standard OPD raises accuracy while mean VA remains nearly flat.Standard OPD moves from 19.6 to 29.1 accuracy with mean VA 0.07→0.10; VA-OPD reaches 31.9 with mean VA 0.07→0.16.
  • 4.5 Training Efficiency: VA-OPD reaches Standard OPD’s final accuracy in ∼6.5 h versus 19.3 h, a ∼3.0× wall-clock speedup, and reaches a higher fixed-time ceiling.On the 8B→2B Geo3K configuration, VA-OPD reaches 31.9, which is +2.8 points above Standard OPD’s endpoint of 29.1.

5 Qualitative Token-Level VA Visualization

Token-level visualization shows that VA-OPD selectively increases visual dependence on diagram-derived values, while Standard OPD leaves those tokens visually unchanged.

  • Token-Level Visualization: VA-OPD develops high-VA hotspots on diagram-extracted values, whereas the initial and Standard OPD students keep those vision-critical tokens pale.The hotspots include specific angle measures and side lengths; language-template tokens remain pale across students.
  • Token-Level Visualization: Language-template tokens such as connectors, fillers, and equation rewrites remain low-VA across all three students.This contrasts with the VA-OPD hotspots on numerical values that must be read from the geometry diagram.

6 Related Work

VA-OPD connects on-policy distillation with visual-dependency modeling by making fine-grained visual reliance part of the VLM training target.

  • Visual Dependency in VLMs: VLMs can rely on language priors instead of fine-grained visual detail, motivating methods that explicitly address visual dependency.Prior mitigations include contrastive decoding, preference-based fine-tuning, and RL-based perception methods.
  • On-Policy Distillation for LLMs: Text-only on-policy distillation methods use token-uniform supervision, whereas VA-OPD targets the sparse concentration of visual dependence in VLM rollouts.This distinction extends on-policy distillation from language models to vision-language models.
  • Positioning: VA-OPD modifies the training target without test-time overhead and uses teacher supervision rather than verifiable rewards.The method is positioned between inference-time visual-dependency methods and RL-based perception methods.

7 Conclusion

The paper concludes that VA-OPD addresses diluted visual supervision by separating visual dependency from surrounding language scaffolding, improving performance across scales while strengthening visual reliance. VA remains a proxy rather than a direct measurement of internal perceptual processes, and gains are strongest on vision-intensive tasks.

  • 7 Conclusion: VA-OPD uses rollout-level reweighting and separate high-VA and low-VA token-level KL averaging to preserve sparse visual supervision.These mechanisms redirect gradient mass toward visually dependent rollouts and prevent high-VA token contributions from being diluted by language scaffolding.
  • 7 Conclusion: VA-OPD consistently improves over standard on-policy distillation across three teacher sizes, two training corpora, and eight benchmarks, with gains increasing as teacher and corpus scale.
  • 7 Conclusion: VA is a useful proxy for visual dependency, but it does not directly measure internal perceptual processes such as attention allocation or feature grounding.
  • 7 Conclusion: The strongest gains occur on vision-intensive tasks, while improvements are more modest on benchmarks containing many questions solvable from text alone.The paper gives MMStar as an example of a benchmark where improvements are more modest.
  • 7 Conclusion: Future work could combine VA-OPD with reinforcement-learning methods and extend VA from token-level to region- or object-level visual dependency.The proposed combination would pair stronger visual dependency during distillation with task-reward optimization.
Loading 2605.21924v1…