PaperlayerView exact revision on arXiv

Source-linked AI summary

ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation

Kanghui Tian, Siyuan Liu, Ziang Yan, Sheng Xia, Shuai Dong, Yi Wang

arXiv:2606.05718v1cs.CVcs.AIcs.LG

TL;DR

Multimodal answer-privileged OPD can create a train-test mismatch because teacher supervision depends on signals unavailable to the student. ViCuR replaces answer-side privilege with recoverable visual cues and uses sink-token cross-attention for internal cue recovery. Across seven benchmarks, it improves answer-based OPSD and stronger-teacher OPD baselines, while cue quality and recovery optimization remain limitations.

  • Problem

    Answer- or rationale-based teacher privilege can depend on inference-unavailable information, encouraging shortcut imitation instead of visually grounded reasoning.

  • Method

    ViCuR replaces answer-side privilege with visual cues derived from the input and adds a sink-token cue recovery module trained through on-policy distillation.

  • Results

    +1.19 and +1.24 overall average gains over answer-based OPSD across seven benchmarks, plus +0.64 and +1.08 over stronger-teacher OPD baselines.

  • Takeaways & Limitations

    Designing teacher privilege around inference-recoverable visual evidence is as important as teacher strength in multimodal distillation.

  • Takeaways & Limitations

    Performance depends on visual-cue quality, and the recovery module adds optimization complexity whose benefit is not monotonic across model scales.

Abstract

from arXiv · show

On-policy distillation (OPD) improves reasoning by training a student on trajectories sampled from its own policy under supervision from a teacher. In multimodal reasoning, a common extension is to use a privileged teacher that observes training-time-only signals such as reference answers or rationales. However, such answer-side privilege creates a train-test mismatch: the teacher's supervision may depend on signals unavailable to the student, encouraging shortcut imitation rather than visually grounded reasoning. We propose ViCuR, a visually grounded privileged-teacher distillation framework that replaces answer-side privilege with visual cues (query-related evidence in the input). Because these cues are derived from the same visual input available at inference, their evidence is recoverable by the student. To support this, ViCuR introduces a lightweight cue recovery module that uses dedicated sink-token cross-attention during prefill to aggregate task-relevant visual evidence into an internal representation, without changing the inference interface or requiring auxiliary cue-generation losses. Across seven benchmarks with Qwen3-VL-2B and 8B students, ViCuR consistently improves over answer-based on-policy self-distillation by +1.19 and +1.24 on overall average performance. It also extends naturally to stronger-teacher OPD, surpassing OPD baselines by +0.64 and +1.08, with consistent out-of-domain gains at the 8B scale. These results show that, in multimodal on-policy distillation, the design of teacher privilege is as important as teacher strength.

1. Introduction

Multimodal OPSD often gives teachers answer-side information unavailable at inference, creating a mismatch that can weaken visually grounded reasoning. ViCuR replaces this privilege with recoverable visual cues and evaluates the approach across multimodal benchmarks and teacher configurations.

  • Motivation: Answer- or rationale-based privilege can create a train-test mismatch because the teacher uses information unavailable to the student at inference.This may encourage answer-aware shortcut patterns rather than reasoning grounded in visual evidence.
  • ViCuR: ViCuR replaces inaccessible answer-side privilege with question-relevant visual cues whose underlying evidence is present in the standard visual-question input.The framework therefore preserves privilege while making its evidence recoverable by the student.
  • Cue recovery: A lightweight sink-token cross-attention module aggregates task-relevant visual information into an internal cue representation during prefill.It is trained under the distillation objective without changing the inference interface or requiring auxiliary cue-generation losses.
  • Empirical results: +1.19 and +1.24 overall average gains over answer-based OPSD across seven benchmarks with Qwen3-VL-2B and 8B students.ViCuR also improves stronger-teacher OPD baselines by +0.64 and +1.08, with consistent out-of-domain gains at 8B.
  • Scope: Teacher scaling and privilege design are treated as orthogonal, allowing ViCuR to extend from privileged-teacher self-distillation to stronger-teacher OPD.The reported stronger-teacher gains are +0.64 and +1.08 over OPD baselines.

2. Related Work

Prior work established on-policy and privileged-information distillation, while attention-sink methods provide a related mechanism for retaining contextual information. ViCuR connects these lines through visually grounded privilege and sink-based cue recovery.

  • On-Policy Distillation and Self-Distillation: OPD supervises trajectories sampled from the student, while OPSD uses one backbone for both teacher and student and strengthens supervision with additional signals.These approaches target alignment with the student’s inference-time distribution while reducing reliance on a stronger external teacher.
  • Privileged Information for Distillation: Distillation with privileged information commonly uses gold answers, rationales, verifier scores, or environment feedback unavailable at test time.Recent alternatives include conciseness instructions, context-conditioned prompts or historical traces, and evidence-centered regional crops.
  • Attention Sinks: Attention sinks are special tokens that attract high attention and can act as persistent information anchors across transformer layers.SinkTrack uses a sink token to aggregate and retain contextual information during long-sequence generation without architectural changes.

3. Method

ViCuR replaces answer- or rationale-based teacher privilege with visually grounded cues whose underlying evidence remains available in the inference input. A sink-token recovery module extracts this evidence during prefill and trains jointly through on-policy distillation without changing the inference interface or adding cue-generation losses.

  • 3.1. Problem Formulation: ViCuR uses visual cues—question-relevant evidence from the input—as privileged teacher information instead of answer- or rationale-based signals.The teacher receives cue text, while the student uses only the visual input and question; the cue’s underlying evidence remains recoverable from that input.
  • 3.1. Problem Formulation: Under the idealized mapping S=f(X), cue-conditioned supervision depends only on information determined by the inference-time multimodal input.The paper notes that real cue generation may leave a residual gap because S need not be strictly deterministic given X.
  • 3.2. ViCuR: Sink-Based Cue Recovery: ViCuR preserves the standard multimodal inference interface by recovering evidence internally rather than appending cue text or requiring cue generation before answering.This avoids the added generation overhead associated with explicit cue production.
  • 3.2. ViCuR: Sink-Based Cue Recovery: The student-side recovery module uses dedicated sink-token cross-attention during prefill to aggregate task-relevant visual states into an internal cue representation.The updated sink state is visible to subsequent question and answer tokens through standard causal attention, and dedicated parameters learn question-relevant visual selection.
  • 3.3. Training Objective: The recovery module is trained jointly with the student through the on-policy distillation objective, without auxiliary cue-generation or text-matching losses.The teacher evaluates tokens on student-sampled trajectories, while gradients through the student representation update the recovery module under the PPO-style distillation surrogate.
  • 3.4. Implementation: The module runs only during prefill, is inserted every five transformer layers, and adds no autoregressive decoding overhead with modest parameter overhead.It uses the full current-layer visual token sequence as keys and values.

4. Experiments

Across seven multimodal benchmarks, ViCuR improves answer-based OPSD and stronger-teacher OPD, with gains varying by task domain, model scale, and teacher strength. Ablations and training analyses attribute the improvements primarily to visually grounded privilege, complementary cue recovery, reduced hint leakage, and better alignment.

  • Main Results: +1.19 at 2B and +1.24 at 8B overall average performance over answer-based OPSD, with consistent improvements across benchmarks.The main results cover in-domain, near-domain, and out-of-domain evaluation at both student scales.
  • Main Results: +0.64 at 2B and +1.08 at 8B overall average performance over stronger-teacher OPD, with gains on five of seven and six of seven benchmarks.MathVista drops by -0.8 and -1.4 because computation-dominated subsets remain challenging.
  • Main Results: +2.30 at 2B and +2.23 at 8B on MMMU-Val under OPD+ViCuR, alongside improvements on Video-MME, indicating stronger out-of-domain transfer.These gains extend beyond the mathematical reasoning training distribution to broader multimodal and video reasoning.
  • Ablation Studies: +4.80 for 2B and +4.65 for 8B from the full component combination on Geometry3K, while visual cues alone provide the largest individual improvement.Visual cues contribute +2.80 and +4.36, whereas cue recovery alone contributes +1.17 and +0.09, showing complementary roles.
  • Scaling Behavior: +6.88 is the largest scaling gain, achieved by an 8B student distilled from a 32B teacher, while extra teacher strength is harder to exploit for a bottlenecked 2B student.The results suggest visual-cue grounding and teacher scaling are complementary when student capacity is sufficient.
  • Training Dynamics: ViCuR reduces both affected samples and total “Hint” occurrences while maintaining higher test accuracy despite initially larger teacher-student gaps.Its smaller back-half gap also indicates improved downstream answer-stage alignment, and the recovery module adds no autoregressive decoding overhead.

5. Conclusion

ViCuR replaces answer-based privilege with visually grounded cues and adds sink-based recovery for internal evidence aggregation. Across seven benchmarks and two student scales, it improves both answer-based OPSD and stronger-teacher OPD, especially on visually grounded reasoning tasks.

  • Conclusion: ViCuR replaces answer-based privilege with visually grounded cues and equips the student with a lightweight sink-based recovery module.The framework centers privilege on evidence recoverable from the inference-time visual input.
  • Conclusion: Across seven benchmarks and two student scales, ViCuR consistently improves answer-based OPSD and extends effectively to stronger-teacher OPD.The strongest gains occur on visually grounded reasoning tasks.
  • Conclusion: Teacher privilege design is as important as teacher strength in multimodal distillation when privilege is based on inference-recoverable visual evidence.This conclusion is stated within the evaluated multimodal distillation setting.

6. Limitations

ViCuR reduces answer-based privilege mismatch by using visually grounded cues, but its effectiveness remains bounded by cue quality and recovery-module optimization.

  • Cue quality limits ViCuR: missed evidence, non-visual information, or ambiguous descriptions can make teacher supervision less reliable.
  • The cue recovery module adds trainable parameters and optimization complexity, with benefits that do not increase monotonically across model scales.For the 8B Geometry3K student, full ViCuR slightly underperforms visual-cue-only supervision despite improving over the corresponding baselines.

A. Training and Evaluation Details

The experiments use standardized training and evaluation configurations across main Vision R1 comparisons and controlled Geometry3K ablations.

  • Table 3 summarizes the key training configurations for the main Vision R1 experiments and controlled Geometry3K ablations.
  • Main comparisons train methods on the same split and evaluate them on the same benchmark suite, varying the learning objective and supervision source.
  • Controlled ablations use Geometry3K, while evaluation covers MathVista_MINI, MathVerse_MINI, WeMath, DynaMath, MMMU_DEV_VAL, and Video-MME_64frame.
  • Distillation experiments use sampled-token on-policy distillation with teacher privileges supplied through teacher_prompt and images supplied through images.
  • Table 4 summarizes the generation and preprocessing settings used during evaluation with VLMEvalKit.

B.1. Main Component Analysis

Component analyses show that visual-cue supervision transfers beyond Geometry3K, scales across teacher–student configurations, and can produce strong reasoning without easier token-level imitation.

  • Cross-Benchmark Evaluation: Visual-cue supervision consistently improves transfer over answer-based supervision on geometry subsets of MathVista and DynaMath.
  • Cross-Benchmark Evaluation: In 2B OPSD, visual cues raise MathVista geometry reasoning from 67.36 to 73.22, geometry problem solving from 68.75 to 75.48, and DynaMath plane geometry from 51.82 to 54.55.
  • Cue Recovery under Cross-Benchmark Evaluation: Cue recovery adds gains in several transfer settings but does not uniformly outperform visual-cue-only supervision, especially across differing target styles and cue granularities.For the 8B student, the recovery module adds 536.9M parameters, or 5.77% of the model, compared with 100.7M and 4.52% for the 2B student.
  • Scaling Behavior across Teacher-Student Configurations: +3.35 to +6.88 gains occur across all four teacher–student configurations, with +6.88 for an 8B student distilled from a 32B teacher.For a 2B student, the stronger 8B teacher yields +3.35 versus +4.80 for self-distillation, indicating that extra teacher strength is harder to exploit when the student is the bottleneck.
  • Token-Level Log-Probability Dynamics: On Geometry3K, answer-based supervision can produce a smaller teacher–student token gap yet much lower accuracy, showing that easier token imitation does not necessarily yield better multimodal reasoning.
  • Segment-Level Log-Probability Analysis: ViCuR often matches or reduces the baseline token gap across rollout segments, and its benefit is not confined to the final answer segment.

B.4. Computational Overhead Analysis

ViCuR adds modest training overhead and no autoregressive decoding overhead, with analyses reporting timing under multiple input resolutions and token-level training dynamics.

  • Training Time: ViCuR introduces only modest wall-clock training overhead over the corresponding baseline at both model scales.Table 6 measures five epochs and 275 steps on a single node with 8 H200 GPUs.
  • Training Dynamics: The computational analysis compares ViCuR against baselines through training-time accuracy and teacher–student log-probability gaps under full and segmented rollouts.
  • Inference Time: Table 7 reports prefill time, per-token decoding time, average output length, and total generation time for small- and large-image inputs.The small-image condition uses 88 total tokens and 64 visual tokens; the large-image condition uses 1,821 total tokens and 1,782 visual tokens.
  • Inference Time: Per-token decoding time remains comparable because the recovery module operates only during prefill and is inactive during autoregressive generation.
  • Inference Time: Because decoding dominates total inference time, per-token decoding time is the comparable metric across configurations.In the 2B large-image case, decoding accounts for more than 98% of total generation time.

D. Hint Leakage Statistics in Student Rollouts

The analysis measures explicit hint reproduction in student rollouts and compares answer-based privilege with visually grounded cues. ViCuR reduces leakage while preserving low absolute counts in stronger-teacher OPD.

  • Hint-leakage statistics scan student rollout outputs for the word “Hint,” reporting both affected samples and total matches across 55 rollout steps per epoch.
  • Qualitative comparisons distinguish answer-bearing reasoning hints from ViCuR cues, which describe only question-relevant visual structure grounded in the image.
  • Answer-based OPSD shows sharply increasing explicit hint reproduction, especially in later epochs, whereas ViCuR substantially reduces this effect at both student scales.
  • In OPD, visual-cue privilege keeps hint leakage at a low absolute level, comparable to the no-privilege OPD baseline.

E.1. Case 1: Geometry3K — Intersecting Chords Theorem

ViCuR improves visual grounding in geometry and chart reasoning by recovering task-relevant relationships from images rather than reproducing answer-bearing reasoning hints. The qualitative cases provide mechanism-consistent evidence across localized and distributed visual cues.

  • Case 1: Geometry3K — Intersecting Chords Theorem: OPSD + ViCuR answers the Geometry3K intersecting-chords example correctly in 10/10 runs, versus 7/10 for answer-based OPSD and 5/10 for the base model.All models invoke the correct theorem, so the decisive difference is recovering the chord-pairing relation between labels and segments.
  • Case 1: Geometry3K — Intersecting Chords Theorem: ViCuR corrects the critical grounding step by binding x and 8 to one chord and 4 and 6 to the other before applying algebra.Its reasoning follows a visual-cue format describing visible segment relations rather than directly exposing the theorem or answer.
  • Case 2: MathVista — Generalization Beyond Geometry: In MathVista, ViCuR identifies RFT U13B as highest at Pretrain Loss = 1.80, whereas the base model and OPSD baseline select LLaMA-2-70B.The chart comparison uses the queried x-axis slice, series identities, and relative vertical positions; RFT U13B reaches 50.0 Acc.
  • Case 1: Geometry3K — Intersecting Chords Theorem: The Geometry3K SinkTrack maps repeatedly attend to numerical labels, the intersection, and adjacent chord segments across layers rather than producing a single-layer spike.This pattern is consistent with internal recovery of the visible relation needed to use the theorem.
  • Case 2: MathVista — Generalization Beyond Geometry: Chart-task attention is distributed across the queried coordinate, legends, and multiple curves, reflecting multi-region evidence aggregation rather than a localized hotspot.The visualizations are mechanism-consistent evidence, not causal proof that the exact target curve was isolated.

F.1. Derivation and Analysis of Proposition 1 (Recoverable Privilege)

The recoverable-privilege analysis models visual cues as information determined by the inference-time input, eliminating the additional conditional information gap under a deterministic abstraction. It then explains how distillation gradients shape the recovery representation, while noting practical and analytical limits.

  • Recoverable Privilege: Under the deterministic abstraction S = f(X), visual cues are determined by inference-time input X and introduce no additional conditional information gap.The result contrasts with answer-dependent privileges that may rely on variables not recoverable from X.
  • Recoverable Privilege: The analysis does not imply that students observe or perfectly reconstruct cue text; it establishes only that the cue is determined by the available input.The remaining challenge is learning to extract and represent relevant visual evidence effectively.
  • Recoverable Privilege: With stochastic cue construction, H(S|X) > 0 and the conditional information term remains nonzero, although it is expected to be smaller than for answer-based privilege.The method shapes internal visual representations through cue-conditioned behavior rather than requiring token-level cue reconstruction.
  • Task-Driven Implicit Alignment: The recovery module receives task-driven gradients through its effect on student predictions under the teacher-shaped token advantage.Positive advantages reinforce evidence representations supporting teacher-preferred tokens, while negative advantages push them away.
  • Task-Driven Implicit Alignment: The gradient analysis uses a simplified unclipped surrogate, whereas implementation uses PPO-style clipping and multilayer sink updates that can alter precise dynamics.The qualitative conclusion about task-driven alignment remains unchanged under these simplifications.
Loading 2606.05718v1…