PaperlayerView exact revision on arXiv

Source-linked AI summary

Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models

Yuanhao Sun, Huawei Ji, Jiaxin Ding, Luoyi Fu, Xinbing Wang

arXiv:2609.38777v1cs.CV

TL;DR

VLM distillation often matches outputs without ensuring that students rely on answer-critical visual evidence. CW-OPD trains on paired visual worlds and matches the teacher’s belief transition between them. Across model scales, it improves benchmark performance, with a 1.16-point gain over OPSD on Qwen3.5-0.8B.

  • Problem

    Matching the teacher’s output distribution in one visual world does not ensure that a student relies on answer-critical evidence, leaving room for world-invariant shortcuts.

    Source: §1 · p. 2, §5 · p. 9

  • Method

    CW-OPD uses semantically symmetric dual-world pairs and supervises per-world predictions and the cross-world belief transition; CWBench measures cross-world consistency through CWPA and CWFR.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    CW-OPD reaches an average score of 68.81 on Qwen3.5-0.8B, 1.16 points above FP-OPD (67.65), and 80.97 on Qwen3.5-4B, 1.16 points above OPSD (79.81).

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    The cross-world objective makes the student’s response to changes in visual evidence an explicit supervision target.

    Source: §3.2 · p. 5, §5 · p. 9

Abstract

from arXiv · show

A central goal of vision-language model (VLM) distillation is to transfer both the teacher's language capabilities and its visual understanding. However, existing methods primarily supervise the student's output, leaving visual understanding implicit. Our analysis reveals that a student can match the teacher's answer without relying on the same visual evidence, raising the question: how can we ensure the student responds to the visual information that actually determines the answer? To this end, we propose \textbf{Cross-World On-Policy Distillation (CW-OPD)}, which explicitly supervises the student's response to changes in visual evidence. For each example, CW-OPD constructs two visual worlds that share the question and scene context but differ in answer-critical evidence, yielding different answers. We perform on-policy distillation in both worlds and distill the teacher's cross-world belief transition, encouraging the student to match not only \emph{what} the teacher predicts but also \emph{why} its prediction changes with the evidence. A gradient analysis shows that this term is invariant to errors shared by both worlds and supplies a corrective signal invisible to endpoint matching alone. In this way, CW-OPD makes reliance on the relevant visual evidence an explicit distillation target rather than an implicit consequence of output matching. To diagnose whether a model truly grounds its answers in visual evidence, we introduce CWBench, which measures cross-world consistency via Cross-World Pair Accuracy (CWPA). Experiments on Qwen3.5-4B show that CW-OPD outperforms the strongest baseline by \textbf{1.2} points on average, and the 4B student exceeds DeepSeek-V4.1 (552B) by \textbf{22.4} CWPA points on CWBench. Code is released in https://github.com/baokou-fw2/CWAD.

1 INTRODUCTION

On-policy distillation can align teacher and student outputs, but it does not ensure that a VLM student relies on the answer-critical visual evidence. CW-OPD makes the student’s response to changes in that evidence an explicit training target.

1 INTRODUCTION

  • On-policy distillation aligns teacher and student state distributions, but transferring visual capabilities through textual generation remains challenging for VLMs.VLMs rely jointly on textual and visual information.Source: §1 · p. 1
  • Cross-modal attention, sensitivity to masked regions, and responses to semantically edited worlds reveal mismatches between teacher and student evidence use.Under semantically edited worlds, the student fails to distinguish between them while the teacher shifts toward the correct answer.Source: Figure 1 · p. 2, §1 · p. 2
  • CW-OPD trains on paired visual worlds that share the question and scene context but differ in answer-critical evidence and answer.It performs on-policy distillation in both worlds and matches the teacher’s cross-world belief transition.Source: §1 · p. 2
  • Existing on-policy distillation can match teacher outputs without ensuring that the student relies on answer-critical visual evidence.Students may instead use linguistic or other evidence shortcuts.Source: §1 · p. 2
  • Experiments across diverse benchmarks show that CW-OPD improves visual grounding and downstream performance, including cross-world consistency over the strongest baseline and frontier models.The supplied passage does not provide the numerical margins for these comparisons.Source: §1 · p. 2

2 RELATED WORK

On-policy distillation supervises teacher responses to student-generated trajectories, while multimodal variants strengthen visual supervision through privileged views or modified objectives. CW-OPD differs by matching the teacher’s belief transition across semantically contrasted worlds.

2.1 ON-POLICY DISTILLATION

  • On-policy distillation evaluates the teacher on student-generated trajectories, providing token-level supervision on visited states.Prior OPD work explores divergences and sampling strategies for output-level knowledge transfer.Source: §2.1 · p. 3

2.2 MULTIMODAL OPD: PRIVILEGED VIEWS AND VISUAL OBJECTIVES

3 METHOD

CW-OPD addresses the evidence ambiguity of standard on-policy distillation by pairing visual worlds and supervising both their predictions and the transition between them. Its transition analysis shows how this adds a corrective signal beyond endpoint matching.

3.1 REVISITING ON-POLICY DISTILLATION

  • Standard OPD gives token-level teacher feedback along student-generated trajectories, but matching marginal outputs leaves the visual evidence underlying a prediction unconstrained.A correct output can therefore arise from different evidence distributions, including non-critical visual cues or linguistic priors.Source: §3.1 · pp. 3–4

3.2 CROSS-WORLD ON-POLICY DISTILLATION

  • CW-OPD pairs worlds that preserve the question and scene but change answer-critical evidence, then matches predictions in both worlds and the teacher’s cross-world belief transition.The transition compares per-token log-likelihood ratios and captures the direction and magnitude of belief change under the visual intervention.Source: §3.2 · pp. 4–5

3.3 WHEN AND WHY DOES TRANSITION SUPERVISION HELP?

  • Transition gradients reject common-mode shifts and act only on parameters whose responses differ across worlds, supplying first-order descent for cross-world mismatch at an endpoint-stationary point.The endpoint loss changes only at second order in this condition.Source: §3.3 · p. 5

4 EXPERIMENTS

CW-OPD improves benchmark performance and cross-world consistency, while CWBench evaluates whether answers remain correct when answer-critical visual evidence changes. Ablations indicate that per-world distillation and full transition matching are both needed to improve evidence-grounded answering.

4.1 EXPERIMENTAL SETUP

  • Training uses 2,365 CW-Bench dual-world pairs; evaluation spans visual grounding, hallucination diagnosis, external-knowledge reasoning, and CWBench.CW-OPD is instantiated atop OPSD with an EMA teacher; all student parameters, including the vision encoder, are trainable.Source: §4.1 · p. 6
  • 80.97 average on Qwen3.5-4B beats Vision-OPD (78.92) by 2.05 points; CWPA rises 18.2 points, from 47.64 to 65.82.On Qwen3.5-0.8B, CW-OPD reaches 68.81, exceeding FP-OPD (67.65) by 1.16 points; gains are reported across both model scales.Source: §4.1 · p. 7

4.2 THE CWPA BENCHMARK

  • CWBench retains 2,959 verified pairs from 5,079 candidates, with answer-critical evidence changed while the question and broader scene context remain fixed.Its edits cover color, presence or absence, shape, object count, spatial relations, and commonsense changes; paired questions focus mainly on fine-grained perception and counting.Source: §4.2 · p. 7, Figure 3 · p. 7
  • A 4B CW-OPD student reaches 65.8 CWPA and 29.3 CWFR, outperforming the compared frontier models on both metrics and achieving 73.6 CWPA-Q.CWPA requires both worlds to be answered correctly, while CWFR measures pairs whose correctness differs between worlds.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 ABLATION STUDY

  • Per-world OPD improves single-world accuracy but leaves cross-world consistency nearly unchanged; adding transition supervision yields gains in accuracy, CWPA, and downstream benchmarks.Extending OPD to both worlds without transition loss can widen the gap between the worlds; the two supervision components address endpoint predictions and their evidence-driven change, respectively.Source: §4.3 · p. 8, Table 3 · p. 8
  • Cosine and magnitude alignment each preserve only part of the transition, while KL alignment matches the teacher’s transition as a distribution and improves ATE, CWFR, and downstream accuracy.The visual-evidence ablations also distinguish answer-relevant supervision from BBox grounding or masked-answer consistency.Source: §4.3 · p. 8, Figure 5 · p. 9, Figure 4 · p. 9
  • CWPA rises under CW-OPD as transition loss decays, while SFT, OPD, and OPSD plateau near or below the pretrained level.The reported pattern indicates that CWPA is optimized by the cross-world objective rather than improving as a by-product of single-world fitting.Source: §4.3 · p. 9, Figure 5 · p. 9

5 CONCLUSION

The conclusion frames CW-OPD and CWBench as addressing shortcut reliance by supervising cross-world evidence changes. The appendix details how dual-world pairs are proposed, edited, and verified.

5 CONCLUSION

  • CW-OPD makes answer-critical evidence reliance an explicit target; the authors propose extending cross-world supervision to reinforcement learning, potentially reducing reliance on image editing.The conclusion also notes that this could cover evidence types that are hard to alter faithfully.Source: §5 · p. 9

A CW-BENCH

  • The initial CW-Bench pool contains 5,079 samples from five VQA sources, restricted to multiple-choice tasks whose ground-truth answer appears among the options.The sources listed include A-OKVQA and Visual7W.Source: A CW-BENCH · p. 11

A.1 COUNTERFACTUAL EDIT INSTRUCTION GENERATION

  • Counterfactual edit instructions target one object or attribute, make the distractor unambiguously true and the ground-truth answer false, and preserve the rest of the scene.Tiny or invisible details and edits requiring a new viewpoint or major scene restructuring are rejected.Source: §A.1 · p. 12

A.2 IMAGE EDITING

  • 4,396 of 5,079 attempted image edits succeeded (86.6%) using the Qwen-Image-3.0 API.The reported editing cost was approximately 643 RMB.Source: §A.2 · p. 13

A.3 QUALITY CONTROL AND VERIFICATION

  • A VLM verifier accepts pairs only when the answer flips and the question and non-critical scene context remain preserved; quality control yields 2,959 verified pairs.Rejected pairs are sent back for instruction rewriting.Source: §A.3 · p. 14

A.4 EDIT AND QUESTION TYPE CLASSIFICATION

The appendix summarizes CW-Bench's edit and question classification, while the algorithm appendix specifies a CW-OPD update that matches per-world predictions and cross-world transitions.

A.4 EDIT AND QUESTION TYPE CLASSIFICATION

  • CW-Bench provides 2,959 verified dual-world pairs spanning diverse edit types and question categories; each pair shares the same question and overall scene while exchanging answer-critical evidence to induce different answers.The 2,959 verified edits were automatically classified using Qwen3.5-9B in a pure text classification setting, with zero parsing errors.Source: §A.4 · pp. 14–15

B ALGORITHM

  • Each CW-OPD update reuses a student rollout prefix across both worlds, computes per-world distillation losses, and matches the teacher’s cross-world belief transition.The final loss weights the two terms; setting λ = 0 recovers DW-OPD with identical rollouts, scoring, masks, and reductions.Source: B ALGORITHM · p. 15

C.1 PROOF OF PROPOSITION 1

The appendix establishes that shared logit shifts leave the transition target unchanged and derives the transition and endpoint loss gradients. It also qualifies the shortcut argument for approximate stationarity.

C.1 PROOF OF PROPOSITION 1

  • Adding the same vector to both worlds’ teacher logits leaves the transition logits and target unchanged, while changing each world’s endpoint target.The transition difference cancels the shared addition; the endpoint target is a softmax of each world’s logits plus that vector.Source: §C.1 · pp. 15–16

C.2 DERIVATION OF EQ. EQUATION 8

  • The transition-loss gradient is driven by the student–teacher transition-logit difference and vanishes when that difference is only a constant shift.The softmax removes constant shifts, so a nonzero gradient requires a difference outside the span of the all-ones vector.Source: §C.2 · p. 16
  • With approximate stationarity, the shortcut argument adds an endpoint-loss term of order ηε; the transition loss still decreases to first order when λ∥∇T∥2 > ε.This qualifies the exact-stationarity argument by allowing a small nonzero gradient.Source: §C.2 · p. 16

C.3 STRUCTURED TEACHER PERTURBATIONS (TARGET-INVARIANCE SANITY CHECK)

Structured teacher-logit perturbations produce opposite robustness orderings for DW-OPD and CW-OPD under shared versus opposite corruption. The experiment tests the target-level invariance, not the gradient-level claim or stationary-point argument.

C.3 STRUCTURED TEACHER PERTURBATIONS (TARGET-INVARIANCE SANITY CHECK)

  • The perturbation check adds a shared random logit direction to both worlds or opposite directions to the two worlds, then reports CWPA averaged over three seeds.The setup uses a Qwen3.5-4B student and frozen Qwen3.5-9B teacher.Source: §C.3 · p. 16
  • Shared corruption degrades DW-OPD while CW-OPD is nearly unaffected; opposite corruption reverses the ordering.Figure 6 reports standard-deviation error bars over three seeds.Source: §C.3 · p. 16, Figure 6 · p. 17
  • The experiment supports target-level invariance only: shared perturbations can still affect CW-OPD through the endpoint loss, and it does not test the gradient-level or stationary-point claims.Source: §C.3 · p. 17

D RESULT WITH STANDARD OPD

With a standard larger teacher, CW-OPD carries over the main-text findings, improving standard-benchmark performance and more substantially improving cross-world diagnostics. The setting uses Qwen3.5-9B to teach Qwen3.5-4B while holding the cross-world training setup fixed.

D RESULT WITH STANDARD OPD

  • The standard-teacher experiment keeps the dual-world pairs, transition objective, and training budget identical to the main experiments.The teacher and student differ in capacity rather than input conditions.Source: D RESULT WITH STANDARD OPD · p. 17
  • CW-OPD improves over other OPD baselines on standard benchmarks and, more substantially, on CWPA and CWFR in the standard-teacher setting.The teacher is Qwen3.5-9B and the student is Qwen3.5-4B.Source: D RESULT WITH STANDARD OPD · p. 17, Table 5 · p. 17
  • The standard-teacher results carry over the main conclusions, indicating that cross-world transition supervision does not rely on the privileged hint or self-distillation.The proposed target is the student's response to evidence changes, regardless of which teacher provides the per-world anchor.Source: D RESULT WITH STANDARD OPD · p. 17

E TRAINING SETTING AND DETAILS

The experiments use a shared on-policy distillation configuration for Qwen3.5-4B and Qwen3.5-0.8B, with an EMA student copy as teacher in OPSD. Method comparisons hold the common training and teacher-update protocol fixed while allowing method-specific hyperparameters to vary.

E TRAINING SETTING AND DETAILS

  • Both model sizes use AdamW, a constant learning rate of 2 × 10^-6, weight decay 10^-2, one epoch, an unfrozen vision encoder, and the same rollout and sequence-length limits.Global batch size is 48 for 4B and 64 for 0.8B.Source: E TRAINING SETTING AND DETAILS · p. 18
  • For OPSD, the teacher is a frozen initial copy of the student updated after each optimizer step as an exponential moving average.The EMA decay is 0.95; buffers are not copied, and CW-OPD uses transition weight λ = 1.0.Source: E TRAINING SETTING AND DETAILS · p. 18
  • Under OPSD, the 4B run takes 2.34 hours (82 PFLOPs) and the 0.8B run takes 1.59 hours (15 PFLOPs) on two NVIDIA RTX PRO 6000D GPUs.The corresponding OPD runs take 3.07 hours (108 PFLOPs) and 2.78 hours (36.2 PFLOPs), respectively.Source: E TRAINING SETTING AND DETAILS · p. 18
  • All methods share the same base training configuration and EMA teacher-update setting, while only method-specific hyperparameters vary.This protocol is intended to make performance differences reflect the distillation objective rather than training or teacher-update differences.Source: E TRAINING SETTING AND DETAILS · p. 18

F ABLATION OF λ

The transition-weight sweep shows that moderate λ improves performance over λ = 0, while larger weights reduce it; the local analysis predicts a qualitative pattern but does not guarantee a peak. The main experiments use λ = 1.0.

F ABLATION OF λ

  • Performance rises from λ = 0 to a moderate-weight peak, then declines as λ increases further, while remaining above the λ = 0 baseline throughout.The peak is at λ = 1.0 for OPD and λ = 0.25–1.0 for OPSD; each entry averages 3 seeds.Source: F ABLATION OF λ · p. 18
  • The local analysis predicts that a small λ adds a corrective transition signal while endpoint error changes only at O(η^2), but does not guarantee an interior optimum or performance-curve shape.As λ grows, the transition loss can dominate and endpoint fit may be sacrificed.Source: F ABLATION OF λ · p. 18
  • The main experiments use λ = 1.0 for CW-OPD.The sweep supports this choice empirically.Source: F ABLATION OF λ · p. 18

G STANDARD BENCHMARK DETAILS

The appendix describes eight standard benchmarks grouped by visual perception and grounding, hallucination diagnosis, and external-knowledge reasoning, alongside their evaluation settings. It also identifies the hyperparameter tables for the two model sizes.

G STANDARD BENCHMARK DETAILS

G.1 VISUAL PERCEPTION AND GROUNDING

  • V*Bench tests detail-finding in cluttered scenes, while HR-Bench 4K assesses perception on true 4K images and the paper uses its 4K split.RealWorldQA measures basic visual understanding in everyday scenes; MMVP probes visual shortcomings using CLIP-blind image pairs.Source: §G.1 · pp. 19–20

G.2 HALLUCINATION DIAGNOSIS

  • POPE probes object hallucination with targeted presence questions across random, popular, and adversarial sampling settings, while HallusionBench targets entangled language hallucination and visual illusion.HallusionBench uses 346 images and 1,129 questions; its paired-question structure tests grounding against language priors.Source: §G.2 · pp. 20–21

G.3 EXTERNAL-KNOWLEDGE REASONING

  • SEED-Bench covers image and video comprehension across 12 dimensions, while OK-VQA tests questions requiring external knowledge beyond the image.OK-VQA questions require extracting visual cues, retrieving knowledge, and integrating perception with knowledge.Source: §G.3 · p. 21

G.4 EVALUATION SETTINGS.

  • Responses are scored with Qwen3.5-9B as a greedy-decoding judge, with rule-based matching first where applicable and the judge used for unparsed or open-ended responses.The prompt asks whether the response expresses the same meaning as the answer and returns Yes or No.Source: §G.4 · p. 21
Loading 2609.38777v1…