CV-OPD: Contrastive Vision On-policy Distillation

1 INTRODUCTION

• On-policy distillation transfers teacher capabilities by matching teacher distributions over student-generated outputs, addressing token-level supervision gaps in reinforcement learning.The introduction describes OPD as alleviating training–inference distribution mismatch.Source: §1 · p. 1
• Under black masking and Gaussian noise, Vision-OPD achieves 43.46% and 48.69% accuracy on V∗, respectively, substantially exceeding the random rate for multiple-choice questions without necessary visual evidence.Source: §1 · p. 2
• CV-OPD uses negative visual samples and a margin-based contrastive loss to steer students away from irrelevant evidence, with DAG selecting reasoning-relevant tokens for constraint.The negative teacher receives an image region unrelated to the answer, while DAG directs optimization toward useful visual content.Source: §1 · p. 2
• CV-OPD consistently improves over vanilla OPD across fine-grained benchmarks and over the base model across visual benchmarks; masked V* shows stronger evidence reliance.The introduction reports substantial improvements and says masked V* experiments show substantially strengthened reliance on visual evidence.Source: §1 · pp. 2–3
• The authors attribute this shortcut primarily to exposing answer-relevant visual information directly to the teacher during distillation.They report that this exposure lets the VLM bypass visual evidence when producing answers.Source: §1 · p. 2

2.1 PRELIMINARY

• OPD matches the teacher’s token-level output distribution along student-sampled trajectories, using an answer-related crop or zoom as the teacher’s visual hint.During optimization, the rollout and teacher distributions are stop-gradient, and only student parameters are optimized.Source: §2.1 · p. 3

2.2 OBSERVATIONS

• 43.46% and 48.69% accuracy under black masking and Gaussian noise exceed the base model’s 42.41% and 42.93%, while attention does not consistently focus on target regions.Thus, correct answers after masking do not establish that the model identified relevant visual evidence.Source: §2.2 · p. 3, Figure 2 · p. 4

2.3 ANALYSIS

• JSD increases the logit of hint-related target tokens, but imposes no constraint on what the student optimizes visually.Because the mapping from the full image to the answer-related hint is neither differentiable nor observable, optimization can fit available conditions rather than identify the positive region.Source: §2.3 · p. 4

3.1 OVERVIEW

• CV-OPD samples multiple student trajectories, pairs an answer-related positive crop with a size-matched negative crop from the same image, and conditions both teachers on the same student prefix.The negative crop's overlap with the positive crop is constrained by a threshold.Source: §3.1 · pp. 4–5

3.2 CONTRASTIVE DIVERGENCE LOSS

• CDL contrasts full-vocabulary Jensen–Shannon divergences to draw the student toward the positive teacher and away from the negative teacher, with a hinge margin that stops pushing once it is met.The analysis also links CDL to amplified sampling discrepancies that help localize positive regions.Source: §3.2 · pp. 5–6

3.3 DISCREPANCY AWARE GATE

• DAG applies the contrastive term only to tokens whose positive discrepancy exceeds the batch average and disables supervision for an example when all rollouts are incorrect.Its sample-level reliability gate uses rollout correctness because incorrect prefixes can yield noisy teacher corrections.Source: §3.3 · p. 6

3.4 TRAINING OBJECTIVE

• The training objective combines standard distillation with gated CDL, enables distillation only when at least one rollout is correct, and uses wi,t = 1/Ti in practice.λ weights the CDL term.Source: §3.4 · p. 6

4.1 SETUP

• Training uses 6.2K Vision-OPD samples on Qwen3.5-4B, while evaluation spans six fine-grained visual benchmarks and uses eight rollouts per sample.Samples with no correct rollouts are excluded from distillation.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• CV-OPD improves across domains, averaging 2.3% higher performance across metrics than Vision-OPD; all distillation methods trail the base model on HR-Bench 4K.KV caching and teacher sharing limit added training time to approximately 10% relative to Vision-OPD.Source: Table 1 · p. 7, §4.2 · p. 7

4.3.1 Q1: HOW TO CONSTRUCT PIPELINE

• Removing CDL lowers Avg6 from 78.22% to 76.81%, removing DAG lowers it to 76.88%, and removing both yields 75.92%.The combined pipeline reaches 78.22% Avg6.Source: Table 2 · p. 8, §4.3.1 · p. 8

4.3.2 Q2: HOW TO SELECT THE NEGATIVE TEACHER

• CV-OPD reaches 78.22% Avg6, compared with 77.20% for Random Crop, 76.76% for All Black, and 76.40% for Neighbor Crop.The authors suggest sharing one negative crop across eight rollouts reduces variation in contrastive gradients and provides more consistent optimization supervision.Source: Table 3 · p. 8, §4.3.2 · p. 8

4.3 ABLATION STUDY

• Controlled ablations examine pipeline construction, negative-teacher selection, and gate control.Source: §4.3 · p. 8

4.3.3 Q3: HOW TO CONTROL THE GATE

• Batch-level thresholding reaches 78.22% Avg6, versus 77.08% for rollout-level thresholding and 76.88% without DAG.The authors note that rollout-level mean discrepancy can be sensitive to a few token positions in short responses.Source: Table 4 · p. 9, §4.3.3 · p. 9

5.1 ON-POLICY DISTILLATION IN LARGE LANGUAGE MODELS

• Language-model OPD matches teacher distributions on student-generated trajectories, while OPSD derives supervision from the student and can use expert demonstrations.Prior work notes that direct access to reference solutions can introduce answer leakage and that provided context can encourage shortcut-like rationalization.Source: §5.1 · p. 9

5.2 ON-POLICY DISTILLATION IN VISION-LANGUAGE MODELS

• VLM distillation approaches use privileged visual information, visual attribution, or positive and negative views; this paper investigates whether crop-guided supervision fosters independent evidence identification.Vision-OPD uses a crop-conditioned teacher to supervise an original-image-conditioned student.Source: §5.2 · p. 9

• CV-OPD uses a randomly sampled negative teacher, margin-based CDL, and DAG combining token-level discrepancies with sample-level reliability filtering to improve distillation supervision.The authors report substantial gains in cross-domain generalization and hallucination mitigation.Source: §6 · p. 10

• The token optimization direction depends on the difference between the positive and negative distributions, degenerating locally to the positive distribution term.Source: §A.1 · p. 12
• For small perturbation jitter, the derivation applies a local second-order expansion of JSD and rewrites the gradient.Source: §A.1 · p. 12
• Under a first-order approximation, CDL uses differences between teacher distributions from matched crop pairs as supervision, unlike original JSD's absolute teacher distribution.Source: §A.1 · p. 12

A.2 TRAINING HYPERPARAMETERS

• Training uses eight A100 GPUs, batch size 96, eight rollouts per sample, one epoch, and AdamW at a learning rate of 2 × 10−6.The maximum prompt and response lengths are 8192 and 1024 tokens; generalized JSD uses β = 0.5 over Top-100+tail distributions, and the EMA update rate is 0.05.Source: §A.2 · p. 12
• Contrastive distillation sets both the margin and loss weight to 0.1, sharing negative crops across rollouts with maximum IoU 0.1.Source: §A.2 · p. 12

A.3 TRAINING ALGORITHM

• At each step, the student generates multiple full-image rollouts; the EMA teacher supplies positive and negative crop-conditioned distributions along the same prefix.Source: §A.3 · p. 12
• Samples with no correct rollouts are excluded; retained trajectories use the main JSD objective and gated Contrastive Divergence Loss, followed by an EMA teacher update.Source: §A.3 · p. 12

A.4 ATTENTION VISUALIZATION

• CV-OPD concentrates attention on answer-relevant regions, while Vision-OPD attends more diffusely across the image.The reported comparison associates this tighter focus with improved fine-grained visual understanding and reduced reliance on irrelevant cues and shortcut predictions.Source: §A.4 · p. 13

A.5 ATTENTION VISUALIZATION IN BLACK MASK

• Under black masking, Vision-OPD predicts the expected answer without localizing the target, whereas CV-OPD describes the visible scene and its ambiguity.Source: §A.5 · pp. 14–15

A.6 RESPONSES UNDER INSUFFICIENT VISUAL EVIDENCE

• With answer-relevant evidence occluded or corrupted, CV-OPD acknowledges what is missing and describes observable content without inventing visual details.Source: §A.6 · p. 15
• In the mailbox example, Vision-OPD follows the dominant mailbox color and predicts red, while CV-OPD attends to the lock and identifies metallic silver.Source: Figure 4 · p. 15
• In the black-mask examples, CV-OPD acknowledges unreliable color or material evidence even when its forced-choice prediction is incorrect.Source: Figure 6 · p. 16, Figure 7 · p. 16