PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Contrastive Self-Distillation

Yijun Liang, Yunjie Tian, Yijiang Li, Yuqi Jia, Furong Huang, Tianyi Zhou, Di Fu

arXiv:2607.21556v1cs.CVcs.AI

TL;DR

On-policy self-distillation still lacks a reliable asymmetry between student and self-teacher without privileged answers or visual evidence. VCSD creates that asymmetry by contrasting EMA-teacher predictions on original and content-erased images, consistently outperforming matched OPSD across evaluated model families and scales.

  • Problem

    On-policy self-distillation needs asymmetric information between student and self-teacher, but existing methods rely on privileged answers or visual evidence.

  • Method

    VCSD contrasts EMA-teacher token distributions conditioned on original and content-erased images to sharpen a full-distribution self-distillation target.

  • Results

    62.27% → 67.04% at 2B on Qwen3-VL’s seven-benchmark aggregate, with VCSD consistently outperforming matched OPSD across Qwen3-VL and Qwen3.5 models.

  • Takeaways & Limitations

    Matched input conditioning can provide an effective self-distillation signal without external teachers, privileged answers, reasoning traces, or visual evidence signals.

  • Takeaways & Limitations

    The theoretical interpretation assumes that content-erased predictions approximate predictions without instance-specific visual information.

Abstract

from arXiv · show

On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be removed, yielding a simpler form of OPSD driven purely by input conditioning. For this purpose, we propose Visual Contrastive Self-Distillation, namely VCSD, which converts image-content removal into an on-policy self-distillation signal. At each student-generated response prefix, the EMA teacher produces two next-token distributions under the same prompt and prefix -- one conditioned on the original image and the other on a content-erased control. Their token-wise log-probability difference highlights candidates whose likelihood is specifically increased by the instance-level visual content. We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student. Using ViRL39K dataset, VCSD consistently outperforms matched OPSD across Qwen3-VL and Qwen3.5 models. For example, on Qwen3-VL, it improves the seven-benchmark aggregate from $62.27\% \rightarrow 67.04\%$ at 2B, $71.30\% \rightarrow 73.16\%$ at 4B, and $72.51\% \rightarrow 76.26\%$ at 8B. Furthermore, VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost.

1 INTRODUCTION

VCSD constructs OPSD’s required teacher–student asymmetry purely through matched input conditioning, contrasting original-image and content-erased teacher predictions to form a plausible full-distribution target. Across Qwen3-VL and Qwen3.5 models, it consistently improves performance on seven vision-language benchmarks.

  • Motivation and contribution: VCSD answers whether OPSD asymmetry can come purely from input conditioning, without privileged answers or visual evidence signals.The method uses a content-erased image as a controlled reference rather than an auxiliary teacher input to imitate.
  • Method: VCSD contrasts EMA-teacher next-token distributions under the original image and a content-erased control at each student-generated prefix.The token-wise log-probability difference measures dependence on instance-level visual content.
  • Method: The original-image distribution identifies plausible candidates, while conditioning contrast sharpens their relative preferences into a full-distribution target distilled through forward KL.Restricting the target to candidates plausible under the original image prevents large relative changes among unlikely tokens from defining the target.
  • Results: 62.27% → 67.04%: Qwen3-VL’s seven-benchmark aggregate improves at 2B with VCSD.The corresponding aggregates are 71.30% → 73.16% at 4B and 72.51% → 76.26% at 8B.
  • Results: VCSD consistently improves Qwen3-VL and Qwen3.5 models from 2B to 9B across seven vision-language benchmarks.The paper reports that VCSD outperforms both base models and matched OPSD baselines across model families and scales.

2 RELATED WORK

Prior on-policy self-distillation methods introduce teacher–student asymmetry through privileged supervision, visual evidence, or trajectory selection. Related work also uses differences between paired predictive distributions for inference-time token selection and same-model training signals.

  • Target asymmetry in on-policy self-distillation: On-policy self-distillation requires asymmetry between the shared model’s target and student to produce an informative learning signal.Existing approaches use privileged answers or reasoning traces, evidence-centered visual views, or paired evidence for trajectory selection.
  • Target asymmetry in on-policy self-distillation: Existing asymmetry mechanisms include privileged answers, reasoning traces, visual evidence, and paired evidence for trajectory selection.These mechanisms are presented as supervision strategies for making same-model targets more informative than the student.
  • Relative predictive distributions under paired conditions: Paired predictive distributions can expose information absent from either prediction alone.Contrastive decoding compares expert–amateur models or original–degraded visual conditions, while paired model behaviors can also define training signals.

3 METHOD

VCSD creates OPSD target asymmetry purely through matched visual conditioning: an EMA teacher contrasts predictions for the original image and a content-erased control at each student-generated prefix. It uses this contrast to shape a plausibility-restricted full-distribution target, which is distilled into the student without external supervision or inference-time overhead.

  • Contrast signal: At every student-generated prefix, the token-wise log-probability contrast identifies tokens whose likelihood changes specifically with instance-level visual content.Positive contrast indicates greater support under the original image, whereas negative contrast indicates greater support after content removal.
  • Matched visual conditioning: VCSD evaluates the EMA teacher under the original image and a same-size black control image while sharing the prompt, prefix, parameters, and multimodal computation path.The control preserves resolution, preprocessing, interface, and visual-token count while removing instance-specific visual content.
  • Contrast-shaped target: VCSD applies contrastive shaping only within the original-image teacher’s relative plausibility support, using the original distribution as an anchor and the contrast to adjust relative probabilities.β controls the support threshold, while α controls shaping strength; the control distribution is used only as a reference, not as a distillation target.
  • Distillation and updates: The contrast-shaped target is distilled at every response position with full-distribution forward KL, while gradients pass only through the student and the teacher is updated by EMA.Both teacher evaluations use the same fixed student-generated prefix and do not create separate response trajectories.
  • Training and inference: Training uses only the original prompt–image pair and the student’s on-policy response, requiring no external teacher, privileged answer, reasoning trace, evidence crop, verifier, or verifiable reward.At inference, only the updated student is retained, with neither the EMA teacher nor the content-erased control adding an inference-time branch.

4 EXPERIMENTS · 4.1 SETUP · 4.2 MAIN RESULTS

VCSD is evaluated against base models and answer-hint OPSD on ViRL39K across six Qwen model configurations and seven visual benchmarks. It consistently outperforms OPSD, with gains spanning capabilities and model families while using input-conditioned image/content-erased targets without reference answers.

  • 4.1 SETUP: VCSD is evaluated on Qwen3-VL and Qwen3.5 at 2B, 4B, and 8B/9B scales using the ViRL39K single-image dataset.Qwen3-VL uses 2B, 4B, and 8B; Qwen3.5 uses 2B, 4B, and 9B.
  • 4.1 SETUP: The comparison includes unmodified base models, answer-hint OPSD, and VCSD, whose target uses paired original and content-erased image conditions without consuming the answer.OPSD conditions its target on the reference answer while the student receives only the question and image; VCSD retains on-policy rollouts.
  • 4.1 SETUP: The seven-benchmark evaluation covers perception, visual mathematics, fine-grained and high-resolution perception, and hallucination, with Acc defined as their unweighted mean.The benchmarks are BLINK, MMStar, MathVista, V∗Bench, HRBench4K/8K, and HallusionBench; HallusionBench first averages aAcc, fAcc, and qAcc.
  • 4.2 MAIN RESULTS: VCSD improves every Qwen3-VL-2B benchmark over OPSD, including +3.00% on MMStar and +4.32% on HallusionBench.The reported gains are BLINK +1.27%, MMStar +3.00%, V∗Bench +2.09%, MathVista +1.40%, HRBench4K +1.00%, HRBench8K +2.00%, and HallusionBench +4.32%.
  • 4.2 MAIN RESULTS: VCSD improves average accuracy by +1.86% to +4.77% over corresponding base models and achieves the best overall performance across nearly all model scales.Table 1 compares base model, OPSD, and VCSD across three Qwen3-VL and three Qwen3.5 scales.
  • 4.2 MAIN RESULTS: +2.15% aggregate gain spans general perception, visual mathematics, fine-grained and high-resolution perception, and hallucination rather than a single benchmark.This result is reported for the primary comparison and indicates broad capability improvements.
  • 4.2 MAIN RESULTS: Across all six configurations, VCSD achieves the highest aggregate accuracy, remaining +2.54% above OPSD on Qwen3-VL-8B and +4.51% above it on Qwen3.5-9B.On Qwen3.5, OPSD provides no consistent gain over base models, whereas VCSD improves every tested scale.

4.3 EFFECT OF PLAUSIBILITY RESTRICTION

Plausibility support limits recursive target distortion by restricting contrast-based target shaping to tokens plausible under the original-image prediction. This prevents visual conditioning contrast from overfavoring tokens that remain unlikely for the original image.

  • Effect of Plausibility Restriction: Restricting target shaping to the original-image prediction’s relative support redistributes probability only among plausible tokens, limiting recursive target distortion.Without plausibility support, conditioning contrast can strongly favor tokens whose probabilities change substantially between visual conditions despite remaining unlikely under the original image.

4.4 EFFECT OF CONTRASTIVE STRENGTH · 4.5 EFFECT OF DISTILLATION DIVERGENCE

Contrastive shaping is most effective at moderate strength, peaking around α = 1, while forward KL best distills the resulting contrast-shaped target. Removing or over-strengthening contrast reduces performance, and forward KL achieves the highest aggregate accuracy among the compared divergences.

  • 4.4 EFFECT OF CONTRASTIVE STRENGTH: α = 1 yields peak performance, with accuracy varying by less than 1% across α ∈[1, 1.5].The contrastive strength is locally robust around its best-performing setting.
  • 4.4 EFFECT OF CONTRASTIVE STRENGTH: 2.33% is the seven-benchmark accuracy decrease when α = 0 is compared with α = 1.At α = 0, conditioning contrast is removed while the rest of target construction remains unchanged.
  • 4.4 EFFECT OF CONTRASTIVE STRENGTH: Approximately the no-contrast performance level results when contrastive strength increases to α = 2.This shows that stronger contrast is not uniformly beneficial.
  • 4.5 EFFECT OF DISTILLATION DIVERGENCE: 67.04% is the highest aggregate accuracy achieved by forward KL among forward KL, reverse KL, and JSD.The comparison holds the contrast-shaped target and all other training settings fixed.
  • 4.5 EFFECT OF DISTILLATION DIVERGENCE: 0.79% is the margin by which forward KL outperforms JSD, while 2.27% is its margin over reverse KL.These are aggregate-accuracy differences under the fixed comparison setup.
  • 4.5 EFFECT OF DISTILLATION DIVERGENCE: Six of the seven benchmarks rank forward KL first among the compared distillation divergences.Forward KL is evaluated against reverse KL and JSD.

4.6 EFFECT OF CONTROL-IMAGE CONSTRUCTION · 4.7 EFFECT OF ORIGINAL-IMAGE ANCHOR

VCSD is robust to how the content-erased control image is constructed, while the original-image anchor mainly regularizes generation rather than improving aggregate accuracy. The anchor reduces language drift during training while preserving the contrastive target’s capability gain.

  • 4.6 EFFECT OF CONTROL-IMAGE CONSTRUCTION: The shared property across control variants is removal of instance-specific image content, rather than any particular degradation process.
  • 4.7 EFFECT OF ORIGINAL-IMAGE ANCHOR: The original-image anchor is analyzed by introducing coefficient λ into the target score.
  • 4.6 EFFECT OF CONTROL-IMAGE CONSTRUCTION: Black, Gaussian-noise, Gaussian-blur, and no-image controls achieve similar aggregate accuracy, indicating robustness to control-image construction.All variants remove instance-specific visual content through different interventions.
  • 4.7 EFFECT OF ORIGINAL-IMAGE ANCHOR: The full method uses λ = 1, whereas the no-anchor variant uses λ = 0 while retaining the same plausibility support.
  • 4.7 EFFECT OF ORIGINAL-IMAGE ANCHOR: With and without the anchor, seven-benchmark accuracy remains comparable, suggesting capability gains mainly come from contrastive shaping.The anchor therefore primarily regularizes generation rather than improving aggregate accuracy.
  • 4.7 EFFECT OF ORIGINAL-IMAGE ANCHOR: The anchor reduces language drift throughout training, with the largest reductions at intermediate checkpoints.Language drift is the fraction of rollouts containing non-target-language tokens.

4.8 TRAINING DYNAMICS

VCSD consistently outperforms OPSD throughout training on the seven-benchmark aggregate, MMStar, and MathVista, while showing less late-stage degradation and narrower accuracy variation.

  • Training dynamics: VCSD stays above OPSD at every evaluated training step on the seven-benchmark aggregate and shows less late-stage degradation.This advantage persists across the training trajectory rather than appearing only at a single checkpoint.
  • Training dynamics: After the first evaluated step, VCSD remains ahead on MMStar throughout training.The comparison is made within the same run.
  • Training dynamics: VCSD is higher than OPSD throughout training on MathVista and varies within a narrower accuracy range.The narrower range indicates more stable accuracy variation across evaluated training steps.

4.9 QUALITATIVE ANALYSIS

VCSD’s qualitative gains arise from more accurate extraction of image evidence, especially visual counting, and from contrast scores that emphasize image-grounded tokens. In the MathVista case study, VCSD corrects the base model and OPSD’s block-counting errors while reallocating probability toward content-dependent evidence.

  • MathVista case study: VCSD correctly counts seven thousand-cubes and predicts 7,519, while the base model predicts 6169 and OPSD predicts 8519.The answer combines 7,000 from seven thousand-cubes, 500 from five flats, 10 from one rod, and 9 units.
  • Image-dependent contrast: VCSD assigns stronger positive contrast to image-grounded concepts such as roof, shingles, food, beige, and sign, unlike the similar patterns of the base model and OPSD.The contrast score is Δ(v) = log p(v | I)−log p(v | Ictrl), and VCSD additionally suppresses less informative tokens over OPSD.
  • Image-dependent contrast: VCSD’s probability shaping reallocates mass toward image-dependent tokens instead of merely amplifying existing language preferences.This behavior is shown by the additional contrast learned over OPSD.

5 CONCLUSION

VCSD constructs the target asymmetry for on-policy self-distillation through matched visual conditioning. It contrasts EMA-teacher predictions under the original image and a content-erased control to sharpen a full-distribution target without external teachers, privileged answers, or visual evidence signals.

  • 5 CONCLUSION: VCSD constructs on-policy self-distillation asymmetry directly from matched visual conditioning.At each student-generated prefix, the EMA teacher is evaluated under the original image and a content-erased control.
  • 5 CONCLUSION: Token-wise log-probability contrast sharpens the original-image distribution within its plausible support.This produces an informative full-distribution target for distillation.
  • 5 CONCLUSION: VCSD requires no external teacher, privileged answers, or visual evidence signals.

A PROOF OF REMARK 1

The proof establishes that the contrast-shaped target is the unique global maximizer of the proposed optimization over the plausibility support. It also shows that α controls the tradeoff between visual-evidence reward and KL regularization, with α → 0 recovering the support-normalized original-image teacher.

  • Existence and uniqueness of the maximizer: The optimization objective has a maximizer because it is finite and continuous on the compact simplex Π(St).Positivity of the teacher distribution on St ensures finiteness and continuity under 0 log 0 = 0.
  • Existence and uniqueness of the maximizer: Strict concavity of the entropy term makes the objective strictly concave, so the maximizer is unique.The remaining term is linear in q, while −Σv∈St q(v) log q(v) is strictly concave.
  • Verifying global optimality: The candidate solution is globally optimal because it is strictly positive, satisfies the KKT conditions, and the problem has a concave objective with affine constraints.Strict positivity makes the omitted nonnegativity constraints inactive; KKT sufficiency then proves global optimality, and strict concavity proves uniqueness.
  • Equivalence with the contrast-shaped target: Substitution and cancellation of the support-normalization constant show that the optimizer exactly equals the contrast-shaped target.Increasing α gives greater weight to visual-evidence reward relative to the KL regularizer, while α → 0 recovers the support-normalized original-image teacher distribution.
Loading 2607.21556v1…