PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation

Sihan Wang, Xiyao Liu, Lianqing Liu, Zhi Han

arXiv:2606.19120v2cs.LGcs.CV

TL;DR

Directly extending OPSD to MLLMs can encourage answer-driven shortcuts that weaken visual grounding. ViGOS separates image-only perception supervision from answer-conditioned reasoning supervision, retaining OPSD’s benchmark gains while improving image-grounded behavior and robustness on prior-sensitive multimodal reasoning tasks.

  • Problem

    Direct multimodal OPSD may let privileged answer text guide responses before image content is checked, producing weakly grounded reasoning.

  • Method

    ViGOS has the student describe the image first, applies an image-only teacher to perception, then uses a privileged teacher for reasoning and answers.

  • Results

    ViGOS keeps OPSD’s main gains while improving robustness on prior-sensitive multimodal reasoning benchmarks and image-grounded answering.

  • Takeaways & Limitations

    Separating visual description from reasoning is a useful approach for adapting OPSD to multimodal reasoning.

  • Takeaways & Limitations

    The PALR diagnostic is not a complete attribution of all possible shortcuts.

Abstract

from arXiv · show

On-policy self-distillation (OPSD) trains a model on its own rollouts and uses a frozen copy to provide dense token-level targets conditioned on a reference target. This works well for LLM reasoning, but a direct extension to multimodal large language models (MLLMs) can create a shortcut: the privileged target may guide tokens mainly based on the text reference target rather than the image. We propose ViGOS, a visually grounded OPSD framework for MLLM post-training. The student first writes a visual description and then reasons toward the final answer. For valid rollouts, an image-only perception teacher supervises the description, while a privileged reasoning teacher supervises the reasoning and final answer on the same student prefix. A reference teacher is used only for invalid rollouts to recover the output format. Across general vision-language, expert reasoning, visual math, spatial grounding, and visual-language-prior benchmarks, ViGOS keeps the main benefits of OPSD and improves image-grounded behavior in shortcut-prone settings.

1 Introduction

The paper identifies a multimodal shortcut in vanilla OPSD, where answer-conditioned supervision can guide MLLM responses before image content is grounded. ViGOS addresses this by separating image-based perception from privileged reasoning supervision while retaining on-policy self-distillation and improving robustness on prior-sensitive benchmarks.

  • Motivation: Vanilla OPSD can make answer-conditioned supervision shape MLLM responses before the model checks the image, producing answer-compatible rationales with weak visual grounding.The privileged teacher sees the reference answer while supervising the rollout, allowing text signals to guide reasoning more easily than visual evidence.
  • Method: ViGOS separates perception from reasoning by having the student first write a visual description supervised by an image-only teacher, then reason with privileged supervision.The visual description serves as a grounding interface, while different teacher contexts supervise different parts of the same student trajectory.
  • Method: ViGOS preserves dense on-policy self-distillation while assigning image-only and answer-conditioned teachers to distinct trajectory segments.This design retains the core OPSD training structure while decoupling image-based perception supervision from answer-conditioned reasoning supervision.
  • Results: Experiments across broad multimodal reasoning benchmarks show that ViGOS keeps OPSD’s main gains and improves robustness on prior-sensitive evaluations.The evaluation spans general vision-language, expert academic reasoning, visual mathematics, spatial grounding, and visual-language-prior stress tests.

2 Preliminaries

The preliminaries define multimodal supervised post-training and on-policy self-distillation, where students generate their own prefixes while teachers may use privileged reference targets. They introduce PALR to diagnose answer leakage and show why multimodal OPSD must control where answer-conditioned supervision applies.

  • Problem setup: The student MLLM receives only the original image and question, while the privileged reference target guides supervision but is never provided as input.Student rollouts are token sequences whose prefixes are generated by the current student.
  • On-policy self-distillation: Vanilla OPSD trains on student-generated prefixes and uses a frozen privileged teacher to provide dense token-level supervision on those visited states.Unlike the proposed visual-description structure, vanilla OPSD can supervise an ordinary reasoning response ending in a final answer.
  • PALR diagnostic: PALR compares answer-driven and image-driven teacher support while keeping the rollout and all student prefixes fixed.Higher PALR indicates that a larger share of dense correction is tied to the privileged answer, but PALR does not capture every possible shortcut.
  • PALR observations: 17.26% for 3B and 26.01% for 7B are the vanilla OPSD PALR(Tra) values on a 1,000-sample Qwen2.5-VL diagnostic subset per scale.Tra denotes tokens parsed as reasoning or final answer; vanilla OPSD still supervises the entire rollout with the privileged target.
  • Shortcut motivation: Multimodal OPSD is vulnerable because answer-conditioned text can compete with visual evidence, allowing reasoning to fit the known answer before needed evidence is explicit.The motivation is to retain valuable answer-guided reasoning while controlling where privileged supervision shapes tokens.

3 ViGOS: Visual Grounding On-Policy Self-Distillation

ViGOS decouples image perception from answer-conditioned reasoning by structuring student rollouts as description, reasoning, and answer segments. Segment-specific teachers supervise valid rollouts on the student’s own prefixes, while a reference teacher handles malformed outputs only.

  • Structured rollout: ViGOS samples a structured trajectory y = (d, r, a), where d is visual description, r is reasoning, and a is the final answer.The description is generated from the image and question and requires no extra annotation at inference time.
  • Segment-specific supervision: For valid rollouts, the image-only teacher supervises description tokens, while the privileged reasoning teacher supervises reasoning and answer tokens on the same student prefix.The image-only context excludes question text, answer options, and the reference target outside the generated prefix.
  • On-policy supervision: Teacher queries remain on-policy because every teacher receives a prefix that the current student actually produced.Teachers provide token-level distributions rather than replacement trajectories.
  • Invalid-rollout recovery: For invalid rollouts, the reference teacher supplies a whole-trajectory recovery signal, while perception and reasoning losses are inactive.Invalidity is determined by missing delimiters, empty description or reasoning segments, or an unparsable final answer.
  • Diagnostic results: Compared with vanilla OPSD, PALR(Tra) falls from 17.26% to 6.33% on 3B and from 26.01% to 7.56% on 7B.Full-rollout PALR also falls from 5.59% to 3.07% on 3B and from 7.55% to 3.72% on 7B.

4 Experiments

Experiments show that ViGOS preserves OPSD’s benchmark gains while improving image-grounded behavior under prior conflict. Ablations indicate that perception supervision, reasoning supervision, and a limited reference fallback each contribute to the final behavior.

  • Main benchmark results: ViGOS raises mean Pass@5 from 60.86% to 71.97% on 3B and from 68.13% to 75.60% on 7B across eight main benchmarks.Mean Avg@5 also increases from 27.91% to 41.35% on 3B and from 45.38% to 50.99% on 7B.
  • Main benchmark results: Compared with OPSD, ViGOS remains close on 3B mean Pass@5, slightly improves mean Avg@5, and achieves the best 7B mean Pass@5 with nearly the same mean Avg@5.The strongest gains occur on concrete image-understanding benchmarks, including RealWorldQA, MMSI, CV-Bench, and MathVista.
  • Prior-sensitive evaluation: ViGOS obtains the best ViLP Score in all settings, with 7B average Score increasing from 39.50 for Baseline to 62.17 for ViGOS.Prior remains high: for 3B it stays around 90-98 across both settings, while 7B ViGOS gives 97.00 on ViLP-F.
  • Prior-sensitive evaluation: With the same prompt, Baseline improves but OPSD falls below Baseline, whereas ViGOS retains strong main-benchmark outcomes and improves ViLP Score while keeping Prior high.This supports separating visual perception supervision from answer-conditioned reasoning supervision rather than attributing gains to the prompt alone.
  • Ablation studies: Removing the perception loss drops ViLP from 69.84 to 67.58 and decreases CV-Bench, while removing the reasoning loss lowers Overall Pass@5 and CV-Bench.The perception teacher ties descriptions to the image, whereas the reasoning loss converts descriptions into final answers; retaining both gives the best balance.
  • Ablation studies: Removing the reference teacher drops ViLP from 69.84 to 63.25, and reverse-KL fallback outperforms forward-KL on CV-Bench and ViLP.The fallback is used only for invalid rollouts to restore parseable output patterns, while step-wise ViLP Score reaches 0.71 for ViGOS versus around 0.63 for OPSD.

5 Conclusion

ViGOS addresses a multimodal OPSD shortcut by separating visual description from reasoning and answer supervision. The results support this separation as a useful approach for adapting OPSD to multimodal reasoning.

  • Conclusion: ViGOS addresses dense answer-conditioned supervision that can guide responses before the model describes the image.The paper frames this as a focused problem in multimodal OPSD.
  • Conclusion: The student first writes a visual description supervised by an image-only perception teacher, then receives reasoning and answer supervision after that prefix is established.This changes the supervision path by separating perception from subsequent reasoning and answering.
  • Conclusion: The results show that separating description and reasoning is useful for adapting OPSD to multimodal reasoning.This conclusion summarizes the reported value of the ViGOS approach.

A Privileged Answer Leakage Rate

The privileged answer leakage rate (PALR) measures how much dense teacher correction is answer-driven rather than image-driven under matched student rollouts. ViGOS lowers harmful pre-grounding answer leakage while retaining answer guidance for later reasoning and final answers.

  • Diagnostic: The diagnostic fixes each student rollout and queries all teachers on identical prefixes, isolating teacher-context differences rather than trajectory differences.Invalid rollouts are excluded because their segment masks are unreliable.
  • Diagnostic: PALR compares answer-driven and image-driven teacher sensitivity, weighted by active token correction strength and reported as a percentage.A high PALR(G) means dense correction on segment G is more driven by the privileged answer under the counterfactual test.
  • Results: 17.26% and 26.01% are vanilla OPSD’s PALR(Tra) values for Qwen2.5-VL 3B and 7B, showing notable answer-driven reasoning supervision.The reasoning-answer segment combines reasoning and final-answer tokens in this diagnostic.
  • Results: 0.00% is ViGOS’s PALR(Td), while PALR(Tra) falls to 6.33% on 3B and 7.56% on 7B.The zero description value is by construction because an image-only teacher supervises and diagnoses the description segment.
  • Token-level interpretation: Token-level examples show ViGOS making visual facts and copied image values more image-driven while retaining answer guidance near arithmetic, final answers, and formatting.The intended behavior is selective: visual evidence should be image-driven, whereas later reasoning and answers may remain partly answer-guided.

B Benchmark Details

The evaluation spans eight multimodal benchmarks covering general vision-language abilities, expert reasoning, visual mathematics, spatial reasoning, and visual grounding. ViLP separately tests whether models rely on image content or visual-language priors using diagnostic and prior-aligned questions.

  • Main evaluation benchmarks: The eight main benchmarks cover integrated vision-language abilities, expert-level multimodal reasoning, visual mathematics, spatial reasoning, and visual grounding.MM-Vet evaluates recognition, OCR, knowledge, spatial understanding, and language generation; MMMU and MMMU-Pro test academic multimodal reasoning; MathVerse and MathVista test visual mathematical reasoning; MMSI, RealWorldQA, and CV-Bench emphasize spatial or vision-centric grounding.
  • Main evaluation benchmarks: MathVerse and MathVista require models to interpret diagrams, charts, geometric structures, or other visual mathematical inputs before answering.MMSI evaluates multi-image spatial intelligence, while RealWorldQA tests real-world spatial understanding.
  • Visual-language prior evaluation: ViLP probes whether a vision-language model answers from image content or visual-language priors when prior-aligned answers can conflict with the image.The benchmark includes examples where the common answer differs from the answer supported by clear image content.
  • Visual-language prior evaluation: ViLP reports Score for visually diagnostic questions and Prior for prior-aligned questions, under ViLP-F with additional facts and ViLP-P with pure questions.A good model should improve Score without greatly hurting Prior, making ViLP suitable for testing whether ViGOS reduces reliance on priors.
  • Benchmark comparison setup: Table A.II compares Baseline, OPSD, and ViGOS under the same structured prompt, reporting Pass@5 / Avg@5 and ViLP Score & Prior as percentages.Baseline is the zero-shot same-prompt Qwen2.5-VL-3B-Instruct control, while OPSD and ViGOS use the structured format in post-training rollouts.

C Same-Prompt Comparison

Under a shared structured prompt, ViGOS outperforms Baseline and retrained OPSD, showing that segment-wise supervision contributes beyond prompting alone. The results associate OPSD’s weaker grounding with answer-conditioned supervision and support ViGOS’s perception-grounded design.

  • Prompt control: The experiment fixes the ViGOS structured prompt across Baseline, retrained OPSD, and ViGOS to test whether segment-wise supervision matters beyond output-format effects.The prompt requires visual description, reasoning, and answer in sequence.
  • Prompt control: The structured prompt alone substantially improves zero-shot image use, with Baseline reaching 66.40 / 23.88 on MMSI, 84.05 / 53.31 on RealWorldQA, and 91.28 / 64.27 on CV-Bench.The experiment therefore tests additional training-objective gains rather than denying the prompt’s effect.
  • Main comparison: 71.97 Pass@5 and 41.35 Avg@5 place ViGOS above Baseline at 71.05 / 39.83 and OPSD at 70.13 / 37.20 across eight main benchmarks.ViGOS achieves the best Avg@5 across all eight main benchmarks.
  • Grounding behavior: On MMSI, RealWorldQA, and CV-Bench, ViGOS reaches 66.40 / 24.90, 86.80 / 55.37, and 91.51 / 64.67, respectively, while OPSD falls below Baseline on grounding-oriented results.OPSD declines from Baseline’s 66.40 / 23.88, 84.05 / 53.31, and 91.28 / 64.27 on those benchmarks.
  • Grounding behavior: In ViLP-F, ViGOS raises Score from 65.83 to 70.17 while maintaining Prior at 97.67; in ViLP-P, OPSD’s Score drops to 62.33, 6.34 points below Baseline.The pattern indicates stronger prior- or answer-driven shortcuts can reduce willingness to follow conflicting image evidence.
  • Mechanistic evidence: PALR and ablations show that ViGOS reduces privileged-answer leakage and that removing perception loss lowers ViLP and CV-Bench, supporting perception-grounded supervision beyond prompting.Vanilla OPSD receives noticeable answer-driven supervision on reasoning and answer tokens, whereas ViGOS removes it from descriptions and reduces it later.

D Qualitative Analysis on ViLP

ViLP examples show that OPSD can recognize relevant visual cues yet let common priors or question-stated facts determine its final answer. ViGOS instead prioritizes image evidence through visual description before reasoning, improving image-grounded responses while retaining high Prior scores.

  • Overall pattern: Across five ViLP examples, OPSD often mentions visual evidence but answers according to a conflicting prior or question fact, whereas ViGOS gives the image-supported answer.These cases motivate ViLP’s Score improvement and ViGOS’s high Prior score.
  • Landmark location prior: In the landmark case, ViGOS uses the Eiffel Tower and city background to answer “Paris” rather than following the Statue of Liberty’s usual location.OPSD instead returns the prior answer, New York City, despite recognizing the visual setting.
  • Text conflict: In the text-conflict case, ViGOS answers “200” from the image’s equation “1 km = 200 m”, while OPSD rejects the visible statement and answers “1000”.The example shows that OPSD can read image text but still follow standard knowledge over the image-indicated answer.
  • Visual relations and priors: For conflicting animal-size, biology, and tool-use cues, ViGOS answers “Elephant”, “Lungs”, and “Saw” from the current image instead of the priors “Giraffe”, “Gills”, and “Beaks”.The cases require comparing visible size, identifying highlighted internal organs, and recognizing the hand saw in use.
  • Mechanism: ViGOS separates perception from reasoning by supervising an initial visual description with an image-only teacher before applying privileged reasoning supervision to the same student prefix.This separation makes later reasoning and final answers more likely to use image evidence rather than ignore it.

E Additional Implementation Details

This section provides the implementation details needed to reproduce training and evaluation, covering prompts, training hyperparameters, and evaluation decoding configuration.

  • The section reports implementation details needed to reproduce both training and evaluation.
  • It lists the prompts used by the student and three teacher roles during training.
  • It gives the training hyperparameters and evaluation decoding configuration.

E.1 Training Prompts · Image-only Perception Teacher Input · Privileged Reasoning Teacher Prompt

The prompts separate image-grounded perception from privileged reasoning by requiring the student to describe the image before reasoning, while restricting reference-target access by teacher role. They also use structured output tags and reserve the full privileged reference teacher for invalid-rollout format recovery.

  • E.1 Training Prompts: For text-containing prompts, the image is placed before the text in the same user message, with {problem} and {reference_solution} replaced by the current question and target.Normalized answers become target strings, while datasets with longer solutions use the solution text.
  • E.1 Training Prompts: The student sees the image and problem but never the reference target, then produces a detailed image description before reasoning and answering.The description must omit the final answer and use <description> tags; reasoning uses <think> tags and the answer appears in \boxed{}.
  • E.1 Training Prompts: The student’s required output is a structured sequence: description, step-by-step reasoning with self-reflection or verification, and a single-word-or-phrase final answer.The format is explicitly represented as <description>...</description> <think>...</think> \boxed{FINAL ANSWER here}.
  • Image-only Perception Teacher Input: The perception teacher receives only the image externally and scores description tokens while conditioning on the student’s generated prefix h_t.It receives neither the problem text, answer options, nor reference target outside that prefix.
  • Privileged Reasoning Teacher Prompt: The privileged reasoning teacher sees the image, problem, and reference target, uses that target to guide reasoning, and does not supervise description tokens.Its prompt asks for independent step-by-step reasoning toward the same final answer rather than copying or paraphrasing the reference solution.
  • Privileged Reasoning Teacher Prompt: The reasoning teacher’s requested output contains step-by-step reasoning followed by a boxed final answer.The prompt explicitly requests reasoning in <think> tags and the final answer in \boxed{}.
  • Privileged Reasoning Teacher Prompt: The reference teacher shares the reasoning teacher’s privileged information but is used only as a fallback loss signal for invalid rollouts.This preserves a format-recovery signal without making the full privileged prompt the default teacher for every token.
  • Privileged Reasoning Teacher Prompt: The full reference-teacher prompt requests a visual description, step-by-step reasoning, and a final answer in the same structured format.It places the reference solution before instructions to reason independently, then requires <description>, <think>, and \boxed{} sections.

E.2 Training Hyperparameters

ViGOS uses a consistent training configuration across its two model scales, sharing the same data, optimization, sampling, loss-weight, and effective-batch-size settings.

  • E.2 Training Hyperparameters: Both model scales use identical training data, optimizer, learning rate, rollout sampling settings, loss weights, and effective batch size.These main hyperparameters are summarized in Table A.IV.

E.3 Evaluation Configuration

Evaluation uses no privileged information at test time: models receive only the image, question, and evaluation prompt. Identical decoding and answer-scoring procedures are applied across models, with five-response metrics for eight benchmarks and single-response scoring for ViLP.

  • Test-time inputs: At evaluation, teachers, reference solutions, and segment masks are removed; the model receives only the image, question, and current evaluation prompt.This separates ViGOS’s training supervision from its test-time information.
  • Decoding controls: The same decoding configuration is used for Baseline, OPSD, ViGOS, and all ablations without model- or benchmark-specific tuning.This focuses comparisons on learned behavior rather than test-time sampling choices.
  • Main-benchmark scoring: For eight main benchmarks, five stochastic responses are generated per example: Pass@5 is correct if any extracted answer is correct, while Avg@5 is mean correctness.No reranking or manual selection is used.
  • ViLP scoring: For ViLP, one response is generated per prompt, then Score and Prior are computed according to the benchmark definition.Score covers visually diagnostic questions, while Prior covers prior-aligned questions.
  • Answer extraction: Unparseable outputs count as incorrect, official benchmark parsers are used when available, and the 4,096-token limit covers ViGOS’s entire generated sequence.For boxed outputs, the content of the last box is used as the final answer.
Loading 2606.19120v2…