PaperlayerView exact revision on arXiv

Source-linked AI summary

From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models

Juncheng Wu, Hardy Chen, Haoqin Tu, Xianfeng Tang, Freda Shi, Hui Liu, Hanqing Lu, Cihang Xie, Yuyin Zhou

arXiv:2605.20177v1cs.CLcs.CV

TL;DR

Visual-language models’ visual reasoning is often limited by perception errors rather than reasoning capacity. The paper decouples post-training into staged perception, textual-reasoning, and visual-reasoning phases, consistently outperforming unified pipelines while producing shorter reasoning traces.

  • Problem

    Visual perception errors can limit visual reasoning in VLMs, while subsequent reasoning often compounds rather than corrects incorrect perceptual assumptions.

  • Method

    The paper uses staged post-training with separate specialized datasets and stages for visual perception, textual reasoning, and visual reasoning.

  • Results

    20.8% shorter reasoning traces accompany consistent improvements over unified training across four model architectures, with RLVR providing a more effective perception-training signal than caption-based SFT.

  • Takeaways & Limitations

    Capability-based staging complements difficulty-based curricula, and combining both produces further additive improvements in VLM post-training.

  • Takeaways & Limitations

    Experiments are limited to 7–8B models, and the perception-data pipeline depends on fine-grained image captions.

Abstract

from arXiv · show

Recent advances in vision-language models (VLMs) emphasize long chain-of-thought reasoning; yet, we find that their performance on visual tasks is primarily limited by a lack of visual perception as opposed to reasoning itself. In this work, we systematically study the interplay between perception and reasoning in VLM post-training by decomposing their capabilities into three separate training stages: visual perception, visual reasoning, and textual reasoning, incorporating specialized training data. We demonstrate that visual perception (a) requires targeted optimization with specialized data; (b) serves as a fundamental scaffold that should be solidified through staged training before refining visual reasoning; and (c) is more effectively learned via RL than caption-based SFT. Our experiments across multiple VLMs demonstrate that staged training consistently improves both visual perception and reasoning performance over merged training. Notably, models trained with our approach achieve 1.5% higher reasoning accuracy with 20.8% shorter reasoning traces, suggesting that superior perception reduces the need for excessive reasoning. Furthermore, we show that this capability-based staging represents a new curriculum dimension orthogonal to traditional difficulty-based curricula, and combining both yields further additive gains. Our staged-training models achieve superior performance among open-weight VLMs, establishing advanced results on several visual math and perception (e.g., +5.2% on WeMath and +3.7% on RealWorldQA) tasks compared with the base counterpart.

1. Introduction

The paper argues that visual perception, rather than reasoning, often limits VLM performance and proposes decoupling post-training into visual perception, textual reasoning, and visual reasoning stages. Staged training improves perception and reasoning, while capability-based and difficulty-based curricula provide complementary gains.

  • Motivation: Visual reasoning failures often originate in incorrect visual perception, which later reasoning tends to compound rather than correct.The paper identifies this failure pattern across visual mathematics, geometry, and diagram-based reasoning tasks.
  • Method: The proposed framework trains visual perception, textual reasoning, and visual reasoning as separate capability stages.It treats visual perception as an independent foundational capability rather than a solved byproduct of pre-training.
  • Visual perception: 7.43-point accuracy gain over the Qwen2.5-VL-7B base model results from adding a visual perception stage during post-training on WeMath.The finding supports targeted optimization with specialized perception data.
  • Curriculum learning: 4.43% improvement over merged training results from combining capability-based staging with difficulty-based ordering.The paper presents capability-dimension curriculum learning as complementary to traditional difficulty-based curricula.
  • Results: The staged-training Qwen3-VL-8B reaches 75.9% on MathVista, 56.1% on WeMath, and 74.5% on RealWorldQA.Compared to OneThinker-8B, it improves accuracy by 1.5% on WeMath and 3.0% on RealWorldQA.

2. Related Work

Recent VLM post-training work increasingly targets visual reasoning through structured-trace SFT and RLVR, covering tasks such as VQA and chart understanding. Existing reasoning-VLM post-training commonly uses merged or curriculum training, with representative methods combining supervision or rewards in single-stage recipes.

  • Visual reasoning post-training: Structured reasoning-trace distillation is a common SFT-based direction for improving visual reasoning in VLMs.The cited works include Xu et al. (2024), Zhang et al. (2024b), Thawakar et al. (2025), Shao et al. (2024a), and Li et al. (2025).
  • Visual reasoning post-training: RLVR trains VLMs on multimodal tasks such as visual question answering and chart or infographic understanding for higher accuracy.The paper positions its approach as RLVR-based tuning of a competent reasoning VLM.
  • Post-training strategies: Merged training combines diverse supervision signals in one phase, exemplified by LLaVA-CoT’s joint use of multiple VQA sources and structured reasoning annotations.For RL-based training, VLAA-Thinker similarly uses single-stage Mixed Reward blending grounding and reasoning rewards.

3. Staged Post-training Pipeline

The pipeline separates post-training into visual perception, textual reasoning, and visual reasoning using three disjoint open-source datasets. It then sequentially optimizes these capabilities with GRPO, contrasting staged training against an identically budgeted merged baseline.

  • Dataset construction: Three disjoint datasets target visual perception, textual reasoning, and visual reasoning, and are synthesized or curated entirely from open-source resources.The visual reasoning set combines multiple sources and retains samples requiring both accurate perception and multi-step reasoning.
  • Visual perception data: 15K DOCCI image-caption pairs support perception-focused question-answer generation emphasizing fine-grained visual details and spatial relations.Qwen2.5-72B generates the questions, while the generated answers serve as ground truth.
  • Visual perception data: Perception filtering retains samples answerable from captions but missed by the base VLM from images, isolating visual-perception deficiencies rather than reasoning difficulty.The resulting Dperc samples are challenging because of insufficient visual perception, and filtering is further applied using two models.
  • Reasoning data: Textual reasoning uses ORZ-Math-13k for multi-step mathematics without visual inputs, while visual reasoning uses multimodal sources requiring accurate perception and multi-step reasoning.The resulting datasets are denoted Dtext and Dvis, respectively.
  • Training procedure: GRPO trains the model sequentially across three stages with equal epochs and identical hyperparameters, whereas merged training combines Dperc, Dtext, and Dvis under the same total-step budget.GRPO uses grouped sampled responses, composite accuracy-and-format rewards, standardized group-relative advantages, clipped optimization, and KL regularization.

4. Experimental Analysis

Staged training consistently improves visual perception and reasoning across model backbones, especially when visual perception is optimized before visual reasoning. RL-based perception training and capability-based curricula further strengthen performance while reducing reasoning costs.

  • Overall Performance Comparison: 42.3%: Staged training raises Qwen2.5-VL-7B’s average visual math score from 37.0% base and 40.7% merged training.It also improves MVerse from 26.4% to 37.9% and WeMath from 30.9% to 38.3%.
  • Performance Superiority of Perception-First Training: 65.8%: Qwen3-VL-8B staged training achieves the highest overall score, with 51.1% average visual math and 80.4% average perception.The model leads WeMath at 56.1%, MathVista at 75.9%, MMStar at 73.1%, and RealWorldQA at 74.5%.
  • Overall Performance Comparison: +0.95% and +3.77%: Staged training outperforms merged training on InternVL3.5-8B and InternVL3-8B, respectively, and wins 14/15 Qwen3-VL-8B benchmarks.These results support generalization across VLM backbones and statistical robustness across three runs.
  • Training Order: 64.8% versus 65.8%: Reversing the order to visual reasoning → textual reasoning → visual perception degrades Qwen3-VL-8B overall accuracy.The preferred sequence is visual perception → textual reasoning → visual reasoning; prematurely training visual reasoning entangles perception.
  • Perception Training Method: 8.2%: RLVR perception training improves Qwen2.5-VL-7B visual math performance over caption-based SFT, alongside perception gains of 77.2% versus 75.7%.RLVR also reaches 80.4% versus 79.2% perception on Qwen3-VL-8B and penalizes unsupported or hallucinated visual interpretations.
  • Curriculum Composition: 62.99%: Combining capability-based and difficulty-based curricula surpasses merged training at 58.56% and either curriculum alone at 60.53% or 60.36%.The results indicate that the two curriculum dimensions address orthogonal optimization aspects and provide additive improvements.

5. Discussion and Conclusion

The paper identifies visual perception as a dominant bottleneck in VLM visual reasoning and proposes staged post-training that separates perception, textual reasoning, and visual reasoning. This capability-dimension curriculum complements difficulty-based curricula, while experiments remain limited to 7–8B models and caption-dependent data.

  • Core finding: Visual perception is a dominant limiting factor for VLM visual reasoning, and longer reasoning alone cannot compensate for perceptual errors.This finding motivates decoupling perception from textual and visual reasoning during post-training.
  • Staged post-training: The staged post-training paradigm consistently outperforms unified training across four model architectures while producing shorter reasoning.It decouples training into visual perception, textual reasoning, and visual reasoning stages.
  • Curriculum perspective: Capability-dimension curriculum learning structures training by functional roles rather than task difficulty and complements existing difficulty-dimension curricula.The framework organizes training around visual perception, textual reasoning, and visual reasoning capabilities.
  • Curriculum perspective: Combining capability- and difficulty-dimension curricula yields further additive improvements.The paper presents this combination as a promising direction for multidimensional training trajectories in future VLM post-training.
  • Limitations: The experiments are limited to the 7–8B parameter scale, caption-dependent perception data, and a potentially coarse three-stage capability decomposition.Validation on 32B+ models, domains without fine-grained image captions, and finer-grained stage decompositions remain future work.

A. Appendix · A.1. Detailed Hyperparameter Setting

The appendix specifies the key hyperparameters for Stage-3 training in Table 6. All unlisted training parameters follow EasyR1 defaults to support controlled comparison and reproducibility.

  • A.1. Detailed Hyperparameter Setting: The appendix provides the full hyperparameter configuration for the training setup.
  • A.1. Detailed Hyperparameter Setting: Unlisted training parameters use EasyR1’s default settings.
  • A.1. Detailed Hyperparameter Setting: Following EasyR1 defaults applies to all remaining training parameters not shown in Table 6.
  • A.1. Detailed Hyperparameter Setting: The default-parameter policy is intended to ensure a controlled comparison.
  • A.1. Detailed Hyperparameter Setting: The same policy is intended to support reproducibility.
  • A.1. Detailed Hyperparameter Setting: Table 6 lists the key hyperparameters used in Stage-3 training.

A.2. More Experimental Results

Additional experiments reinforce visual perception training as the key contributor to staged VLM performance. They also show that encoder-freezing choices have limited impact, while perception data improves visual-math results without harming perception capabilities.

  • Ablation of each training stage: Stage 1 perception training before Stage 3 raises MVision from 26.64% to 29.28% and WeMath from 56.10% to 58.76%.The overall average also increases from 67.33% to 68.27%.
  • Ablation of each training stage: Stage 1 perception training increases the overall average from 67.33% to 68.27%, whereas adding Stage 2 before Stage 3 produces only marginal changes.
  • Impact of Training Vision Encoder: Encoder freezing strategy causes relatively small performance differences, indicating that freezing alone is not a dominant determinant of final performance.
  • Impact of Training Vision Encoder: Staged training consistently outperforms merged training under comparable encoder configurations.This pattern is reported across the evaluated Qwen model settings.
  • Perception and reasoning data: 42.26% to 43.78%: perception+reasoning improves Qwen3-VL-8B MVerse (VI) performance over reasoning-only training.On the same model, MVista improves from 73.80% to 75.90%, while A-OKVQA and POPE remain comparable.

A.3. Visual Perception Data Example

The generated visual perception data targets difficult competencies through representative scenes involving low-light object counting and fine-grained attribute discrimination. Together, these examples exercise perception skills that can bottleneck downstream visual reasoning.

  • Representative examples: One example requires detecting and counting seven streetlamps and their river reflections under low-light conditions.It tests robust object detection, counting, and reflection understanding.
  • Representative examples: A second example requires identifying the most recently painted letter in weathered graffiti using color intensity and paint texture.It targets fine-grained appearance comparison and material aging cues.
  • Targeted competencies: Together, the examples exercise object counting, reflection understanding, fine-grained appearance comparison, and material aging cues that often bottleneck downstream visual reasoning.The generated data explicitly targets these core visual competencies.

A.4. Prompt Settings

This section documents the prompts used throughout the experiments, covering visual-perception data generation, error assessment, and model training.

  • A.4. Prompt Settings: The experiments use a prompt for generating visual perception question-answering data.This prompt is associated with Figure 7.
  • A.4. Prompt Settings: A separate prompt assesses visual perception errors in the model’s reasoning.This prompt is associated with Figure 8.
  • A.4. Prompt Settings: The setup also specifies a system prompt used for model training.This system prompt is associated with Figure 9.

A.5. Extended Benchmark Results Across Four Model Families · A.6. Statistical Robustness: Three-Run Averaged Results · A.7. Response Length on Test Sets

Staged training generalizes across four VLM families, remains statistically robust across repeated evaluations, and produces shorter responses while improving accuracy. These results support staged training as a broadly effective approach that reduces excessive visual reasoning.

  • A.5. Extended Benchmark Results Across Four Model Families: Staged training outperforms merged training across four model families, with gains ranging from +1.62% to +3.77% overall.InternVL3-8B shows the largest gain (+3.77%), followed by Qwen3-VL-8B (+3.37%), Qwen2.5-VL-7B (+1.62%), and InternVL3.
  • A.5. Extended Benchmark Results Across Four Model Families: The staged-training paradigm generalizes beyond the Qwen family to architecturally distinct VLMs and is especially impactful for weaker base models.
  • A.6. Statistical Robustness: Three-Run Averaged Results: Three-run averages show staged training wins on 14/15 benchmarks for Qwen3-VL-8B and 12/15 for Qwen2.5-VL-7B.The corresponding overall AVG gains are +2.79% and +1.59%, respectively.
  • A.6. Statistical Robustness: Three-Run Averaged Results: Merged-training leads on a few Qwen2.5-VL-7B benchmarks, but the differences remain within 0.6%, supporting statistical robustness.The cited examples are WeMath and RWQA for Qwen2.5-VL-7B.
  • A.7. Response Length on Test Sets: Shorter responses occur alongside higher accuracy across the benchmarks.The passage attributes this combination to stronger perception reducing excessive reasoning and repeated image re-checking.
  • A.7. Response Length on Test Sets: Staged training produces 6.6–12.6% shorter responses across all visual math test benchmarks than merged training.This pattern is consistent with the training-time observation in Figure 5.
Loading 2605.20177v1…