PaperlayerView exact revision on arXiv

Source-linked AI summary

Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards

Honghao Chen, Xingzhou Lou, Xiaokun Feng, Kaiqi Huang, Xinlong Wang

arXiv:2509.19003v1cs.CV

TL;DR

Vision-language models often produce coarse reasoning that is difficult to evaluate at the intermediate-step level. This paper introduces Chain-of-Step reasoning, step-level data, and PRM-based rewards for training and inference-time scaling. Across multiple benchmarks, the approach consistently improves on strong baselines, while the authors identify step quality as more important than reasoning length.

  • Problem

    Current VLM reasoning is often coarse-grained and lacks clear progression, making intermediate reasoning quality difficult to evaluate.

    Source: §1 · pp. 1–2

  • Method

    Chain-of-Step structures reasoning into steps, uses step-annotated data to train a PRM, and applies its fine-grained rewards in reinforcement learning and inference-time scaling.

    Source: §1 · p. 2

  • Results

    Across multiple benchmarks, CoS consistently improves on strong baselines; average gains are 4.9% on LLaVA-NeXT and 3.2% on InternVL-2.5-MPO.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The analysis finds that step-level evaluation is more effective than answer-only evaluation and that step quality matters more than reasoning length.

    Source: §1 · p. 2

    The process data from Monte Carlo estimation and LLM-as-Judge is not guaranteed to be fully correct, and the authors do not use human annotation because of its cost and limited scalability.

    Source: G Limitations · p. 17

Abstract

from arXiv · show

Chain of thought reasoning has demonstrated remarkable success in large language models, yet its adaptation to vision-language reasoning remains an open challenge with unclear best practices. Existing attempts typically employ reasoning chains at a coarse-grained level, which struggles to perform fine-grained structured reasoning and, more importantly, are difficult to evaluate the reward and quality of intermediate reasoning. In this work, we delve into chain of step reasoning for vision-language models, enabling assessing reasoning step quality accurately and leading to effective reinforcement learning and inference-time scaling with fine-grained rewards. We present a simple, effective, and fully transparent framework, including the step-level reasoning data, process reward model (PRM), and reinforcement learning training. With the proposed approaches, our models set strong baselines with consistent improvements on challenging vision-language benchmarks. More importantly, we conduct a thorough empirical analysis and ablation study, unveiling the impact of each component and several intriguing properties of inference-time scaling. We believe this paper serves as a baseline for vision-language models and offers insights into more complex multimodal reasoning. Our dataset, PRM, and code will be available at https://github.com/baaivision/CoS.

1 Introduction

Vision-language models often produce coarse, lengthy reasoning that lacks structured progression and makes intermediate reasoning quality difficult to evaluate. The paper introduces Chain-of-Step reasoning, structured step-level data, and a process reward model to support fine-grained evaluation, reinforcement learning, and inference-time scaling.

1 Introduction

  • Coarse, unstructured VLM reasoning makes systematic inference and evaluation of intermediate reasoning quality difficult.This also complicates selecting high-quality trajectories for training or inference.Source: §1 · pp. 1–2
  • Chain-of-Step decomposes reasoning into consecutive Name, Thought, and Reflection components that connect detailed reasoning with visual content and prior steps.The authors construct ShareGPT-Step-300K with GPT-4o, using 17 datasets spanning varied reasoning tasks.Source: §1 · p. 2
  • Monte Carlo estimation and LLM-as-Judge annotations train a PRM to assess intermediate-step quality and provide fine-grained rewards.The PRM is presented as a basis for reinforcement learning and inference-time scaling.Source: §1 · p. 2
  • Across multiple benchmarks, CoS improves on strong baselines; analysis finds step-level evaluation more effective than answer-only evaluation and step quality more important than reasoning length.The authors also report that longer reasoning is not necessarily better for VLMs.Source: §1 · p. 2

2 Related Work

Prior VLM work explores complex visual reasoning and inference-time scaling, while this paper positions its approach as fine-grained structured reasoning supported by PRM-based step rewards.

VLM Reasoning

  • Prior VLM reasoning approaches include datasets and fine-tuning, visual question-to-region correspondence, and multi-agent long-chain reasoning with summary judgments.The paper contrasts these approaches with its fine-grained structured reasoning approach.Source: §2 · p. 2

Inference-Time Scaling

  • Inference-time scaling uses additional reasoning time, with prior methods including majority voting, best-N search, and sentence-level beam search.The passage notes that majority voting suits standard-answer problems, while assessing best-N and sentence-level search can be challenging.Source: §2 · pp. 2–3
  • The paper uses PRM granular rewards for step-level beam search, distinguishing its VLM inference-time approach from prior VLM explorations.The passage also describes verifiers and Math-Shepherd as methods for supervising or scoring reasoning paths and steps.Source: §2 · p. 3

3 Method

The method combines structured step-wise reasoning, a generated and filtered dataset, step-level annotations, and PRM-guided iterative preference optimization. Its stages are presented as a pipeline from supervised fine-tuning through reward modeling to reinforcement learning.

3 Method

  • The pipeline first applies supervised fine-tuning, then trains a PRM, and uses its rewards to select reasoning paths for iterative DPO.The PRM is described as trainable once for use with different models.Source: §3 · p. 3, Figure 1 · p. 3, §3.4 · p. 5

3.1 Structured Reasoning

  • CoS organizes reasoning into consecutive steps with Name, Thought, and Reflection components, using special tokens to support stable step splitting and evaluation.Reflection connects visual content with previous steps; step content and the number of steps are self-generated.Source: §3.1 · p. 3

3.2 ShareGPT-Step-300K

  • GPT-4o generates step-by-step reasoning from questions and reference answers, and filtering yields 300K ShareGPT-Step instances across 17 multimodal tasks.The task categories include mathematical and scientific reasoning, chart and document analysis, and world knowledge.Source: §3.2 · p. 4

3.3 Step-Level Annotation

  • Monte Carlo estimation and GPT-4o judging provide step annotations used to train a PRM for fine-grained rewards.Monte Carlo scoring compares how many sampled continuations reach the golden answer; the judging method assigns Good, Neutral, or Bad ratings.Source: §3.3 · pp. 4–5

3.4 RL with Fine-grained Rewards

  • The SFT model generates 16 reasoning paths per question, which the PRM scores by average step and final-answer quality for iterative DPO selection.Positive-negative pairs must exceed a score-difference threshold, and the procedure repeats across rounds.Source: §3.4 · p. 5

4 Experiment

The experiments evaluate CoS across six reasoning benchmarks with two base VLMs, then analyze step scoring, inference scaling, reward design, PRM quality, reasoning length, and training patterns. CoS improves over open-source baselines, while ablations examine the framework’s components.

4.2 Main Results

  • CoS yields average increases of 4.9% on LLaVA-NeXt and 3.2% on InternVL-2.5-MPO across six benchmarks.Both SFT and DPO improve LLaVA-NeXt, whereas InternVL-2.5-MPO gains relatively little from SFT alone and reaches state-of-the-art after iterative DPO.Source: §4.2 · p. 6, Table 1 · p. 6

4.3.1 Reasoning Step Matters

  • Best-of-16 M3CoT accuracy first rises and then falls as step-score weight increases from 0% to 100%, favoring joint step-and-answer evaluation.The default step-score weight is 20%.Source: §4.3.1 · p. 7, Figure 3 · p. 7

4.3.2 Inference-Time Scaling

  • As N increases from 1 to 64, PRM selection outperforms Self-Consistency, and PRM-BS consistently improves over PRM at the same sampling efficiency.Step-level beam search samples and scores steps iteratively, continuing from the best-scoring step until it generates a final answer.Source: §4.3.2 · p. 7, Figure 5 · p. 8

4.3.3 Ablation on RL training

  • Selecting DPO pairs using both reasoning-step and answer quality performs best, surpassing outcome-based selection; answer-only scoring underperforms outcome-based selection.The comparison uses offline-selected training data for iterative DPO.Source: §4.3.3 · pp. 7–8, Table 2 · p. 8

4.3.4 Ablation on PRM

  • Among the tested base models, stronger models judge intermediate reasoning steps more accurately, leading the authors to select InternVL2.5-38B for the PRM.The assessment includes 5K questions from the PRM training dataset and 5K unseen questions.Source: §4.3.4 · p. 9, Table 3 · p. 9

4.3.5 Reasoning Length

  • Outcome DPO progressively lengthens reasoning, while PRM DPO first shortens it and later slowly lengthens it; longer reasoning is not necessarily better.The authors report that PRM DPO performs better than outcome DPO and emphasize high-quality steps for visual complex reasoning.Source: §4.3.5 · p. 9, Figure 6 · p. 9

4.4 Ablation on Reasoning Pattern

  • Reasoning-augmented SFT models outperform QA-only models on reasoning benchmarks, and CoS gains more during RL than outcome-only reward settings.No Reason uses question-answer pairs without intermediate reasoning; Direct Prompt concatenates reasoning thoughts, and neither supports strict step-level assessment during RL.Source: §4.4 · p. 10, Table 4 · p. 10

4.5 GRPO-based ablation

  • Fine-grained reward RL consistently outperforms outcome-reward RL in the GRPO-based ablation.The experiments use a 38B PRM as the reward server and start from the SFT model.Source: §4.5 · p. 10, Table 5 · p. 10

5 Conclusion

The paper introduces Chain of Step reasoning for vision-language models, using fine-grained rewards to assess reasoning-step quality and support reinforcement learning and inference-time scaling. Experiments across multiple benchmarks demonstrate its effectiveness, while empirical analyses and ablations examine its properties.

  • Chain of Step reasoning for vision-language models enables accurate assessment of reasoning-step quality and supports reinforcement learning and inference-time scaling with fine-grained rewards.Source: §5 · p. 10
  • Experiments across multiple benchmarks demonstrate the effectiveness of Chain of Step reasoning.Source: §5 · p. 10
  • Empirical analyses and ablations examine Chain of Step’s properties, with the paper aiming to offer insights into more complex multimodal reasoning.Source: §5 · p. 10

A Special Tokens

The appendix describes CoS’s special tokens and the prompts used to construct its SFT and PRM data.

A Special Tokens

  • CoS uses 11 special tokens, initialized from the average embeddings of their corresponding definitions, to realize structured step-by-step reasoning.Table 6 lists the tokens and definitions; Fig. 7 illustrates the reasoning structure.Source: A Special Tokens · p. 15

B Prompts

  • The appendix presents the prompts used to create ShareGPT-Step-300K and to have GPT-4o label process-level annotations.The prompts are illustrated in Figs. 8 and 9, respectively.Source: B Prompts · p. 15

C Training Recipes

This appendix section reports training hardware and selected data-construction settings, alongside tables of special-token definitions and training recipes.

C Training Recipes

  • SFT and iterative DPO experiments use 8 × NVIDIA-A800 GPUs by default, while PRM training uses 16 × NVIDIA-A800 GPUs.Source: C Training Recipes · p. 15
  • Positive/negative pair generation and Math-Shepherd sampling use temperature 1.0, 16 return sequences, and top p 0.95.Source: C Training Recipes · p. 15
  • The appendix includes Table 6 for special-token definitions and Table 7 for hyperparameter settings and training recipes.Source: Table 6 · p. 15, Table 7 · p. 16

D Dataset Sources

The SFT dataset draws from diverse multimodal tasks, spanning 17 tasks grouped into four broad reasoning categories.

D Dataset Sources

E Attempt for Step-wise DPO

The paper builds preference pairs at every reasoning step for step-wise DPO, but reports that the method significantly harms performance because the chosen and rejected steps are too similar.

E Attempt for Step-wise DPO

F Discussion of Table 3

The discussion reports that InternVL-2.5-38B does not achieve higher Answer Accuracy than smaller counterparts, and relates model size to differences in step-level optimization.

F Discussion of Table 3

  • InternVL-2.5-38B does not achieve higher Answer Accuracy than its smaller counterparts.Source: F Discussion of Table 3 · p. 16
  • Step-level evaluation is harder because it requires holistic semantic judgments, supervises many steps rather than one answer, and is less amenable to answer memorization.Source: F Discussion of Table 3 · p. 17
  • The authors attribute the reported pattern to smaller PRMs favoring final-answer accuracy, while 38B PRM better optimizes intermediate reasoning quality and reduces total loss.They describe step-level prediction as a greater optimization challenge for smaller models.Source: F Discussion of Table 3 · p. 17

G Limitations

The process-labeled data are not guaranteed to be fully correct, and the authors identify higher-quality process labels as a future need.

G Limitations

  • The process-labeled data generated by Monte Carlo estimation and LLM-as-Judge are not guaranteed to be fully correct.Source: G Limitations · p. 17
  • Human annotation is also not guaranteed to be accurate and high-quality, and the authors avoid it because of its high cost and limited scalability.Source: G Limitations · p. 17
  • The authors hope that higher-quality process-labeled data could further benefit their method.Source: G Limitations · p. 17
Loading 2509.19003v1…