PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast-Slow Thinking GRPO for Large Vision-Language Model Reasoning

Wenyi Xiao, Leilei Gan

arXiv:2504.18458v2cs.CLcs.AIcs.CV

TL;DR

LVLM reinforcement learning can struggle to scale reasoning length or produce verbose outputs with only marginal accuracy gains. FAST-GRPO estimates question difficulty and adapts length rewards and KL regularization; across seven benchmarks, FAST reports higher accuracy and shorter responses than prior slow-thinking approaches.

  • Problem

    LVLM reinforcement learning methods may fail to scale reasoning length or produce verbose outputs with only marginal accuracy gains, motivating adaptive fast-slow reasoning.

    Source: §1 · pp. 1–2

  • Method

    FAST-GRPO estimates question difficulty and uses adaptive length-based rewards and difficulty-aware KL regularization to adjust reasoning depth.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Across seven benchmarks, FAST reports state-of-the-art accuracy, over 10% average accuracy improvement over the base model, and 32.7–67.3% shorter reasoning than slow-thinking models.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The reported results indicate that FAST balances reasoning length and accuracy across the evaluated LVLM reasoning benchmarks.

    Source: §7 · p. 10

    Evaluation was limited to models up to 32B parameters, and the approach’s effectiveness at 70B+ scale may differ.

    Source: A Limitations · p. 15

Abstract

from arXiv · show

When applying reinforcement learning--typically through GRPO--to large vision-language model reasoning struggles to effectively scale reasoning length or generates verbose outputs across all tasks with only marginal gains in accuracy. To address this issue, we present FAST-GRPO, a variant of GRPO that dynamically adapts reasoning depth based on question characteristics. Through empirical analysis, we establish the feasibility of fast-slow thinking in LVLMs by investigating how response length and data distribution affect performance. Inspired by these observations, we introduce two complementary metrics to estimate the difficulty of the questions, guiding the model to determine when fast or slow thinking is more appropriate. Next, we incorporate adaptive length-based rewards and difficulty-aware KL divergence into the GRPO algorithm. Experiments across seven reasoning benchmarks demonstrate that FAST achieves state-of-the-art accuracy with over 10\% relative improvement compared to the base model, while reducing token usage by 32.7-67.3\% compared to previous slow-thinking approaches, effectively balancing reasoning length and accuracy.

1 Introduction

FAST-GRPO targets LVLM reasoning methods that either fail to scale response length or overthink across tasks, by adapting reasoning depth to question characteristics. Across seven benchmarks, FAST reports higher accuracy with shorter responses than prior approaches.

1 Introduction

  • RL-only methods struggle to scale reasoning length, while SFT-RL methods can produce verbose responses with marginal accuracy gains and lower accuracy on simpler questions.On Geometry, R1-OneVision’s chains are approximately 2× longer than its base model across difficulty levels; on simple questions, accuracy is 69.5% vs. 72.7%.Source: §1 · p. 2
  • Prior approaches to concise reasoning in LLMs use training-time length rewards or inference-time prompts, but the cited work says their effectiveness for visual inputs and question characteristics remains largely unexplored.The authors state that, to their knowledge, no existing work effectively balances fast and slow thinking in LVLMs.Source: §1 · p. 2
  • FAST-GRPO adapts reasoning depth to multimodal question characteristics using difficulty estimates, adaptive length-based rewards, and difficulty-aware KL divergence.The rewards encourage concise or detailed reasoning by question characteristics, while KL regularization varies with estimated difficulty.Source: §1 · p. 2
  • FAST reports state-of-the-art reasoning accuracy, over 10% average accuracy improvement over the base model, and 32.7–67.3% shorter reasoning than slow-thinking models.Figure 1 summarizes the higher average accuracy and shorter average response lengths across seven benchmarks.Source: Figure 1 · p. 1, §1 · p. 2

2 Background: Group Relative Policy Optimization

The background describes GRPO as an extension of PPO that replaces the value model with group-relative reward estimation. Its objective uses accuracy and format rewards, alongside clipping and KL regularization.

2 Background: Group Relative Policy Optimization

  • GRPO extends PPO by replacing the value model with group-relative reward estimation and optimizes the objective in Equation 1.The passage identifies the method as Group Relative Policy Optimization.Source: §2 · p. 2
  • The objective’s ε is the clipping hyperparameter, while β controls the KL regularization coefficient..Source: §2 · p. 2
  • The described GRPO setup uses an accuracy reward for responses equivalent to the answer and a format reward for adherence to the specified format..Source: §2 · p. 2

3 Pilot Experiments

Pilot experiments on Geometry 3K examine how length rewards and training-data difficulty distributions affect LVLM response length and accuracy. They report that length can vary substantially with modest accuracy changes, while difficulty distributions are associated with different length–accuracy patterns.

3.1 Length Rewards Analysis

  • Length rewards produced responses from 180 to 700 tokens, while Naive GRPO remained around 330 tokens; accuracy changed only ±3%.The authors interpret this decoupling as evidence that LVLMs can maintain reasoning performance across different response lengths.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Data Distribution Analysis

  • Training exclusively on Hard samples produced significantly longer responses with only marginal accuracy improvements, whereas Easy samples produced shorter responses while improving accuracy.Models trained on Medium samples showed a modest length increase and the highest accuracy.Source: §3.2 · pp. 3–4, Figure 3 · p. 3
  • The pilot findings characterize question difficulty as an implicit regulator of reasoning length and suggest using data distribution to pursue adaptive fast-slow thinking..Source: §3.2 · p. 4

4 Fast-Slow Thinking GRPO

FAST-GRPO estimates multimodal question difficulty and uses it to guide sampling, response-length rewards, and KL regularization. The design encourages more exploration for difficult questions and efficient responses for simpler ones.

4.1 Multimodal Question Difficulty Estimation

  • The difficulty estimate combines model-relative difficulty, measured online from 1 − pass@k, with intrinsic visual complexity measures including GLCM and ViT classification entropy.The combined score is described as reflecting both visual complexity and difficulty for the policy model.Source: §4.1 · p. 4
  • Slow-to-Fast Sampling has binary and continuous variants that shift training from harder toward easier questions over time.The binary version uses distinct phases; the continuous version gradually shifts sampling probability.Source: §4.1 · pp. 4–5

4.2 FAST-GRPO

  • FAST-GRPO combines accuracy and format rewards with a difficulty-aware length reward that adjusts response-length incentives by question difficulty.For less complex questions, the reward favors shorter correct responses; for complex questions, it favors longer incorrect responses, with the reward capped at 1.Source: §4.2 · p. 5
  • Difficulty-aware KL regularization varies the coefficient by question difficulty because no single static β optimally serves all difficulty levels.Lower KL constraints allow broader exploration on challenging questions, while stronger regularization helps maintain performance on simpler tasks.Source: §4.2 · p. 5
  • The gradient update combines an advantage signal with adaptive KL regularization; the length-normalization term also affects per-token updates for correct and incorrect responses.The passage describes weaker KL regularization for high-difficulty questions and stronger regularization for low-difficulty questions.Source: §4.2 · p. 6

5 Experiments

Across seven benchmarks, FAST improves reasoning accuracy while adapting response length, with ablations and analyses examining its data sampling, rewards, and difficulty-aware optimization. Error analysis finds visual misinterpretation remains the dominant bottleneck.

5.1 Experimental Setup

  • Experiments evaluate accuracy and response length on seven multimodal benchmarks, using baselines spanning SFT, RL-only, and combined SFT-RL approaches.The benchmarks cover mathematical reasoning and general multimodal abilities; additional evaluations include science, open-domain VQA, visual perception, and calibrated evaluation.Source: §5.1 · p. 6

5.2 Main Results

  • FAST reaches 73.8 on MathVista and 50.6 on MathVerse, improves Qwen2.5-VL-7B accuracy by over 10% on average, and shortens reasoning 32.7–67.3% versus slow-thinking methods.It also improves Qwen2.5-VL-3B accuracy by over 14%; on more challenging questions, responses are longer than on easy ones.Source: §5.2 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7

5.3 Ablations

  • FAST’s ablations show that data sampling matters for accuracy, thinking rewards cut response length, and difficulty-aware regularization improves benchmark performance.Without data sampling, accuracy degrades across benchmarks; thinking rewards reduce relative response length by 42%, with MathVision changing from 31.5 to 30.6, while regularization adds 1.8 absolute points on MathVista.Source: §5.3 · p. 8, Table 5 · p. 8

5.4 Analyses

  • Slow-to-Fast sampling outperforms alternatives in the reported analysis: Fast-to-Slow degrades on challenging MathVision, while Dynamic Sampling produces 80% longer responses without better accuracy.FAST with 18K samples also performs better across benchmarks than the tested SFT approaches using 260K GPT-4o or 200K DeepSeek-R1 examples.Source: §5.4 · p. 8
  • The 80th-percentile difficulty threshold balances accuracy and conciseness; medium-difficulty training gives the best overall performance, while difficulty-specific training transfers unevenly across test tiers.The 50th-percentile threshold has comparable accuracy but 37% longer responses; hard-sample training improves hard-test performance to 15.1% versus a 5.5% base, but degrades easy performance.Source: §5.4 · pp. 8–9, Figure 4 · p. 9
  • Difficulty-dependent KL effects and image-complexity checks support adaptive optimization: lower β helps hard questions but risks forgetting, while higher β supports easy questions and MM-Vet generalization.The combined image-complexity metric correlates moderately with human judgments (SRCC=0.49, PLCC=0.54); a weighted-sum difficulty alternative differs by less than 0.5% accuracy.Source: §5.4 · pp. 8–9, Figure 4 · p. 9

5.5 Case Studies and Failure Mode Analysis

  • FAST-7B’s recurring MATHVISTA errors include visual perception failures, reasoning-error propagation, and knowledge conflicts or gaps; it reduces the latter two categories by approximately 27% and 19% versus its base model.The reported categories include misreading visual cues, mid-chain mistakes, and language priors overriding visual evidence.Source: §5.5 · pp. 9–10
  • Visual misinterpretation remains the dominant bottleneck, accounting for over half of FAST-7B’s errors.The authors point to fine-grained OCR, calibrated scale reading, chart and graph extraction, and spatial grounding as possible input-stage improvements.Source: §5.5 · p. 10

6 Related Work

Related work covers LVLM slow-thinking methods and LLM approaches to overthinking, noting that text-only fast-slow methods remain largely unexplored for LVLM reasoning.

Slow-thinking methods for LVLMs

  • LVLM slow-thinking work includes SFT-RL methods that use distilled reasoning trajectories and RL-only methods that apply reinforcement learning to curated visual questions.The cited discussion notes that SFT-RL can behavior-clone overthinking, while RL-only methods can struggle to scale response length.Source: §6 · p. 10

Fast-Slow thinking methods for LLMs

  • LLM overthinking approaches shorten or adapt reasoning through inference-time prompts, token budgets, training objectives, varied-length chains, and length-based rewards.The passage says these approaches remain largely unexplored for LVLM reasoning.Source: §6 · p. 10

7 Conclusion

FAST adapts LVLM reasoning depth to question characteristics and balances accuracy with response length. Evaluation was limited to models up to 32B parameters, leaving its behavior at larger scales uncertain.

7 Conclusion

  • FAST combines question-characterization metrics, adaptive thinking rewards, and difficulty-aware KL regularization to dynamically adjust reasoning depth.The framework is designed to address overthinking in LVLMs.Source: §7 · p. 10
  • FAST achieves state-of-the-art accuracy, improves over the base model by more than 10%, and reduces token use versus prior slow-thinking approaches.The conclusion describes the result as balancing reasoning length and accuracy.Source: §7 · p. 10

A Limitations

  • Evaluation covered models only up to 32B parameters; the fast-slow mechanism may behave differently in models with 70B+ parameters.The stated reason for this scope limit is computational resource constraints.Source: A Limitations · p. 15

B Implementation Details

The appendices describe FAST’s training setup, dataset composition, length-reward comparison, and image-complexity rating prompt.

B Implementation Details

  • FAST uses Qwen2.5-VL-3B and 7B base models, with 10 training epochs, batch size 512, and a maximum sequence length of 4096.Training uses an 18K dataset, learning rate 1e-6, and BF16 precision; the reported training run required approximately 600 H100 GPU hours.Source: B Implementation Details · p. 15

C Datasets

  • The training data spans mathematics, visual QA, science, and figure understanding, with mathematics the largest category.Figure understanding is next largest, followed by science and visual QA.Source: C Datasets · p. 16

D Length Rewards

E Human Evaluation Prompt

  • The human-evaluation prompt rates image complexity from 1 to 5 using factors including object count, clarity, information, spatial relationships, and reasoning steps.The scale ranges from very simple to very complex images.Source: E Human Evaluation Prompt · p. 17

F Naive GRPO Results

Naive GRPO’s training accuracy continued rising while its validation accuracy peaked and then fell, a pattern the authors identify as overfitting. Later outputs showed reasoning that did not support the final answer.

F Naive GRPO Results

  • Validation accuracy peaked at 0.40 around step 150, then declined despite continued training-accuracy gains.The authors identify the divergence between training and validation performance as overfitting.Source: F Naive GRPO Results · pp. 17–18
  • Later naive-GRPO outputs could give a correct final answer while presenting chain-of-thought reasoning that contradicted it.The geometry example’s reasoning concluded 70° while its final answer was 110°; the authors interpret such inconsistency as memorization rather than generalizable reasoning.Source: F Naive GRPO Results · p. 18, Figure 9 · p. 18
  • The authors recommend selecting naive-GRPO checkpoints by validation performance, around step 150 in their experiments, before severe overfitting.They attribute validation deterioration to optimizing for the training distribution at the expense of generalization.Source: F Naive GRPO Results · p. 18

G Statistical Significance Analysis

FAST-7B significantly improved accuracy on six of seven benchmarks and reduced response length across all benchmarks. The appendix also reports robustness checks for difficulty scoring and a trade-off in continuous sampling.

G Statistical Significance Analysis

  • FAST-7B significantly improved accuracy on 6 of 7 benchmarks and reduced response length by 13.9% to 53.8% across all benchmarks.DynaMath’s 0.3 percentage-point accuracy improvement was not significant; Figure 10 visualizes accuracy gains and length reductions, and Table 11 reports benchmark results with standard deviations.Source: Figure 10 · p. 19, G Statistical Significance Analysis · p. 19, Table 11 · p. 19

H Analysis of Multiplicative Difficulty Formulation and Weighted-Sum Alternative

  • The multiplicative difficulty score can be low when model-relative hardness is high but visual complexity is low, potentially signaling fast thinking for a challenging question.The authors identify this as a concern with the product formulation.Source: H Analysis of Multiplicative… · p. 19

Reward design avoids conflict

  • Length-reward shaping is non-zero only for aligned cases: correct and not complex prompts shorter responses, while incorrect and complex prompts longer responses.Misaligned cases receive zero length reward, avoiding a penalty or incorrect encouragement.Source: Reward design avoids conflict · p. 20

Empirical rarity of corner cases

  • High model-relative hardness combined with low image complexity occurred in less than 5% of the 1,000 randomly sampled training examples.The analysis ranked those samples and computed correlations among the difficulty measures.Source: Empirical rarity of corner cases · p. 20

Weighted-sum alternative

  • FAST’s performance was nearly identical with multiplicative and weighted-sum difficulty formulations, supporting robustness to the combination strategy.The comparison is identified in Table 12.Source: Weighted-sum alternative · p. 20, Table 12 · p. 20

I Continuous Slow-to-Fast Sampling

  • Continuous Slow-to-Fast sampling increased average output length by 26% while yielding an example accuracy gain of +0.6pp on MATHVISTA.The authors hypothesize that the chosen Pmax sampled too few easy questions in later epochs; additional tuning may improve the trade-off.Source: I Continuous Slow-to-Fast Sampling · p. 20, Table 13 · p. 21

J Scalability to Larger Models

FAST-32B uses fewer output tokens than two slow-thinking baselines while matching or slightly exceeding their accuracy. Cross-domain evaluations also report gains over the base model in physics, chemistry, and biology.

J Scalability to Larger Models

  • Training FAST-32B stopped after three epochs (∼1,200 GPU hours) due to compute constraints, which the authors say likely yielded a sub-optimal checkpoint.They leave evaluation on ultra-large (≥70B) LVLMs for future work because these models may scale differently.Source: J Scalability to Larger Models · p. 21
  • At 32B, FAST reduces average output length by ∼40% versus Vision-R1-32B (422.1 vs. 710.6 tokens) and ∼22% versus MM-Eureka-32B (422.1 vs. 546.0 tokens).Table 14 reports accuracy and average length for FAST-GRPO and baselines.Source: Table 14 · p. 21, J Scalability to Larger Models · p. 21
  • Average accuracy is comparable to Vision-R1-32B (64.3% vs. 64.6%) and slightly higher than MM-Eureka-32B (64.3% vs. 64.1%).The authors conclude FAST-GRPO scales effectively to larger LVLMs, maintaining its accuracy-efficiency trade-off.Source: J Scalability to Larger Models · p. 21

K Cross-Domain Evaluation

  • The cross-domain evaluations cover science reasoning, open-domain VQA, hallucination analysis, and low-level visual perception beyond math-intensive benchmarks.MM-K12 comprises 2,000 questions evenly covering mathematics, physics, chemistry, and biology.Source: K Cross-Domain Evaluation · p. 21, §K.1 · p. 21

K.1 MM-K12 Scientific Reasoning

  • FAST-7B improves over its base model by +8.4pp in physics, +7.0pp in chemistry, and +8.8pp in biology, while reducing output length by ∼33.8%.Against MM-Eureka-7B, accuracy remains comparable with ∼30% fewer tokens.Source: §K.1 · p. 21

K.2 Additional General Benchmarks

On open-domain VQA and hallucination-robustness benchmarks, FAST matches or slightly outperforms strong slow-thinking baselines while producing shorter outputs.

K.2 Additional General Benchmarks

  • FAST matches or slightly outperforms strong slow-thinking baselines in open-domain VQA and hallucination-robustness benchmarks, while producing shorter outputs.Table 16 reports performance across diverse benchmarks.Source: Table 16 · p. 22, §K.2 · p. 22
  • The additional evaluation includes Bingo and MMHal for open-domain VQA and hallucination analysis, MMVP for low-level visual perception, and MMEval-Pro for calibrated multimodal evaluation.The benchmarks span hallucination, informativeness, visual perception, and math, science, and general VQA.Source: §K.2 · pp. 21–22
  • The reported outcome supports the observation that adaptive reasoning length mitigates hallucination risk in multimodal reasoning [59].Source: §K.2 · p. 22

L Case Study

This section presents case studies of FAST’s response-length adjustments and error modes, and states that LLM usage was not applicable to the method development.

L Case Study

  • FAST uses 59 tokens for simple coordinate identification versus R1-OneVision’s 349, and solves complex geometry correctly in 375 tokens versus R1-OneVision’s verbose 676-token solution.The base model makes a critical angle-calculation error on the complex problem.Source: L Case Study · p. 22, Figure 11 · p. 23
  • The case studies present three representative failure categories: visual perception failure, reasoning error propagation, and knowledge conflict and gap.Figures 12–14 provide task-specific examples of these errors.Source: L Case Study · p. 22
  • The model answers 600 ml instead of 400 ml after confusing a beaker’s total capacity with its highest marked measurement.The case identifies the error as misinterpretation of visual context.Source: L Case Study · p. 23, Figure 12 · p. 23
  • The model selects 34° instead of 58° after correctly applying the tangent ratio but mapping the computed angle to the wrong target.The example shows an intermediate mistake propagating to the final answer.Source: L Case Study · p. 24, Figure 13 · p. 24
  • The model answers 451 instead of 839 for a TCP ACK number after applying general protocol knowledge over the diagram’s sequence data.It misidentifies message 3’s sequence range.Source: L Case Study · p. 24, Figure 14 · p. 24

2. Limitations

  • The paper notes that resource constraints limited evaluation to models up to 7B parameters, and that effectiveness may differ for models of 70B or more.The checklist also reports that the paper does not explicitly discuss potential positive and negative societal impacts.Source: §2 · pp. 25, 28

NeurIPS Paper Checklist

  • The checklist reports that experimental results include error bars and that implementation details are available; it also states the code and model checkpoint are open source.It points to Figure 10 and Table 11 for statistical significance information and to Sections 5.1 and B for reproducibility details.Source: §2 · pp. 26–27

2. Limitations

  • The paper marks LLM usage as not applicable, stating that its core method development does not involve LLMs as an important, original, or non-standard component.Source: §2 · p. 31

Related papers

Loading 2504.18458v2…