Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models

1 Introduction

• Prior RL approaches such as GRPO improve reasoning but can generate unnecessarily long, redundant trajectories, motivating a model-controlled choice about when to think.Existing responses to inefficiency include rule-based penalties and external controls.Source: §1 · p. 1
• 51% of AITZ questions can be answered correctly with the entire reasoning trace omitted, indicating potential for thought-token savings without accuracy loss.The authors present this as empirical evidence that reasoning is not always necessary.Source: §1 · p. 1
• Prompting models to skip reasoning on easier queries does not reliably produce adaptive omission, as math-enhanced VLMs tend to generate full traces regardless of difficulty.The authors frame the ability to decide whether to think as distinct from reasoning capacity and propose activating it through SFT format-following.Source: §1 · p. 2
• TON combines thought dropout during SFT with GRPO self-exploration to learn when to skip reasoning and when to retain detailed inference.In SFT, reasoning traces are replaced with empty thought segments; GRPO then refines the policy through rewards without extra regularization.Source: §1 · p. 2
• Across GSM8K, CLEVR, SuperCLEVR, GeoQA, and AITZ, TON reduces completion length without compromising performance, including up to 17% higher GeoQA accuracy than vanilla GRPO.Reported savings include 87% of tokens on CLEVR, 65% on GeoQA, and an AITZ task-level output reduction from 3.6K to 0.9K tokens.Source: §1 · p. 2

Reinforcement Learning for Vision–Language Models

• GRPO uses group-sampled reasoning processes, answer-verification rewards, and relative advantages to promote diverse reasoning paths without a value function [5].Follow-up approaches still assume every question needs full reasoning, leading to lengthy decoding [28] [29].Source: §2 · pp. 2–3
• Existing VLM post-training includes RLHF with PPO [24] [22] and DPO preference optimization [27], alongside RL methods such as GRPO [5].The related work describes these as approaches to improving model outputs or reasoning capabilities.Source: §2 · p. 2

Thinking in Language Models

• Prior language-model work includes long2short compression [36], terminating unproductive reasoning [37], and token-budget-aware policies, while this work asks whether reasoning is needed at all.The paper frames its selective-reasoning approach as distinct from deciding how to reason once reasoning has started.Source: §2 · p. 3

Task Definition

• The task is modeled as an MDP over visual context, query, state, policy, and reward, with the policy generating an answer and positive reward assigned to correct predictions.The objective is to learn an adaptive policy that maximizes expected reward across input settings.Source: §3 · p. 3

Reward Function

• Responses use a required think/answer tag format that separates the reasoning process from the predicted answer for parsing.The format reward is shared, while task outcome rewards may be designed for different tasks or requests.Source: §3 · p. 3

4 TON: Selective Reasoning via Policy Optimization

• 52.1% of answers remained correct without “think,” and 14.5% were correct only without it, suggesting explicit thinking is not always necessary.The observation compares inference with and without a thinking process.Source: §4 · p. 3

4.1 First SFT stage: Thought Dropout

• Thought Dropout randomly injects empty thought segments during SFT, treating think and non-think as output formats so the model can learn to decide later.The authors use random dropout because which samples should skip reasoning is difficult to determine across models.Source: §4.1 · p. 4
• The cold-start thought corpus is generated by the base policy itself, which is prompted to infer intermediate reasoning from visual context, a query, and the ground-truth answer.The authors report that this reverse-thinking approach curates thought data without external models.Source: §4.1 · p. 4

4.2 Second RL stage: Group Relative Policy Optimization

• GRPO lets the model explore whether to retain or skip thoughts after SFT teaches the skip-thought format.The method samples candidate responses and evaluates them with a reward function; group-relative advantages guide policy updates toward higher-advantage responses.Source: §4.2 · pp. 4–5
• TON can sample both empty-think and think responses, shifting attention to response-distribution diversity relative to vanilla GRPO’s think responses.The paper contrasts this with DAPO’s emphasis on advantage distribution.Source: §4.2 · p. 5
• Task-aware outcome rewards combine discrete answer matching with continuous matching for spatial outputs, including point-in-box and distance-threshold criteria.The applicable reward components are summed as ro = rd + rc.Source: §4.2 · p. 5

5.2 Q1: Performance and Efficiency Comparison between TON and GRPO

• On AITZ OOD domains, TON retains comparable generalization while reducing task-level output length from 3K to 0.9K, a 70% token saving.Table 3 reports step-level action-type and exact-match accuracy across the OOD domains.Source: Table 3 · p. 7, §5.2 · p. 7
• TON reduces average completion length by up to 90% versus GRPO while achieving comparable or superior performance.The comparison covers CLEVR and GeoQA under both 3B and 7B settings.Source: §5.2 · p. 7
• On GSM8K, TON reduces response length while maintaining high accuracy, extending the evaluation to a text-only task.Source: §5.2 · p. 7

5.3 Q2: Skip Thought Ratio Analysis

• On GeoQA, the skip ratio rises alongside training reward, while think-output lengths remain comparable to vanilla GRPO.Source: §5.3 · p. 8
• Across 20%, 50%, and 80% thought-dropout settings on AITZ, skip ratios increase during training, though the 80% setting remains relatively stable.The 20% setting shows a rapid increase, and the three variants have close reward curves.Source: §5.3 · p. 8
• TON keeps skip ratios high for easy counting and low for harder questions; random dropout slightly outperforms difficulty-aware dropout (51.0% vs. 50.5%).The authors suggest hand-crafted difficulty heuristics may introduce noise or unintended bias.Source: §5.3 · pp. 8–9

5.4 Q3: Emprical Verfication of SFT Significance in TON

• Hybrid-prompt GRPO produced only 2 skip cases in GeoQA and none in AITZ; the authors identify thought-dropout SFT as necessary to establish skip-format behavior.Source: §5.4 · p. 9

5.5 Qualitative Examples

• In examples, TON omits thoughts for simple questions but provides focused reasoning for complex visual-occlusion cases, unlike GRPO's reasoning in both cases.The AITZ comparison also describes TON as saving 60% of tokens without sacrificing task accuracy.Source: §5.5 · pp. 9–10, Table 4 · p. 10

• TON combines thought dropout in SFT with reward-guided refinement through GRPO to make selective reasoning a controllable, trainable behavior.Source: §6 · p. 10
• Across diverse reasoning tasks, TON reduces completion length by up to 90% without sacrificing performance and sometimes improves it.Source: §6 · p. 10
• The authors argue these findings challenge the assumption that full reasoning traces are always beneficial and point toward more efficient, human-like reasoning strategies.Source: §6 · p. 10

• Using the think source during SFT results in higher performance but longer output lengths.Source: A Motivation Experiments · p. 15
• Excluding the think source results in lower performance with shorter outputs.Source: A Motivation Experiments · p. 15
• Table 5 compares agent-dataset accuracy and step-level output length across think-source and GRPO-reward-format settings.Source: Table 5 · p. 15, A Motivation Experiments · p. 15

B Rewards for Downstream Tasks

• General and math VQA reward format compliance plus whether the predicted answer exactly matches the ground-truth number.Both task types use the same stated reward components.Source: B Rewards for Downstream Tasks · p. 15
• Agent VQA rewards format compliance, correct action type, and continuous coordinate accuracy for clicks.Coordinates are normalized to 0–1, with θ = 0.14 following [19].Source: B Rewards for Downstream Tasks · p. 15

C Limitations

• The evaluation covers 3B and 7B models; larger models such as 235B and proprietary VLMs were not evaluated.The authors attribute these boundaries to computational resources and lack of access to proprietary source code.Source: C Limitations · p. 15

D Broader Impact

• TON combines SFT and RL through thought dropout, and the authors describe it as informing flexible injection of prior knowledge rather than hand-crafted reward rules.They also state that the work may affect RL research in VLMs and LLMs.Source: D Broader Impact · p. 15

E Dataset

• The datasets span synthetic counting, geometric problem solving, and GUI navigation, with selected 1,000-example training sets and task-specific test settings.CLEVR testing includes 200 unseen samples and 200 OOD Super-CLEVR counting problems; GeoQA uses 1,000 angle and side-length problems; AITZ uses 1,000 General-domain examples and evaluates other OOD domains.Source: E Dataset · pp. 15–16

F Setup

• SFT uses full-parameter LlamaFactory training for Qwen2.5-VL-3B/7B and takes no longer than 15 minutes; GRPO is run with vLLM.The setup also specifies θ = 0.14 following [19] and zero1_no_optimizer GRPO settings.Source: F Setup · p. 16, Table 6 · p. 16, Table 7 · p. 16

G.1 TON on Math–GeoQA

• The GeoQA training visualizations track variables over training for both Qwen2.5-VL-3B and 7B.Figures 8 and 9 show the progression, without a specific outcome stated in these passages.Source: §G.1 · p. 17, Figure 8 · p. 17, Figure 9 · p. 18

G.2 TON on Counting–CLEVR

• The CLEVR visualization tracks training-variable progression for Qwen2.5-VL-3B.Figure 10 is identified as the TON-versus-GRPO visualization.Source: §G.2 · p. 18, Figure 10 · p. 18

G.3 TON on Mobile Agent–AITZ

• The AITZ visualizations track training-variable progression for Qwen2.5-VL-3B and 7B.Figures 11 and 12 show TON and GRPO during training.Source: §G.3 · p. 18, Figure 11 · p. 19, Figure 12 · p. 19

G.4 OOD Performance of TON on CLEVR

• TON performs better than vanilla GRPO on IID and especially OOD CLEVR scenarios under easy reasoning tasks.The comparison is described as helping avoid vanilla GRPO overfitting to the training set; Table 9 covers CLEVR and Super-CLEVR.Source: §G.4 · p. 19, Table 9 · p. 20

G.5 Different Thought Dropout Probabilities

• The dropout-probability visualization compares GRPO training progress at 20%, 50%, and 80% thought dropout.The passages identify the compared settings but state no specific outcome.Source: §G.5 · p. 20, Figure 13 · p. 20

G.6 Skip-thought Ratio on Different benchmarks

• The appendix plots TON’s skip-thought ratios across VQA benchmarks during training.Figure 14 is captioned as showing skip ratio on three benchmarks, while Figure 21 is described as showing ratios under different VQA benchmarks.Source: §G.6 · p. 20, Figure 14 · p. 20

G.7 Prompt v.s. SFT on different benchmarks

• Figures 15–17 compare training progress when skip-thought is injected in the prompt versus during SFT across CLEVR, AITZ, and GeoQA.The passages identify the comparison and tasks but state no specific outcome.Source: §G.7 · p. 21, Figure 15 · p. 21, Figure 16 · p. 21

G.8 Visualization Examples

• A CLEVR counting example contrasts TON’s selective reasoning on differently difficult images with GRPO’s reasoning on both; a Super-CLEVR example contrasts TON’s concise answer with GRPO’s output exceeding 500 tokens.The Super-CLEVR example is out of distribution.Source: Figure 18 · p. 22, Table 10 · p. 23

G.9 Prompt for AITZ task

• The AITZ prompt uses an observation and action-history sequence to produce the next action, then waits for another observation; actions use a dictionary with action, value, and position fields.Actions include CLICK, TYPE, and PRESS HOME; click positions are required and relative screenshot coordinates are scaled to 0-1.Source: §G.9 · p. 24

H Comprehensive Comparison of Length, Rewards, and Skip Ratio Across Three Benchmarks

• TON reduces completion length and increases skip ratio as rewards increase across three benchmarks.Figures 19–21 visualize rewards, completion length, and skip ratio, respectively.Source: H Comprehensive Comparison of Length,… · p. 25, Figure 19 · p. 25, Figure 20 · p. 25

I Reward for length rather than SFT

• With a length reward, the length reward remains zero during the first 100 training steps.The appendix describes rewarding outputs with an empty thought and shows the training process in Figures 22 and 23.Source: I Reward for length rather than SFT · p. 25, Figure 22 · p. 26, Figure 23 · p. 26

J More cases

• An OOD Super-CLEVR example shows GRPO producing over 500 tokens while TON gives a concise response; another figure reports TON performing well on AITZ app-uninstallation tasks without extensive reasoning.The GeoQA comparison figures are identified, but these passages state no specific outcome for them.Source: Figure 24 · p. 27, Figure 25 · p. 27, Figure 26 · p. 28