PaperlayerView exact revision on arXiv

Source-linked AI summary

SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuning

Fangxun Shu, Yongjie Ye, Yue Liao, Zijian Kang, Weijie Yin, Jiacong Wang, Xiao Liang, Shuicheng Yan, Chao Feng

arXiv:2511.02280v2cs.CVcs.CL

TL;DR

Existing approaches can reward correct answers without sound reasoning, while uniform thinking strategies risk overthinking simple tasks and underthinking complex ones. SAIL-RL combines rewards for reasoning quality and adaptive decisions about when to reason. Experiments report stronger benchmark performance, competitive results against proprietary systems, and reduced hallucinations.

  • Problem

    Outcome-focused and uniform reasoning approaches leave reasoning quality and task-adaptive effort insufficiently addressed.

    Source: §1 · pp. 1–2

  • Method

    SAIL-RL post-trains MLLMs with a Thinking Reward for reasoning quality and a Judging Reward for choosing between direct answering and deep reasoning.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    SAIL-RL reports state-of-the-art results among comparable models, competitive performance with leading proprietary systems such as GPT-4o, and substantially reduced hallucinations.

    Source: §1 · p. 2, §5 · pp. 8–9

  • Takeaways & Limitations

    The findings support SAIL-RL as a robust and scalable paradigm for training reliable, meta-cognitive MLLMs.

    Source: §5 · p. 9

Abstract

from arXiv · show

We introduce SAIL-RL, a reinforcement learning (RL) post-training framework that enhances the reasoning capabilities of multimodal large language models (MLLMs) by teaching them when and how to think. Existing approaches are limited by outcome-only supervision, which rewards correct answers without ensuring sound reasoning, and by uniform thinking strategies, which often lead to overthinking on simple tasks and underthinking on complex ones. SAIL-RL addresses these challenges with a dual reward system: the Thinking Reward, which evaluates reasoning quality through factual grounding, logical coherence, and answer consistency, and the Judging Reward, which adaptively determines whether deep reasoning or direct answering is appropriate. Experiments on the state-of-the-art SAIL-VL2 show that SAIL-RL improves reasoning and multimodal understanding benchmarks at both 4B and 8B scales, achieving competitive performance against commercial closed-source models such as GPT-4o, and substantially reduces hallucinations, establishing it as a principled framework for building more reliable and adaptive MLLMs. The code will be available at https://github.com/BytedanceDouyinContent/SAIL-RL.

1. Introduction

The introduction argues that outcome-only rewards can accept flawed reasoning, while uniform strategies can overthink easy tasks and underthink hard ones. SAIL-RL pairs rewards for reasoning quality and adaptive effort, and experiments report gains across reasoning and multimodal benchmarks.

1. Introduction

  • Outcome-only supervision rewards correct final answers without evaluating reasoning, allowing flawed or fabricated reasoning to be reinforced as false correctness.The introduction links incoherent or redundant traces to inaccurate answers and exacerbated hallucinations.Source: §1 · pp. 1–2
  • Uniform reasoning strategies can waste effort and produce noisy traces on simple tasks, yet shallow reasoning and inaccurate answers on complex tasks.Figure 1 illustrates overthinking on a simple problem and a correct answer reached through flawed reasoning.Source: Figure 1 · p. 1, §1 · p. 2
  • SAIL-RL assesses reasoning quality through logical coherence, factual grounding, and trace-to-answer consistency, while its Judging Reward selects direct answers for simple tasks and full reasoning for complex ones.The framework is presented as a post-training approach for MLLMs.Source: §1 · p. 2
  • SAIL-VL2-Thinking gains over the baseline and conventional RL approaches, reaches state-of-the-art results on multiple 8B reasoning benchmarks, and substantially reduces hallucinations.The introduction also reports leading OpenCompass performance and competitive accuracy on general multimodal understanding tasks.Source: §1 · p. 2, Figure 2 · p. 2

2. Related Work

Related work distinguishes direct, perception-oriented System 1 models from deliberative System 2 models, while noting the cost of extended reasoning and the open challenge of switching modes. Prior multimodal RL methods commonly target efficiency or reasoning quality separately; SAIL-RL combines both aims.

2. Related Work

  • System 1 models prioritize rapid visual perception but can struggle with complex logical derivations, whereas System 2 models use explicit thinking tokens and can improve visual math and logic at high inference cost.Routing mechanisms have begun to switch between modes, but optimizing when and how to switch remains open.Source: §2 · pp. 2–3
  • SAIL-RL’s reward overview evaluates Format, Answer, Thinking, and Judging, with Gemini providing nuanced semantic rewards for Thinking and Judging.Source: Figure 3 · p. 3
  • Multimodal RL methods address overthinking simple questions or incorrect reasoning chains, with prior approaches often optimizing efficiency or reasoning quality in isolation.R-4B uses bi-mode policy optimization; VisualPRM and URSA use visual process reward models for step-by-step verification.Source: §2 · p. 3

3. SAIL-RL

SAIL-RL combines rewards for reasoning quality and adaptive decisions about when to reason. Its cascading reward couples judgment, reasoning, and answer correctness, and a two-stage post-training strategy first establishes a structured response format, then optimizes quality and efficiency.

3. SAIL-RL

  • SAIL-RL unifies reasoning effectiveness and efficiency through a dual-reward mechanism: Thinking Reward assesses reasoning quality, and Judging Reward enables adaptive reasoning.Source: §3 · p. 3

3.1. Thinking Reward: How to Think

  • Thinking Reward evaluates reasoning through logical coherence, factual grounding, and answer consistency, including whether the final answer follows from the reasoning trace.Logical checks cover problem formulation and deductive soundness; factual checks use visual, textual, and, when needed, world-knowledge sources.Source: §3.1 · pp. 3–4

3.2. Judging Reward: When to Think

  • Judging Reward scores a model’s decision against task-complexity labels, rewarding reasoning for complex tasks and direct answering for simple ones.Source: §3.2 · p. 4

3.3. Cascading Reward System

  • The Cascading Reward System combines judging, thinking quality, and answer accuracy multiplicatively, with format reward weighted separately; the paper describes it as a logical AND gate.With α = 0.9, a zero in any of the three multiplied components nullifies that component’s reward.Source: §3.3 · p. 4

3.4. Post-training Strategy

  • Post-training first uses LongCoT SFT to teach a judgment-reasoning-answer sequence, then tunes the model with RL and the reward system on difficulty-filtered verifiable tasks.The SFT objective uses next-token prediction over the structured sequence; RL uses DAPO (Yu et al., 2025).Source: §3.4 · pp. 4–5

4. Experiments

Experiments evaluate SAIL-RL on reasoning and general multimodal benchmarks, then test its rewards, training behavior, and generalization. The results show stronger benchmark performance, adaptive reasoning, and robustness across reward models and model scales.

4.1. Experimental Setups

  • SAIL-VL2 is trained with full-parameter SFT followed by DAPO-based RL, using 400K LongCoT SFT samples and 70K mixed RL samples.Evaluation covers advanced reasoning and general multimodal understanding benchmarks, using VLMEvalKit and GPT-4o-Mini as judge.Source: §4.1 · p. 5

4.2. Benchmark Performance

  • SAIL-VL2-8B-Thinking scores 59.3 on multimodal reasoning and 80.4 on multimodal understanding, with open-source state-of-the-art averages in both categories.The reasoning score improves by +20.0 over the SAIL-VL2-8B baseline (39.3); the understanding results include 61.5 on HallusionBench and 93.6 on ChartQA.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.3. Thinking Reward Enhances Reasoning Capability

  • Adding the thinking reward improves STEM reasoning over answer-only training, while training dynamics show rising thinking scores and preserved reasoning-answer consistency.The ablation reports gains of +3.2 on WeMath, +1.6 on DynaMath, +2.4 on LogicVista, and +2.1 on MathVision; the answer-only baseline's consistency degrades into reasoning collapse.Source: §4.3 · pp. 5–6, Table 3 · p. 6, Figure 4 · p. 7

4.4. Judging Reward Enables Adaptive Reasoning

  • SAIL-RL adapts thinking to task complexity, and its judging reward outperforms always-thinking on perception-heavy benchmarks.LogicVista and MMMU trigger reasoning at 100.0 and 99.3, while OCRBench triggers at 7.5; SAIL-RL uses 1.2× tokens on OCRBench versus 2.9× for Judge-w/o-reward, which triggers on 47.6% of samples.Source: §4.4 · p. 7, Figure 5 · p. 7, Table 4 · p. 7

4.5. Reward Mechanism Analysis

  • The cascading product beats additive reward aggregation, discrete rewards outperform continuous scores, and equal weighting avoids cross-task trade-offs.The product gains +3.3 on MathVision, discrete rewards yield a +3.2 average improvement, and equal weights balance logic, hallucination, and consistency components.Source: §4.5 · pp. 7–8, Table 6 · p. 7, Table 7 · p. 8

4.6. Robustness and Generalization

  • SAIL-RL outperforms the SFT baseline across tested reward models and improves reasoning and general benchmarks across model scales.For the 7B model, MathVision rises from 25.4 to 30.2, exceeding the answer-only baseline of 27.1; results across GPT-5, Gemini-2.5-Pro, and Qwen2.5-VL-32B are comparable.Source: §4.6 · p. 8, Table 10 · p. 8, Table 9 · p. 8

5. Conclusion

SAIL-RL is an RL framework that teaches MLLMs when and how to think through a dual-reward mechanism. The conclusion reports state-of-the-art results among comparable models and performance competitive with leading proprietary systems such as GPT-4o.

5. Conclusion

  • SAIL-RL combines rewards for reasoning quality and adaptive reasoning to guide MLLMs on how and when to think.The framework is presented as moving beyond outcome-only reward.Source: §5 · p. 8
  • SAIL-RL achieves state-of-the-art results among comparable models and performance competitive with leading proprietary systems like GPT-4o.Source: §5 · p. 9
  • The authors present SAIL-RL as a robust and scalable paradigm for training reliable, meta-cognitive MLLMs.They characterize it as democratizing advanced reasoning capabilities through efficient post-training.Source: §5 · p. 9

A.1. Data Curation

The data curation pipeline builds separate datasets for long-chain reasoning SFT and mixed-domain RL, with samples structured or filtered to support adaptive reasoning.

A.1. Data Curation

  • LongCoT data are organized in a judge-think-answer format to train models to assess problem complexity and select a reasoning path.The source data span complex logical problems and simple perception questions.Source: §A.1 · p. 12
  • The LongCoT cleaning pass removes extraneous content and deduplicates samples by image-question pairs.Source: §A.1 · p. 12
  • Complex samples receive generated chain-of-thought and a judge label for thinking, while simple perceptual samples receive a no-thinking label and an empty thought section.Answers are standardized within the boxed tag.Source: §A.1 · p. 12
  • LongCoT quality filtering removes trivial reasoning using thought-answer token overlap and balances reasoning-chain lengths.Source: §A.1 · p. 12
  • The LongCoT pipeline produces 400K samples for training when and how to reason.Source: §A.1 · p. 12
  • The RL data pipeline filters STEM questions by difficulty using the SFT model’s pass@4, then combines them with general QA samples.The resulting 70K RL dataset contains 50K STEM problems and 20K General QA instances.Source: §A.1 · p. 12

A.2. Training strategy

Training uses SAIL-VL2 as its base, first applying LongCoT supervised fine-tuning and then RL optimization with the proposed reward system.

A.2. Training strategy

  • The model builds on SAIL-VL2, and experiments use VeOmni for SFT, VeRL for RL, and Gemini-2.5-Pro as reward judger.Experiments run on 64 NVIDIA A100 GPUs with AdamW and a cosine learning-rate schedule.Source: §A.2 · p. 12
  • The LongCoT SFT stage fine-tunes all model parameters for one epoch on 400K samples with a 20K maximum sequence length.The global batch size is 1024 and the learning rate is 1e-6.Source: §A.2 · p. 12
  • The RL stage trains the SFT model for three epochs on 70K mixed samples using DAPO and SAIL-RL rewards.It removes standard KL divergence and dynamically sets clipping ε within [0.20, 0.28].Source: §A.2 · p. 12

B. Qualitative Analysis: Case Study

Two qualitative cases illustrate the distinct roles of the Thinking Reward and Judging Reward: supporting logically sound reasoning and avoiding unnecessary reasoning, respectively.

Case I: Thinking Reward Ensures Logical Fidelity

  • In the geometric reasoning case, the answer-only baseline reaches the correct result with flawed, disjointed reasoning, while SAIL-RL produces a sound step-by-step derivation.The comparison is presented as an example of the Thinking Reward’s role in logical fidelity.Source: B. Qualitative Analysis: Case Study · p. 13, Figure 6 · p. 13

Case II: Judging Reward for Mitigating Computational Redundancy

  • In the high-resolution OCR case, the forced-thinking baseline generates a noisy, unnecessarily complex chain, while SAIL-RL chooses a direct response.The case illustrates the Judging Reward’s adaptive allocation of reasoning effort.Source: B. Qualitative Analysis: Case Study · p. 13, Figure 7 · p. 13

C. Reward Prompt

The reward prompts assess reasoning quality and whether a question requires reasoning. They cover logical consistency, factual accuracy, and structural soundness.

C. Reward Prompt

  • Thinking reward prompts assess reasoning quality through logical consistency, factual accuracy, and structural soundness.Source: C. Reward Prompt · p. 14
  • A judge reward prompt evaluates whether the model can determine if a question requires reasoning.Source: C. Reward Prompt · p. 14
  • Together, the prompts assess both the reasoning process and the model’s decision about when to reason.Source: C. Reward Prompt · p. 14

Judge Reward Prompt

The judge prompt assesses whether a question requires reasoning and whether the model judges that need correctly. It distinguishes direct perception from tasks requiring inference or synthesis.

Judge Reward Prompt

  • The judge evaluates whether the model correctly decides if a question requires reasoning.Source: Judge Reward Prompt · p. 14
  • The evaluation compares an independent task-complexity assessment with the model’s stated judgment.Source: Judge Reward Prompt · p. 14
  • The prompt takes an image, a question, and the model’s judgment as inputs and returns whether the judgment is correct and reasoning is required.Source: Judge Reward Prompt · p. 14
  • It classifies direct observation, recognition, retrieval, counting, and simple OCR as tasks that do not require reasoning.Source: Judge Reward Prompt · p. 14
  • Inference, calculation, synthesis, and combining information from multiple image parts are classified as requiring reasoning.Source: Judge Reward Prompt · p. 14
  • Examples contrast a fire-hydrant color question judged simple with a product-value comparison judged complex.Source: Judge Reward Prompt · pp. 14–15

Thinking Logic Reward Prompt

The reward prompts separately assess logical soundness, consistency between reasoning and answer, and factual grounding. Their checks distinguish validity of reasoning from whether an answer follows that reasoning.

Thinking Logic Reward Prompt

  • Logical soundness requires both a valid problem formulation and error-free deductive execution.The checks cover mapping the image and question into a valid model, then checking calculations, algebra, and contradictions.Source: Thinking Logic Reward Prompt · p. 15
  • The logic prompt evaluates a solution using the image, question, and step-by-step reasoning as its inputs.Source: Thinking Logic Reward Prompt · p. 15
  • A reasoning trace is judged sound only if both structural and deductive checks pass; the prompt does not fact-check empirical constants unless the problem states them.Source: Thinking Logic Reward Prompt · p. 15

Thinking Consistency Reward Prompt

Thinking Hallucination Reward Prompt

Loading 2511.02280v2…