PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning

Ruifeng Yuan, Chenghao Xiao, Sicong Leng, Jianyu Wang, Long Li, Weiwen Xu, Hou Pong Chan, Deli Zhao, Tingyang Xu, Zhongyu Wei, Hao Zhang, Yu Rong

arXiv:2507.22607v2cs.CVcs.AIcs.CL

TL;DR

Multimodal reasoning spans heterogeneous tasks and difficulty levels, while common length rewards can impose a uniform target across prompts. VL-Cogito trains with PCuRL, which progressively adjusts task difficulty and adapts reasoning length; it reports state-of-the-art or highly competitive results across multimodal benchmarks.

  • Problem

    Multimodal tasks vary in semantic content, formulation, and complexity, while uniform length rewards do not account for prompt-specific reasoning needs.

    Source: §1 · p. 1, §4.2.2 · p. 6

  • Method

    PCuRL progressively trains VL-Cogito on harder tasks using online difficulty soft weighting and prompt-specific dynamic reasoning-length rewards.

    Source: §1 · p. 2, §4.2.2 · p. 6

  • Results

    VL-Cogito achieves state-of-the-art or highly competitive performance across multimodal benchmarks spanning mathematical, scientific, and general domains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported approach balances deeper reasoning on complex tasks with concise responses on simpler ones.

    Source: §1 · p. 2

Abstract

from arXiv · show

Reinforcement learning has proven its effectiveness in enhancing the reasoning capabilities of large language models. Recent research efforts have progressively extended this paradigm to multimodal reasoning tasks. Due to the inherent complexity and diversity of multimodal tasks, especially in semantic content and problem formulations, existing models often exhibit unstable performance across various domains and difficulty levels. To address these limitations, we propose VL-Cogito, an advanced multimodal reasoning model trained via a novel multi-stage Progressive Curriculum Reinforcement Learning (PCuRL) framework. PCuRL systematically guides the model through tasks of gradually increasing difficulty, substantially improving its reasoning abilities across diverse multimodal contexts. The framework introduces two key innovations: (1) an online difficulty soft weighting mechanism, dynamically adjusting training difficulty across successive RL training stages; and (2) a dynamic length reward mechanism, which encourages the model to adaptively regulate its reasoning path length according to task complexity, thus balancing reasoning efficiency with correctness. Experimental evaluations demonstrate that VL-Cogito consistently matches or surpasses existing reasoning-oriented models across mainstream multimodal benchmarks spanning mathematics, science, logic, and general understanding, validating the effectiveness of our approach.

1 Introduction

Multimodal reasoning spans heterogeneous tasks and difficulty levels, challenging models to learn across diverse problem types. VL-Cogito uses progressive curriculum reinforcement learning with difficulty weighting and adaptive reasoning-length rewards, and reports competitive or state-of-the-art benchmark performance.

1 Introduction

  • Heterogeneous multimodal problem types and complexities make it challenging to learn reasoning skills across tasks.The cited discussion ranges from chart interpretation and geometry to scientific analysis.Source: §1 · p. 1
  • VL-Cogito achieves state-of-the-art or highly competitive performance across multimodal benchmarks in mathematical, scientific, and general domains.The model is trained directly from its backbone via PCuRL with GRPO, without a cold-start SFT phase.Source: §1 · p. 2
  • PCuRL progressively exposes the model to harder tasks using online difficulty soft weighting and adapts reasoning length to each problem’s complexity.The framework aims to balance depth on complex tasks with concise, efficient responses on simpler ones.Source: §1 · p. 2
  • Ablations report that progressive curriculum learning increases reasoning depth and improves performance on complex tasks.The dynamic length reward encourages concise answers for simple problems and longer reasoning for harder ones.Source: §1 · p. 2

2 Related Work

Prior multimodal RL work addresses specific domains and training challenges through approaches such as data selection, reward design, and advanced RL recipes. This paper contrasts its curriculum-based recipe, combining progressive difficulty weighting with prompt-adaptive reasoning length.

2 Related Work

  • Prior multimodal RL research includes data selection, reward design, and advanced RL recipes aimed at training stability and effectiveness.Examples include difficulty estimation, variance-aware sampling, step-wise rewards, selective replay, and stage-specific skill rewards.Source: §2 · pp. 2–3
  • PCuRL combines training stages based on task difficulty with dynamic reasoning-length rewards, rather than indiscriminately extending reasoning.Figure 1 describes online soft weighting across Easy, Medium, and Hard stages.Source: §2 · p. 3, Figure 1 · p. 3

3 Preliminaries

The preliminaries describe GRPO as a group-relative policy optimization method and RLVR as reward-based training using objective correctness criteria. The latter can combine accuracy and output-format rewards.

3 Preliminaries

  • GRPO estimates generation advantages by comparing candidate responses sampled for the same input.The behavior policy samples a group of G responses, and the method also uses a clipped surrogate objective to limit policy deviation.Source: §3 · p. 3
  • RLVR evaluates outputs against predefined correctness criteria using rule-based rewards.The reward can combine binary accuracy and strict binary format rewards.Source: §3 · pp. 3–4

4 Methodology

The methodology curates multimodal training data and applies PCuRL, a staged reinforcement-learning framework combining difficulty-based weighting with prompt-specific reasoning-length rewards. Its curriculum moves from easy to hard tasks, introducing the length reward in the hard stage.

4.1 Data Curation

  • The training collection spans 23 datasets across six task categories, with open-ended reformulation and difficulty filtering used to select RL samples.The filter removes questions answered with above 50% accuracy over eight trials by Qwen2.5-VL-7B-Instruct.Source: §4.1 · p. 4

4.2 Progressive Curriculum Reinforcement Learning Framework

  • PCuRL uses a GRPO-based, multi-stage curriculum to increase task difficulty while adapting reasoning-chain length to task complexity.It aims to extend reasoning on harder tasks while retaining concise responses on simpler ones.Source: §4.2 · p. 4

4.2.1 Online Difficulty Soft weighting (ODSW)

  • ODSW weights prompt-level advantages using rollout accuracy, allowing higher-weight prompts to contribute more to gradient updates rather than discarding prompts outside a difficulty range.Its continuous piecewise weighting uses a 0.5 threshold and sine and constant components; variants target easy, medium, or hard prompts, and the curriculum uses simpler tasks before harder ones.Source: §4.2.1 · pp. 4–6

4.2.2 Dynamic Length Reward (DyLR)

  • DyLR sets each prompt’s target reasoning length to the average length of its correct rollout responses, or to a preset maximum when none are correct.The prompt-specific target adapts as accuracy improves; a hyperparameter limits the advantage for prompts with zero accuracy to mitigate excessive length growth.Source: §4.2.2 · p. 6

4.2.3 Progressive Curriculum RL

  • The curriculum runs three consecutive easy, medium, and hard stages on the same dataset, using stage-specific difficulty weighting and independently shuffled data order.DyLR is introduced only in the hard stage; that stage takes more steps to converge than the easy and medium stages.Source: §4.2.3 · p. 7

5 Experiments

The experiments assess VL-Cogito across multimodal benchmarks in mathematics, logic, science, and general understanding, comparing it with general-purpose and reasoning-oriented models. Evaluation uses shared prompts and benchmark-specific protocols, with training details specified for the Qwen2.5-VL-Instruct-7B backbone and staged PCuRL.

5.1 Experimental Settings

  • The benchmark suite covers mathematical, logical, chart, scientific, and general image-understanding tasks.It includes Geometry@3K, MathVision, MathVista, MathVerse, LogicVista, ChartQA, ScienceQA, MMMU, EMMA, and MMStar.Source: §5.1 · p. 7
  • VL-Cogito is compared with general-purpose and reasoning-oriented MLLMs, including models of similar size.Baselines include Qwen2.5-VL-Instruct-7B, InternVL variants, LLaVA-OneVision-7B, and multiple reasoning-oriented models.Source: §5.1 · p. 7
  • Table 1 compares models across mathematical, scientific, and general-domain benchmarks, with baselines reevaluated under identical conditions.Parentheses mark scores reported in original papers, dashes mark results excluded due to benchmark contamination, and bold and underline mark best and second-best scores.Source: Table 1 · p. 8
  • Evaluations use a unified prompt, follow official protocols where available, and apply Math-Verify to mathematics questions on other benchmarks.Non-mathematical questions use strict exact matching; supplementary procedures address multiple-choice answers outside the options and some open-ended answers or parser failures.Source: §5.1 · p. 8
  • Training uses Qwen2.5-VL-Instruct-7B with GRPO, progressing through 100-step easy and medium stages and an approximately 200-step hard stage.The best validation checkpoint from each of the first two stages initializes the next; the hard stage incorporates the dynamic length reward.Source: §5.1 · p. 8

5.2 Performance Comparison on Diverse Multimodal Benchmarks

VL-Cogito improves on its backbone across the evaluated domains and achieves the best or highly competitive results against reasoning-oriented models. Ablations and training analyses examine how progressive curriculum, online difficulty weighting, and dynamic length rewards relate to performance and response length.

5.2 Performance Comparison on Diverse Multimodal Benchmarks

  • VL-Cogito improves on Qwen2.5-VL-Instruct across all evaluated benchmarks, with absolute gains including 7.6% on Geometry@3K and 4.9% on LogicVista.It achieves the highest performance on 6 of 10 benchmarks and is lower than VL-Rethinker only on MathVerse, ChartQA, and MMMU.Source: §5.2 · p. 9

5.3.1 Component-wise Performance Decomposition of PCuRL

  • The progressive curriculum alone yields gains on nearly every task, while adding dynamic length reward at the final stage raises the overall average score to 59.5%.Applying dynamic length reward directly to vanilla GRPO produces unstable and inconsistent performance; the authors hypothesize that early misclassification of ill-structured queries can prompt excessive reasoning length and hinder convergence and generalization.Source: §5.3.1 · pp. 9–10

5.3.2 Impact of Online Difficulty Soft Weighting on Model Performance

  • Online difficulty soft weighting outperforms binary weighting on most benchmarks, with larger advantages when target difficulty is skewed or imbalanced.The analysis reports suboptimal performance when training disproportionately emphasizes easier samples and motivates prioritizing challenging tasks in later curriculum stages.Source: §5.3.2 · p. 10

5.3.3 Impact of Dynamic Length Reward on Reasoning Efficiency and Performance

  • Dynamic length rewards achieve higher average accuracy than fixed-length rewards while producing shorter overall responses and adapting response length to task complexity.Raising the target from 500 to 1,000 tokens increases response lengths on mathematics datasets, while ScienceQA and MMStar remain concise.Source: §5.3.3 · pp. 10–11, Figure 3 · p. 11

5.3.4 Comparison of Training Dynamics between PCuRL and Vanilla GRPO

  • At the hard-stage onset, PCuRL’s average reward briefly falls below vanilla GRPO before recovering to match or exceed it as responses meet the growing target length.Validation accuracy rises from about 0.3 to around 0.7 in the easy and medium stages for both methods; PCuRL then surpasses the baseline as its hard-stage accuracy continues rising beyond the baseline’s approximately 0.7 plateau.Source: §5.3.4 · pp. 11–12, Figure 4 · p. 11
  • In the hard stage, PCuRL response length rises from approximately 280 tokens toward the 500-token target, while vanilla GRPO stays around 250–300 tokens.The paper reports that PCuRL modulates response length with the curriculum and that the hard-stage length increase accompanies validation-accuracy gains.Source: §5.3.4 · pp. 12–13

5.4 Case Study

The case studies illustrate different reasoning trajectories across general, scientific, and mathematical questions, including a self-correction during a geometry solution.

Reasoning Trajectories

  • For multiple-choice general and science questions, VL-Cogito evaluates options; for math, it typically derives a solution before mapping it to the choices.The examples contrast a direct 30-60-90 triangle solution with a more challenging problem handled in four reasoning steps.Source: §5.4 · p. 13

Self-Reflective Reasoning

  • In the fourth example, VL-Cogito recognizes an erroneous use of the Pythagorean theorem and invokes “re-evaluate” to correct the calculation.The corrected calculation supports the remaining steps in the example.Source: §5.4 · p. 13

6 Conclusion

VL-Cogito uses a progressive curriculum reinforcement-learning framework to improve multimodal reasoning. It combines difficulty adjustment with response-length rewards and achieves state-of-the-art or highly competitive benchmark performance.

  • VL-Cogito achieves state-of-the-art or highly competitive performance across various benchmarks.Source: §6 · p. 13
  • PCuRL progressively guides the model through increasingly complex tasks and dynamically adjusts task difficulty as its abilities evolve.Source: §6 · p. 13
  • A dynamic length reward adapts response length to problem-specific demands, balancing reasoning depth and efficiency.Source: §6 · p. 13

A Data Statistics

Table 4 summarizes the distribution of reinforcement-learning training data by dataset, including sample counts and filter rates.

B System Prompt

The RL training prompt encourages a reasoning format and boxed final answers, while evaluation uses a simple prompt intended to generalize across models rather than the training system prompt.

Loading 2507.22607v2…