Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning

Huilin Deng; Ding Zou; Rui Ma; Hongchen Luo; Yang Cao; Yu Kang

Source-linked AI summary

Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning

Huilin Deng, Ding Zou, Rui Ma, Hongchen Luo, Yang Cao, Yu Kang

arXiv:2503.07065v1Mar 10, 2025cs.CV

TL;DR

Small-scale VLMs are more practical than heavily scaled models but face OOD generalization, reasoning, and stability challenges under SFT. Curr-ReFT uses progressive curriculum reinforcement learning followed by rejected-sample self-improvement, and experiments report state-of-the-art visual-task performance across in-domain and OOD settings, with small models matching larger counterparts.

Problem
Small-scale VLMs face limited OOD generalization and shallow reasoning under SFT, while many existing reasoning approaches focus on large models and non-visual tasks.
Method
Curr-ReFT combines difficulty-aligned curriculum reinforcement learning with rejected-sampling self-improvement from multimodal and language examples.
Results
Curr-ReFT-trained models achieve state-of-the-art performance across visual tasks in in-domain and OOD settings, with enhanced small-scale models matching much larger counterparts.
Takeaways & Limitations
The experiments support reinforcement learning as a post-training approach that improves both reasoning and OOD generalization in small-scale VLMs.
Takeaways & Limitations
Small-scale VLMs can hit a “Brick Wall” on complex examples, with training instability and degradation on previously mastered tasks.

Abstract

from arXiv · show

While state-of-the-art vision-language models (VLMs) have demonstrated remarkable capabilities in complex visual-text tasks, their success heavily relies on massive model scaling, limiting their practical deployment. Small-scale VLMs offer a more practical alternative but face significant challenges when trained with traditional supervised fine-tuning (SFT), particularly in two aspects: out-of-domain (OOD) generalization and reasoning abilities, which significantly lags behind the contemporary Large language models (LLMs). To address these challenges, we propose Curriculum Reinforcement Finetuning (Curr-ReFT), a novel post-training paradigm specifically designed for small-scale VLMs. Inspired by the success of reinforcement learning in LLMs, Curr-ReFT comprises two sequential stages: (1) Curriculum Reinforcement Learning, which ensures steady progression of model capabilities through difficulty-aware reward design, transitioning from basic visual perception to complex reasoning tasks; and (2) Rejected Sampling-based Self-improvement, which maintains the fundamental capabilities of VLMs through selective learning from high-quality multimodal and language examples. Extensive experiments demonstrate that models trained with Curr-ReFT paradigm achieve state-of-the-art performance across various visual tasks in both in-domain and out-of-domain settings. Moreover, our Curr-ReFT enhanced 3B model matches the performance of 32B-parameter models, demonstrating that efficient training paradigms can effectively bridge the gap between small and large models.

1. Introduction

Small-scale VLMs are more deployable than heavily scaled models but struggle with OOD generalization, reasoning, and training stability under SFT. Curr-ReFT addresses these challenges by progressively increasing task difficulty with aligned rewards and selectively preserving capabilities through rejected-sample self-improvement.

Models above 32B parameters create deployment barriers, motivating efficient training paradigms for 1B–7B small-scale VLMs.
SFT can cause overfitting, severe OOD degradation, and shallow reasoning in smaller VLM architectures.
RL methods maintain robust OOD generalization in experiments where SFT suffers significant performance degradation on out-of-domain data.
Small-scale VLMs encounter a “Brick Wall” when complex examples require simultaneous visual understanding and reasoning.Challenging cases can also degrade previously mastered tasks and produce unstable, oscillating learning curves.
Curriculum Reinforcement Learning progressively increases task complexity with difficulty-calibrated rewards, advancing from basic concept recognition to complex reasoning.The curriculum uses binary decisions, multiple-choice selection, and open-ended responses with increasingly complex rewards.
Curr-ReFT combines curriculum reinforcement learning with rejected-sampling self-improvement to enhance reasoning and OOD generalization while maintaining fundamental capabilities.
Curr-ReFT-trained models achieve state-of-the-art performance across visual tasks in both in-domain and out-of-domain settings, with small models matching much larger counterparts.

2. Related Work

VLM research evolved from dual encoders toward LLM-integrated architectures, while reasoning research increasingly uses post-training reinforcement learning. Existing reasoning approaches mainly target large LLMs and overlook smaller VLMs in computer-vision tasks.

Vision-Language Models: CLIP pioneered contrastive visual-textual alignment, but early dual-encoder models had limitations in fine-grained alignment.
Vision-Language Models: LLaVA advanced VLMs by projecting visual features into an LLM’s embedding space, following unified architectures such as BEIT-3.
Reasoning Models: Reasoning research has incorporated MCTS, Tree of Thoughts, process supervision, and post-training methods to improve model reasoning.
Reasoning Models: GRPO compares response performance within groups and avoids dependence on additional critic networks, challenging conventional RL-plus-SFT combinations.
Reasoning Models: Current reasoning models focus mainly on mathematical and coding tasks with LLMs, while approaches for smaller VLMs and computer-vision tasks remain underexplored.

3. Method

Curr-ReFT is organized as a two-stage post-training paradigm: curriculum reinforcement learning for difficulty-aligned task progression, followed by rejected-sample self-improvement for capability preservation.

Curriculum Reinforcement Learning: Curr-ReFT first orchestrates task progression through Curriculum Reinforcement Learning with difficulty-aligned reward mechanisms.
Rejected Sample based Self-improvement: Its second stage uses Rejected Sample based Self-improvement and quality-guided learning to preserve fundamental capabilities.
Training data organization includes three-stage progressive response formats for Curriculum Reinforcement Learning and data sources for the reject-sampling SFT phase.

3.1. Preliminary

The preliminary discussion motivates reinforcement learning for reasoning by contrasting critic-dependent feedback, rule-based verification, and GRPO’s relative response comparison.

RLHF relies heavily on critic models, whereas RLVR directly verifies correctness but depends on scenario-specific rules and expert knowledge.
GRPO evaluates responses through relative performance comparisons within groups rather than absolute reward values.
For each query, GRPO generates N distinct responses from the current policy and evaluates them through group-wise comparison.
The normalized relative quality A_i represents the quality of the i-th response within its response group.

3.2. Curriculum Reinforcement Learning

Curriculum Reinforcement Learning progressively increases task complexity from binary decisions to multiple-choice and open-ended responses, using rewards tailored to visual understanding and reasoning. Its visual-task rewards provide graded signals for classification and detection accuracy while also checking response format.

Curriculum design: Curriculum learning advances through Binary Decision, Multiple Choice, and Open-ended Response stages with difficulty-aware rewards.The progression is task-level rather than based on ambiguous sample-level difficulty.
Stage 1: Binary Decision: Binary decision training restricts outputs to “yes” or “no,” simplifying learning of basic visual understanding and reasoning patterns.The reward is binary: one for matching the ground-truth answer and zero otherwise.
Stage 2: Multiple Choice: Multiple-choice learning introduces more sophisticated decisions while retaining structured response formats and scenario-specific reward mechanisms.Single-choice questions use binary rewards, whereas multiple-choice questions receive more nuanced treatment.
Stage 2: Multiple Choice: Partial-correctness rewards give credit for identifying correct options while preserving an incentive for complete multiple-choice answers.The reward assigns 0.2 when selected options overlap with the correct set but are incomplete.
Stage 3: Open-ended Response: Classification rewards use category intersection-over-union to provide continuous partial credit, while detection rewards combine localization accuracy with format compliance.Detection localization filters matches below IoU threshold τ, averages valid IoUs, and further discretizes the reward at 0.5.

3.3. Rejected Sample based Self-improvement

Rejected Sample-based Self-improvement preserves model competencies while maintaining reasoning capabilities by sampling high-quality responses and optimizing on the resulting curated dataset.

Method overview: The method combines High-Quality Data Sampling with Self-Improvement Training to enhance capabilities while preserving fundamental model competencies.It is grounded in curriculum reinforcement learning principles.
High-Quality Data Sampling: GPT-4-O scores generated responses for accuracy, logical consistency, format compliance, and linguistic fluency, retaining responses above 85.The curated dataset contains 1,520 high-quality examples alongside their corresponding queries.
Self-Improvement Training: Self-improvement optimization trains on the curated dataset to enhance fundamental capabilities while preserving reasoning capabilities.The objective is defined over model parameters, samples, answer-token positions, vocabulary classes, and self-generated labels.

4. Experiments

The experiments compare reinforcement learning and Curr-ReFT with SFT and mainstream VLMs across visual tasks, scaling settings, and diverse multimodal benchmarks. Evaluation covers in-domain and out-of-domain data using accuracy-based task metrics and broader benchmark assessments.

Research questions: The study asks how RL compares with SFT, how Curr-ReFT compares with mainstream VLMs, which components matter, and whether effectiveness scales with parameters.These questions span traditional computer-vision tasks, model comparisons, ablations, and scaling.
Evaluation framework: The evaluation framework covers visual detection, visual classification, and additional in-domain and out-of-domain multimodal data.Detection and classification use sampled training, in-domain testing, and out-domain testing datasets.
Evaluation framework: Accuracy is the unified evaluation metric; detection counts predictions correct above IoU 0.5, while classification requires matching ground-truth labels.These criteria define correctness for the task-specific evaluations.
Benchmarks: The benchmark suite includes MathVista, MATH, AI2D, MMVet, MMBench, OCRBench, and LLaVABench, covering mathematical, diagrammatic, reasoning, perception, OCR, and generalization tasks.The listed benchmarks range from 6,141 MathVista examples to LLaVABench’s 24K images and 60K questions.
Baselines and scale: Models from 3B to 32B parameters are compared, including Qwen, InternVL, and LLaVA families.The baselines span small-scale, medium-scale, and large-scale models.

4.2. Generalization Verification of RL (RQ 1)

The evaluation compares SFT and RL variants on in-domain and out-of-domain visual tasks. RL-based methods generalize more robustly, with Curr-RL also improving explanation quality and localization.

Evaluation setup: Table 1 compares Qwen2.5-VL-3B variants trained with SFT, RL, and Curriculum Reinforcement Learning on math, classification, and detection tasks.The evaluation uses in-distribution and out-of-distribution test sets.
In-domain results: RL-based methods outperform SFT in in-domain scenarios, supporting reinforcement learning during VLM post-training.This observation is reported across the evaluated tasks.
Out-of-domain results: RL-based methods retain stronger out-of-domain performance, whereas SFT improves less and worsens in some cases.Curr-RL shows particularly strong generalization in the reported comparisons.
Qualitative and task-specific findings: Curr-RL produces more detailed explanations and more accurate localization across out-of-domain visual tasks.The comparison is qualitative for explanations and performance-based for localization.
Training trends: With increasing training steps, both approaches improve in-domain, but only RL maintains convincing out-of-domain performance.Figure 5 reports a decreasing out-of-domain trend for SFT-based methods.

4.3. Performance Comparation (RQ2)

Curr-ReFT is evaluated against baseline models on visual tasks and public benchmarks. The reported comparisons show strong performance across domains, benchmarks, and model scales, with generalization gains especially prominent.

Overall performance: Curr-ReFT consistently performs strongly on in-domain and out-of-domain datasets and recognized public benchmarks.The comparisons include visual tasks and benchmarks such as AI2D and MMVet.
Comparison with larger models: Curr-ReFT-3B frequently outperforms the 26B InternVL and 32B LLaVA-Next models in the reported comparisons.This comparison is made across the evaluated visual and benchmark tasks.
Reasoning and benchmark results: Curr-ReFT-3B improves math and logic capabilities relative to Qwen2.5-VL-3B.The reported gains include benchmarks such as AI2D and MMVet in the broader comparison.
Ablation study: Removing both reinforcement learning and rejected-sample SFT degrades performance, while removing Curriculum RL produces the worst performance.The ablation identifies Curriculum RL as a critical component of the post-training design.
Ablation study: Rejected-sample self-improvement slightly reduces detection and classification performance but substantially improves reasoning and generalization benchmarks.The ablation describes this as a trade-off between task-specific and general capabilities.

4.5. Scaling Analysis (RQ4)

Scaling experiments show that Curr-ReFT remains effective when applied to the 7B model. Larger-model training improves visual-task performance, benchmark generalization, and especially complex reasoning.

Scaling results: Curr-ReFT-7B improves over Curr-ReFT-3B on detection, classification, MMVet, and MathVista.Reported changes are detection: 89.8% →92.2%, classification: 71.5% →73.1%, MMVet: 29.95% →36.78%, and MathVista: 58.60% →92.2%.
Scaling results: The gains are particularly pronounced on complex reasoning tasks, indicating that larger models better leverage Curr-ReFT.This conclusion is drawn from the reported scaling experiments.

5. Conclusion

The paper targets reasoning and out-of-domain generalization in small-scale VLMs. It proposes Curr-ReFT, combining progressive curriculum learning with rejected sampling to support stable optimization and both capabilities.

Conclusion: The paper focuses on improving reasoning and out-of-domain generalization in small-scale VLMs.Its empirical findings report reinforcement learning improvements in both areas.
Conclusion: Curr-ReFT combines progressive curriculum learning with rejected sampling in a post-training paradigm.The method gradually increases task complexity and selectively learns from high-quality examples.
Conclusion: The proposed design is intended to support stable optimization while maintaining reasoning and generalization capabilities.This is the paper’s stated conclusion about the role of gradually increasing complexity and selective learning.