PaperlayerView exact revision on arXiv

Source-linked AI summary

Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning

Jiaer Xia, Yuhang Zang, Peng Gao, Sharon Li, Kaiyang Zhou

arXiv:2505.14677v3cs.CV

TL;DR

Reinforcement learning can lead visual language models to rely on question-text shortcuts rather than image content, limiting generalization to unseen examples. Visionary-R1 trains models to caption images before reasoning using CoT-free visual question-answer pairs, and outperforms strong multimodal models on multiple visual reasoning benchmarks.

  • Problem

    When reinforcement learning is applied to visual reasoning, models may answer easy training questions through textual shortcuts without grounding in images, then fail to generalize to harder unseen questions.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    Visionary-R1 uses reinforcement learning on visual question-answer pairs without explicit CoT supervision, training models to caption images before reasoning and answering.

    Source: §3 · p. 3, §3.3 · p. 5

  • Results

    7.9% improvement on MathVista over the base model accompanies gains on MathVision and MMBench; Visionary-R1 also surpasses GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro on MathVista and MMBench.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The results indicate that understanding image context through captioning is important for enhancing VLM reasoning with CoT-free question-answer pairs.

    Source: §5 · p. 9

Abstract

from arXiv · show

Learning general-purpose reasoning capabilities has long been a challenging problem in AI. Recent research in large language models (LLMs), such as DeepSeek-R1, has shown that reinforcement learning techniques like GRPO can enable pre-trained LLMs to develop reasoning capabilities using simple question-answer pairs. In this paper, we aim to train visual language models (VLMs) to perform reasoning on image data through reinforcement learning and visual question-answer pairs, without any explicit chain-of-thought (CoT) supervision. Our findings indicate that simply applying reinforcement learning to a VLM -- by prompting the model to produce a reasoning chain before providing an answer -- can lead the model to develop shortcuts from easy questions, thereby reducing its ability to generalize across unseen data distributions. We argue that the key to mitigating shortcut learning is to encourage the model to interpret images prior to reasoning. Therefore, we train the model to adhere to a caption-reason-answer output format: initially generating a detailed caption for an image, followed by constructing an extensive reasoning chain. When trained on 273K CoT-free visual question-answer pairs and using only reinforcement learning, our model, named Visionary-R1, outperforms strong multimodal models, such as GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro, on multiple visual reasoning benchmarks.

1 INTRODUCTION

Visionary-R1 uses reinforcement learning on visual question-answer pairs without chain-of-thought supervision, requiring the model to caption an image before reasoning and answering. The approach targets shortcut learning and aims to improve generalization across data distributions.

1 INTRODUCTION

  • The work addresses the high cost and time demands of existing visual-reasoning pipelines, which also rely heavily on chain-of-thought data distilled from proprietary models.The introduction frames the lack of large-scale human-annotated reasoning data as a broader challenge for training AI to reason.Source: §1 · p. 1
  • With a reason–answer format, GRPO can exploit shortcuts on easy training questions, then produce incoherent reasoning and incorrect answers on harder, unseen examples.The paper identifies this failure as limiting general-purpose reasoning and generalization.Source: §1 · pp. 1–2
  • The caption–reason–answer format requires a detailed image caption before reasoning, with an AI-feedback caption reward added to accuracy and format rewards during policy optimization.The authors say this encourages visual understanding before reasoning and produces longer, more meaningful reasoning tokens than GRPO alone.Source: §1 · p. 2
  • Visionary-R1 outperforms GPT-4o, Claude3.5-Sonnet, Gemini-1.5-Pro, and recent supervised-pretraining and reinforcement-fine-tuning competitors on challenging visual reasoning benchmarks.The paper evaluates it on MathVista, MathVision, MMBench, MMMUPro, MMStar, and CV-Bench.Source: §1 · p. 3
  • Training uses 272.6K CoT-free question-answer pairs aggregated from 11 datasets spanning scene understanding, chart analysis, mathematical problem-solving, and document processing.The evaluation covers MathVista, MathVision, MMBench, MMMUPro, MMStar, and CV-Bench.Source: §1 · p. 3

2 RELATED WORK

Prior visual-reasoning work relies on chain-of-thought supervision or SFT-heavy training, whereas Visionary-R1 uses pure reinforcement learning on simple question-answer pairs without CoT supervision. The related work contrasts this approach with supervised and reinforcement-learning methods that depend on annotated data or SFT.

Supervised Learning for Visual Reasoning

  • Visionary-R1 uses simple question-answer pairs without CoT supervision and achieves stronger reasoning performance than the supervised models discussed.Source: §2 · p. 3
  • LLaVA-CoT (Xu et al., 2024) uses GPT-4o to label 100K visual question-answer examples, while MMCR (Yan et al., 2025) creates 310k multi-turn reasoning data.CoMCTS (Yao et al., 2024a) introduces Mulberry-260k for training tree-structure reasoning models; the passage notes CoT-label collection costs and scalability limits.Source: §2 · p. 3

Reinforcement Learning for Visual Reasoning

  • Insight-V (Dong et al., 2024) selects preference data from self-generated reasoning paths, while R1-VL (Zhang et al., 2025) uses step-wise rewards but relies on SFT labels.Source: §2 · p. 3
  • Vision-R1 (Huang et al., 2025) and R1-Onevision (Yang et al., 2025) use 10K RL samples but focus mainly on SFT with over 200K samples.Pixel Reasoner (Su et al., 2025) and VL-Rethinker (Wang et al., 2025) encourage deeper reasoning but still rely heavily on SFT and complex data selection and annotation.Source: §2 · p. 3
  • Visionary-R1 departs from SFT-followed-by-RL by adopting pure RL, removing reliance on large-scale SFT annotations and enabling flexible, autonomous reasoning through RL-from-AI-feedback.Source: §2 · p. 3

3 METHODOLOGY

Visionary-R1 trains VLMs with reinforcement learning on visual question-answer pairs without explicit CoT supervision, using caption-first outputs and a caption reward to encourage image-grounded reasoning.

3.1 MOTIVATION: THE SHORTCUT PHENOMENON IN VISUAL REASONING

  • GRPO can exploit textual question patterns instead of visual evidence, producing correct answers on simple training questions without grounding them in the image.This shortcut risks poor generalization when answers depend on image details such as embedded text, numerical values, object relationships, or chart patterns.Source: §3.1 · p. 4

3.2 VISIONARY-R1: GROUNDING REASONING VIA CAPTIONING

  • The caption-reason-answer format asks the model to describe salient image features, reason from that caption, and then answer; binary format and caption rewards encourage useful, structured captions.The caption reward uses reinforcement learning from AI feedback (Bai et al., 2022): the policy model’s LLM answers based solely on the caption, which is rewarded if correct.Source: §3.2 · p. 5

3.3 TRAINING OBJECTIVE WITH CAPTION REWARD

  • The training objective adds a caption reward to accuracy and format rewards, while cosine-annealing the KL penalty to stabilize early training and allow longer outputs later.The caption reward checks whether the policy model’s LLM can answer correctly using only the generated caption; the KL coefficient starts high and gradually decreases.Source: §3.3 · pp. 5–6, §3.2 · p. 5
  • For each question-image pair, GRPO samples multiple responses and computes group-normalized rewards to determine advantages, avoiding a critic model.The objective regularizes policy deviation from a reference policy; its KL coefficient is large early in training and decreases over time to support longer reasoning outputs.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments compare Visionary-R1 with supervised and reinforcement-learning baselines across visual reasoning benchmarks, then test captioning components and KL-coefficient strategies. The results show stronger benchmark performance with caption-before-reason training, while ablations identify conditions that affect training stability and generalization.

4.1 EXPERIMENTAL SETTINGS

  • Training combines 272.6K CoT-free visual question-answer pairs from 11 datasets without preprocessing or filtering, spanning scenes, charts, tables, diagrams, math, documents, and 3D data.The evaluation includes MathVista, MathVision, and MMBench, with MMMUPro, MMStar, and CV-Bench results provided in Appendix A.4.Source: §4.1 · p. 6
  • The study compares same-backbone, same-data SFT and GRPO baselines with proprietary and open-source state-of-the-art models.The named comparisons include GPT-4o and Claude3.5, and InternVL2.5 and LLaMA3.2, respectively.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • SFT underperforms the base model on three of four datasets, while GRPO gains 0.3% on MathVista and 1.2% on MathVision but loses 1.5% on MMBench.The authors attribute this weakness to overfitting to the training-data distribution and report shortcuts on easy samples alongside short, useless reasoning on unseen samples.Source: §4.2 · pp. 6–7
  • Visionary-R1 improves over the base model by 7.9% on MathVista, 5.6% on MathVision, and 2% on MMBench, and surpasses GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro on MathVista and MMBench.The table compares methods on three challenging visual reasoning benchmarks; the prose identifies the specific improvements and benchmark wins.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 ABLATION STUDY AND ANALYSES

  • Ablations using ChartQA and A-OKVQA training show that adding the caption output format and caption reward improves results, whereas a simple length reward increases redundancy and reduces performance.The evaluation tests ChartQA-trained models on MathVista and MathVision, and A-OKVQA-trained models on MMStar and MMBench; Figure 4 illustrates improved reasoning from the caption format and deeper reasoning from the caption reward.Source: §4.3 · pp. 7–8, Figure 4 · p. 8, Table 2 · p. 8
  • Static KL coefficients yield the worst results, linear decay significantly improves them, and cosine annealing performs slightly better, but applying cosine annealing to GRPO brings no performance gain.This indicates the dynamic coefficient chiefly affects Visionary-R1’s captioning component.Source: §4.3 · p. 8, Table 3 · p. 9
  • With β = 0.04, output length spikes around 700 steps before falling to 100 tokens as format and caption rewards decline; with β = 0.008, the model instead exploits the caption slot and omits reasoning.Linear decay and cosine annealing alleviate the reward-hacking behavior associated with the smaller coefficient.Source: §4.3 · pp. 8–9, Figure 5 · p. 9

5 CONCLUSION AND FUTURE WORK

The paper identifies shortcut learning as a challenge when applying RL to VLM reasoning and presents Visionary-R1, which performs strongly on challenging visual reasoning benchmarks using CoT-free question-answer pairs. Its findings emphasize image captioning and dynamic KL tuning, while larger-scale models remain future work.

5 CONCLUSION AND FUTURE WORK

  • Visionary-R1, trained on CoT-free question-answer pairs, surpasses strong commercial AI models on challenging visual reasoning benchmarks, though those models likely benefit from larger-scale, higher-quality training data.The paper argues that understanding image context through captioning is essential to enhancing VLM reasoning.Source: §5 · p. 9
  • Applying RL to VLMs reveals shortcut learning, and VLM reasoning is more difficult to train without annotated data than LLM reasoning.Source: §5 · p. 9
  • The results highlight the importance of dynamically tuning the KL coefficient to stabilize RL training, and suggest cosine annealing may apply more broadly to other RL applications.Source: §5 · p. 9
  • Investigating larger-scale models is left as future work, with the authors suggesting that larger models could significantly amplify RL training effectiveness.Source: §5 · p. 9

A TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL · A.1 COMPLETE LIST OF TRAINING DATA

The training set aggregates 11 popular question-answer datasets spanning diverse visual formats and tasks.

A.1 COMPLETE LIST OF TRAINING DATA

  • The training data combines 11 popular question-answer datasets, spanning general scene understanding, chart understanding, mathematical problem-solving, and document processing.Examples include A-OKVQA and TextVQA for scene understanding, ChartQA and RoBUT SQA for charts, GeoQA+ for mathematical problem-solving, and DocVQA for document processing.Source: §A.1 · p. 13, Table 4 · p. 13

A.2 POLICY MODEL PROMPT

The policy model’s system prompt instructs it to describe the image before reasoning, adding an <info> section ahead of <think> and <answer>.

A.2 POLICY MODEL PROMPT

  • The system prompt places an image description labeled <info> before the model’s <think> reasoning and <answer>.The prompt is intended to ensure the model interprets the image before beginning its thought process.Source: §A.2 · p. 13

A.3 CAPTION REWARD PROMPT

Caption detail is evaluated through question answering, because sufficiently detailed image descriptions provide information needed to answer accurately.

A.3 CAPTION REWARD PROMPT

  • The policy model’s language model answers questions based on the caption to judge its detail.A sufficiently detailed caption should provide the information needed to answer those questions accurately.Source: §A.3 · p. 13

Policy Model Prompt

The policy prompt directs the model to answer questions based on the image caption and filters out thought process and answers in the information section to prevent reward hacking.

Policy Model Prompt

  • The prompt directs the language model to answer from the caption and adds a filtering command to prevent its thought process and answer from appearing in the information section.The passage identifies Figure 7 as showing the complete caption reward prompt.Source: Policy Model Prompt · p. 14

Caption Reward Prompt … A.6 VISUALIZATION OF THE VISIONARY-R1 OUTPUT

The appendix reports consistent benchmark improvements and model-scaling results, describes the caption-based answer prompt and reward-weight choice, and notes that no LLMs were used while showing outputs across visual formats.

Caption Reward Prompt

  • Figure 7 provides a system prompt for answering a question based on the given caption.Source: Figure 7 · p. 14

A.4 ADDITIONAL EXPERIMENTAL RESULTS

  • Using caption-reward weights of 0.1 and 0.5 yields no significant difference; the authors choose 0.1 for training stability and recommend it to practitioners.Accuracy and format rewards receive equal weights, following the original GRPO recipe.Source: §A.4 · pp. 14–15, Table 7 · p. 15
  • Visionary-R1 achieves consistent improvements across MMMUPro (Yue et al., 2024), MMStar, and CV-Bench (Tong et al., 2024), unlike prior methods that can regress on some datasets.The authors interpret this stability as evidence that learned reasoning extends beyond dataset-specific adaptations.Source: §A.4 · p. 14, Table 5 · p. 15
  • With A-OKVQA’s 17.1K samples for training, Visionary-R1 outperforms the base model across all benchmarks for both 3B and 7B variants.The experiments use Qwen2.5-VL-7B as the base model.Source: §A.4 · p. 14, Table 6 · p. 15

A.5 THE USE OF LARGE LANGUAGE MODELS (LLMS)

  • The authors report that no LLMs were used in the paper’s ideation, writing, or editing.Source: §A.5 · p. 15

A.6 VISUALIZATION OF THE VISIONARY-R1 OUTPUT

Related papers

Loading 2505.14677v3…