PaperlayerView exact revision on arXiv

Source-linked AI summary

Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning

Jiaer Xia, Yuhang Zang, Peng Gao, Sharon Li, Kaiyang Zhou

arXiv:2505.14677v3cs.CV

TL;DR

The paper addresses shortcut learning when reinforcement learning trains VLMs to reason from visual question-answer pairs without annotated chain-of-thought data. It introduces Visionary-R1, which requires captioning before reasoning and adds caption-based rewards to GRPO. Trained on CoT-free data, Visionary-R1 outperforms strong multimodal models across challenging visual reasoning benchmarks.

  • Problem

    Direct reinforcement learning can make VLMs exploit textual shortcuts on easy questions instead of grounding reasoning in image content, limiting generalization.

  • Method

    Visionary-R1 trains VLMs with a caption–reason–answer format and augments GRPO with a caption reward that tests whether captions support answering.

  • Results

    Visionary-R1 outperforms strong multimodal models, including GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro, on multiple visual reasoning benchmarks.

  • Takeaways & Limitations

    The results indicate that understanding image context through captioning is essential for enhancing VLM reasoning.

Abstract

from arXiv · show

Learning general-purpose reasoning capabilities has long been a challenging problem in AI. Recent research in large language models (LLMs), such as DeepSeek-R1, has shown that reinforcement learning techniques like GRPO can enable pre-trained LLMs to develop reasoning capabilities using simple question-answer pairs. In this paper, we aim to train visual language models (VLMs) to perform reasoning on image data through reinforcement learning and visual question-answer pairs, without any explicit chain-of-thought (CoT) supervision. Our findings indicate that simply applying reinforcement learning to a VLM -- by prompting the model to produce a reasoning chain before providing an answer -- can lead the model to develop shortcuts from easy questions, thereby reducing its ability to generalize across unseen data distributions. We argue that the key to mitigating shortcut learning is to encourage the model to interpret images prior to reasoning. Therefore, we train the model to adhere to a caption-reason-answer output format: initially generating a detailed caption for an image, followed by constructing an extensive reasoning chain. When trained on 273K CoT-free visual question-answer pairs and using only reinforcement learning, our model, named Visionary-R1, outperforms strong multimodal models, such as GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro, on multiple visual reasoning benchmarks.

1 INTRODUCTION

The paper identifies shortcut learning as a failure mode when reinforcement learning is applied directly to VLMs and proposes Visionary-R1, which captions images before reasoning. Using CoT-free visual question-answer data, the method aims to improve generalization and achieves strong benchmark performance.

  • Motivation: Reasoning is difficult to train because large-scale human-annotated reasoning data is scarce, motivating reinforcement learning from question-answer pairs.Prior LLM work shows that reinforcement learning can induce reasoning without explicit step-by-step supervision.
  • The shortcut problem: Directly prompting a GRPO-trained VLM to reason before answering encourages shortcuts on easy questions and poor generalization to harder unseen examples.The model may answer correctly by exploiting textual patterns rather than understanding subtle visual features.
  • Visionary-R1: Visionary-R1 enforces a caption–reason–answer format so the model first generates a detailed image description before constructing its reasoning chain.The caption is intended to cover objects, numbers, text, spatial relations, and other salient visual features.
  • Experiments: 272.6K CoT-free question-answer pairs from 11 datasets cover scene understanding, chart analysis, mathematical problem-solving, and document processing.The model is evaluated on MathVista, MathVision, MMBench, MMMUPro, MMStar, and CV-Bench.
  • Results: Visionary-R1 beats strong multimodal models including GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro on challenging visual reasoning benchmarks.The paper attributes this result to learning to interpret images before reasoning.

2 RELATED WORK

Prior work improves visual reasoning through supervised reasoning traces and reinforcement learning. Supervised approaches require costly labeled thinking data, while reinforcement-learning methods seek broader self-developed reasoning processes.

  • Supervised Learning for Visual Reasoning: Supervised fine-tuning for visual reasoning relies on labeled thinking processes, often generated by GPT-4o because human annotation is costly.GPT-4o-generated labels also limit scalability and impose a performance upper bound.
  • Reinforcement Learning for Visual Reasoning: Reinforcement learning is presented as a way to develop general-purpose reasoning by exploring a broader language space and allowing models to develop their own thinking processes.Related methods include preference learning and step-wise rewards for visual reasoning.

3 METHODOLOGY

Visionary-R1 adapts GRPO to visual reasoning by requiring captioning before reasoning and by rewarding informative captions. Its training objective combines answer accuracy, format compliance, caption quality, group-relative advantages, and KL regularization.

  • Framework: Visionary-R1 trains VLMs with only visual question-answer pairs by using the caption–reason–answer output format.The model first generates an informative caption to understand image context, then produces an extensive reasoning chain.
  • Shortcut phenomenon: Direct GRPO can ignore visual input and rely on textual question patterns, producing correct easy-sample answers without image grounding.This shortcut is especially harmful when answers depend on embedded text, numerical values, object relationships, or chart structure.
  • Caption-Reason-Answer Output Format: The required output sequence is caption, reasoning, then answer, with the caption describing salient visual content before reasoning begins.The format is checked by a binary format reward.
  • Caption Reward: The caption reward evaluates whether an LLM can answer the question using only the generated caption, rewarding captions that contain sufficient information.The policy model’s LLM component performs this caption-based answer check.
  • Training Objective: The combined reward is Ri = ra + rf + αrc, adding answer accuracy, format compliance, and weighted caption quality.α controls the caption reward’s contribution.
  • Training Objective: GRPO samples multiple responses per question-image pair, computes normalized group rewards as advantages, and optimizes the policy without a critic model.Visionary-R1 adds caption rewards and a cosine-annealed KL penalty to adapt GRPO to visual reasoning.
  • Cosine Annealing KL Coefficient: The KL penalty regularizes deviation from a reference policy, balancing stable training against the need for longer, more detailed reasoning.A large coefficient can impede in-depth thinking, whereas a small coefficient can destabilize training and enable reward hacking.

4 EXPERIMENTS

Experiments evaluate Visionary-R1 across diverse visual reasoning benchmarks and show that caption-before-reasoning reinforcement learning improves generalization. Ablations further examine captioning, caption rewards, and KL-coefficient schedules.

  • Experimental Setup: The training data combines 11 visual question-answer datasets into 272.6K CoT-free pairs spanning scenes, charts, tables, diagrams, mathematics, documents, and 3D data.
  • Experimental Setup: Evaluation covers MathVista, MathVision, MMBench, MMMUPro, MMStar, and CV-Bench, representing varied visual formats and question types.
  • Experimental Setup: Baselines include SFT and GRPO trained with the same backbone and data, alongside proprietary and open-source state-of-the-art models.
  • Main Results: SFT underperforms the base model on three of four datasets, while GRPO improves MathVista by 0.3% and MathVision by 1.2% but decreases MMBench by 1.5%.
  • Main Results: 7.9% on MathVista, 5.6% on MathVision, and 2% on MMBench are Visionary-R1’s improvements over the base model.
  • Ablation Study and Analyses: The caption output format and caption reward improve reasoning, whereas a simple length reward increases redundancy and reduces performance.
  • Ablation Study and Analyses: Dynamic KL schedules outperform static coefficients, with cosine annealing slightly better than linear decay and no observed gain when applied to GRPO alone.

5 CONCLUSION AND FUTURE WORK

The paper identifies shortcut learning as a challenge for reinforcement learning in VLMs and reports strong benchmark performance from Visionary-R1 using CoT-free question-answer pairs. It also highlights dynamic KL tuning and larger models as directions for improving RL training.

  • Visionary-R1 achieves strong performance on challenging visual reasoning benchmarks despite using only CoT-free question-answer pairs.
  • Applying reinforcement learning to VLMs exposes shortcut learning, making visual reasoning harder to train without annotated data.
  • Understanding image context through captioning is essential for enhancing visual reasoning in VLMs.
  • Dynamic tuning of the KL coefficient is important for stabilizing reinforcement learning training.
  • The effectiveness of reinforcement learning with larger-scale models remains an open direction for future work.

A.1 COMPLETE LIST OF TRAINING DATA

The training data aggregates 11 question-answer datasets spanning diverse visual formats and reasoning tasks, including scene understanding, charts, mathematics, and documents.

  • The training data combines 11 popular question-answer datasets covering diverse visual formats and tasks.
  • A-OKVQA and TextVQA provide data for general scene understanding.
  • ChartQA and RoBUT SQA cover chart understanding, while GeoQA+ addresses mathematical problem-solving.
  • DocVQA contributes data for document processing.

A.2 POLICY MODEL PROMPT

The policy prompt instructs the model to interpret the image before thinking by placing an image description before the reasoning and answer stages.

  • The system prompt instructs the policy model to generate an image description before engaging in the thought process.
  • The output format places an <info> image description before the existing <think> and <answer> components.

A.3 CAPTION REWARD PROMPT

Caption quality is judged by whether the model can answer questions from the generated caption, making sufficient image detail a requirement for accurate answers.

  • The caption reward assesses detail by having the model answer questions based on the generated caption.
  • A sufficiently detailed image description is essential for providing the information needed to answer questions accurately.

Policy Model Prompt

The caption reward prompt has the language model answer from the generated caption while filtering out reasoning and answer content from the information section.

  • An additional filtering command removes thought-process and answer content from the information section to prevent reward hacking.

Caption Reward Prompt

Additional results report stable benchmark gains, scaling benefits, and reward-weight robustness, while the appendix visualizes outputs across formats and documents the authors’ disclosure.

  • Caption Reward Prompt: The caption reward prompt instructs the language model to answer using the provided caption.
  • A.4 ADDITIONAL EXPERIMENTAL RESULTS: Visionary-R1 achieves consistent improvements on MMMUPro, MMStar, and CV-Bench.The reported stability contrasts with inconsistent performance and regressions from prior methods.
  • A.4 ADDITIONAL EXPERIMENTAL RESULTS: Visionary-R1 outperforms the base model across all benchmarks for both 3B and 7B variants.The scaling experiment trains on 17.1K A-OKVQA samples.
  • A.4 ADDITIONAL EXPERIMENTAL RESULTS: Caption-reward weights of 0.1 and 0.5 produce no significant difference in the reported results.The authors select 0.1 to maintain training stability and recommend it for practitioners.
  • A.5 THE USE OF LARGE LANGUAGE MODELS (LLMS): The paper states that no LLMs were used for research ideation, writing, or editing.
  • A.6 VISUALIZATION OF THE VISIONARY-R1 OUTPUT: The appendix includes a document-format example describing a black-and-white informational seminar flyer.
  • A.6 VISUALIZATION OF THE VISIONARY-R1 OUTPUT: The appendix includes math-format visualization with Chinese input and output translated directly without modification.
Loading 2505.14677v3…