PaperlayerView exact revision on arXiv

Source-linked AI summary

R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization

Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, Wei Chen

arXiv:2503.10615v2cs.CV

TL;DR

Multimodal reasoning remains difficult because models struggle to organize and deeply reason over visual information, while existing benchmarks provide incomplete coverage. R1-Onevision formalizes visual content for language-based reasoning, trains on step-by-step data with supervised fine-tuning and reinforcement learning, and reports superior performance across multimodal benchmarks.

  • Problem

    Multimodal reasoning remains challenging because visual-language models often fail to organize, extract, and deeply reason over image information, while existing benchmarks cover limited aspects of capability.

  • Method

    R1-Onevision transforms images into visual formal representations, builds step-by-step reasoning data, and applies supervised fine-tuning followed by rule-based reinforcement learning.

  • Results

    R1-Onevision demonstrates superior multimodal reasoning performance across several challenging benchmarks and significantly improves over state-of-the-art open-source models.

  • Takeaways & Limitations

    The framework unifies cross-modal data construction, model training, and evaluation through fine-grained alignment and structured reasoning pathways.

Abstract

from arXiv · show

Large Language Models have demonstrated remarkable reasoning capability in complex textual tasks. However, multimodal reasoning, which requires integrating visual and textual information, remains a significant challenge. Existing visual-language models often struggle to effectively analyze and reason visual content, resulting in suboptimal performance on complex reasoning tasks. Moreover, the absence of comprehensive benchmarks hinders the accurate assessment of multimodal reasoning capabilities. In this paper, we introduce R1-Onevision, a multimodal reasoning model designed to bridge the gap between visual perception and deep reasoning. To achieve this, we propose a cross-modal reasoning pipeline that transforms images into formal textural representations, enabling precise language-based reasoning. Leveraging this pipeline, we construct the R1-Onevision dataset which provides detailed, step-by-step multimodal reasoning annotations across diverse domains. We further develop the R1-Onevision model through supervised fine-tuning and reinforcement learning to cultivate advanced reasoning and robust generalization abilities. To comprehensively evaluate multimodal reasoning performance across different grades, we introduce R1-Onevision-Bench, a benchmark aligned with human educational stages, covering exams from junior high school to university and beyond. Experimental results show that R1-Onevision achieves state-of-the-art performance, outperforming models such as GPT-4o and Qwen2.5-VL on multiple challenging multimodal reasoning benchmarks.

1 Zhejiang University 2 WeChat Vision, Tencent Inc. 3 Renmin University of China

Figure 1 contrasts baseline failures with R1-Onevision’s accurate visual interpretation, structured reasoning, and correct solution.

  • R1-Onevision accurately interprets the image, applies structured reasoning, and arrives at the correct solution.

1. Introduction

The introduction identifies weaknesses in multimodal reasoning and evaluation, then presents R1-Onevision’s pipeline, dataset, model training, and benchmark. Experiments report superior performance across multimodal reasoning benchmarks.

  • Multimodal reasoning requires models to iteratively extract, structure, and verify image information, but existing visual-language models often fail at this process.
  • Predefined reasoning templates can limit robustness and creative potential, while direct imitation may bypass trial-and-error reasoning.
  • Existing multimodal benchmarks remain specialized and cover only limited aspects of reasoning, motivating a comprehensive evaluation resource.
  • The proposed pipeline converts images into visual formal representations, and the R1-Onevision dataset supplies detailed step-by-step reasoning across diverse domains.
  • R1-Onevision uses supervised fine-tuning to cultivate coherent reasoning patterns and reinforcement learning to improve reasoning performance and generalization.
  • Extensive experiments demonstrate superior multimodal reasoning performance, including comparisons with Qwen2.5-VL and closed-source models such as GPT-4o.
  • R1-Onevision-Bench evaluates multimodal reasoning across educational levels and subject areas in a course-aware manner.

2. Related Work

Related work covers multimodal language models, language-model reasoning, and visual reasoning benchmarks, while the framework overview presents R1-Onevision’s training pipeline.

  • Multimodal Large Language Models: Multimodal large language models integrate visual and textual data for visual understanding and increasingly use plan-based chain-of-thought prompting.
  • Large Language Model Reasoning: Chain-of-thought prompting guides language models by decomposing complex questions into intermediate reasoning steps.
  • R1-Onevision Framework: The framework overview combines visual formal descriptions, role-playing data generation, supervised fine-tuning, and rule-based reinforcement learning.
  • Large Language Model Reasoning: Step-by-step reasoning capabilities are important for improving multimodal foundational-model performance.
  • Visual Reasoning Benchmarks: Visual reasoning benchmarks include mathematical problems with visual contexts and datasets designed to assess reasoning robustness.

3. Method

R1-Onevision bridges visual perception and language reasoning by converting image content into formal representations and generating structured multimodal reasoning data. It then applies supervised fine-tuning and rule-based reinforcement learning to stabilize reasoning, ensure valid outputs, and improve generalization.

  • Cross-Modal Reasoning Pipeline: The cross-modal pipeline converts visual content into formal representations so language reasoning models can process and reason over images precisely.The dataset is generated by using DeepSeek R1 to produce reasoning processes on LLaVA-OneVision.
  • Dataset Construction: Formal annotations combine structured descriptions, spatial information, OCR text, and reasoning strategies across charts, scenes, documents, and mathematical images.GPT-4o, Grounding DINO, and EasyOCR support the annotation process for different image types.
  • Dataset Construction: R1-Onevision aggregates multimodal data spanning science, mathematics, charts, and general real-world scenarios, totaling over 155k curated samples.Its source data covers natural images, OCR text extraction, charts, mathematical expressions, and scientific reasoning problems.
  • Post-Training: The training strategy uses supervised fine-tuning to stabilize reasoning and standardize outputs, followed by rule-based reinforcement learning to improve generalization.SFT provides the foundation for subsequent RL-based performance gains.
  • Post-Training: Rule-based reinforcement learning optimizes structured reasoning and output validity with accuracy and format rewards, updated through GRPO.GRPO clipping limits extreme policy shifts, while KL regularization keeps the updated policy aligned with the baseline.

4. Multimodal Reasoning Benchmark

R1-Onevision-Bench evaluates multimodal reasoning across scientific and logical domains using educationally graded difficulty levels. Its diverse task structure supports comparisons across subjects, categories, and stages of reasoning complexity.

  • Benchmark Scope: R1-Onevision-Bench evaluates multimodal reasoning across mathematics, physics, chemistry, biology, and logical deduction.The benchmark is designed to cover a wide spectrum of reasoning tasks.
  • Difficulty Structure: The benchmark organizes tasks into educational levels spanning middle school, high school, university, and social tests.This structure reflects graded complexity and includes both academic and practical reasoning skills.
  • Difficulty Structure: R1-Onevision-Bench comprises 38 subcategories grouped into five major domains and five difficulty levels.The levels range from Junior High School to Social Test challenges.
  • Evaluation Purpose: The benchmark enables evaluation of the grade at which multimodal models exhibit reasoning capabilities and identification of knowledge or experience needing supplementation.Its design integrates diverse problem types and difficulty levels for rigorous benchmarking.
  • Task Examples: Its examples include multimodal questions and answers across math, physics, chemistry, biology, and deduction at multiple educational grades.The task distribution spans categories, subcategories, and grade levels.

5. Experiment

R1-Onevision is evaluated on public mathematical reasoning benchmarks and the newly introduced R1-Onevision-Bench. Results show broad gains over multimodal baselines, with performance varying by educational level, discipline, training strategy, and model size.

  • 5.2. Main Results: R1-Onevision consistently outperforms state-of-the-art multimodal methods across the majority of evaluated benchmarks.
  • 5.2. Main Results: 29.9% accuracy on MathVision makes R1-Onevision comparable to GPT-4o.
  • 5.2. Main Results: R1-Onevision surpasses GPT-4o by 5.2% on MathVerse ALL, 5.5% on MathVerse Vision Only, and 4.1% on MathVista.
  • 5.3. Benchmark Analysis: The R1-Onevision-Bench spans four difficulty levels and five disciplines, enabling evaluation across educational stages and academic categories.
  • 5.3. Benchmark Analysis: Gemini-2.0-Flash exceeds GPT-4o by about 10% average accuracy, while Qwen2.5-VL-72B reaches 52% average accuracy and is on par with Claude-3.5.
  • 5.3. Benchmark Analysis: All evaluated models struggle with Deduction questions, with none exceeding 40% accuracy, and performance generally declines on university-level and professional certification exams.
  • 5.4. Ablation Study: SFT on the R1-Onevision dataset improves MathVision and MathVerse Vision Only performance, while the method also scales to Qwen2.5-VL-3B.
  • 5.4.2. Ablation Study on Model Parameters Variants: R1-Onevision-3B achieves 23.6% accuracy on MathVision and 38.6% on MathVerse ALL.

6. Conclusion

The paper concludes with a cross-modal formalization framework that unifies data construction, training, and evaluation for multimodal reasoning. Its dataset, model, and benchmark support detailed reasoning annotations, generalization, and systematic assessment across diverse tasks.

  • The framework unifies multimodal data construction, model training, and evaluation through cross-modal formalization.
  • The cross-modal pipeline uses fine-grained alignment and structured reasoning pathways to bridge visual and textual modalities.
  • The R1-Onevision dataset provides detailed step-by-step reasoning annotations across diverse domains.
  • R1-Onevision demonstrates strong multimodal reasoning and robust generalization across visual question answering and complex problem-solving tasks.
  • R1-Onevision-Bench assesses multimodal reasoning performance across multiple dimensions.
  • Experiments report significant improvements over state-of-the-art open-source models.

A.1. Data Details

The data pipeline combines curation, formal image description, reasoning-process generation, and quality assurance. Its prompts encourage direct image-based reasoning and filter generated chains of thought for correctness and relevance.

  • The cross-modal reasoning pipeline consists of data curation and filtering, image formal description, reasoning-process generation, and quality assurance.
  • Reasoning prompts ask models to simulate image reasoning while framing observations as direct visual perception.
  • The pipeline removes visualization steps and rewrites caption-based references as direct references to the image.
  • Quality assurance checks reasoning correctness, answer consistency with the standard answer, image-based grounding, and relevance of each reasoning step.
  • Model training uses full fine-tuning followed by reinforcement learning, with a fixed inference prompt separating thinking and final-answer tags.

A.3. Evaluation Details

Evaluation uses a unified framework to extract final answers from chain-of-thought responses and compare them with ground truth. Supplementary examples illustrate formal descriptions across circuits, music sheets, tables, natural scenes, print, mathematics, and memes.

  • A.3. Evaluation Details: The unified evaluation framework covers both public benchmarks and R1-Onevision-Bench.
  • A.3. Evaluation Details: GPT-4o-mini extracts final choices or free-form answers from chain-of-thought responses before scoring them against ground truth.
  • A.3. Evaluation Details: Multiple-choice extraction returns answer options, whereas free-form extraction returns the final numerical value or text without explanation.
  • B.1. Image Formal Description: The formal-description examples cover circuits, music sheets, tables, natural scenes, print, mathematics, and memes.
  • B.1. Image Formal Description: A geometric example describes triangle ABC, including AB perpendicular to BC and an exterior angle measuring 90°.

B.2. Model Reasoning Capabilities and Qualitative Examples

This section presents practical R1-Onevision examples spanning mathematical, physical, chemical, logical, and biological reasoning. The figures highlight processing across several specialized subdomains.

  • Physical Sciences: Physical reasoning examples include physics, mechanics, and optics.The figures separately identify physics, physical mechanics, and physical optics as processing areas.
  • Other Domains: Additional examples address chemistry, chemical reactions, logic, and biology.These areas are represented by figures on chemistry, chemical reaction, logic, and biology.
Loading 2503.10615v2…