PaperlayerView exact revision on arXiv

Source-linked AI summary

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, Ziwei Liu

arXiv:2411.14432v2cs.CV

TL;DR

Long-chain visual reasoning in MLLMs lacks scalable, high-quality data and effective training strategies. Insight-V combines automated reasoning-data generation with multi-agent training, improving average performance across seven challenging benchmarks. The authors describe structured long-chain reasoning as a promising direction for MLLMs.

  • Problem

    Long-chain visual reasoning remains underexplored, with a shortage of large-scale, high-quality datasets and efficient, effective training strategies, while detailed visual-data annotation and validation require substantial human labor.

    Source: §1 · p. 1

  • Method

    Insight-V pairs automated progressive generation and multi-granularity assessment of long-chain reasoning data with reasoning-and-summarization agents and iterative DPO.

    Source: §1 · p. 2

  • Results

    Insight-V improves average performance by 7.0% on LLaVA-NeXT and 2.9% on the strong base MLLM across seven challenging visual reasoning benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors characterize organizing single-step reasoning into structured long-chain paths as a promising approach to MLLM reasoning.

    Source: C. Discussion and Limitations · p. 9

    The data-generation process depends on other models for multi-granularity assessment, and the authors identify sampling efficiency as needing improvement.

    Source: C. Discussion and Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) demonstrate enhanced capabilities and reliability by reasoning more, evolving from Chain-of-Thought prompting to product-level solutions like OpenAI o1. Despite various efforts to improve LLM reasoning, high-quality long-chain reasoning data and optimized training pipelines still remain inadequately explored in vision-language tasks. In this paper, we present Insight-V, an early effort to 1) scalably produce long and robust reasoning data for complex multi-modal tasks, and 2) an effective training pipeline to enhance the reasoning capabilities of multi-modal large language models (MLLMs). Specifically, to create long and structured reasoning data without human labor, we design a two-step pipeline with a progressive strategy to generate sufficiently long and diverse reasoning paths and a multi-granularity assessment method to ensure data quality. We observe that directly supervising MLLMs with such long and complex reasoning data will not yield ideal reasoning ability. To tackle this problem, we design a multi-agent system consisting of a reasoning agent dedicated to performing long-chain reasoning and a summary agent trained to judge and summarize reasoning results. We further incorporate an iterative DPO algorithm to enhance the reasoning agent's generation stability and quality. Based on the popular LLaVA-NeXT model and our stronger base MLLM, we demonstrate significant performance gains across challenging multi-modal benchmarks requiring visual reasoning. Benefiting from our multi-agent system, Insight-V can also easily maintain or improve performance on perception-focused multi-modal tasks.

1. Introduction

Insight-V addresses the shortage of scalable, high-quality data and effective training strategies for long-chain visual reasoning. It combines automated data generation with multi-agent training and improves average performance across seven challenging benchmarks.

1. Introduction

  • Long-chain visual reasoning remains difficult because large-scale, high-quality datasets and effective training strategies are lacking, while detailed data collection requires substantial human labor.Existing chain-of-thought training approaches have shown limited effectiveness for improving visual reasoning.Source: §1 · pp. 1–2
  • Insight-V generates structured, diverse long-chain reasoning data through progressive generation and multi-granularity assessment, using automated generation, assessment, and ranking to scale without human labor.Its multi-agent system separates detailed reasoning from summarization, and iterative DPO further refines reasoning quality.Source: §1 · p. 2
  • 7.0% for LLaVA-NeXT and 2.9% for the strong base MLLM are the reported average performance improvements across seven challenging visual reasoning benchmarks.The evaluation is presented as evidence of Insight-V’s effectiveness and generalizability.Source: §1 · p. 2
  • The system comprises a scalable data-generation pipeline, a reasoning-and-summarization multi-agent system, and a two-stage training pipeline for visual reasoning.Together, these contributions address challenges in visual reasoning and provide a foundation for future MLLM reasoning research.Source: §1 · p. 2

2. Related Work

Prior vision-language reasoning work often emphasizes chain-of-thought data quality over structured, extended reasoning paths, while complex tasks can challenge a single model. The paper also reviews preference-alignment methods and uses iterative DPO for stronger alignment and reasoning.

2.1. Vision-Language Reasoning

  • Prior MLLM reasoning approaches [11] [56] [57] commonly use Chain-of-Thought, but often overlook structured reasoning paths and extended chains; complex tasks also challenge a single model.The paper positions scalable reasoning-data generation and reasoning–summarization decomposition as responses to these challenges.Source: §2.1 · p. 2

2.2. Vision-Language Alignment

  • Iterative DPO [5] refreshes preference pairs at each iteration using the updated model and reward-model evaluation, and this paper uses it to strengthen alignment and reasoning.The section contrasts this iterative procedure with traditional DPO’s primarily offline setting, whose effectiveness may diminish as the model evolves.Source: §2.2 · pp. 2–3

3. Methodology

Insight-V addresses MLLM reasoning challenges and the shortage of structured, high-quality training data with a system for long-chain data generation, specialized agents, and staged training. Its methodology combines progressive data construction, reasoning and summarization, and iterative preference optimization.

3.1. Overview

  • Insight-V separates reasoning from summarization, with the summary agent assessing a detailed reasoning process as supplementary evidence for answering.The paper frames this decomposition as a way to leverage MLLMs’ reasoning strengths.Source: §3.1 · p. 3

3.2. Construction of Structured Reasoning Data

  • The data pipeline iteratively generates structured reasoning paths, filters incorrect answers, and scores surviving paths for step accuracy and detail.A scoring agent assigns each path a score from 1 to 100; the selected dataset supports model training.Source: §3.2 · pp. 3–4

3.3. Model Design

  • The reasoning agent produces detailed steps, while the summary agent selectively uses or rejects them and trains on both sound and flawed reasoning examples.Flawed examples span different score ranges to encourage evaluation of reasoning quality at varying granularities.Source: §3.3 · p. 4

3.4. Training Pipeline

  • Training uses supervised fine-tuning followed by direct preference optimization of the reasoning model.The paper describes these two stages as building a robust system with enhanced visual reasoning capabilities.Source: §3.4 · pp. 4–5

3.4.1. Supervised Fine-tuning for Multi-agent System

  • Both agents start from a trained base MLLM; agent-specific fine-tuning uses curated reasoning data and summary data that includes general image-text pairs.The summary-agent data also includes standard question-answering examples to preserve direct perception capabilities.Source: §3.4.1 · p. 5

3.4.2. Enhanced Reasoning with RL

  • Iterative DPO refreshes preference data from successive reasoning models to reduce divergence from the model’s current distribution during training.The authors apply this process to the fine-tuned reasoning agent to support structured, detailed reasoning steps.Source: §3.4.2 · p. 5

4. Experiments

The experiments evaluate Insight-V on visual-reasoning and general image-understanding benchmarks, then analyze its design choices and qualitative behavior. Results cover both LLaVA-NeXT and a separately trained base MLLM.

4.1. Implementation Details

  • The baseline is trained in two stages, then used to initialize reasoning and summary agents; the reasoning agent also receives iterative DPO training.Baseline pretraining uses 558K captioning examples, followed by supervised fine-tuning on approximately 4 million images. DPO uses approximately 15K preference examples over three rounds.Source: §4.1 · pp. 5–6

4.2. Main Results on Visual Reasoning

  • Insight-V improves average results on visual-reasoning benchmarks for both LLaVA-NeXT and the base model, with further gains after DPO.Average gains are 4.9% on LLaVA-NeXT and 3.2% on the base model; DPO adds 2.6% and 1.0%, respectively. Reported task gains include 9.1% on MME, 5.8% on ChartQA, 4.5% on MMStar, and 3.0% on MathVista.Source: Table 1 · p. 6, §4.2 · pp. 6–7
  • The evaluation spans expert-level and general benchmarks, chart reasoning, mathematical problem-solving, and tasks with varying difficulty.MMMU and MMMU-Pro assess expert-level perception and reasoning; MMBench covers image evaluation, MME includes 14 subtasks, and ChartQA and MathVista target chart and math reasoning.Source: §4.2 · p. 6

4.3. Further Analysis

  • The multi-agent configuration outperforms alternatives that omit the reasoning process, combine reasoning with summarization, or use multi-turn conversations.The authors report that direct SFT yields modest gains and that multi-turn models remain sub-optimal; they associate the multi-agent design with separating reasoning and summarization.Source: Table 3 · p. 6, §4.3 · p. 7

4.2. Main Results on Visual Reasoning

  • On general visual-perception benchmarks, Insight-V maintains strong performance and improves results for both evaluated models.The benchmarks include TextVQ, DocVQA, OCRBench, and AI2D; the authors attribute improved understanding to guiding attention toward relevant regions.Source: Table 2 · p. 6, §4.2 · p. 7

4.3. Further Analysis

  • Curated DPO data and additional training rounds each improve average benchmark performance.The curated dataset yields approximately 0.6% improvement versus approximately 0.2% for size-matched RLAIF-V data; two additional DPO rounds yield a further 0.6% gain.Source: Table 4 · p. 7, §4.3 · p. 8
  • Reasoning-agent training benefits from scaling: larger datasets support step-by-step reasoning and more useful input to the summary agent.The ablation compares training sets of 50K, 100K, 150K, and 200K samples using the same summary agent across six benchmarks; limited data can perform worse than baseline models.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4. Qualitative Results

  • In a table-based multiple-choice case, Insight-V produces structured reasoning and a summary agent that evaluates whether to use the reasoning to answer.Direct Chain-of-Thought and single-model fine-tuning reach incorrect answers in the described example; the single model falters as the reasoning chain lengthens.Source: Figure 5 · p. 8, §4.4 · p. 8

5. Conclusion

Insight-V combines scalable generation of long-chain reasoning data with multi-agent training to improve MLLM reasoning, and evaluation across benchmarks demonstrates the approach’s effectiveness.

5. Conclusion

  • Insight-V combines scalable generation of long-chain, high-quality reasoning data with a multi-agent training pipeline for MLLMs.The system is presented as a scalable approach to model training aimed at improving reasoning performance.Source: §5 · p. 9
  • Extensive evaluation across various benchmarks demonstrates the effectiveness of the approach.Source: §5 · p. 9
  • The paper presents the system as a foundation for future research on MLLM reasoning.Source: §5 · p. 9

A. More Implementation Details

Preference data is collected from sampled reasoning paths scored by an advanced LLM, and DPO training uses selected positive and moderately scored rejected examples with supervised fine-tuning for stability.

A. More Implementation Details

  • For each image-text pair, the authors sample 16 outputs and have an advanced LLM score all reasoning paths in one forward pass.Qwen2.5-72B is given the question, ground-truth answer, and corresponding reasoning processes.Source: A. More Implementation Details · p. 9
  • Paths scoring above 85 become positive examples, while rejected examples are chosen around a score of 25 rather than from the lowest-scoring path.This choice is intended to increase task complexity and avoid overfitting specific data patterns.Source: A. More Implementation Details · p. 9
  • DPO uses β = 0.1 and incorporates a standard supervised fine-tuning loss to stabilize training.Source: A. More Implementation Details · p. 9

B. Analysis Experiments of Multi-agent System

The multi-agent analysis reports more accurate reasoning paths and the ability to answer correctly even when a reasoning path is incorrect, indicating selective use of reasoning results.

B. Analysis Experiments of Multi-agent System

C. Discussion and Limitations

Insight-V is an initial exploration of o1-like MLLM reasoning, organizing single-step reasoning into long chains and separating reasoning from summarization. The discussion identifies sampling efficiency, model scalability, and reasoning quality as areas for improvement.

C. Discussion and Limitations

  • Insight-V organizes MLLM single-step reasoning into structured, long-chain reasoning paths.The paper describes this as a promising approach and reports that fine-tuning on the dataset yields long-chain reasoning ability.Source: C. Discussion and Limitations · p. 9
  • A multi-agent system separates question answering into reasoning and summarization stages, allowing selective incorporation of reasoning results.Source: C. Discussion and Limitations · p. 9
  • The authors identify sampling efficiency as a limitation because multi-granularity assessment currently depends on other models.They propose evaluating reasoning at each step and pruning redundant samples to streamline the process.Source: C. Discussion and Limitations · p. 9
  • Training two models of the same size may not scale well.The authors suggest that improving the reasoning agent could permit a smaller, cost-effective summarization agent because summarization is less complex.Source: C. Discussion and Limitations · pp. 9–10
  • The authors identify improving overall reasoning quality as another priority, including reflection on prior steps or inference scaling.These approaches could provide a stronger foundation for the summary model to answer questions.Source: C. Discussion and Limitations · p. 10
  • The paper frames its method as a foundational attempt to inspire and guide future research in the emerging field.Source: C. Discussion and Limitations · p. 10
Loading 2411.14432v2…