PaperlayerView exact revision on arXiv

Source-linked AI summary

MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale

Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, Xiang Yue

arXiv:2412.05237v2cs.CLcs.CV

TL;DR

Open-source MLLMs struggle with reasoning-heavy tasks, and many instruction datasets provide simplistic examples with phrase-level answers rather than rationales. The paper constructs a 12-million-pair multimodal dataset with open models and detailed rationales; training on it improves performance across reasoning benchmarks, including gains over open SoTA models.

  • Problem

    Open-source MLLMs struggle with reasoning-heavy tasks, while existing instruction datasets often provide simplistic examples and phrase-level answers without intermediate rationales.

    Source: §1 · pp. 1–2

  • Method

    The authors use open models to build a 12-million-pair multimodal instruction dataset with diverse tasks and detailed CoT-style rationales, using collection, rewriting, and filtering.

    Source: §1 · p. 2

  • Results

    MAmmoTH-VL-8B improves over open SoTA models by 8.1% on MathVerse, 7% on MMMU-Pro, and 13.3% on MuirBench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that open-source models can generate diverse, high-quality training data that improves MLLMs’ practical capabilities while reducing reliance on costly proprietary models.

    Source: §7 · p. 10

    A performance gap relative to Qwen2-VL-7B persists, which the authors link to having only 1M multi-image and video training samples because of computational constraints.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Open-source multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks. However, their reasoning capabilities remain constrained by existing instruction-tuning datasets, which were predominately repurposed from academic datasets such as VQA, AI2D, and ChartQA. These datasets target simplistic tasks, and only provide phrase-level answers without any intermediate rationales. To address these challenges, we introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales designed to elicit CoT reasoning. Using only open models, we create a dataset containing 12M instruction-response pairs to cover diverse, reasoning-intensive tasks with detailed and faithful rationales. Experiments demonstrate that training MLLMs on this dataset significantly improves reasoning capabilities, achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%), MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation studies further highlight the importance of key components, such as rewriting and self-filtering, in the dataset construction process.

1 Introduction

Open-source multimodal models struggle with reasoning-heavy tasks, while many instruction datasets offer simplistic tasks and phrase-level answers rather than intermediate rationales. The paper introduces a 12-million-pair dataset built with open models and reports improved benchmark performance after training on it.

Motivation

  • Existing multimodal instruction datasets often focus on simple VQA and phrase-based answers, which do not elicit deliberate reasoning.The paper identifies human-authored CoT data as costly and proprietary tools as costly and licensing-constrained for open-source use.Source: §1 · pp. 1–2

Method

  • The dataset contains 12 million instruction-response pairs built with open models through collection, CoT-style rewriting, and filtering for coherence and accuracy.Its task coverage includes mathematical problem-solving, OCR, and domain-specific reasoning.Source: §1 · p. 2

Results

  • MAmmoTH-VL-8B, trained on a curated dataset using the LLaVA-OneVision architecture, improves over open SoTA models by 8.1% on MathVerse, 7% on MMMU-Pro, and 13.3% on MuirBench.Source: §1 · p. 2

2 Method

The method builds a 12-million-sample dataset from publicly available multimodal sources, then uses task-aware rewriting and model-based filtering to enrich and check examples.

2 Method

  • The pipeline produces 12 million samples through open-source data collection, task-specific rewriting with open models, and quality filtering.Filtering targets hallucinated or irrelevant content.Source: §2 · p. 3, §2.1 · p. 3, §2.2 · p. 4

2.1 Dataset Collection and Categorization

  • The authors screened 153 public datasets, retaining 58 as-is, rewriting 60, removing 35, and organizing data into ten task categories.The categories include OCR, Chart, Code&Math, Detection, Multi-Image, and Video.Source: §2.1 · p. 3

2.2 Instruction Data Rewriting

  • Task-aware rewriting addresses sparse rationales and limited real-world task coverage by turning image-linked data into diverse, detailed instruction-response pairs.Caption data uses a text-only model, while other data types use a multimodal model to align generated instructions with images.Source: §2.2 · p. 4

2.3 Self-data Filtering

  • The same InternVL2-Llama3-76B model judges each question-answer pair against its image for logical consistency, with the aim of minimizing hallucinations.The approach assumes the model performs better at verification than generation.Source: §2.3 · p. 4

3 Analysis of MAmmoTH-VL-Instruct

Rewritten data scores higher on content and relevance, has a longer token-length tail, and extends beyond the original instruction distribution while retaining overlap. Filtering shows category-specific patterns and agreement with human judgments.

3.1 Quality Verification

  • Rewritten data consistently surpasses original data on both information-content and relevance scores.The comparison evaluates 1,000 sampled examples from each dataset on 1–5 scales.Source: §3.1 · p. 4

3.2 Distribution Comparison

  • Rewritten examples show a broader, more evenly distributed token length, with a longer tail at larger token counts.The paper suggests the added length likely reflects more detail or rationales for explanation and clarity.Source: §3.2 · p. 4
  • The rewritten data significantly overlaps the original distribution while extending beyond its boundaries; experimental validation trains the model on a mixture of 70% rewritten data and 30% original data.The instructions were projected onto a two-dimensional plot using t-SNE.Source: §3.2 · p. 5

3.3 Analysis of the Filtering Step

  • 0.64 average Kappa measures agreement between the model and two human evaluators, compared with 0.55 average Kappa among the three human evaluators.The paper describes 0.60–0.80 as good consistency.Source: §3.3 · p. 5
  • OCR and Chart data receive more extensive filtering than GeneralQA and Math, and the text reports high hallucination rates in visually complex categories.The paper also points to a filtering ablation with performance improvements.Source: Figure 6 · p. 5, §3.3 · p. 5

4 Experiments

MAmmoTH-VL-8B is evaluated across single-image, multi-image, and video tasks after staged visual instruction tuning. It performs strongly across single-image benchmarks and surpasses the previous open-source leader on multi-image and video benchmarks, while a gap to Qwen2-VL-7B remains.

4.1 Model Training

  • The three training stages progress from language-image alignment to single-image instruction tuning, then to mixed single-image, multi-image, and video data.Stage 2 uses 10M single-image instructions; Stage 3 uses 2M mixed-modality samples, including about 1M single-image samples without Stage 2 overlap.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Model Evaluation Setup

  • Evaluation spans single-image, multi-image, and video scenarios, using standardized zero-shot greedy decoding across 23 benchmarks unless specified otherwise.The authors primarily report results from original papers and evaluate unavailable results under consistent settings.Source: §4.2 · pp. 5–6

4.3 Single-Image Performance

  • MAmmoTH-VL-8B achieves state-of-the-art performance on 9 benchmarks and remains robust overall despite some declines from adding multi-image and video training.The single-image results cover multidisciplinary knowledge, mathematical reasoning, chart, diagram, document, and real-world interaction benchmarks.Source: Table 2 · p. 6, §4.3 · p. 7, Table 3 · p. 7

4.4 Multi-Image and Video Performance

  • Experimental results show that MAmmoTH-VL-8B significantly advances the state of the art among open-source models, surpassing previous leader LLaVA-One-Vision-7B with a 13-point improvement on MuirBench.Source: §4.4 · p. 7
  • A performance gap remains relative to Qwen2-VL-7B; the authors attribute it likely to computational constraints limiting multi-image and video training data to 1M samples.The paper identifies expanded modality coverage as potential headroom for improvement.Source: §4.4 · p. 8

5 Ablation Study

The ablations examine filtering, data mixing, training scale, and rewrite-model size using Qwen2.5-1.5B-Instruct as the backbone. Results favor filtering and rewriting, with effects varying by task category and data scale.

5.1 Effect of Data Filtering

  • Filtering improves results, particularly on chart, diagram, and document benchmarks, where filtered data removes more OCR and chart examples associated with hallucinations.The comparison uses 1M randomly sampled examples before and after filtering.Source: §5.1 · p. 8

5.2 Effect of Data Mixing Ratio

  • A 3:7 original-to-rewritten mix performs marginally better than rewritten-only data, while rewritten-only data outperforms original-only data.The figure reports generally improving performance as the rewritten-data share increases, with 70% rewritten data selected as a balance.Source: §5.2 · p. 8, Figure 8 · p. 8

5.3 Effect of Training Data Scale

  • Performance generally rises with training-data scale, although MMStar and SeedBench decline late in Stage 2, mainly in scene understanding and instance attributes.The authors suggest models may prioritize object relationships over detailed attribute recognition as task complexity increases.Source: §5.3 · p. 8

5.4 Impact of Rewrite Model Size

  • Rewrite-model size has task-dependent effects: smaller rewrites are comparable for knowledge and reasoning, but degrade chart and document results and fare worse on multi-interaction and preference tasks.Larger models provide modest chart and document improvements and perform better on multi-interaction and preference tasks.Source: §5.4 · p. 9, Figure 9 · p. 9

5.5 Qualitative Analysis

  • The qualitative analysis presents real-world question-and-answer examples to assess complex, unconstrained queries and contextually appropriate responses.The authors say the examples also offer insights into current strengths and areas for improvement.Source: §5.5 · p. 9

6 Related Work

Prior multimodal instruction-data approaches trade off annotation quality, scalability, diversity, cost, and accessibility. The paper positions open-source data generation and quality filtering as a way to address these constraints.

Multimodal Instruction Data

  • Human annotation can produce precise, context-rich data but is expensive and labor-intensive, limiting scalability.The paper cites Xu et al. (2024c), Deitke et al. (2024), McKinzie et al. (2024), and Sun et al. (2023).Source: §6 · p. 9
  • Repurposed academic VQA datasets offer a cost-effective alternative but often lack diversity and nuanced reasoning support.The paper cites Tong et al. (2024) and Liu et al. (2024d).Source: §6 · p. 9
  • Proprietary-model querying can generate high-quality, diverse data but brings substantial computational costs and licensing issues; the paper instead uses open-source models, augmentation, and filtering.The paper cites Chen et al. (2023a), Luo et al. (2024), Liu et al. (2024b), Wang et al. (2023), and Chen et al. (2024a).Source: §6 · p. 9

Multimodal Large Language Models

  • Open-source MLLMs face a shortage of high-quality supervised fine-tuning data, while connector-based systems link visual encoders to language models with lightweight projection modules.The paper situates its work within this connector-training paradigm.Source: §6 · p. 9

7 Conclusion

The conclusion presents open-model data generation as a scalable route to diverse, high-quality multimodal training data and reports broad task performance gains.

7 Conclusion

  • The MAmmoTH-VL-Instruct dataset contains 12 million multimodally enriched entries and underpins a model reported to achieve state-of-the-art performance across varied multimodal tasks.The conclusion says the approach reduces reliance on costly proprietary models and improves practical capabilities in diverse, realistic scenarios.Source: §7 · p. 10

A.1 Breakdown performance on each benchmark before and after filtering

Filtering improves model performance compared with unfiltered data, particularly on chart, diagram, and document understanding benchmarks.

A.1 Breakdown performance on each benchmark before and after filtering

  • Table A1 compares benchmark performance for models trained on filtered and unfiltered data, with gains particularly on chart, diagram, and document understanding.The authors report that OCR and chart data make up the majority of the filtered data.Source: Table A1 · p. 19, §A.1 · p. 19
  • Table A2 reports the retention rates for different data types after filtering.Source: Table A2 · p. 19, §A.1 · p. 19

A.2 Specific performance on each benchmark for different mix ratio

The authors compare original, rewritten, and mixed datasets, finding that a 3:7 original-to-rewritten mix slightly outperforms rewritten-only training.

A.2 Specific performance on each benchmark for different mix ratio

  • A 3:7 original-to-rewritten mix shows a slight performance gain over rewritten-only training.Across five schedules, rewritten-only training also achieves better average performance than original-only training.Source: §A.2 · p. 21, Table A3 · p. 21
  • Table A3 compares benchmark performance for models trained on five original, rewritten, and mixed-data schedules.Source: Table A3 · p. 21

A.3 Specific performance on each benchmark as trained data increased

The appendix reports performance trends with increasing training data, rewriting-model retention patterns, filter agreement, and prompt designs for rewriting multimodal examples.

A.3 Specific performance on each benchmark as trained data increased

  • Expanding training data is associated with improved performance across single-image benchmarks.Figure A1 tracks results in 2-million-data-point increments against LLaVA-OneVision-7B, Llava-CoT-11B, and LLaVA-OneVision-72B.Source: §A.3 · p. 21

A.4 Additional Details of different models for data rewriting

  • Larger rewriting models have higher data retention rates, while all three models retain very little OCR data.The authors note that the low OCR retention highlights problems facing current multimodal models.Source: §A.4 · p. 22, Table A4 · p. 22

A.5 Additional Details of Verifying the consistency of model filter and manual filter

  • Replacing one human evaluator with the model raises average pairwise Kappa from 0.55 to 0.64.The authors characterize 0.60–0.80 as good consistency and describe model/manual agreement as strong.Source: §A.5 · p. 23, Table A5 · p. 23

B.1 Prompts For Rewriting

  • Rewriting prompts ask models to create more complex tasks and detailed responses grounded in image or caption content.The appendix includes formats for step-by-step math solutions, caption-based tasks, and multi-round dialogues from VQA pairs.Source: §B.1 · pp. 25–27

B.2 Score Details Before and After Rewriting

The appendix compares original and rewritten data quality scores and presents dataset-level score details.

B.2 Score Details Before and After Rewriting

  • Rewritten data scores higher than original data on content and relevance, indicating improved quality.Source: §B.2 · p. 27, Table A6 · p. 28
  • Table A6 presents original and rewritten scores across individual datasets.The listed dataset rows include content and relevance score pairs.Source: Table A6 · p. 28, §B.2 · p. 28

C.1 Incorrect Cases

Incorrect rewrites can introduce image-unsupported details or questions with little image relevance, whereas good rewrites add complexity, diversity, and knowledge. The examples contrast faulty reasoning and unsupported elaboration with richer image-based tasks.

C.1 Incorrect Cases

  • The computer-advice rewrite invents graphics capabilities; the image supports OS/2 compatibility and keyboard and hardware details, not those graphics claims.The analysis says “OS/2 READY!” indicates compatibility but does not establish high-performance graphics.Source: §C.1 · p. 32
  • The angle explanation wrongly treats vertical opposite angles ∠1 and ∠2 as supplementary, despite arriving at 42° (option D).The analysis identifies the relationship between the angles—not the final answer—as the key reasoning error.Source: §C.1 · p. 34
  • A smartphone-armband prompt asks for market predictions largely unrelated to the image, and its answer projects sales and revenue figures.The analysis explicitly says the generated questions have very little relevance to the image.Source: §C.1 · p. 35

C.2 Good Cases

  • Good rewrites add more complex and diverse instructions, contain more knowledge, and expand monotonous datasets.Examples include extending an angle question and asking for a genre inference supported by visual evidence.Source: §C.2 · pp. 36, 38, 39
  • A good rewrite turns musicians’ instruments, attire, and setting into a genre-inference question with supporting visual evidence.The answer infers folk or indie from acoustic instruments, casual attire, rustic decor, and the absence of electric instruments.Source: §C.2 · p. 39

D Model Capabilities and Qualitative Examples

The qualitative examples span geometry, graph and table reasoning, OCR, document understanding, image interpretation, and video description. They also show differing model answers on some comparison cases.

Geometry

  • The maximum-angle example derives 30° as the maximum angle of sight and selects option G; Llava-OneVision-7B gives option D.The supplied passages present this as a model comparison without explaining whether the answers refer to identical options.Source: D Model Capabilities and Qualitative… · p. 40

Graph reasoning

  • For the Roman and Han population graph, MAmmoTH-VL-8B selects pathogen spread along the Silk Road, while Llava-OneVision-7B selects political instability.Both responses describe declining populations, but offer different explanations for the trend.Source: D Model Capabilities and Qualitative… · p. 41

Table reasoning

  • On the HIV rapid-test table, MAmmoTH-VL-8B reports sensitivity 0.995, while Llava-OneVision-7B reports 0.985.The MAmmoTH response explains sensitivity using true positives and false negatives.Source: D Model Capabilities and Qualitative… · p. 42

Chart & Doc Understanding

  • Document examples include listing Monday-to-Friday weather and transcribing repeated 12-point text from an image.The weather response gives a condition for each weekday; the OCR response reproduces the displayed sentences.Source: D Model Capabilities and Qualitative… · p. 44

Multi-Discipline Knowledge and Mathematical Reasoning

  • Image examples identify U.S. planes using visual clues and map grass, grasshopper, mouse, and hawk into a food chain.The food-chain response also identifies grass as a producer and mushroom as a decomposer.Source: D Model Capabilities and Qualitative… · pp. 43, 45

Multimodal Interactions & Preferences

  • Other examples describe a cartoon’s characters and speech bubble, assess a room image as rendered, and narrate a bacon-preparation video.The photorealism response gives a score around 1 or 2 for the image it judges to be rendered.Source: D Model Capabilities and Qualitative… · pp. 43, 47, 48

Related papers

Loading 2412.05237v2…