Source-linked AI summary
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, Jifeng Dai
TL;DR
Open-source MLLMs can lose multimodal reasoning performance under CoT because supervised fine-tuning creates a training–inference distribution shift, while suitable multimodal preference data remains limited. The paper constructs MMPR and proposes MPO to learn response preferences, response quality, and preferred-response generation. InternVL2-8B-MPO reaches 67.0 on MathVista, 8.7 points above InternVL2-8B and comparable to the 10× larger InternVL2-76B.
Problem
Open-source MLLMs have limited multimodal reasoning, particularly under CoT, while multimodal reasoning preference data is scarce and costly to annotate.
Method
The paper constructs MMPR with an automated preference-data pipeline and proposes MPO to learn relative preference, absolute response quality, and preferred-response generation.
Results
67.0 MathVista accuracy from InternVL2-8B-MPO surpasses InternVL2-8B by 8.7 points and is comparable to the 10× larger InternVL2-76B.
Takeaways & Limitations
Preference optimization enhances multimodal reasoning abilities for both InternVL2-8B and InternVL2-76B.
Takeaways & Limitations
Because MMPR may differ from benchmark distributions, the paper uses M3CoT training and validation sets for optimization-algorithm ablations.
Abstract
from arXiv · showhide
Existing open-source multimodal large language models (MLLMs) generally follow a training process involving pre-training and supervised fine-tuning. However, these models suffer from distribution shifts, which limit their multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance. To address this, we introduce a preference optimization (PO) process to enhance the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data side, we design an automated preference data construction pipeline to create MMPR, a high-quality, large-scale multimodal reasoning preference dataset; and (2) on the model side, we explore integrating PO with MLLMs, developing a simple yet effective method, termed Mixed Preference Optimization (MPO), which boosts multimodal CoT performance. Our approach enhances the multimodal reasoning abilities of both InternVL2-8B and InternVL2-76B. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance comparable to the 10$\times$ larger InternVL2-76B. We hope this study could inspire further advancements in MLLMs. Code, data, and model are released.
1. Introduction
Open-source MLLMs have limited multimodal reasoning, with distribution shift making long CoT generation especially problematic. The paper addresses this through automated preference data construction and MPO, reporting improved reasoning performance.
- Motivation: InternVL2-8B scores 58.3 with direct answers but 56.8 with CoT on MathVista.The paper attributes this decline to distribution shift between training and inference.
- Motivation: Teacher forcing creates a training–inference distribution shift that becomes more severe when models generate long CoT rationales.During inference, models condition on their own prior outputs rather than ground-truth tokens.
- Challenges: Multimodal preference datasets largely target hallucination reduction and natural-image perception, leaving scientific-image reasoning data scarce and costly to annotate.The paper identifies limited preference data and high annotation cost as central challenges.
- Contributions: MPO combines preference optimization with multimodal training to improve reasoning without requiring a reward model.The method is presented as a simple yet effective approach for boosting multimodal CoT performance.
- Contributions: The automated pipeline constructs MMPR, a high-quality multimodal reasoning preference dataset containing approximately 3 million samples.It is designed for instructions with and without clear ground truths.
- Results: 67.0 MathVista accuracy from InternVL2-8B-MPO exceeds InternVL2-8B by 8.7 points and is comparable to the 10× larger InternVL2-76B.The experiments also report improved reasoning for both InternVL2-8B and InternVL2-76B.
2. Related Work
MLLM research has advanced through architectural designs and improved training data, but prevailing pre-training plus supervised fine-tuning remains associated with distribution shift and limited multimodal reasoning. This work studies preference optimization as an alternative improvement direction.
- MLLM Architectures: MLLMs commonly connect pretrained language and vision models or add fusion layers to align visual and textual representations.Other work explores vision-encoder-free architectures that jointly process visual and textual information.
- Training Data: Recent studies also construct higher-quality training data to improve multimodal reasoning abilities.These efforts complement architectural changes rather than replacing them.
- Research Gap: Despite these advances, MLLMs typically rely on pre-training and supervised fine-tuning, which suffer from distribution shift and limited multimodal reasoning abilities.The paper positions preference optimization as the focus of its systematic study.
3. Scalable Multimodal Preference Dataset Generation
The paper introduces a scalable pipeline for constructing multimodal preference data from instructions with either clear or unclear ground truths. The resulting MMPR dataset combines diverse preference samples at million-level scale.
- Dataset Construction: MMPR is a million-level multimodal preference dataset constructed with a scalable data pipeline.The dataset addresses scarcity of multimodal preference data.
- Dataset Structure: Each MMPR sample contains an image, an instruction, a chosen response, and a rejected response, with the chosen response preferred over the rejected one.Images and instructions are collected from existing datasets.
- Correctness-Based Pipeline: For clear-ground-truth instructions, responses matching the ground-truth answer form the positive set, while incorrect or unclear-final-answer responses form the negative set.The model is prompted to provide reasoning followed by a final answer.
- DropoutNTP: For unclear-ground-truth samples, DropoutNTP uses an image-conditioned response as chosen and an image-free completion of its truncated prefix as rejected.The image-free completion is expected to introduce more hallucinations, establishing the preference ordering.
- Dataset Statistics: MMPR contains approximately 750K samples without clear ground truths and 2.5M samples with clear ground truths.This totals approximately 3.25M samples across the two categories described.
4. Improved Multimodal Large Language Model with Preference Optimization
MPO combines supervised fine-tuning with preference objectives to improve multimodal CoT learning. It uses complementary losses and diverse CoT data-generation strategies to learn preferred reasoning responses.
- Mixed Preference Optimization: MPO blends SFT loss with preference optimization losses to enhance multimodal reasoning capabilities.The method is designed to improve training effectiveness for multimodal CoT.
- Training Objective: MPO combines preference, quality, and generation losses to learn response preferences, absolute response quality, and preferred-response generation.The objective is defined as a combination of Lp, Lq, and Lg.
- Preference and Quality Losses: DPO teaches relative preference between chosen and rejected responses, while BCO models the absolute quality of individual responses.BCO trains a binary classifier that maps chosen responses to 1 and rejected responses to 0.
- Preference Data Construction: For clear-ground-truth instructions, multiple solutions are separated into chosen and rejected responses according to answer correctness.The example category is explicitly labeled as instructions with clear ground truths.
- Generation Loss: SFT serves as the generation loss, helping the model learn how to generate preferred responses.This loss targets the generation process rather than only pairwise or absolute response quality.
- CoT Data Construction: The data pipeline samples step-by-step CoT responses and mixes background-knowledge, visual, and grounded reasoning approaches to diversify multimodal training data.Grounded CoT links referenced objects to image regions and is applied to general VQA samples; the approaches also improve negative-response quality in DropoutNTP.
5. Experiments
Experiments evaluate MPO and its data-construction pipeline across multimodal reasoning, VQA, hallucination, text-only, and optimization-algorithm settings. MPO improves reasoning and related capabilities over pre-MPO and SFT baselines, while SFT-augmented preference losses are especially effective for CoT.
- 5.1. Main Results: MPO models outperform their pre-MPO counterparts across multimodal reasoning, VQA, and hallucination benchmarks.InternVL2-8B-MPO and InternVL2-76B-MPO show significant improvements over their corresponding baselines.
- 5.1. Main Results: 67.0% accuracy on MathVista lets InternVL2-8B-MPO surpass InternVL2-8B by 8.7 points and match the performance of the 10× larger InternVL2-76B.InternVL2-76B-MPO also reaches 30.6% accuracy on MathVision, reported as state-of-the-art.
- 5.2.1. Effects of MPO and SFT: 79.2 on M3CoT makes MPO outperform SFT by 11.4 points, with MPO also performing better on general and hallucination benchmarks.The SFT-trained model performs worse with CoT than direct answers on MMVet and POPE.
- 5.2.2. Effects of Data Construction: DropoutNTP achieves comparable performance to RLAIF-V while requiring only one continuation per sample instead of decomposing responses into atomic claims.Its Object HalBench response-level hallucination rate is 7.6 versus 7.3 for RLAIF-V, and its MMHal-Bench score is 3.6 versus 3.5.
- 5.2.3. Effects of Optimization Algorithms: Almost all preference optimization methods outperform SFT in Direct and CoT settings, but DPO variants need SFT loss to improve CoT reasoning.DPO+ and BCO+ achieve the best CoT performance among existing algorithms; MPO combines DPO and BCO losses.
- 5.2.4. Text-only Performance: MPO improves text-only performance despite MMPR containing no text-only data, with the largest gains reported on TheoremQA and IFEval.On TheoremQA, MPO reaches 20.8 accuracy, exceeding the baseline by 5.2 points and SFT by 5.0 points.
6. Conclusion
The paper introduces preference optimization for multimodal reasoning through an automated preference-data pipeline and MPO. The resulting models improve reasoning and reduce hallucinations, with InternVL2-8B-MPO matching the 10× larger model on MathVista.
- Data Contribution: The paper creates MMPR, a high-quality, large-scale multimodal reasoning preference dataset, using an automated pipeline.The pipeline handles instructions both with and without clear ground truths.
- Model Contribution: MPO learns response preferences, individual-response quality, and the process for generating preferred responses.The method is designed as a simple yet effective preference optimization approach for MLLMs.
- Findings: InternVL2-8B-MPO and InternVL2-76B-MPO improve multimodal reasoning and reduce hallucinations relative to their pre-MPO counterparts.InternVL2-8B-MPO reaches performance comparable to the 10× larger InternVL2-76B on MathVista.
7. Implementation Details
Implementation details specify preference-data sampling and MPO optimization settings. The pipeline uses bounded reasoning-process and preference-pair generation, while training uses AdamW with warmup, cosine decay, and a KL penalty.
- Data Construction: For clear-ground-truth samples, the pipeline samples at most 32 reasoning processes and constructs at most 15 preference pairs per query.For DropoutNTP, the original response is truncated by half before completion without image input.
- Optimization: MPO training uses global batch size 256, AdamW, learning rate 5e-6, 5% linear warmup, cosine decay, and KL coefficient β=0.1.AdamW uses β1=0.9, β2=0.999, and weight decay 0.05.
- Optimization: The mixed objective weights are set to w_p=0.8, w_q=0.2, and w_gt is specified in the MPO training configuration.The supplied implementation passage truncates the final weight value.
8. More Ablation Studies
The ablations show that MPO’s performance depends on preference-optimization design, data scale, loss weighting, and multimodal CoT data. Across comparisons, adding SFT or generation loss and using balanced dropout improve reasoning outcomes.
- PO variants: Adding an SFT loss significantly improves CoT performance over each preference-optimization algorithm’s vanilla counterpart.
- PO variants: MPO outperforms other loss combinations, while DPO+cDPO underperforms DPO alone and remains below MPO even after adding generation loss.
- PO variants: MPO achieves superior overall performance to DPO+ and BCO+ when trained with MMPR.
- DropoutNTP: A dropout ratio of 0.50 performs best, whereas 0.75 performs worst and 0.25 underperforms 0.50 because chosen and rejected response quality differences become less effective for training.
- Data scale: 78.9 accuracy is achieved with CoT at 100K preference pairs, the highest tested data volume, while direct answering reaches 76.4.Both direct-answer and CoT accuracy positively correlate with data scale, and CoT is higher at every tested scale.
- Hyper-parameters: The optimal reported coefficients are wp=0.8, wq=0.2, and wg=1; reducing wg to 0.01 makes CoT worse than direct answering.
- Multimodal CoT: Including multimodal CoT samples improves complex VQA performance by 1.6 points on MMVet and 5.3 points on LLaVA-Bench.
9. More Data Examples in MMPR
This section presents additional MMPR examples spanning the dataset’s task types and preference-data construction methods.
- Figures 7a–7f show DropoutNTP examples, Figures 7g–7j show correctness-based examples, and Figures 7k–7m show multimodal CoT examples.
10. Qualitative Results
The qualitative comparison contrasts InternVL2-8B, DPO, and MPO outputs, showing that DPO more often produces repetitive or invalid responses. MPO also provides qualitative gains in image recognition and image-grounded reasoning.
- 16.4% of InternVL2-8B-DPO responses are unparsable gibberish or repetitive outputs, versus 0.4% for DPO+ and 0.3% for MPO.
- The qualitative examples indicate that InternVL2-8B-MPO better recognizes image information and reasons from it than the pre-MPO model.
11. Discussion
The discussion distinguishes SFT loss from MPO’s generation loss by their objectives: generation loss corrects response probabilities to mitigate distribution shift. The examples illustrate how preferred responses preserve image-grounded reasoning over hallucinated or incorrect alternatives.
- SFT trains on positive samples to mimic high-quality responses, whereas PO compares positive and negative samples to raise positive-response probability.
- Generation loss decreases negative-sample probability while increasing positive-sample probability, thereby correcting distribution shift and preventing model collapse.
- The paired magnet examples select “repel” because both facing pole pairs are alike, illustrating explicit reasoning from visual relations.
- The MMPR examples include chosen and rejected responses that differ in object counting, graph interpretation, and receipt-total reasoning.
- MPO’s preference examples favor image-grounded answers, such as identifying exponential rather than logistic growth from a J-shaped graph.