mDPO: Conditional Preference Optimization for Multimodal Large Language Models

1 Introduction

• DPO can achieve similar performance when all preference-data images are removed, indicating that it may neglect visual conditions during optimization.This controlled comparison suggests the issue is not solely preference-data quality.Source: §1 · p. 1, Figure 1 · p. 1
• Language-only preferences can cause multimodal models to ignore images, producing question-driven responses, suboptimal performance, and increased hallucination.Qualitative examples show standard DPO accepting adversarial image premises or guessing without consulting the image.Source: §1 · pp. 1–2
• MDPO addresses multimodal DPO's unconditional visual preferences by combining conditional preference optimization with anchored preference optimization.The method emphasizes both visual and language cues while preventing the likelihood of chosen responses from decreasing.Source: §1 · p. 2
• MDPO is evaluated on Bunny-v1.0-3B and LLaVA-v1.5-7B using MMHalBench, Object HalBench, and AMBER.Both automatic and human evaluations cover different model and data scales.Source: §1 · p. 2
• MDPO consistently outperforms standard DPO and reduces hallucinations across multimodal benchmarks and varying model and data scales.Fine-grained and qualitative analyses also report improved image comprehension and reduced language bias.Source: §1 · p. 2

2.1 Background: Preference Optimization

• Preference optimization trains models to prefer evaluator-selected responses, while DPO maximizes the reward difference between chosen and rejected responses.In multimodal settings, preferences are intended to depend on both the image and the question.Source: §2.1 · p. 3

2.2 Problem: Unconditional Preference

• Multimodal DPO expects preferences conditioned on images and questions, but its objective may practically prioritize language-only preferences.This gap can lead models to disregard image information and increase hallucination.Source: §2.2 · pp. 3–4
• DPO (No Image) removes visual signals from preference data, testing whether multimodal DPO actually uses images during preference optimization.The experiment uses Bunny-v1.0-3B and 10K Silkie preference instances.Source: §2.2 · pp. 3–4
• DPO (No Image) performs similarly or slightly better than DPO on MMHalBench, indicating that learned preferences may disregard image information.The comparison challenges explanations based solely on preference-data quality.Source: §2.2 · pp. 3–4

3.1 Conditional Preference Optimization

• Conditional preference optimization constructs pairs differing in visual information while holding the question and chosen response fixed, forcing preference decisions to use image content.The rejected image should be less compatible but retain shared features as a hard negative; random cropping less than 20% performed best in the reported comparison.Source: §3.1 · p. 4
• The conditional objective contrasts the chosen response under a compatible image with the same response under a less compatible image, complementing DPO’s response comparison.Together, the objectives encourage preferences based on both visual and language cues.Source: §3.1 · p. 4

3.2 Anchored Preference Optimization

• Anchored preference optimization addresses chosen-response likelihood decreases by forcing the chosen response’s reward above a specified anchor value.Standard relative preference optimization can enlarge the chosen–rejected gap by lowering the chosen response likelihood, despite chosen responses often being high quality.Source: §3.2 · p. 4
• The anchor introduces absolute reward regularization, with values adjustable by data properties, expected behavior, or separate purposes.The anchor may also be applied oppositely to rejected responses, and alternative anchors are compared in §4.4.Source: §3.2 · p. 4

3 MDPO

• MDPO combines standard DPO, conditional preference optimization, and anchored preference optimization in a single objective.The combined objective is L_MDPO = L_DPOm + L_CoPO + L_AncPO.Source: §3.2 · p. 4

4.2 Main Results

• MDPO consistently outperforms standard DPO across three benchmarks for Bunny and LLaVA, with particularly strong hallucination reduction despite a minor object-coverage decrease.A 3B Bunny trained with 10K preference examples becomes comparable to a stronger 7B Qwen-VL-Chat trained with DPO on 80K examples.Source: Table 1 · p. 5, §4.2 · p. 6

4.3 Human Evaluation

• Human evaluators judged MDPO responses better or equal to DPO responses on 89% of MMHalBench instances, while DPO was better on 11%.The comparison used domain experts selecting the better response from Bunny models trained with DPO or MDPO.Source: §4.3 · p. 6, Figure 4 · p. 6

4.4 Analysis

• MDPO improves with increasing preference-data scale and outperforms DPO across scales, whereas standard DPO shows no multimodal scaling effect.The analysis indicates that DPO’s neglect of visual information cannot be remedied merely by increasing preference-data size.Source: §4.4 · pp. 6–7, Figure 5 · p. 6
• Ablations show that both anchored and conditional preference improve MDPO, but conditional preference produces the larger gain and is crucial for using visual information.The table reports only slight improvement from anchored preference alone over DPO.Source: §4.4 · p. 7, Table 2 · p. 7
• Cropping 0-20% of chosen images is the strongest rejected-image strategy because it preserves similarity while removing visual information; random images are too easy to reject.MoCo v2 augmentation may fail to produce clearly worse images.Source: §4.4 · p. 7, Table 3 · p. 7
• The chosen-response reward anchor is sufficient: adding anchors for rejected responses or images yields no obvious improvement and may complicate training.MDPO regularizes the chosen response-image reward r(m_w, q, y_w) to be positive by default.Source: §4.4 · p. 7, Table 4 · p. 7

4.5 Fine-grained Results

• MDPO outperforms DPO on six of eight MMHalBench question categories, with especially strong gains on adversarial questions containing false image premises.MDPO identifies incorrect premises using the image, whereas DPO often accepts them.Source: §4.5 · p. 7, Table 5 · p. 8

4.6 Qualitative Study

• Qualitative examples show MDPO answering conditionally on the image, while standard DPO may trust adversarial descriptions or disregard the image and guess.This behavior connects the qualitative results to the observed unconditional-preference problem.Source: §4.6 · p. 8

• Textual preference optimization advances address algorithmic efficiency, reward overfitting, unpaired data, or on-policy sampling through methods such as ORPO, IPO, KTO, NCA, and Iterative DPO.Source: §5 · p. 8
• This work instead identifies a multimodal DPO challenge in which preference optimization neglects images and proposes conditional preference optimization to address it.Source: §5 · p. 8
• Multimodal preference studies primarily create enhanced preference data, while others apply DPO, reinforcement learning, or contrastive learning objectives.Source: §5 · p. 8

• MDPO encourages multimodal LLMs to use both visual and language cues through conditional preference optimization.Anchored preference optimization prevents the likelihood of preferred responses from decreasing.Source: §6 · p. 8
• MDPO consistently enhances multimodal LLM performance and reduces hallucination across different model sizes on three widely used benchmarks.The authors state that the method can extend to multiple images, videos, in-context learning, and risk-sensitive domains.Source: §6 · p. 9

• Experiments on more multimodal LLMs across varied sizes and architectures are needed to further assess MDPO’s advantages and disadvantages.Source: Limitation · p. 9
• The study focuses on unconditional preference, leaving combinations with DPO improvements from other perspectives for future work.Source: Limitation · p. 9
• Evaluation on three benchmarks covers a limited range of tasks and settings compared with real-world scenarios, motivating broader benchmark testing.Source: Limitation · p. 9