OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

1 Introduction

• The authors’ preliminary study found that adding text-only SFT data did not improve multimodal preference alignment and harmed foundational multimodal skills.They argue that improving human alignment in multimodal settings calls for specialized multimodal instruction-tuning datasets.Source: §1 · p. 2
• OmniAlign-V contains about 200K multimodal instruction samples designed around open-ended questions, broad topics, varied response formats, and instruction adherence.It includes natural images and infographics, with complex knowledge-based, creative, and reasoning tasks.Source: §1 · p. 2
• Using OmniAlign-V for SFT and DPO significantly improves human preference alignment, while VQA performance remains comparable or improves.The reported SFT and DPO results include gains across InternLM2.5 and Qwen2.5 encoders; the Qwen2.5-32B LLaVA-NeXT baseline after both stages surpasses Qwen2VL-72B finetuned on extensive proprietary datasets.Source: §1 · p. 2
• MM-AlignBench provides 252 human-annotated samples with diverse image sources and carefully crafted questions for evaluating MLLMs’ human preference alignment.The authors report that existing multimodal preference benchmarks have limited image-source diversity, repetitive questions, and unclear questions.Source: §1 · p. 2
• The paper investigates MLLM human-alignment degradation by analyzing the effects of text-only and multimodal tuning data.This investigation is listed as a study contribution.Source: §1 · p. 2

Alignment of LLMs

• Prior work focused on generating high-quality SFT training data to enhance LLM alignment; recent benchmarks use open-ended, challenging questions, yet benchmarks designed to evaluate MLLM alignment are notably lacking.Source: §2 · p. 3

Visual Question Answering Datasets

• Traditional VQA and visual classification datasets often use simple questions or brief descriptions that elicit short answers focused on basic image semantics or main objects.The passage cites VQA examples (Hudson and Manning, 2019) and visual datasets (Lin et al., 2014; Schuhmann et al., 2021).Source: §2 · p. 3

Instruction Tuning data

• Multimodal instruction-tuning datasets expanded to captions, complex QA, OCR, math, and document images, yet remained oriented toward foundational capabilities and short factual exchanges.The passage cites LLaVA (Liu et al., 2023b) and later data efforts, including Chen et al. (2023a), Chen et al. (2024a), and Gu et al. (2024).Source: §2 · p. 3

3 Human Alignment of MLLMs: The Preliminary Study

• After multimodal SFT, MLLMs’ text-only open-ended alignment degrades significantly relative to the corresponding LLMs.The authors suggest possible contributors: insufficient or low-quality text-only samples, or simplistic multimodal tuning data with short factual answers.Source: §3 · p. 3, Table 1 · p. 3
• Higher-quality text-only SFT samples improve language-alignment benchmarks but reduce multimodal alignment and common VQA performance.The study replaced or increased the proportion of text-only data using samples from Magpie-Llama3.3 and Condor; the reported table describes declines on WildVision and MMVet.Source: §3 · pp. 3–4, Table 2 · p. 4
• The authors conclude that language-alignment capability does not directly transfer to multimodal alignment and argue for high-quality multimodal human-aligned training data.The tested text-only data mixtures did not improve multimodal preference alignment despite improving text-only results.Source: §3 · p. 4

4.1 Task Taxonomy: A Big Picture

• Natural-image tasks are Knowledge, Inferential, and Creation; infographic tasks cover Arts, Charts, Diagrams, and Posters.The natural-image tasks use complex questions and comprehensive, reasoned responses; infographic categories target human-designed visuals.Source: §4.1 · p. 4

4.2 Image Selection Strategy

• Natural-image selection filters for semantic richness by combining IC9600 complexity scores with object recognition to exclude complex but low-content images.High IC scores alone do not guarantee semantic richness; the Recognize Anything Model further filters images with minimal meaningful content.Source: §4.2 · p. 4

4.3 Data Generation Pipeline

• Knowledge and inferential QAs use task-specific few-shot prompts, while creative QA generation selects caption-relevant seed questions before sampling examples for GPT-4o.A lightweight MLLM generates image captions, and an LLM selects relevant seed questions based on each caption.Source: §4.3 · p. 5
• Infographic QA uses information-rich image sources and specialized prompts, while chart-answer refinement merges complementary model responses and adds human review.The refinement filters questions with substantial response-trend discrepancies; merged answers are reviewed for quality and consistency.Source: §4.3 · pp. 5–6
• Post-refinement augments Knowledge QAs with instructions and enriches Inferential answers with explanations, background information, and reasoning.The added instructions are selected to avoid dependence on visual content, and corresponding answers are adjusted to fit the modified question and original context.Source: §4.3 · p. 6

4 OmniAlign-V

• 205K high-quality SFT samples span knowledge, inferential, creative, instruction-following, infographic, and image-detail questions.The dataset includes 39K Knowledge, 37K Inferential, 10K Creative, 38K Instruction-Following (Knowledge), 44K Infographic, and 35K image-detail QAs.Source: §4 · p. 4, §4.3 · p. 6

4.3 Data Generation Pipeline

• For DPO, OmniAlign-V QAs provide positive samples, while an LLM judge selects the generated response that most deviates from the question’s intent and context as the negative.A LLaVANext baseline generates multiple high-temperature responses for each question.Source: §4.3 · p. 6

4 OmniAlign-V

• OmniAlign-V SFT significantly improves MLLMs’ multi-modal preference alignment while delivering comparable or better performance on common VQA benchmarks.The reported comparisons are in Table 3.Source: Table 3 · p. 7

5 MM-AlignBench

• MM-AlignBench targets repetitive, simplistic question formats in existing multi-modal preference benchmarks with human-curated images and questions.The benchmark is intended to support more nuanced evaluation of alignment with human preferences.Source: §5 · p. 7
• 252 high-quality image-question pairs were selected from natural-image and infographic sources, then reviewed to remove low-quality, repetitive, or contextually weak samples.The process began with 2,000 semantically rich natural images and 1,000 selected infographic images.Source: §5 · p. 7
• Evaluation uses GPT-4o to compare model responses with Claude3V-Sonnet reference responses and reports winning rates and reward scores.Source: §5 · p. 7

6.1 SFT with OmniAlign-V

• OmniAlign-V significantly improves alignment across MM-AlignBench, WildVision, and MIA-Bench, while also improving general multimodal capabilities, particularly on MMVet and MMMU.Even without language samples in training, models retain stronger language alignment than models trained on LLaVA-Next-778k.Source: §6.1 · p. 7

6.2 DPO with OmniAlign-V-DPO

• DPO improves WildVision performance across all tested models, but substantial MM-AlignBench gains follow prior OmniAlign-V SFT rather than DPO alone.InternVL2-8B also improves on MM-AlignBench after DPO; the authors attribute the gains to prior exposure to human-preference-aligned SFT data.Source: §6.2 · p. 8, Table 5 · p. 8

6.3 MM-AlignBench

• A 72.6 winning rate and +33.5 average reward put LLaVA-OA-32B-DPO above Qwen2VL-72B on MM-AlignBench.The model was trained with OmniAlign-V and OmniAlign-V-DPO.Source: §6.3 · p. 8, Table 6 · p. 8

6.4 Ablation Study

• Instruction-following data significantly improves results across MM-AlignBench, WildVision, and MMVet, while creation data improves MM-AlignBench alone.The authors relate the latter difference to the absence of multimodal creative question types in WildVision and MMVet.Source: §6.4 · p. 8

• The paper introduces OmniAlign-V for human-preference alignment and MM-AlignBench as a specific-purpose benchmark for evaluating that alignment.The conclusion emphasizes multimodal open-ended training data and investigates the impact of language and multimodal training data.Source: §7 · pp. 8–9
• Progressively incorporating OmniAlign-V tasks is associated with gradual improvement in model alignment performance.This conclusion is reported in the Table 7 ablation caption.Source: Table 7 · p. 9
• Incorporating OmniAlign-V into SFT and DPO stages yields significant improvements in MLLM human-preference alignment.Source: §7 · p. 9

8 Limitation

• The dataset scale used in this paper may be insufficient for large-scale training because of cost limitations, although the pipeline can scale to larger datasets.The authors state that deeper exploration of MLLM alignment is still needed.Source: §8 · p. 9

A Alignment of MLLMs with Human Preference

• Open-source MLLMs can match or surpass proprietary models on common VQA tasks, yet remain less aligned with human preferences on open-ended questions.InternVL2-76B struggles with comprehensive, readable answers, while GPT-4o provides structured responses with contextual knowledge.Source: A Alignment of MLLMs with Human… · p. 12

B Image sources

• Image sources include WikiArt, ChartQA, TextbookVQA, and InfographicsVQA, selected for diverse art styles and information-rich charts, diagrams, and posters.The authors uniformly sample 2,000 WikiArt images across painting styles, filter simplistic two-column charts, and use question and teaching images from TextbookVQA.Source: B Image sources · p. 12

C Chart Post-Refine Pipeline

• Chart answers are cross-checked across model-generated responses, with conflicting facts flagged for review or discarded and minor differences merged for expert review.Qwen2.5-72B extracts chart facts; human experts review the merged answers for quality and consistency.Source: C Chart Post-Refine Pipeline · pp. 12–13

D Training Details

• Pretraining freezes both the visual encoder and LLM, using CLIP-Large-336-Patch14 and the LLaVA-pretrain-558k and ALLaVA-pretrain-728k datasets.Pretraining uses batch size 256, learning rate 1e-3, and 336×336 images without image splitting.Source: D Training Details · p. 13
• During SFT, LLaVA unfreezes the LLM, while LLaVA-NeXT also unfreezes the visual encoder and applies image splitting up to 3×3.Reported training configurations range from 8×A800 GPUs for 12 hours to 32×A800 GPUs for 24 hours.Source: D Training Details · p. 13
• Table 8 reports an ablation study on the impact of using an image filter.Source: Table 8 · p. 13

E Ablation on Image Filter

• Applying the image filter yields richer semantic images, higher-quality image-question pairs, and improved human-preference alignment.The comparison uses sampled LLaVA-Next-778K images as a baseline and CC3M images selected with or without the filter to generate Knowledge and Inferential pairs.Source: E Ablation on Image Filter · p. 13
• The ablation compares LLaVA-Next-InternLM2.5-7B performance after training on pairs generated from baseline images and filtered or unfiltered CC3M images.The generated pairs follow the pipeline in Sec. 4.3; results are presented in Tab. 8.Source: E Ablation on Image Filter · p. 13

F Human Experts

• Two computer-science PhD students review and refine MM-AlignBench questions and filter incorrect merged cases after Chart Post-Processing Refinement.Source: F Human Experts · p. 13

G License

• The InternLM and Qwen models use Apache-2.0, ChartQA uses GNU GPL v3.0, and the remaining datasets use CC BY-NC 4.0 for non-commercial use.Source: G License · p. 13
• Figure 6 reports that GPT-4o shows superior human-preference alignment to InternVL2-76B.Source: Figure 6 · p. 14
• A Knowledge-task example asks for a landmark’s history and a culturally enriching visitor activity plan, with a response describing the Colosseum and suggested itinerary.Source: G License · p. 16
• A sample image question asks what occasion people are celebrating, and its response infers Christmas from gifts, stockings, and festive decorations.Source: G License · p. 16
• A Creative-task example invites an athlete to describe pre-game thoughts and motivation, followed by a first-person response from a Vancouver hockey player.Source: G License · p. 16
• The appendix includes prompt examples for Knowledge, Creative, Inferential, Chart, and Diagram tasks, as well as examples of each OmniAlign-V task.Source: Figure 10 · p. 17, G License · pp. 20–22, Figure 16 · p. 23