PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, Xue Wang, Yibo Hu, Bin Wen, Fan Yang, Zhang Zhang, Tingting Gao, Di Zhang, Liang Wang, Rong Jin, Tieniu Tan

arXiv:2502.10391v1cs.CLcs.CV

TL;DR

MLLMs have seldom received comprehensive human-preference alignment, leaving its broad effect on capability insufficiently explored. MM-RLHF provides a large, fine-grained preference dataset and introduces critique-based reward modeling with dynamic reward scaling. Evaluations across ten dimensions and 27 benchmarks report significant, consistent improvements, including gains in conversational ability and safety.

  • Problem

    MLLM alignment research has mainly addressed specific domains, while comprehensive evidence that human-preference alignment improves overall capability remains limited.

  • Method

    MM-RLHF combines a 120k-pair fine-grained human-annotated dataset with a Critique-Based Reward Model and Dynamic Reward Scaling for DPO.

  • Results

    Across ten dimensions and 27 benchmarks, the approach produces significant and consistent performance improvements, including an 11% average conversational gain and 57% reduction in unsafe behavior.

  • Takeaways & Limitations

    The results support comprehensive alignment as a way to improve MLLMs across diverse capabilities and trustworthiness dimensions.

  • Takeaways & Limitations

    Standard reward models underuse detailed human feedback and provide opaque scalar rewards, motivating critique-based intermediate reasoning.

Abstract

from arXiv · show

Despite notable advancements in Multimodal Large Language Models (MLLMs), most state-of-the-art models have not undergone thorough alignment with human preferences. This gap exists because current alignment research has primarily achieved progress in specific areas (e.g., hallucination reduction), while the broader question of whether aligning models with human preferences can systematically enhance MLLM capability remains largely unexplored. To this end, we introduce MM-RLHF, a dataset containing $\mathbf{120k}$ fine-grained, human-annotated preference comparison pairs. This dataset represents a substantial advancement over existing resources, offering superior size, diversity, annotation granularity, and quality. Leveraging this dataset, we propose several key innovations to improve both the quality of reward models and the efficiency of alignment algorithms. Notably, we introduce a Critique-Based Reward Model, which generates critiques of model outputs before assigning scores, offering enhanced interpretability and more informative feedback compared to traditional scalar reward mechanisms. Additionally, we propose Dynamic Reward Scaling, a method that adjusts the loss weight of each sample according to the reward signal, thereby optimizing the use of high-quality comparison pairs. Our approach is rigorously evaluated across $\mathbf{10}$ distinct dimensions and $\mathbf{27}$ benchmarks, with results demonstrating significant and consistent improvements in model performance. Specifically, fine-tuning LLaVA-ov-7B with MM-RLHF and our alignment algorithm leads to a $\mathbf{19.5}$% increase in conversational abilities and a $\mathbf{60}$% improvement in safety. We have open-sourced the preference dataset, reward model, training and evaluation code, as well as reward modeling and safety benchmarks. For more details, please visit our project page: https://mm-rlhf.github.io.

1 Introduction

MM-RLHF addresses the limited and fragmented alignment of MLLMs by combining a broad human-preference dataset with improved reward modeling and alignment optimization. Across ten dimensions and 27 benchmarks, the approach produces consistent gains in capabilities and safety.

  • Current state-of-the-art MLLMs rarely undergo rigorous human-preference alignment beyond supervised fine-tuning, leaving truthfulness, safety, and preference alignment insufficiently addressed.
  • The paper asks whether human-preference alignment can enhance MLLMs comprehensively rather than only on limited task domains.
  • MM-RLHF spans image understanding, video understanding, and MLLM safety, using a rigorous pipeline to produce high-quality, fine-grained annotations.
  • The dataset contains more than 120k high-quality ranked comparison pairs and advances diversity, response quality, and annotation granularity over existing datasets.
  • The Critique-Based Reward Model generates critiques before scores, while Dynamic Reward Scaling weights comparison pairs according to reward margins within DPO.
  • 11% average conversational gain and 57% reduction in unsafe behavior were reported across evaluations spanning ten dimensions and 27 benchmarks.

2 MM-RLHF-Dataset

MM-RLHF constructs a diverse multimodal alignment dataset by filtering large-scale data, generating responses with leading models, and collecting fine-grained human annotations. Its design combines broad task coverage with ranking, scoring, explanations, and quality-control strategies.

  • Data collection and filtering: The pipeline starts from 10 million instruction samples, clusters images by similarity, and samples across categories spanning image-based and video question answering.
  • Dataset scope: The dataset covers three domains: image understanding, video understanding, and multimodal safety.
  • Data collection and filtering: Sampling weights classify image-understanding tasks as multiple-choice, long-text, or short-text questions according to question type and response length.
  • Data collection and filtering: Cluster-based sampling uses CLIP image encodings, first-frame video features, KNN clustering with 100 centers, and random sampling from each cluster.
  • Response generation: Responses are generated by leading open- and closed-source models, including Qwen2-VL-72B, LLaVA-OV-72B, GPT-4o, Claude 3.5-Sonnet, and LLaVA-Video-72B.
  • Human annotation: Human annotation evaluates Helpfulness, Faithfulness, and Ethical Considerations, while rankings, justifications, and tie-handling strategies improve comparison quality.
  • Human annotation: More than 50 annotators and 8 multimodal research experts complete quality checks, interactive reviews, and re-annotation of low-quality samples.

3 MM-RLHF-Reward Model

The paper develops a critique-based reward model that combines human-aligned critique generation with response scoring, addressing the limited interpretability and feedback utilization of scalar reward models. It evaluates the resulting reward signals with a dedicated benchmark and positions the approach as a way to improve reward-model supervision for MLLM alignment.

  • 3.1 Background and Limitations of Standard Reward Models: Standard reward models use scalar outputs that underuse detailed human feedback and provide limited transparency into how rewards are generated.These limitations motivate using critiques as intermediate reasoning steps.
  • 3.2 Critique-Based Reward Model Training: The Critique-Based Reward Model first generates critiques for preferred and less-preferred responses, then assigns scalar rewards using those critiques.Its critique head handles critique generation, while its scoring head evaluates responses based on the generated critiques.
  • 3.2 Critique-Based Reward Model Training: Human-provided scoring reasons are augmented with GPT-4o-generated detail and fluency before training the reward model’s language head.The augmentation is constrained to reduce hallucinated or irrelevant analysis in the enhanced critiques.
  • 3.2 Critique-Based Reward Model Training: Teacher forcing trains the scoring head with ground-truth critiques, avoiding noise from model-generated critiques during early training.The scoring head computes rewards from the query, response, and critique.
  • 3.2 Critique-Based Reward Model Training: The joint objective combines critique-generation loss and scoring loss, while inference generates a critique before computing the final reward score.This two-step inference process explicitly reasons about critiques before scoring.
  • 3.3 Discussion: The MM-RLHF-RewardBench evaluates reward-model signals on 170 comparison pairs derived from human-ranked responses.The benchmark is sampled from the MM-RLHF dataset and includes multiple responses with corresponding rankings.

4 MM-DPO

MM-DPO extends DPO with Dynamic Reward Scaling, using reward margins to adjust update strength and prioritize informative comparison pairs. It also incorporates all valid ranked pairs to capture nuanced preference information while bounding scaling to support stable optimization.

  • Dynamic Reward Scaling: Dynamic Reward Scaling adjusts update strength according to each comparison pair’s reward margin or confidence.The mechanism is designed to give stronger influence to high-confidence samples while reducing the impact of noisy or low-confidence data.
  • Outcomes: MM-DPO’s reward-scaling mechanism is reported to improve optimization stability, robustness, efficiency, and overall effectiveness.The framework uses reward-signal quality and confidence-dependent optimization steps to support these aims.
  • Training Pair Selection: MM-DPO uses all comparison pairs with differing ranks for a query instead of training only on the hardest pairs.For queries with multiple responses, every pair with different rankings is treated as a valid comparison.
  • Dynamic Reward Scaling: Dynamic scaling bounds β(δ) between βori and (1 + w)βori to prevent overly aggressive updates.The bounded function 1 −e^(-kδ) controls sensitivity to reward-margin changes; larger k produces faster growth toward the maximum.
  • Dynamic Reward Scaling: Larger reward margins receive higher training weights, giving more informative comparison pairs stronger influence on model updates.The reward margin is computed from the positive and negative response scores produced by MM-RLHF-Reward-7B.

5 Experiments

Experiments show that MM-RLHF alignment improves MLLM performance across diverse capabilities, while reward modeling and optimization analyses reveal important strengths and boundaries.

  • Alignment training consistently improves performance across nearly all benchmarks, with reward signals in MM-DPO further amplifying gains.
  • The MM-RLHF-Reward-7B model achieves state-of-the-art reward-model performance among open-source models, surpassing several 72B models.
  • Conversational benchmarks improve by over 10%, unsafe behaviors decline by at least 50%, and WildVision win rates increase by at least 50%.
  • Alignment also improves hallucination, mathematical reasoning, multi-image, and video-understanding performance, despite no dedicated multi-image data.
  • Performance varies by model and task, with model-specific hyperparameter preferences and weaker relative OCR gains for InternVL-1B than LLaVA-OV models.
  • High-resolution gains remain limited because the dataset contains few ultra-high-resolution images and filtering does not explicitly optimize resolution.

6 Conclusion and Future Work

The paper concludes that MM-RLHF enables broad, consistent MLLM improvements across evaluation benchmarks. Future work will better exploit annotation granularity, add high-resolution data, and scale dataset construction.

  • MM-RLHF and its alignment approach produce significant, consistent gains across almost all evaluation benchmarks and diverse performance dimensions.
  • Future work will exploit per-dimension scores and ranking rationales, integrate high-resolution data, and scale the dataset through semi-automated strategies.

MM-RLHF

MLLMs have advanced substantially, but alignment research remains fragmented and most state-of-the-art models lack rigorous human-preference alignment. Existing alignment efforts and evaluation resources motivate broader, unified assessment and annotation procedures.

  • MLLM Alignment: MLLM alignment remains fragmented across image, video, and specialized domains, with limited evidence of improved general capabilities.Existing work has reduced hallucinations and improved conversational capabilities, but has not produced notable gains in general capabilities.
  • MLLM Alignment: Existing manually annotated MLLM alignment datasets generally contain fewer than 10K samples, limiting unified multimodal alignment research.
  • MLLM Evaluation: MLLM evaluation benchmarks span perception, cognition, instruction-following, and multiple ability dimensions using manually designed or large-scale question sets.Examples include MME’s 14 perception and cognition tasks, MMBench’s 20 ability dimensions, and Seed-Bench’s larger question collections.
  • Annotation: The annotation interface combines image or video display, questions, model outputs, scoring criteria, and reviewer verification of score accuracy.

B.1 I. Visual Faithfulness Evaluation

Visual Faithfulness evaluates whether a generated response accurately reflects the image’s objects, relationships, and attributes. The rubric distinguishes degrees of accuracy from severe inaccuracy to high accuracy.

  • Visual Faithfulness: Visual Faithfulness measures consistency between the generated response and the image’s true objects, relationships, and attributes.
  • Visual Faithfulness: Object Description Accuracy checks whether responses correctly describe existing objects without introducing nonexistent ones.
  • Visual Faithfulness: Object Relationship Accuracy evaluates the correctness of spatial, structural, or functional relationships described between objects.
  • Visual Faithfulness: Object Attribute Accuracy verifies that described object attributes match the true answer.
  • Scoring: The rubric ranges from severely inaccurate to highly accurate, reflecting the extent of errors in descriptions, relationships, and attributes.

B.2 II. Helpfulness Evaluation

Helpfulness assesses whether responses are relevant, instruction-following, and useful for understanding the image or task. Ethical evaluation separately covers safety, privacy, fairness, and harmful content.

  • Helpfulness: Helpfulness evaluates whether a response provides relevant insights and valuable information that improve the user’s understanding of the task.
  • Helpfulness: Helpfulness also checks compliance with the user’s specific instructions and the response’s overall contribution to the user experience.
  • Helpfulness: Helpfulness is scored from Not Helpful through Very Helpful according to the response’s relevance, usefulness, and insight.
  • Ethical Considerations: Ethical Considerations evaluate safety, privacy protection, fairness, and avoidance of harmful content.
  • Ethical Considerations: The ethical rubric ranges from highly unsafe or harmful to highly ethical, safe, privacy protected, fair, and non-harmful.
  • Annotation Requirements: Annotators read the prompt and model response, record reasons for scores, consider all criteria, and manually rank the responses.
  • Annotation Requirements: Ties are recorded when users perceive no significant difference between model outputs, with a negative example required when a tie occurs.

C Safety and Trustworth Dataset and Benchmark Construction

The safety and trustworthiness resources combine human-reviewed multimodal safety data with adversarial examples and tasks spanning attacks, risk identification, jailbreaks, and harmful-content robustness. Human annotation is emphasized for difficult or confusing cases that model annotators may mishandle.

  • Dataset Construction: The self-constructed safety data contains 850 safety samples and 500 adversarial samples sourced from established datasets and AnyAttack-generated examples.
  • Dataset Construction: Safety samples cover jailbreak, privacy and discrimination, hacking, violence, and self-injury categories.The category counts are 200, 200, 150, 200, and 100 samples, respectively.
  • Adversarial Data: Adversarial examples pair clean images with generated adversarial images using ϵ = 8/255, followed by manual verification that LLaVA-OV-7B produces hallucinated outputs.
  • Query Construction: Safety and adversarial queries are generated from VLGuard prompts or GPT-4o variations of an image-description question to maintain prompt diversity.
  • Benchmark Construction: The benchmark selects nine Multitrust tasks and two VLGuard tasks covering adversarial evaluation, risk identification, jailbreaks, cross-modal attacks, and NSFW robustness.
  • Human Annotation: Human annotation is more accurate and adaptable for confusing or incomplete questions that models cannot reliably annotate.
  • Benchmark Evaluation: MM-RLHF-SafetyBench summarizes task data by attack type and uses attack success rates or harmful-output rejection rates as evaluation metrics.

D.2 Difficult-to-Distinguish Answers

Model-based annotation becomes unreliable when questions are confusing, incomplete, beyond model capabilities, or contain subtle response differences. Human annotators can identify flawed premises and fine-grained errors that automated ranking misses.

  • Question-quality failures: Some complex reasoning questions exceed every model’s capabilities, making model-based ranking unreliable.When no model produces a correct response, ranking their outputs introduces further annotation problems.
  • Question-quality failures: Confusing or incomplete questions can cause models to select preferred choices or generate answers despite flawed premises.Human annotators can reject all model-generated answers and provide a more accurate response.
  • Annotation support: Human-comment augmentation is instructed to expand the comment while retaining its strengths and weaknesses without answering the question.The prompt format supplies the question, answer, and human comment as separate fields.
  • Question-quality failures: Models may answer questions whose required information is absent from the image, whereas human annotators recognize the limitation and give the correct answer.This difficulty appears in the high-resolution perception task illustrated in Figure 10.
  • Response-comparison failures: Long, specialized responses can differ only in subtle sections, requiring substantial effort even from multimodal annotators.Annotators take an average of 6 minutes to assess a single long-response question accurately.

E Comparison to Existing Methods on Beta Adjustment in LLMs and MLLMs

The paper examines dynamic beta adjustment for multimodal alignment, where methods developed for LLMs cannot be directly transferred because MLLM data are more complex. It argues that high-quality external rewards make instance-level adjustment effective and improve robustness and optimization stability.

  • Motivation and distinction: Dynamic beta adjustment has been relatively unexplored in large multimodal language models compared with its use in LLMs.Existing LLM studies commonly address noisy preference samples through implicit rewards during DPO training.
  • Motivation and distinction: Existing dynamic beta methods for LLMs cannot be directly adapted to MLLMs because multimodal data are more complex.The paper identifies this as its first exploration of dynamic beta adjustment in MLLMs.
  • Reward-guided adjustment: Instance-level beta adjustment can improve performance when guided by a high-quality external reward model.This finding contrasts with the usual view that instance-level adjustments are unstable and suboptimal.
  • Reward-guided adjustment: Dynamic beta scaling based on reward margins improves robustness and optimization stability, particularly with noisy data.The paper reports notable improvements over existing methods in handling noisy data and algorithmic performance.

F More Ablation and Analysis

Ablation analyses show that MM-RLHF improves the baseline across benchmarks, especially in OCR and conversation tasks, while the proposed dynamic scaling remains robust across hyperparameter choices.

  • Dataset ablation: MM-RLHF produces general improvement across benchmarks, with particularly significant gains in OCR and conversation tasks.The comparison uses the baseline model with and without MM-RLHF.
  • Algorithm ablation: Implicit Reward was tested to adjust each sample’s beta during training, following methods from the LLM domain.The experiment was motivated by differing quality across preference samples.
  • Hyperparameter analysis: Hyperparameter k controls reward-margin mapping, while w controls the strength of beta correction.The analysis evaluates combinations of k and w using the same benchmarks as the preceding comparison.
  • Hyperparameter analysis: The method shows robustness across different selections of k and w.Figure 12(b) reports their effects on final average performance.
Loading 2502.10391v1…