CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

1 INTRODUCTION

• MLLM hallucinations remain a significant challenge because outputs may not be based on visual input.Source: §1 · p. 1
• Multimodal DPO struggles to align image and description representations and to distinguish hallucinated from non-hallucinated descriptions.The paper analyzes these limitations through image and text representation distributions.Source: §1 · pp. 2–3
• 52.7% and 55.5% relative-point improvements over DPO in hallucination reduction on Object HalBench for Muffin and LLaVA-1.6, respectively.Source: §1 · p. 2
• CHiP combines visual preference optimization with textual preferences at response, segment, and token levels.Visual preference pairs let the model learn from both modalities; the hierarchical module captures preferences at multiple textual granularities.Source: §1 · p. 2
• The paper evaluates CHiP on four hallucination benchmarks under LLaVA-1.6 and Muffin frameworks.The introduction reports that CHiP outperforms GPT-4V on the evaluated benchmarks.Source: §1 · p. 2

2 RELATED WORK

• MLLMs process images and text for multimodal understanding and reasoning, building on open-source LLMs and image encoders.Source: §2 · p. 3
• Hallucination mitigation approaches include training-free output postprocessing and training-based instruction fine-tuning or preference learning.Examples include REVERIE for reflective instruction tuning and FDPO for fine-grained preference learning.Source: §2 · p. 3
• CHiP addresses visual-language preference misalignment by simultaneously optimizing image- and text-modality preferences from a fine-grained perspective.Source: §2 · p. 3

3 PRELIMINARIES

• DPO aligns language models with human preferences without explicit reward modeling or reinforcement learning.Source: §3 · p. 3
• The reward function is derived by maximizing a KL-constrained reward objective and mapping the reward model to the optimal policy.The passage defines x as the instruction, y as the response, β as a constant, and Z(x) as the partition function.Source: §3 · p. 3
• Given a preferred response yw and rejected response yl, DPO learns to maximize their reward difference using whole-response rankings.Source: §3 · pp. 3–4
• The action score uses the log probability of a response token conditioned on the instruction and preceding tokens; the reference model stays fixed while the policy is updated.yi denotes the i-th token of response y.Source: §3 · p. 4

4.1 HIERARCHICAL TEXTUAL PREFERENCE OPTIMIZATION

• Hierarchical textual preference optimization addresses the limits of response-only rankings by assigning preference information at response, segment, and token levels.Response-level optimization otherwise gives one reward to an entire response, making it difficult to identify hallucinated segments or tokens and assign credit precisely.Source: §4.1 · pp. 4–5
• Segment-level scoring gives higher rewards to segments that differ between chosen and rejected responses, with normalization intended to avoid favoring longer responses.The paper identifies corrected segments, particularly entity nouns, as important for eliminating hallucinations.Source: §4.1 · p. 4

4.2 VISUAL PREFERENCE OPTIMIZATION

• Visual Preference Optimization compares images paired with the same chosen response, using a preferred image that better matches that response than the rejected image.The rejected image may be generated by rotating, cropping, or adding noise to the chosen image.Source: §4.2 · p. 5
• CHiP combines hierarchical textual and visual preference objectives, fully weighting response-level and visual preferences while partially weighting segment- and token-level preferences.Source: §4.2 · p. 5, §4.1 · p. 5

5.1 EXPERIMENTAL SETTINGS

• CHiP uses Muffin (13B) and LLaVA-1.6 (7B), trained on 5k RLHF-V preference samples and compared primarily with same-model DPO.Evaluation covers ObjHal, MMHal, HallusionBench, and AMBER, with their listed hallucination, quality, accuracy, and coverage metrics; training runs for three epochs.Source: §5.1 · pp. 5–6

5.2 RESULTS AND OBSERVATIONS

• Across four benchmarks, CHiP consistently outperforms DPO; versus base Muffin, ObjHal response- and mention-level hallucinations fall 71.2% and 66.4%.The corresponding LLaVA reductions are 65.3% and 56.7% relative points. On AMBER, Cog falls 57.1% (Muffin) and 61.9% (LLaVA), while Hal falls 45.2% and 49.6%.Source: §5.2 · p. 6, Table 1 · p. 7

5.3 ABLATION STUDY

• On ObjHal and MMHal, both hierarchical textual preference optimization and visual preference optimization outperform DPO; combining them reduces ObjHal response- and mention-level rates by 49.6% and 41.3%.The component analysis attributes reduced response-level label ambiguity to finer-grained preference signals and improved image-text alignment to visual preference optimization.Source: §5.3 · p. 7
• In expert MMHal evaluation, CHiP and DPO tie on 63.5% of samples, while CHiP wins 24% overall.Among the 36.5% of samples where a distinction was possible, CHiP outperformed DPO in 31.6%.Source: §5.3 · p. 7, Figure 3 · p. 7
• The best segment-level weight is λ = 1 for Muffin and λ = 3 for LLaVA, and these settings are used in all experiments.The figure caption identifies these settings as best on AMBER for CHAIR and Hallucination Rate.Source: §5.3 · p. 8, Figure 4 · p. 8
• Training the visual encoder lowers DPO’s hallucination rate, but does not produce the expected reduction for CHiP.The authors suggest that CHiP’s multiple optimization objectives may dilute focus on image-text semantic alignment during joint visual-encoder training.Source: §5.3 · p. 8, Table 3 · p. 8

6.1 GENERAL CAPABILITY ANALYSIS

• LLaVA+CHiP outperforms LLaVA on five of six general-capability datasets, with slight gains on MMMU, LLaVA-Wild, and MMB-CN and comparable performance on others.Table 4 reports general-capability evaluations; its caption notes the evaluator used for LLaVA-Wild and missing official LLaVA-1.6 results for MMMU-test.Source: §6.1 · p. 9, Table 4 · p. 9

6.2 IMPACT OF REJECTION IMAGE CONSTRUCTION STRATEGY

• CHiP performs best with diffusion-based rejection images; retaining chosen-image information benefits preference optimization, while rotation, blackness, and randomness perform poorly.Diffusion and cropping retain substantial visual information; rotation can preserve information but produces large differences in visual tokens after tokenization.Source: Table 5 · p. 9, §6.2 · pp. 9–10
• T=500 yields the best ObjHal performance, while fewer or more noise steps can respectively make preference labels ambiguous or erase useful image information.Figure 6 reports response- and mention-level hallucination rates across noise-step values.Source: Figure 6 · p. 9, §6.2 · p. 10

6.3 REPRESENTATION VISUALIZATION

• CHiP brings image and ground-truth description representations closer than CMDPO while retaining separation between hallucinated and non-hallucinated text.The visualization uses last-token representations from 150 COCO-2017 image-description pairs, with GPT-4-generated descriptions and PCA.Source: §6.3 · p. 10, Figure 7 · p. 10

7 CONCLUSION

• CHiP combines visual and hierarchical textual preference optimization, and experiments on four datasets show reduced hallucinations and improved image–non-hallucinatory-description alignment and text discrimination.The conclusion compares these representation findings with standard multimodal DPO.Source: §7 · pp. 10–11

A DIFFERENCES BETWEEN TOKEN- AND RESPONSE-LEVEL OPTIMIZATION

• Response-level optimization scores an entire response, whereas token-level optimization accumulates token-wise rewards and sequential KL distances across the response.The appendix introduces these differences before giving their respective derivations.Source: A DIFFERENCES BETWEEN TOKEN- AND… · p. 16, §A.1 · p. 16

A.2 RESPONSE-LEVEL PREFERENCE OPTIMIZATION

• Response-level DPO uses a preferred-versus-rejected response ranking to maximize the reward difference, with the reference model usually fixed during training.The response-level reward is derived from policy and reference generation probabilities; x is the instruction, m the image, y the response, and β a constant.Source: §A.2 · p. 17

A.3 TOKEN-LEVEL PREFERENCE OPTIMIZATION

• Token-level optimization conditions each next-token prediction on the instruction, image, and preceding tokens, then aggregates rewards over tokens.Its sequential KL term compares policy and reference distributions; the appendix also describes token-level optimization as avoiding the policy cancellation possible for sentence-level preference pairs.Source: §A.3 · pp. 17–18
• The token-level objective uses sequential KL differences between chosen and rejected responses, with a stop-gradient operator in the stated LPOk term.The appendix distinguishes this token-level KL optimization term from reward differences used in response- and segment-level optimization.Source: §A.3 · p. 18

B.1 DOES CHIP MAKE THE MODEL LESS TALKATIVE?

• Across AMBER, MMHal, and ObjHal, LLaVA, LLaVA+DPO, and LLaVA+CHiP have comparable output lengths, while CHiP lowers hallucination rates.The authors omit HallusionBench because it is a multiple-choice task where response length is not meaningful.Source: §B.1 · p. 18, Table 6 · p. 19
• When images contain ambiguous or uncertain objects or attributes, CHiP tends to omit them, which the authors report effectively reduces hallucinations.This observation comes from manual analysis of LLaVA+CHiP responses.Source: §B.1 · p. 18
• On LLaVA-7B, CHiP's fA is lower than DPO's but higher than the original model's; the authors suggest its multiple optimization objectives may dilute focus on logical consistency.fA evaluates logical consistency, including whether answers are based on random guesses.Source: §B.1 · p. 19

B.2 ANNOTATOR BACKGROUND IN HUMAN EVALUATION

• Three experts specializing in multimodal hallucination manually evaluated MMHal; their labeling prioritized object hallucinations as more severe than attribute and environment hallucinations.The manual evaluation was introduced to mitigate GPT-evaluation errors associated with incomplete MMHal annotations.Source: §B.2 · p. 19

B.3 EFFECT OF TOKEN-LEVEL PREFERENCE OPTIMIZATION

• Across four datasets, token-level preference optimization alone yields lower hallucination rates than both DPO and original LLaVA.Table 7 compares LLaVA after token-level preference optimization (TDPO) with DPO.Source: §B.3 · p. 19, Table 7 · p. 19

B.4 FINE-GRAINED ANALYSIS

• The fine-grained evaluation compares Muffin, LLaVA, DPO, and CHiP on AMBER, MMHal-Bench, and Object HalBench.Source: §B.4 · p. 19
• CHiP achieves the best AMBER Score and MMHal-Bench Overall score and Hallucination rate under both Muffin and LLaVA, and leads on Object HalBench metrics except Object Recall.The evaluation compares the base models, DPO, and CHiP across AMBER, MMHal-Bench, and Object HalBench.Source: §B.4 · p. 19, Table 9 · p. 20, Table 8 · p. 20

B.5 PROMPT

• For 150 COCO-2017 validation images, Gemini generated detailed hallucinated and non-hallucinated descriptions, with the latter manually verified.Table 11 presents the prompts used to guide description generation.Source: §B.5 · p. 21, Table 11 · p. 21
• Figure 8 compares LLaVA+CHiP with LLaVA+DPO and LLaVA on MMHal-Bench, highlighting correct answers in green and hallucinations in red.Source: Figure 8 · p. 22

• LLaVA+CHiP demonstrates a significant reduction in hallucinations on MMHal-Bench, attributed to multi-level preference optimization and improved image-text semantic alignment.Source: C QUALITATIVE RESULT · p. 22