PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrated Self-Rewarding Vision Language Models

Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, Huaxiu Yao

arXiv:2405.14622v4cs.LGcs.CLcs.CV

TL;DR

LVLMs may generate plausible text that contradicts images, while preference data from external models or human annotations may not reflect the target model’s preferences. CSR iteratively generates and rewards candidate responses using sentence-level image relevance, then curates preference data for fine-tuning. The authors report improved performance and reduced hallucinations across ten benchmarks and tasks, with up to a 7.62% average improvement.

  • Problem

    LVLMs can generate linguistically plausible responses that contradict images, and externally curated preference data may not reflect the target model’s preferences.

    Source: §1 · pp. 1–2

  • Method

    CSR iteratively generates sentence-level candidate responses, calibrates their rewards with image-response relevance, and uses preference pairs for fine-tuning.

    Source: §3.1 · p. 4, §3.2.1 · p. 4

  • Results

    CSR improves performance and reduces hallucinations across ten benchmarks and tasks, with up to a 7.62% average improvement over competitive approaches.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CSR’s reported gains include compatibility with different LVLM backbones and continued performance improvements over iterations.

    Source: §1 · p. 2, §7 · p. 10

    The theoretical analysis assumes infinite data and noise vectors with well-conditioned covariance matrices and sub-Gaussian norms bounded by a universal constant.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) have made substantial progress by integrating pre-trained large language models (LLMs) and vision models through instruction tuning. Despite these advancements, LVLMs often exhibit the hallucination phenomenon, where generated text responses appear linguistically plausible but contradict the input image, indicating a misalignment between image and text pairs. This misalignment arises because the model tends to prioritize textual information over visual input, even when both the language model and visual representations are of high quality. Existing methods leverage additional models or human annotations to curate preference data and enhance modality alignment through preference optimization. These approaches may not effectively reflect the target LVLM's preferences, making the curated preferences easily distinguishable. Our work addresses these challenges by proposing the Calibrated Self-Rewarding (CSR) approach, which enables the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. In the reward modeling, we employ a step-wise strategy and incorporate visual constraints into the self-rewarding process to place greater emphasis on visual input. Empirical results demonstrate that CSR enhances performance and reduces hallucinations across ten benchmarks and tasks, achieving substantial improvements over existing methods by 7.62%. Our empirical results are further supported by rigorous theoretical analysis, under mild assumptions, verifying the effectiveness of introducing visual constraints into the self-rewarding paradigm. Additionally, CSR shows compatibility with different vision-language models and the ability to incrementally improve performance through iterative fine-tuning. Our data and code are available at https://github.com/YiyangZhou/CSR.

1 Introduction

LVLMs can produce plausible descriptions that contradict images, a problem linked to models favoring textual knowledge over visual input. CSR addresses this alignment challenge through iterative self-rewarding with visual constraints.

1 Introduction

  • LVLM hallucinations can persist even with capable language and vision models, and are linked to image-text misalignment that can lead models to prioritize training-text knowledge over images.Source: §1 · p. 1
  • Preference fine-tuning approaches may require additional models or human annotations, whose preferences may not reflect the target LVLM and can be easy for it to distinguish.Source: §1 · p. 2
  • CSR iteratively generates sentence-level candidates, calibrates their language-decoder rewards with image-response relevance, and uses the highest- and lowest-reward responses for preference fine-tuning.Source: §1 · p. 2
  • 7.62% average improvement is reported across comprehensive LVLM benchmarks and VQA tasks, alongside reduced hallucination.The paper also reports stronger modality alignment compared with competitive approaches.Source: §1 · p. 2

2 Preliminaries

The preliminaries introduce LVLM inputs and outputs and describe preference optimization as learning from preferred and dispreferred responses. DPO is presented as a representative optimization method.

2 Preliminaries

  • Figure 1 is labeled as a benchmark performance comparison.Source: Figure 1 · p. 2
  • An LVLM takes an image-text pair as input and generates a text response.Source: §2 · p. 2
  • Preference optimization uses data pairing prompts with preferred and dispreferred responses to optimize language models.Source: §2 · p. 2
  • DPO models the probability of a preference pair with a sigmoid of the reward difference and optimizes using a classification loss.Source: §2 · p. 3

3 Calibrated Self-Rewarding Vision Language Models

CSR combines sentence-level candidate generation and visual-relevance-calibrated rewards to create preference pairs for iterative fine-tuning. Its reward is assigned step by step, and preference learning uses each iteration’s data.

3 Calibrated Self-Rewarding Vision Language Models

  • CSR alternates candidate-response generation with preference curation and fine-tuning, using image-calibrated sentence rewards to select responses.Source: §3 · p. 3

3.1 Step-Level Reward Modeling and Calibration

  • Step-wise calibrated reward modeling combines the LVLM’s instruction-following score with image-response relevance for each generated sentence.The image-response relevance score uses CLIP-score, and a hyperparameter balances it with the instruction-following score.Source: §3.1 · p. 4

3.2 Iterative Fine-Tuning

  • CSR iteratively generates candidates and curates preference data for optimization.Source: §3.2 · p. 4

3.2.1 Step-Level Candidate Response Generation

  • Sentence-level beam search selects top-k and bottom-k sentences by calibrated reward at each step, then sums sentence rewards to score each completed response.Source: §3.2.1 · p. 4

3.2.2 Preference Curation and Optimization

  • For each prompt, CSR selects the responses with the highest and lowest cumulative calibrated rewards as preferred and dispreferred, then fine-tunes with DPO using the previous iteration’s model as reference.Source: §3.2.2 · p. 5

4 Experiment

The experiments evaluate CSR across LVLM benchmarks, preference-tuning baselines, and iterative updates, then analyze reward components, modality alignment, and case examples. Results report improved benchmark performance, stronger visual attention, and fewer hallucinations across iterations.

4.1 Experimental Setups

  • Experiments used LLaVA-1.5 7B and 13B, three CSR iterations, and about 13,000 sampled LLaVA150k examples across comprehensive, VQA, and hallucination benchmarks.Baselines included self-rewarding, data-driven preference methods, and other open-source LVLMs.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 Results

  • Approximately 7.62% for LLaVA-1.5 7B and 5.25% for 13B: average performance improved across benchmarks through online iterative updates.The reported gains gradually shrink, which the authors interpret as convergence.Source: Figure 3 · p. 6, §4.2 · p. 6
  • CSR outperformed existing self-rewarding methods by an average of 2.43% and outperformed other open-source LVLMs on 9 of 10 benchmarks.The cross-model comparison used LLaVA-1.5 after three CSR rounds.Source: §4.2 · pp. 6–7

4.3 Analysis

  • Combining self-generated instruction-following and image-response relevance scores improved performance over variants using either score alone or no CSR.The ablation also reports that incorporating external visual scores through λ enhances performance.Source: §4.3 · p. 7
  • Vila 7B improved by 3.37% overall after three CSR rounds, with increases of 8.48% on VisWiz and 14.0% on MM-Vet.The authors report gradual improvements across all ten evaluated benchmarks and tasks.Source: §4.3 · p. 7, Figure 4 · p. 7
  • Across three iterations and 500 LLaVA-150k examples, chosen and rejected responses both gained image relevance, while their average relevance scores moved closer together.The authors say this makes the self-generated preference data more challenging to distinguish while further strengthening learning.Source: §4.3 · p. 7, Figure 5 · p. 7
  • CSR strengthened attention to selected visual tokens and distributed attention between image and text tokens more effectively than the self-rewarding approach.The attention-map analysis presents the original LLaVA-1.5 7B, self-rewarding, and CSR models.Source: §4.3 · pp. 7–8, Figure 6 · p. 8

4.4 Case Study

  • In LLaVA-1.5 13B case examples, hallucinations noticeably decreased with each CSR iteration, and the model generated preference pairs without human annotation.The sampled responses and preference data are shown in Figures 7 and 9, respectively.Source: §4.4 · p. 8

5 Theoretical Explanation

The theoretical analysis models image and text prompts separately and examines how an image-response relevance score calibrates self-rewarding. Under its stated assumptions, the analysis finds that incorporating this score can improve generation accuracy over self-rewarding without it.

5 Theoretical Explanation

  • The analysis represents image and text prompts as separate continuous-valued random vectors, following CLIP theory literature [38] [39] [40].It assumes orthonormal decoders, identity-covariance latent signals, sub-Gaussian noise, and an infinite-data setting.Source: §5 · pp. 8–9
  • The calibrated reward combines image-response relevance and self-generated instruction-following scores, weighted by λ and 1 − λ, respectively.The image-response score uses image and text embeddings; λ = 1 corresponds to omitting that score.Source: §5 · p. 9
  • When the model prioritizes text over visual input, using the image-response relevance score (λ < 1) yields a better solution than omitting it (λ = 1).The analysis frames the score as increasing attention to image signals during generation.Source: §5 · pp. 8, 10

6 Related Work

Prior work addresses LVLM hallucination through data, decoding, and external-model interventions, while preference optimization tackles modality alignment using human- or model-curated data. CSR instead applies calibrated self-rewarding to LVLMs with visual constraints.

6 Related Work

  • LVLM hallucination mitigation includes fine-tuning data improvements, decoding interventions, and external closed-source models, but these approaches do not directly target modality alignment.The passage cites dataset and fine-tuning work [8] [61] [9], decoding approaches [37] [62] [64] [65] [66], and external-model approaches [58] [67] [68] [69] [70].Source: §6 · p. 10
  • Preference optimization for LVLM alignment uses human-annotated or additional-model-curated data, which can be resource-intensive and introduce biases [8] [30] [9] [10].The passage says these methods may not capture LVLMs’ inherent preferences, making the curated data less effective.Source: §6 · p. 10
  • Unlike LLM self-improvement methods [74] [12], CSR targets LVLM modality misalignment by adding visual constraints to preference modeling.The cited LLM methods select consistent self-generated answers or distinguish model-generated from human-annotated responses.Source: §6 · p. 10

7 Conclusion

The conclusion reports that CSR improves image-text alignment, reduces hallucination, and continues improving LVLM capabilities over iterations. The appendix specifies beam-search and calibrated-reward settings.

7 Conclusion

  • CSR improves image-text alignment, reduces hallucination, and supports continued capability gains over iterations, with better utilization of visual information.The paper reports empirical results across LVLM evaluation benchmarks and says theoretical findings support them.Source: §7 · p. 10

A.1.1 Hyperparameter Settings

  • Beam search uses 5 beams and 5 token beams, a period end marker, a 1,024-token maximum length, and a 74-token new-content limit.Group beam search uses 5 groups and a diversity penalty of 3.0; calibrated rewarding weights the CLIP score at 0.9 and language score at 0.1.Source: §A.1.1 · p. 16

A.2 Evaluation Metrics and Benchmarks

The evaluation section describes benchmarks spanning multimodal perception, reasoning, VQA, and hallucination assessment, alongside comparisons with baseline LVLMs and preference-learning methods.

A.2 Evaluation Metrics and Benchmarks

  • The benchmark suite includes MME, SEED-Bench, LLaVAW, MMBench, MM-Vet, ScienceQA, VizWiz, and GQA for multimodal capabilities and VQA.The passage describes POPE as a yes-or-no object-hallucination assessment and CHAIR as measuring object hallucination in captions; the CHAIR evaluation sampled 500 COCO validation images.Source: §A.2 · pp. 16–17

A.3 Overview of the Baselines

  • Comparisons include LLaVA-1.5, self-rewarding [29], and preference-learning baselines based on annotated or external-model preference data [8] [9] [10] [30].The baselines include LLaVA-RLHF [8], POVID [10], RLHF-V [30], and Silkie [9].Source: §A.3 · pp. 17–18
  • Other open-source LVLM comparisons include InstructBLIP, Qwen-VL-Chat, mPLUG-Owl2, BLIP-2, and IDEFICS.These passages describe models with varied architectures, training setups, and visual-language capabilities.Source: §A.3 · p. 17

A.4 Do Different Sources of Preference Data Have Different Impacts?

The authors compare self-generated dispreferred responses with negatives modified by GPT-4. The modified negatives are less confusable to the target LVLM, partly supporting the concern that external preferences may be easier for it to distinguish.

A.4 Do Different Sources of Preference Data Have Different Impacts?

  • The analysis sampled 500 original LLaVA-1.5 7B examples, selected reward extremes as preferred and dispreferred, then used GPT-4 to transform preferred responses into negatives.The self-reward prompt asks the model to judge whether a response should be chosen or rejected.Source: §A.4 · p. 18, Table 3 · p. 18
  • GPT-4-modified dispreferred responses were less confusable to the target LVLM than its self-generated dispreferred responses, partially supporting their lower effectiveness.The comparison uses the target LVLM’s sampling-probability scores.Source: §A.4 · p. 18

A.5 Additional Experiments

Additional experiments compare CSR with state-of-the-art models, evaluate iterative performance and hallucinations, test another backbone, and examine reward-score relationships and the effect of λ. The reported λ ablation finds improved performance as λ increases.

A.5 Additional Experiments

  • The experiments cover CSR versus open-source state-of-the-art LVLMs, online iteration on comprehensive and hallucination benchmarks, and evaluation with Vila 7B.The corresponding tables describe comparisons with other models and online-iteration results using LLaVA-1.5 and Vila 7B backbones.Source: §A.5 · p. 18, Table 5 · p. 19, Table 6 · p. 19
  • Increasing λ improves performance across various benchmarks in the three-round ablation, suggesting visual-score calibration can enhance preference learning.The ablation uses training settings consistent with Table 1.Source: §A.5 · p. 18, Table 9 · p. 20
  • The appendix also examines reward scores against average performance across CSR iterations and provides a GPT-4 prompt for producing negative responses based on an image.Table 10 covers LLaVA-1.5 7B; Table 4 describes the prompt.Source: §A.5 · p. 18, Table 4 · p. 19, Table 10 · p. 20

B Proofs

The proofs include a result that the global minimizer converges to a point-mass distribution under the stated choice of response. They also show that CSR up-weights the image-input signal.

B Proofs

  • Under the choice y_w = arg max_y R(y), the response distribution is a point mass, and the global minimizer to (5) converges to π*θ_t(y | x).The passage states this conclusion in the proof of Theorem 5.1.Source: B Proofs · p. 20
  • The CSR procedure up-weights the image-input signal, as indicated by its stated transformation.The passage introduces the transformation as part of the CSR procedure.Source: B Proofs · p. 20
  • The proof’s argument invokes independence between image and text inputs and assumes a well-conditioned covariance for ξ1.The passage says this assumption implies Cov(x_t) is well-conditioned.Source: B Proofs · p. 21

C Limitations

The paper reports computational and model-size limits on its experiments, which leave scaling-law behavior undetermined. It also notes that hallucinations remain despite reductions and calls for safety measures in real-world use.

C Limitations

  • CSR experiments were limited to three iterations and 7B and 13B models, leaving whether the method follows a scaling law undetermined.The authors hope to train larger models and continue iterative training with more computing resources.Source: C Limitations · pp. 21–22

D Broader Impacts

  • Although the approach reduces LVLM hallucinations, they still occur, so the authors call for safety and stability measures in real-world applications.The authors also state that the approach requires no additional human annotations.Source: D Broader Impacts · p. 22
Loading 2405.14622v4…