PaperlayerView exact revision on arXiv

Source-linked AI summary

ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling

Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou, Qixing Huang, Li Erran Li

arXiv:2402.06118v4cs.CVcs.AI

TL;DR

LVLM descriptions can hallucinate, omit scene content, and misstate object attributes or relationships, limiting their visual grounding. ViGoR uses fine-grained human and automated reward modeling to select and refine descriptions for fine-tuning. It improves LLaVA’s accurate and relevant image-conditioned text generation while retaining creative reasoning, and the authors report gains across grounding evaluations.

  • Problem

    LVLM-generated text can hallucinate objects, omit important scene content, and infer incorrect attributes or relationships.

    Source: §1 · p. 2

  • Method

    ViGoR combines sentence-level human preference rewards with automated visual-perception rewards to select and refine descriptions for LVLM fine-tuning.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    ViGoR significantly improves LLaVA’s accurate and relevant image-conditioned text generation while preserving creative scene reasoning; on MMHal-Bench it gains +0.3 over LLaVA-7B.

    Source: Detailed Description Generation… · p. 11, §5 · p. 14

  • Takeaways & Limitations

    The approach improves visual grounding with limited human evaluation data and automated scoring, and the released annotation set contains 15,440 image-text pairs.

    Source: §1 · p. 3

    The automated reward component is limited to detector-suitable objects, and training preference data may need to be recollected for LVLMs with substantially different outputs or failure modes.

    Source: §5 · p. 14

Abstract

from arXiv · show

By combining natural language understanding, generation capabilities, and breadth of knowledge of large language models with image perception, recent large vision language models (LVLMs) have shown unprecedented visual reasoning capabilities. However, the generated text often suffers from inaccurate grounding in the visual input, resulting in errors such as hallucination of nonexistent scene elements, missing significant parts of the scene, and inferring incorrect attributes of and relationships between objects. To address these issues, we introduce a novel framework, ViGoR (Visual Grounding Through Fine-Grained Reward Modeling) that utilizes fine-grained reward modeling to significantly enhance the visual grounding of LVLMs over pre-trained baselines. This improvement is efficiently achieved using much cheaper human evaluations instead of full supervisions, as well as automated methods. We show the effectiveness of our approach through a variety of evaluation methods and benchmarks. Additionally, we released our human annotation (https://github.com/amazon-science/vigor) comprising 15,440 images and generated text pairs with fine-grained evaluations to contribute to related research in the community.

1 Introduction

LVLMs can generate detailed descriptions but still hallucinate, omit scene content, or misstate attributes and relationships. ViGoR combines fine-grained human feedback and automated rewards to improve grounding with limited annotation effort.

1 Introduction

  • LVLM training data is limited, and synthetic descriptions can include hallucinations, omissions, and inaccurate attributes or relations.The cited examples contrast BLIP-2’s coarse descriptions with more detailed but sometimes inaccurate LLaVA outputs.Source: §1 · p. 2, Figure 1 · p. 2
  • The method uses sentence-level human evaluations and reward-model predictions to fine-tune an LVLM with 16K samples.The resulting dataset contains image-text-evaluation trios.Source: §1 · p. 3
  • Automated reward methods improve LVLM visual grounding without additional human effort.They are presented as a low-cost complement to human feedback.Source: §1 · p. 3
  • The released human evaluation dataset contains 15.4K image-text pairs with fine-grained assessments.The contribution list describes these as human evaluation data.Source: §1 · p. 3

2 Related Work

Prior LVLM research emphasizes architecture and instruction tuning, while visual perception models offer grounding capabilities that are not fully realized in LVLMs. ViGoR applies fine-grained reward modeling and rejection sampling to address this gap.

2 Related Work

  • LVLMs have largely focused on aligning image features and instruction tuning, whereas ViGoR proposes a general framework for improving visual grounding.The related-work discussion contrasts ViGoR with architectural and data-centered approaches.Source: §2 · p. 4
  • GroundingDINO and related open-set perception models can detect scene elements, motivating their use to score LVLM descriptions.The paper describes GroundingDINO as detecting object presence and counts.Source: §2 · p. 4
  • LVLM reward modeling remains underexplored; unlike LLaVA-RLHF’s sparse reward, ViGoR uses fine-grained rewards with rejection sampling.LLaVA-RLHF compares responses to identify the more hallucinated one.Source: §2 · p. 4
  • Figure 2 depicts two reward sources—perception models and annotator-feedback-trained LVLM scoring—combined to select and refine a caption for fine-tuning.The selected sample is used for supervised fine-tuning.Source: Figure 2 · p. 5

3 ViGoR: Visual Grounding Improvement Framework

ViGoR fine-tunes a base LVLM through rejection sampling, using complementary human and automated fine-grained scores to select and refine training descriptions. Its components cover human preference collection, automated object verification, score aggregation, and supervised fine-tuning.

3 ViGoR: Visual Grounding Improvement Framework

  • ViGoR combines a human-feedback reward model and an open-set detector to rank sampled descriptions, then refines the best candidate for supervised fine-tuning.Its aim is to improve grounding while preserving the base model’s intuitive reasoning and creative thought.Source: §3 · pp. 5–6

3.1 Reward Modeling via Fine-Grained Human Feedback

  • Sentence-level human evaluations label error types and creativity, while also assessing overall detail and missing scene elements.The reward model scores the last sentence with preceding generated text as context.Source: §3.1 · p. 6

3.2 Reward Modeling with Automatic Methods

  • The automated branch extracts caption nouns and asks GroundingDINO to verify their presence, rewarding correct detections and penalizing hallucinated objects.This branch has limited ability to assess attributes and relationships, so human-preference rewards remain necessary.Source: §3.2 · p. 7

3.3 Reward Score and Rejection Sampling

  • The system normalizes negative signals by their variances, combines them linearly, and selects the lowest-penalty sample, using positive scores as tie-breakers.Error signals are the primary selector.Source: §3.3 · p. 7

3.4 Refinement Module

  • The refinement module removes any sentence containing noun phrases judged nonexistent by the reward model.This produces a polished candidate description for use as a regression target.Source: §3.4 · p. 8

3.5 Model Training

  • The refined description becomes the supervised fine-tuning target, and ablations indicate the human and automated signal sources are complementary.Training uses the standard autoregressive objective from the original LLaVA.Source: §3.5 · p. 8

4 Experiments and Results

The experiments evaluate ViGoR against baselines across detailed descriptions, short answers, and programmatic benchmarks, and compare fine-grained with holistic reward modeling. Results favor the complete model, while the later comparison reports an advantage for fine-grained reward modeling.

4.1 Testing Framework

  • The experiments use LLaVA with a frozen ViT-L/14 CLIP image encoder and a 7B Vicuna v1.3 language model, without modifying the architecture.Grounding DINO uses its smallest Swin-T variant and default 0.25 box and text thresholds for open-set detection.Source: §4.1 · p. 8

4.2 LVLM Model Fine-tuning

  • For LVLM fine-tuning, the authors select the highest-reward output from five samples per ADE20K training image, then refine it for supervised fine-tuning.They save refined outputs offline using the initial model state to reduce computational complexity; online rejection sampling is described as a more computationally expensive possibility.Source: §4.2 · p. 9

4.3 Reward Model Training

  • Reward-model training uses 15,440 detailed captions from MS COCO, evaluated by 15 professional annotators, and trains on the resulting sample set for five epochs.The authors chose MS COCO rather than ADE20K for human feedback because ADE20K’s scene complexity created high annotator cognitive load.Source: §4.3 · p. 9

4.4 Quantitative Results

  • The evaluation spans detailed captions, short answers requiring some reasoning, and concise answers to specific questions.For detailed captions, GPT-4Vision ranks candidate responses on multiple criteria rather than assigning an absolute score to a long response.Source: Detailed Description Generation… · p. 10
  • ViGoR-All achieves the best average preference ranking across all detailed-description metrics, while either supervision source outperforms the original LLaVA baseline.GPT-4Vision ranks four candidates on hallucination, counting, attributes, relationships, relevance, reasoning, and detail; lower average preference rank is better.Source: Table 1 · p. 10, Detailed Description Generation… · p. 11
  • ViGoR gains +0.3 on MMHal-Bench over LLaVA-7B, versus +0.1 for LLaVA-RLHF, and improves consistently across POPE and MME categories.The authors report transfer to MMHal-Bench queries and images outside the fine-tuning sources; POPE and MME results improve across evaluated categories.Source: Detailed Description Generation… · p. 11, Table 2 · p. 11, Short Answer Programmatic Benchmark… · pp. 12–13

4.5 Qualitative Results

  • Qualitative examples show fewer hallucinations and invalid observations, more detailed descriptions, and retained plausible intuitive reasoning after fine-tuning.The examples compare ViGoR with the original LLaVA.Source: Figure 3 · p. 13

4.6 Fine-grained vs Holistic Evaluation by Reward Model

  • Fine-grained reward modeling performs better than holistic evaluation, which assigns one positive or negative reward to an entire description.The holistic reward is negative if a description contains at least one erroneous sentence and positive otherwise; the authors attribute the fine-grained advantage to denser, more informative signals.Source: §4.6 · p. 14, Table 5 · p. 14

5 Conclusion, Limitations, and Future Work

The conclusion reports that ViGoR improves LLaVA’s visual grounding while preserving intuitive scene reasoning. It also identifies constraints in the detector-based automatic rewards and the architecture-specific human feedback data, and outlines future work.

Conclusion

  • ViGoR significantly improves LLaVA’s generation of accurate and relevant image-conditioned text while preserving creative and intuitive scene reasoning.The framework combines fine-grained human-preference reward modeling with existing open-set visual perception models.Source: §5 · p. 14

Limitations

  • Automatic rewards rely on GroundingDINO and are limited to detector-suitable objects, while human preference data may need to be recollected for substantially different LVLM outputs.The authors specify that the current automatic component does not apply to stuff regions, attributes, or layouts; collecting additional preference data may incur further cost.Source: §5 · p. 14

Future Work

  • Future work may apply RLHF and explicitly link visual entities with phrases in generated text to further improve grounding.The authors also hope to address the stated limitations.Source: §5 · p. 14

Related papers

Loading 2402.06118v4…