FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback

1 Introduction

• Vision-language misalignment is associated with three hallucination types: nonexistent objects, inaccurate attributes, and inaccurate object relationships.The paper identifies faithful responses as important for practical LVLM applications.Source: §1 · pp. 1–2
• Existing RL alignment methods provide general feedback, sequence-level rewards that can miss segment errors, and human annotations that are costly and labor-intensive.The paper identifies scalability as a constraint when methods require massive accurate feedback.Source: §1 · p. 2
• FGAIF collects AI-based sub-sentence labels for three hallucination types, trains three specialized reward models, and integrates their fine-grained rewards into PPO.The reward models learn from the collected fine-grained reward data.Source: §1 · p. 2
• AI-based labeling detects object existence, attribute, and relation hallucinations at the sub-sentence level without manual annotation.The paper also claims its method provides multiple types and segment-level feedback for LVLM modality alignment.Source: §1 · p. 3
• Experiments on several hallucination benchmarks and one general benchmark demonstrate FGAIF’s effectiveness, while ablations show each module is necessary.The supplied passage gives no benchmark-specific values.Source: §1 · p. 3

2.1 Large Vision-Language Model

• LVLMs combine text encoders, image encoders, and crossmodal alignment modules, and are generally trained in multiple stages for visual comprehension.Examples include aligning image features with pretrained LLM word embeddings before instruction-data fine-tuning.Source: §2.1 · p. 3

2.2 Hallucinations in LVLMs

• LVLM hallucinations are content inconsistent with the input image; prior work measures them and explores decoding, post-processing, datasets, and reinforcement learning.The cited example LLaVA-RLHF uses human-annotated preferences to minimize hallucinations in LLaVA.Source: §2.2 · p. 3, §2.1 · p. 3
• FGAIF differs from prior approaches by learning fine-grained rewards from automatic AI feedback rather than human annotation and coarse-grained feedback.Figure 2 presents the method as feedback collection, reward-model training, and reinforcement learning with fine-grained rewards.Source: §2.2 · p. 3, Figure 2 · p. 4

3 Problem Formulation

• Given image I and prompt P, LVLM M with parameters Θ_M produces sampled response R.The formulation omits image and prompt indices for simplicity.Source: §3 · p. 4
• An AI method identifies existence, attribute, and relation hallucinations in R and trains corresponding reward models to predict labels for each sub-sentence.The labels indicate whether a sub-sentence contains each hallucination type.Source: §3 · p. 4
• The trained reward models and corresponding images and prompts are used to fine-tune M to produce a faithful response.The final optimized parameters are denoted Θ_f.Source: §3 · p. 4

4.1 AI-based Feedback Collection

• ChatGPT extracts existence, attribute, and relation facts from response sub-sentences, and LLaVA 1.5 checks each fact against the image to derive type-specific labels.The atomic-fact labels are aggregated per sub-sentence; segments with no extracted facts receive label 2.Source: §4.1 · p. 5

4.2 Fine-grained Reward Model Training

• Three specialized reward models predict existence, attribute, and relation hallucinations from each sub-sentence’s final-token representation, trained with cross-entropy.The models use the feature for the last token of each sub-sentence and an MLP classifier to predict hallucination labels.Source: §4.2 · pp. 5, 7

4.3 Reinforcement Learning with Fine-grained Reward

• PPO fine-tunes the LVLM using per-sub-sentence rewards from the trained models, assigned at each segment’s final token.The policy model is optimized during PPO training; the reward models are not optimized during this stage.Source: §4.3 · pp. 7–8

5.2 On Model Comparison (RQ1)

• FGAIF consistently outperforms prior methods on most POPE and MMHal-Bench metrics, beats LLaVA-RLHF7B across all metrics, and leads across MMHal-Bench question categories.The reported category split covers object existence, object attribute, and object relation.Source: Table 1 · p. 8, Table 2 · p. 9, §5.2 · p. 10
• On CHAIR and FaithScore, InstructBLIP models achieve the best results, but FGAIF outperforms LLaVA-RLHF7B despite producing longer answers.The authors note these metrics assess faithfulness precision without recall, which may favor shorter answers.Source: §5.2 · pp. 10–11, Table 3 · p. 10
• FGAIF yields greater gains than LLaVA-RLHF7B on LLaVA-Bench’s Conversation, Detail, Complex, and full-set results.LLaVA-RLHF7B scores 94.1 on the full benchmark, compared with 81.0 for LLaVA7B.Source: Table 2 · p. 9, §5.2 · p. 11

5.3 On Ablation Study (RQ2)

• The ablation study finds that removing any specialized reward model or replacing them with one coarse reward model underperforms FGAIF; removing object-existence feedback hurts most.Removing reinforcement learning also performs substantially worse, while attribute- and relation-feedback removals have similar effects.Source: Table 4 · p. 11, §5.3 · p. 12
• Reward-model accuracy is 79.2% on a new test set with 15–20 sub-segments per response, versus 80.4% on the original test set.The authors interpret the comparable scores as indicating minimal impact of sub-segment count on accuracy.Source: §5.3 · p. 12
• On 200 Foggy images containing unseen object types, reward-model accuracy is 76%, below the original test-set result but described as reasonably robust.The training set uses a fixed set of hallucination labels.Source: §5.3 · p. 12

5.4 On Case Study (RQ3)

• In two examples, FGAIF gives a faithful image description where LLaVA13B hallucinates a seagull and a boat.The figure marks generated hallucinations in red.Source: Figure 5 · p. 11, §5.4 · p. 13

5.5 Discussion

• Manual review of atomic-fact labels finds 85.07% accuracy, leaving room to improve the AI-based feedback.The evaluation sampled 90 MSCOCO validation images.Source: §5.5 · p. 13

6 Conclusion

• FGAIF combines AI-based feedback collection, fine-grained reward-model training, and reinforcement learning with fine-grained rewards.The authors report superiority on hallucination and general benchmarks and say ablations show each component is necessary.Source: §6 · p. 13
• The authors plan to incorporate additional reward models, including soundness and fluency, to provide more training feedback.Source: §6 · p. 13

A Detailed Results

• Table 5 reports detailed MMHal-Bench results, while Table 6 reports POPE evaluation results.The supplied passages do not provide the tables’ numerical cell values.Source: A Detailed Results · p. 18, Table 5 · p. 18, Table 6 · p. 18
• FGAIF achieves the best performance across MMHal-Bench object-existence, object-attribute, and object-relation categories.The categories map respectively to adversarial object, object attribute and counting, and spatial relation.Source: A Detailed Results · p. 18, Table 5 · p. 18

B Error Cases

• The ChatGPT example misclassifies a white car as an object-existence rather than object-attribute issue, while LLaVA misjudges an atomic fact’s consistency with the image.Figure 6 presents the two error cases.Source: B Error Cases · p. 18, Figure 6 · p. 19

C PPO with Fine-Grained Rewards

• The PPO procedure takes initial policy and value models, three AI-feedback-trained reward models, prompts, and hyperparameters, then outputs an updated policy.It samples responses, computes rewards and advantages, updates the policy with a clipped surrogate objective, and updates the value model with a squared-error objective.Source: C PPO with Fine-Grained Rewards · p. 19

• ChatPT is configured with one beam and temperature 0 to avoid randomness.Source: D Experimental Setups · p. 19
• The feedback model is LLaVA 1.5 with 13B parameters.Source: D Experimental Setups · p. 19
• The ChatPT version is gpt-3.5-turbo-0125.Source: D Experimental Setups · p. 19

E Abliation on Hallucination Labeler

• 89.47% accuracy was achieved by InternVL 2.5 (26B) on the manually annotated hallucination-labeling set.Source: E Abliation on Hallucination Labeler · p. 20
• Using InternVL 2.5 labels led to better performance across multiple benchmarks than the prior LLaVA-based labels.The comparison is reported in Table 7.Source: E Abliation on Hallucination Labeler · p. 20, Table 7 · p. 20

F Details of Sentence Splitter

• The splitter first uses spaCy for coarse sentences, then starts new sub-sentences after commas, semicolons, exclamation marks, or question marks.Source: F Details of Sentence Splitter · p. 20

• Despite imperfect feedback accuracy, noisy-label training reached 91.2 FaithScore, and RL showed smooth reward curves with no mode collapse.The authors characterize the framework as stable and effective with moderately noisy reward signals.Source: G Detailed Discuss · p. 20
• FaithScore rose 91.2→91.5, FaithScore_S 74.7→75.8, POPE F1 83.4→84.3, CHAIR_I 3.9→3.3, and CHAIR_S 6.2→5.9 with InternVL-collected feedback.The feedback accuracies were 85.07% for LLaVA-13B and 89.47% for InternVL 2.5.Source: G Detailed Discuss · p. 20
• Figure 7 compares FGAIF and LLaVA13B responses on two test samples, marking generated hallucinations in red.Source: Figure 7 · p. 21
• GPT-3.5 and GPT-4o yielded 98% overlap in atomic facts across 10 sampled points, while GPT-4o achieved POPE F1 of 83.5.Source: G Detailed Discuss · p. 21
• The three reward models can run in parallel, so the authors report no large additional computational overhead beyond the standard framework.The pipeline has reward-model training and RL stages, like the other RLHF method discussed.Source: G Detailed Discuss · p. 21

• LLaVA-13B hallucinates an absent chair and bowl in two examples, while FGAIF suppresses these co-occurrence-driven errors.Both models correctly identify the couch in the first example and the spoon in the second.Source: H More Qualitative Analysis · p. 21