Source-linked AI summary
Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, Cao Xiao
TL;DR
Existing approaches to modality alignment often rely on external models or data, which can make results unstable and costly. SIMA uses LVLM-generated responses, in-context self-criticism, and preference tuning; it improves performance across 14 benchmarks, including a 7.5% gain for LLaVA-1.5-7B.
Problem
Many preference-tuning methods rely on external models or human-labeled data, raising concerns about distribution shifts, unstable optimization, and high costs.
Source: §1 · pp. 1–2Method
Results
SIMA significantly improves performance across 14 hallucination and comprehensive benchmarks, including a 7.5% increase for LLaVA-1.5-7B.
Source: §1 · pp. 2–3Takeaways & Limitations
Avoiding third-party models for data generation and response evaluation makes SIMA more scalable and cost-effective.
Source: §5 · p. 8SIMA depends on current LVLM capabilities and does not address biases in vision instruction tuning data, with less significant gains on some benchmarks such as TextVQA.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large vision-language models (LVLMs) have achieved impressive results in visual question-answering and reasoning tasks through vision instruction tuning on specific datasets. However, there remains significant room for improvement in aligning visual and language modalities. Existing methods often depend on external models or data, leading to uncontrollable and unstable alignment results. In this paper, we propose SIMA, a self-improvement framework that enhances visual and language modality alignment without external dependencies. SIMA leverages existing vision instruction tuning datasets to self-generate responses, incorporating an in-context self-critic mechanism that constructs preference pairs for tuning. Crucially, our approach allows LVLMs to act as critics by designing effective critic prompts, eliminating the need for additional fine-tuning with external instruction data. We introduce three novel visual metrics within the self-critic process to guide judgment, significantly improving the accuracy of self-critic. Through extensive experiments across 14 hallucination and comprehensive benchmarks, we demonstrate that SIMA significantly improves LVLM's performance and outperforms previous approaches, achieving superior modality alignment.
1 Introduction
SIMA addresses the challenge of aligning visual and language modalities while avoiding external models and data used by many preference-tuning approaches. It uses self-generated responses and an in-context critic to form preference pairs, improving results across multiple benchmarks.
1 Introduction
- Existing preference-tuning approaches can rely on external models or human-labeled data, raising distribution-shift, stability, and cost concerns.External models may generate hallucinations unrepresentative of the optimized LVLM, while human annotation or third-party feedback incurs costs.Source: §1 · p. 2
- SIMA samples prompts from existing visual instruction datasets, generates diverse responses, has the LVLM critic rank them into preference pairs, and tunes on those pairs.This three-stage process avoids external data or models for response generation and evaluation.Source: §1 · p. 2
- Its in-context self-critic evaluates responses without fine-tuning, using object-description, relationship, and attribute accuracy metrics.The critic prompt incorporates three visual metrics to guide evaluation of visual content.Source: §1 · p. 2
- Across 14 benchmarks, SIMA improved LLaVA-1.5-7B by 7.5%, LLaVA-1.5-13B by 4.5%, and VILA-7B by 5.3%, outperforming external-data preference-tuning approaches.Figure 1 reports a 7.5% average increase for LLaVA-1.5-7B.Source: Figure 1 · p. 1, §1 · pp. 2–3
2 Self-Improvement Modality Alignment
SIMA generates response candidates with the current LVLM, has that model rank them using an in-context critic, then tunes the model on the resulting preference pairs. Its stages are designed to avoid external-model distribution shift and direct ranking toward visual comprehension.
2 Self-Improvement Modality Alignment
- SIMA comprises response self-generation, in-context self-criticism, and preference tuning.The framework uses the model itself to critique candidates and then updates it using the self-ranked responses.Source: §2 · p. 3
2.1 Response self-generation
- External-model preference data may introduce distribution shift, so SIMA generates two candidates with the current LVLM using greedy decoding and temperature sampling.The authors note that external models’ hallucinations may not reflect errors the current LVLM would make, making such data potentially unhelpful for its training.Source: §2.1 · p. 3
2.2 In-context self-critic
- The current LVLM ranks its self-generated responses as positive and negative without an additional reward model, using the ground-truth response as a reference.The authors justify this reference because the sampled prompts and corresponding ground truths come from the visual instruction-tuning dataset.Source: §2.2 · pp. 3–4
- Three critic metrics guide ranking toward visual comprehension: object descriptions, object relationships, and attributes.The prompt also provides two ranking demonstrations; the criteria penalize unsupported objects and inaccuracies in described objects, relationships, and attributes.Source: §2.2 · p. 4
2.3 Preference tuning
- SIMA applies DPO to preference pairs containing an image, question, positive response, and negative response.The current LVLM and reference model are initialized with visual instruction-tuning weights.Source: §2.3 · pp. 4–5
3 Experiment
Experiments compare SIMA with preference-tuning baselines across hallucination and comprehensive benchmarks, then examine the critic metrics and training choices. Results cover hallucination reduction, comprehension, critic judgment, and ablations of iteration count and response-generation temperature.
3.1 Benchmark evaluation
- Evaluation covered 14 hallucination and comprehensive benchmarks, with preference-tuning baselines including LLaVA-RLHF, HA-DPO, POVID, and GT-DPO.The study used 17k prompts sampled from each backbone’s visual instruction-tuning dataset to form preference pairs; LLaVA training used LoRA, with epoch counts varying by model due to overfitting concerns.Source: §3.1 · p. 5
- Across CHAIRs, CHAIRi, and Mementos-Object, average gains were 16.1%, 7.1%, and 8.4% for LLaVA-1.5-7B, LLaVA-1.5-13B, and VILA-7B; MM-Hal gains were 12.7%, 10.1%, and 5.1%.The values correspond respectively to the three named models in each benchmark group.Source: Table 1 · p. 6, §3.1 · p. 6
- On nine comprehensive and VQA benchmarks, average improvements were 3.5%, 2.1%, and 4.4% over LLaVA-1.5-7B, LLaVA-1.5-13B, and VILA-7B, respectively.The reported improvements were superior to other preference-tuning methods.Source: Table 2 · p. 6, §3.1 · p. 6
- SIMA improved Mementos-Behavior performance by 13.1% with LLaVA-1.5-7B, and the paper relates behavior and object hallucination in sequential-image understanding.The authors note a correlation between the hallucination types and say reducing object hallucination increases the likelihood of correctly inferring corresponding behavior.Source: §3.1 · p. 6
3.2 Importance of our critic metric
- Removing the three visual critic metrics still improved performance over the original LLaVA, but left a significant gap versus metric-inclusive SIMA, especially on MM-Hal and Mementos.The comparison used preference pairs generated with metric-inclusive versus metric-free critic prompts.Source: Table 3 · p. 7, §3.2 · p. 7
- Metric-inclusive critic judgments were 89.8% consistent with human evaluations, up 11.2% from 78% without the metrics.The study compared 500 sampled response candidates with human judgments and GPT-4v evaluations.Source: Table 4 · p. 7, §3.2 · p. 7
3.3 Ablation studies
- On LLaVA-1.5-13B, the largest average performance gain came in the first SIMA iteration; average performance saturated by the third, though some benchmarks continued improving.Each iteration resampled 17k prompts from the visual instruction-tuning dataset.Source: Figure 6 · p. 7, §3.3 · p. 8
- Increasing response-generation temperature improved SIMA performance; the authors attribute this to more diverse, more hallucination-prone responses that differ further from greedy-decoded outputs.The proposed explanation links that distribution shift to better preference-tuning improvements.Source: Figure 6 · p. 7, §3.3 · p. 8
4 Related Work
Prior work has advanced LVLMs through large language models and vision instruction tuning, while modality alignment remains a challenge. SIMA applies self-improvement to LVLMs without external models or data.
4 Related Work
- LVLMs combine large language models and image encoders, supporting visual-language instruction following and multimodal understanding.The paper focuses on further enhancing their visual understanding and reasoning through self-improvement on visual instruction tuning data.Source: §4 · p. 8
- SIMA applies self-improvement to LVLMs using their vision instruction tuning data.The paper presents this as the first exploration of self-improvement in LVLMs.Source: §4 · p. 8
- Prior preference-learning approaches use human annotation or third-party model feedback, and may be resource-intensive or introduce external hallucinations and unstable results.SIMA addresses these issues without introducing external models or data.Source: §4 · p. 8
5 Conclusion
SIMA improves visual-language alignment through self-generated responses, in-context self-critique, and preference tuning. The paper reports improved comprehension and reduced hallucinations across benchmarks.
5 Conclusion
- SIMA combines self-generated responses, in-context self-critique, and preference tuning to improve visual-language alignment.It bypasses third-party AI models for data generation and response evaluation.Source: §5 · p. 8
- SIMA improves model comprehension and reduces hallucinations across various benchmarks.The paper also describes the approach as more scalable and cost-effective because it bypasses third-party models.Source: §5 · p. 8
- The framework uses the model itself to enhance alignment between visual and language modalities.The process evaluates self-generated responses before preference tuning.Source: §5 · p. 8
Limitations
SIMA relies on self-generated responses and self-critique, tying its performance to the LVLM’s current capabilities. It does not address potential biases in vision instruction tuning data.
Limitations
- SIMA’s performance is tied to the current capabilities of the LVLM because it relies on self-generated responses and self-critique.This is identified as a limitation of the approach.Source: Limitations · p. 9
- The method does not address potential biases caused by the vision instruction tuning dataset.The limitation concerns possible dataset biases rather than a stated result on every benchmark.Source: Limitations · p. 9
- Improvements are less significant for LLaVA-1.5-7B and LLaVA-1.5-13B on TextVQA.The paper gives this as an example of the limitation’s benchmark-specific effect.Source: Limitations · p. 9
Broader Impacts
The paper associates SIMA with reduced hallucinations and improved reasoning, while noting that self-generated and self-critic data may reinforce training-data biases.
Broader Impacts
- SIMA is reported to reduce hallucinations and improve reasoning capabilities, with the stated aim of increasing LVLM reliability.The paper describes this as an impact of applying self-rewarding in LVLMs.Source: Broader Impacts · p. 9
- Self-generated and self-critic data may unintentionally reinforce biases arising from training-data distribution shifts.The paper notes that SIMA has not eliminated potential risks within LVLMs.Source: Broader Impacts · p. 9
- The paper calls for ethical guidelines and safeguards to mitigate these risks and support responsible use.This recommendation accompanies its caution about residual LVLM risks.Source: Broader Impacts · p. 9
B Detailed ablation studies
The section reports benchmark performance for LLaVA-1.5-7B across training epochs and SIMA across decoding temperatures.
B Detailed ablation studies
- Performance across benchmarks is reported for LLaVA-1.5-7B at different training epochs.The results are presented in Table 5.Source: §B.1 · p. 12
- SIMA’s benchmark performance is reported for different temperature coefficients during decoding.The results are presented in Table 6.Source: §B.1 · p. 12
B.2 LLaVA-1.5-13B
For LLaVA-1.5-13B, results vary across training epochs and iterations, while comparisons with other open-source LVLMs show SIMA outperforming all except Qwen-VL-Chat on MM-Vet.
B.2 LLaVA-1.5-13B
- LLaVA-1.5-13B achieves its best results after one epoch, while third-iteration performance improves on some benchmarks and declines on others due to overfitting.Source: §B.2 · p. 12
B.3 Comparison with other open-source LVLMs
- SIMA significantly outperforms the compared open-source LVLMs except Qwen-VL-Chat on MM-Vet.The comparison is reported across comprehensive benchmarks.Source: §B.3 · p. 12
C Benchmark details
The benchmark descriptions cover visual reasoning, multimodal question answering, and related LVLM capabilities; the appendix also identifies training and evaluation settings.
C Benchmark details
- LLaVAW covers visual reasoning with 24 varied images and 60 questions, while ScienceQA provides about 21,000 multimodal multiple-choice science questions.Source: C Benchmark details · p. 12
- TextVQA evaluates reading and reasoning over text in images, whereas MME tests perception and cognition across 14 subtasks.Source: C Benchmark details · p. 12
- MMBench combines a diverse curated evaluation set with CircularEval, which uses ChatGPT to convert free-form predictions into structured choices.Source: C Benchmark details · p. 12
- MM-Vet organizes complex multimodal tasks into 16 ensembles derived from six core visual-language features.Source: C Benchmark details · p. 12
- SEEDBench spans 12 dimensions of spatial and temporal understanding across image and video, while VizWiz contains over 31,000 questions from blind people’s images and verbal queries.Source: C Benchmark details · p. 12
D Hyperparameters
- The appendix identifies training hyperparameters and the GPT version used for evaluation, and includes the critic prompt and performance tables for epoch and temperature comparisons.The relevant materials include Figure 7 and Tables 5–7.Source: D Hyperparameters · p. 12, Figure 7 · p. 13, Table 5 · p. 13
E.1 SIMA case study
The case studies illustrate fewer hallucinations in SIMA-generated descriptions and more visually grounded critic judgments when critic metrics are used. The appendix also records the study’s IRB exemption determination.
E.1 SIMA case study
- In the Figure 8 example, LLaVA invents chair placements absent from the image, while SIMA captures the visual details accurately.Across both demonstrations, SIMA is described as generating more accurate descriptions with fewer hallucinations.Source: §E.1 · p. 14
E.2 Critic prompt case study
- With critic metrics, the example evaluation focuses more on visual details and reaches a correct judgment.The passage identifies critic metrics as crucial to improving response-evaluation accuracy during in-context self-critic.Source: §E.2 · p. 15
F Status of Exemption from Institutional Review Board
- The IRB granted an “Exempt” determination, classifying the proposal as “Not Human Subjects Research.”Source: F Status of Exemption from… · p. 15