CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning

• VLM hallucinations include misrecognizing objects or attributes, and distribution bias in tuning data can prompt groundless positive answers [13].Source: §1 · pp. 1–2
• Prior POPE and MME evaluations require human annotation, focus only on measurement, and provide yes/no answers without detailed justification [19] [20].Source: §1 · p. 2
• CIEM generates factual and contrastive QA pairs with Chain-of-Thought justifications from image captions, while CIT generates such data for instruction tuning.CIEM uses an LLM and supports QA metrics such as accuracy and recall; CIEM is presented as a benchmark for systematically evaluating VLM perception.Source: §1 · p. 2
• CIT-tuned VLMs show advantages on both the CIEM setting and public datasets.Source: §1 · p. 3

2.1 Large Vision-Language Models

• LLM-based VLMs such as BLIP-2 [11] and MiniGPT-4 [12] support multimodal tasks, but hallucination and inaccurate information restrict their further application.Source: §2.1 · p. 3

2.2 Evaluation of VLMs

• Existing benchmarks cover selected tasks, while POPE [19] focuses on object hallucination and leaves fine-grained attributes and mitigation unaddressed.Other VLM benchmarks include MME [20] and LAMM-Benchmark [26].Source: §2.2 · p. 3

2.3 Instruction Tuning

• Multimodal instruction-tuning data from approaches such as LLaVA [29] and InstructBLIP [15] provides positive samples, which can introduce factual bias.Source: §2.3 · pp. 3–4

3 Methodology

• The framework separates CIEM evaluation on annotated test data from CIT training on generated examples from the training set, avoiding data leakage.The overall methodology has CIEM generate QA pairs for evaluation and CIT generate factual/contrastive training data with justifications.Source: §3 · p. 4

3.1.1 Automatic Generation of CIEM

• CIEM prompts an LLM with caption-grounded entities, attributes, and relations to generate factual questions and contrastive questions about non-existent information.The generated QA pairs cover object existence and fine-grained attributes and actions, and the pipeline does not require human labeling.Source: §3.1.1 · pp. 4–5

3.1.2 Verifying CIEM

• About 5% of CIEM-generated QA pairs from the COCO test set were inaccurate in a three-round blind review.The cited main causes were factual errors and incomplete information in caption annotations.Source: §3.1.2 · p. 5, Table 1 · p. 5, Figure 3 · p. 6

3.1.3 Evaluating VLMs on CIEM

• CIEM evaluation compares VLM answers with generated answers for factual and contrastive questions, treating the yes/no responses as binary classification.It reports Accuracy, Precision, Recall, Specificity, and F1-score; Precision directly reflects the hallucination issue.Source: §3.1.3 · p. 5

3.2 Contrastive Instruction Tuning

• CIT prompts generate factual “yes” and contrastive “no” questions with detailed explanations, then use the contrastive samples for instruction tuning.Contrastive alternatives are similar, easily confused, or co-occurring objects, attributes, or actions; explanations provide a factual basis for answers.Source: §3.2 · pp. 6–7

4.1 CIEM & CIT on COCO

• On COCO Caption, InstructBLIP has the best F1-score, while LLaVA and Mini-GPT4 show high recall but poor precision and F1-score.The authors interpret the latter pattern as a tendency to answer positively regardless of the question.Source: §4.1 · p. 7, Table 2 · p. 7
• Removing CoT harms overall performance despite improving precision, supporting the value of explanations beyond final answers.The authors argue that CoT lets the VLM learn from details rather than merely memorize the final answer.Source: §4.1 · pp. 7–8
• CIT improves InstructBLIP precision, specificity, F1-score, and accuracy by a large margin, with only a slight drop in recall.The table compares CIT with zero-shot and LLaVA-instruction-tuned InstructBLIP baselines.Source: §4.1 · p. 7, Table 3 · p. 8

4.2 How does CIT Affect Downstream Tasks?

• After CIT, image captioning improves slightly on NoCaps and COCO Caption, while VQAv2 performance drops slightly.The authors attribute the pattern to caption-derived questions emphasizing primary image content while potentially missing details useful for VQA.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Qualitative Results

• Qualitative examples show CIT helping the VLM reject objects absent from images, while CoT adds detailed explanations and justifications.Figure 5 visualizes the authors' claim that CIT alleviates visual hallucination and CoT further enables correction of wrong information.Source: §4.3 · p. 8, Figure 5 · p. 9

4.4 Limitation and Future Work

• CIEM depends on annotated multimodal data, and annotation quality can affect generated QA accuracy; it cannot process raw images without annotations.The authors suggest using other large models to generate confident captions as a possible solution.Source: §4.4 · p. 9
• CIEM and CIT generate yes-or-no questions, which the authors say should become more flexible and diverse for general scenarios.Source: §4.4 · p. 9
• CIEM focuses on perception and visual hallucination; integrating knowledge retrieval and reasoning into a unified benchmark remains future work.Source: §4.4 · p. 9

5 Conclusion

• CIEM automatically generates factual and contrastive question-answer pairs with explanations and reasoning paths from training data to evaluate visual hallucination.The pipeline uses an external LLM and image captions to query VLMs about entities and attributes.Source: §5 · p. 9
• The authors report that CIEM experiments reveal hallucination issues in VLMs and that CIT is effective at reducing them.Source: §5 · p. 9
• CIT uses generated question-answer pairs with explanations and reasoning paths to tune VLMs to hallucinate less.Source: §5 · p. 9