PaperlayerView exact revision on arXiv

Source-linked AI summary

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, Ping Luo

arXiv:2306.09265v1cs.CVcs.AI

TL;DR

LVLM-eHub addresses the lack of holistic evidence about LVLM efficacy by combining broad quantitative evaluation with an online human-judgment arena. Across its evaluations, the study finds overfitting, object hallucination, and metric limitations, while showing that multi-turn reasoning can mitigate hallucination.

  • Problem

    Existing studies evaluate only portions of LVLM capabilities or limited model sets, leaving systematic evaluation insufficient.

  • Method

    The paper builds LVLM-eHub, evaluating 8 representative LVLMs across 6 capability categories and 47 visual benchmarks, alongside an open-world online arena.

  • Results

    The evaluation reveals poor open-world generalization after massive in-domain tuning, object hallucination with moderate instruction data, and mitigation through multi-turn reasoning.

  • Takeaways & Limitations

    Effective LVLM assessment should combine broad task coverage with open-world human feedback and evaluation pipelines that address hallucination.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) have recently played a dominant role in multimodal vision-language learning. Despite the great success, it lacks a holistic evaluation of their efficacy. This paper presents a comprehensive evaluation of publicly available large multimodal models by building a LVLM evaluation Hub (LVLM-eHub). Our LVLM-eHub consists of $8$ representative LVLMs such as InstructBLIP and MiniGPT-4, which are thoroughly evaluated by a quantitative capability evaluation and an online arena platform. The former evaluates $6$ categories of multimodal capabilities of LVLMs such as visual question answering and embodied artificial intelligence on $47$ standard text-related visual benchmarks, while the latter provides the user-level evaluation of LVLMs in an open-world question-answering scenario. The study reveals several innovative findings. First, instruction-tuned LVLM with massive in-domain data such as InstructBLIP heavily overfits many existing tasks, generalizing poorly in the open-world scenario. Second, instruction-tuned LVLM with moderate instruction-following data may result in object hallucination issues (i.e., generate objects that are inconsistent with target images in the descriptions). It either makes the current evaluation metric such as CIDEr for image captioning ineffective or generates wrong answers. Third, employing a multi-turn reasoning evaluation framework can mitigate the issue of object hallucination, shedding light on developing an effective pipeline for LVLM evaluation. The findings provide a foundational framework for the conception and assessment of innovative strategies aimed at enhancing zero-shot multimodal techniques. Our LVLM-eHub will be available at https://github.com/OpenGVLab/Multi-Modality-Arena

1 Introduction

LVLM-eHub addresses the lack of systematic LVLM evaluation with broad quantitative and open-world human assessments. Its findings expose generalization, hallucination, and evaluation-pipeline issues while motivating better zero-shot evaluation strategies.

  • Evaluation gap and benchmark: LVLM-eHub evaluates 8 representative LVLMs through quantitative capability testing and an online arena platform.The benchmark covers models including InstructBLIP and MiniGPT-4.
  • Evaluation gap and benchmark: The quantitative evaluation spans 6 multimodal capability categories across 47 standard text-related visual benchmarks.The categories include visual perception, knowledge acquisition, reasoning, commonsense, object hallucination, and embodied intelligence.
  • Findings: Instruction-tuned LVLMs trained on massive in-domain data can overfit existing tasks and generalize poorly to open-world scenarios.InstructBLIP is identified as an example of this pattern.
  • Findings: Moderate instruction-following data can produce object hallucination, making answers incorrect and metrics such as CIDEr ineffective.Hallucinated objects are inconsistent with target images in generated descriptions.
  • Findings: A multi-turn reasoning evaluation pipeline can mitigate object hallucination and supports more effective LVLM assessment.The authors present this finding as motivation for developing better evaluation pipelines.
  • Evaluation gap and benchmark: The online arena supplements benchmark scores with anonymous, randomized, user-level pairwise comparisons in open-world scenarios.The arena is designed to provide human feedback beyond limited quantitative evaluations.

2 LVLM Evaluation Hub

The LVLM Evaluation Hub combines six-category zero-shot benchmarking with an online arena for human-judged, open-world comparisons. It evaluates diverse visual capabilities using task-specific prompting, scoring, multi-turn reasoning, and user feedback.

  • Capability coverage: Visual perception is measured through image classification, multi-class identification, and object counting.Classification and identification assess high-level semantics, while counting targets fine-grained object recognition.
  • Capability coverage: Visual knowledge acquisition uses OCR, key information extraction, and image captioning benchmarks.The supplied passage specifies twelve OCR benchmarks and KIE datasets including SROIE and FUNSD.
  • Capability coverage: Visual reasoning is evaluated through VQA, knowledge-grounded image description, and visual entailment tasks.The VQA-related benchmarks include DocVQA, TextVQA, OKVQA, GQA, VSR, and Visual Dialog, while SNLI-VE supports entailment.
  • Capability coverage: The hub also evaluates visual commonsense, object hallucination, and embodied intelligence using ImageNetVC, VCR, POPE on MSCOCO, and embodied-task benchmarks.Embodied benchmarks include Minecraft, VirtualHome, Meta-World, and Franka Kitchen.
  • Online arena: The LVLM Arena samples model pairs, supports anonymous side-by-side chats, and updates Elo ratings from user votes.Users can select Model A, Model B, Tie, or Both are bad; the current platform supports single-round chats.
  • Evaluation methods: Zero-shot evaluation uses task-specific prompt engineering with question answering, prefix-based scores, multi-turn reasoning, and user studies.Prefix-based scoring estimates image-text likelihood after visual prompts are prefixed into text embeddings.

3 Experiment and Analysis

The evaluation covers six multimodal capability categories and an online arena, revealing strengths, overfitting, hallucination, and evaluation-scheme effects across LVLMs. Results show that model performance depends strongly on training data and whether tasks use multi-turn reasoning.

  • Evaluation scope: The study evaluates visual perception, visual knowledge acquisition, visual reasoning, visual commonsense, object hallucination, and embodied intelligence through zero-shot tasks.The arena evaluation is presented separately for open-world visual questions.
  • Visual perception and knowledge acquisition: InstructBLIP performs strongly on fine-grained perception and visual knowledge tasks, but the paper attributes this to overfitting massive VQA data.It performs best on fine-grained image classification, object counting, multi-class identification, and almost all visual knowledge acquisition tasks.
  • Visual reasoning: Instruction-tuned models generally underperform BLIP2 on visual reasoning unless tasks use multi-turn reasoning, where they surpass BLIP2 on SNLI-VE.The paper links conventional metric limitations to common words in instruction data that influence generated content.
  • Visual commonsense: All evaluated LVLMs partly solve visual commonsense problems, while InstructBLIP performs best on ImageNetVC and VCR.The paper attributes this result to InstructBLIP’s 1.6M fine-grained VQA fine-tuning and notes that VCR uses multi-turn reasoning.
  • Object hallucination: Instruction-tuned models except InstructBLIP perform worse than BLIP2 on hallucination evaluation because they tend to answer “Yes” to object-presence questions.The evaluation uses the POPE pipeline on MSCOCO with accuracy, precision, recall, F1-Score, and yes-answer ratio.

4 Discussion and Conlcusion

The paper combines quantitative evaluation with human-feedback evaluation to assess LVLMs across capabilities and open-world use. It argues that this framework exposes evaluation weaknesses and supports more realistic model comparison.

  • Evaluation Framework: LVLM Arena uses one-versus-one model battles with human judgment for open-world evaluation.The approach is intended to produce more accurate and realistic evaluations, but reliable ratings require substantial human effort.
  • Evaluation Framework: CIDEr and generic answer-similarity methods can be unreliable for diverse LVLM responses, while GPT-based judges cannot directly assess image content.These limitations motivate human-judged arena evaluation and continued development of fast, accurate, generalized metrics.
  • Evaluation Framework: The platform evaluates six capabilities through multiple tasks and datasets, while allowing users to contribute datasets and models.It also provides online inference services and an Elo rating system based on human feedback.
  • Evaluation Framework: LVLM-eHub combines quantitative performance evaluation on 16 tasks across more than 40 text-related visual datasets with continual human-feedback evaluation.The online arena supplies open-world rankings through Elo ratings.

A.1 Overall Evaluation Results

The evaluation finds that training-data scale and instruction tuning affect LVLM generalization and answer reliability. In-domain gains can coexist with poor open-world performance, hallucinated objects, and biased responses.

  • Overall Findings: InstructBLIP achieves the best results in five capability categories but lags behind LLaMA-Adapter V2 and mPLUG-Owl in embodied AI and the LVLM arena.Its fine-tuning on 16M visual question-answering pairs is associated with strong in-domain performance but poor generalization to real-world tasks.
  • Overall Findings: Instruction-tuned LVLMs with moderate high-quality data may generate objects inconsistent with target images, causing wrong answers or ineffective CIDEr evaluation.The paper identifies this as an object hallucination issue in image descriptions and answers.
  • Overall Findings: Instruction-tuned LVLMs using moderate high-quality data generally show higher accuracy and recall but lower precision than BLIP2 and InstructBLIP.The evaluated models include LLaMA-Adapter V2, LLaVA, MiniGPT-4, mPLUG-Owl, Otter, and VPGTrans.
  • Overall Findings: Moderate high-quality instruction data can make LVLMs prone to answering “Yes” regardless of the underlying answer’s accuracy.This response tendency is reported for several instruction-tuned models and helps explain their lower precision.

B Evaluation Details

The evaluation details define task-specific datasets, prompts, metrics, and procedures across visual perception, knowledge acquisition, reasoning, commonsense, and prompt-sensitive model responses. Several tasks use tailored prompts or multi-turn reasoning instead of naive question answering.

  • Visual Perception: Image classification uses ImageNet1K, CIFAR10, Pets37, and Flowers102 with top-1 or per-class accuracy.The study uses a default prompt for coarse-grained benchmarks and specialized prompts for fine-grained categories, counting a response correct when it contains the correct class name.
  • Visual Perception: Object counting and multi-class identification query object quantities or presence and evaluate the responses as classification accuracy.The prompts ask how many objects are present or whether a specified object exists in the image.
  • Visual Knowledge Acquisition: Optical character recognition uses twelve text-image datasets, while key information extraction uses SROIE and FUNSD with entity-level F1 scores.KIE employs information-specific prompts for fields such as company name and invoice address.
  • Visual Knowledge Acquisition: Image captioning uses NoCaps and Flickr30K, evaluating generated descriptions with CIDEr under an image-description prompt.The benchmarks provide images paired with corresponding captions.
  • Visual Reasoning: Visual question answering spans nine benchmarks covering question-image pairs and uses task-specific metrics including mean reciprocal rank for Visual Dialog.The tasks require visual understanding together with question comprehension and reasoning.
  • Visual Reasoning: SNLI-VE and VCR use multi-turn reasoning because naive question-answering prompts do not produce meaningful predictions.For SNLI-VE, 500 development samples are selected; VCR similarly uses 500 validation samples.
  • Visual Commonsense: ImageNetVC evaluates zero-shot visual commonsense by converting question-answer pairs into token sequences and selecting the answer with maximum prefix-based likelihood.The evaluation covers generic concepts such as color and shape.

C More Experiments

LVLM performance is highly sensitive to prompt choice, so evaluation outcomes can vary substantially even when the image and model remain fixed.

  • Prompt Sensitivity: VPGTrans performance on NoCaps varies from 19.66 to 48.13 when the evaluation prompt changes.Figure A.2 illustrates response variation for the same image under different prompts.

C.2 More Results on Object Hallucination

LVLMs can hallucinate objects inconsistent with the image, while multi-turn reasoning evaluation can mitigate this problem.

  • A multi-turn reasoning evaluation pipeline can mitigate object hallucination issues.
  • Reasoning may encourage instruction-tuned models to reorganize acquired knowledge before producing answers.
  • Many LVLMs generate content that does not exist in the given image, demonstrating object hallucination.

C.3 More Results on Embodied Tasks.

Embodied-task evaluation covers representative scenes from Minecraft, Franka Kitchen, and Meta-World, combining quantitative results with participant ratings of generated plans.

  • Embodied-task results use representative scenes from Minecraft, Franka Kitchen, and Meta-World benchmarks.The corresponding task results are provided in the appendix sections.
  • Minecraft: In Minecraft, all models recognized the boat, but only LLaMA-Adapter V2, InstructBLIP, and MiniGPT-4 used it in plans to reach the opposite shore faster.
  • Franka Kitchen: In Franka Kitchen, all models except BLIP produced feasible and reasonable plans, while LLaVA recognized that the goal was already achieved.
  • Meta-World: In Meta-World, low-resolution inputs contributed to visual reasoning errors, including mistaking a basketball or robotic arm for other entities.

C.3.1 Minecraft

Minecraft evaluation asks LVLMs to describe a scene and generate a feasible plan for reaching the opposite shore with a boat.

  • InstructBLIP described the scene with additional objects, including a boat, tree, and rock, while discussing whether to use the boat.
  • LLaMA-Adapter V2 identified the boat as the primary transportation mode for reaching the opposite shore quickly.
  • LLaVA described a boat, sandy beach, island, and shore-crossing objective but did not provide the same boat-use planning outcome reported for the successful models.
  • MiniGPT-4 described a detailed boat scene but also added unsupported visual details such as a cabin, mast, sail, and multiple shore types.
  • mPLUG-Owl and Otter introduced scene descriptions involving a person or raft, while VPGTrans described an unobstructed boat route toward an island.

C.3.2 Franka Kitchen

Franka Kitchen tests whether LVLMs can plan moving a kettle to the top-left burner while recognizing when the goal is already achieved.

  • The task prompt asks models to move the kettle to the top-left burner, with some outputs explicitly decomposing the action into sequential steps.
  • LLaVA recognized that the goal state had already been achieved in the provided image.
  • mPLUG-Owl generated conditional steps for both locating and moving the kettle, including skipping actions if it was already on the target burner.
  • Otter interpreted the image as showing the robot already lifting the teapot and proposed positioning it safely on the burner.
  • VPGTrans proposed identifying both locations, moving and centering the kettle, then turning on the burner.

C.3.3 Meta World

The Meta World examples show LVLMs describing basketball scenes and proposing action plans, often adding details beyond the concise visual description. Their outputs vary in how much planning, scene interpretation, and unsupported detail they include.

  • BLIP provides a concise description of a robot dunking a basketball in a pool.
  • InstructBLIP describes a 3D basketball game and expands it into a six-step dunking plan.
  • LLaMA-Adapter V2 similarly combines scene description with a proposed sequence of actions for dunking.
  • LLaVA interprets the scene as a basketball simulation involving a person controlling the basketball with a remote.
  • MiniGPT-4 adds multiple objects and scene details before proposing a robot-dunking plan.
  • mPLUG-Owl, Otter, and VPGTrans also generate embodied-action descriptions involving positioning, ramps, timing, or spatial measurements.Their outputs differ in the specific actions and environmental details they emphasize.

D Summary of Evaluation Datasets

The evaluation datasets span visual recognition, knowledge acquisition, and embodied-AI settings. They include benchmark-specific validation, test, caption, form, text-instance, and simulated-environment samples.

  • Visual perception: VCR-OC and VCR-MCI use 10,000 validation-set images from VCR for object counting and multi-class identification.
  • Visual knowledge acquisition: The visual knowledge acquisition datasets include Total-Text, FUNSD, and NoCaps for text recognition, form understanding, and novel-object captioning.
  • Visual knowledge acquisition: Total-Text contains 1,555 images and 2,551 cropped text instances in its test set.
  • Visual knowledge acquisition: FUNSD contains 199 annotated scanned forms, split into 50 test and 149 training examples.
  • Embodied AI: Embodied-AI evaluation uses selected samples from Minecraft, VirtualHome, Meta-World, and Franka Kitchen.
Loading 2306.09265v1…