LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

Motivation

• Prior evaluations examined specific tasks or capabilities, leaving no overall understanding of LVLM capabilities.The introduction cites work on object hallucination, visual commonsense, visual recognition with text recognition, and visual semantic understanding and fine-grained perception.Source: §1 · p. 2

Evaluation benchmark

• LVLM-eHub evaluates 8 representative models across six multimodal capability categories and 47 text-related visual benchmarks, alongside an anonymous, randomized pairwise arena for open-world user rankings.The capability categories include visual perception, knowledge acquisition, reasoning, commonsense, object hallucination, and embodied intelligence.Source: §1 · p. 2

Findings

• Massive in-domain instruction tuning can overfit and generalize poorly in open-world use, while moderate instruction-following data can induce object hallucinations that undermine answers and CIDEr.A multi-turn reasoning evaluation pipeline can mitigate hallucination. Figure 1 contrasts InstructBLIP’s in-domain performance with moderate-data models’ arena performance.Source: §1 · p. 2, Figure 1 · p. 2

2 LVLM Evaluation Hub

• The hub compares eight LVLMs using checkpoints under 10B parameters, alongside quantitative evaluation and an online arena.The models include BLIP2 [6], LLaVa [9], LLaMA-Adapter V2 [7], MiniGPT-4 [10], mPLUG-Owl [11], Otter [12], InstructBLIP [13], and VPGTrans [8]; they differ in training-data scale and configuration.Source: §2 · p. 3, Figure 2 · p. 4

2.1 Quantitative Capability Evaluation

• Quantitative evaluation covers visual perception, visual knowledge acquisition, visual reasoning, visual commonsense, object hallucination, and embodied intelligence.Benchmarks span image classification, counting and identification; OCR, key-information extraction and captioning; question answering, knowledge-grounded descriptions and entailment; commonsense tasks; POPE on MSCOCO; and embodied tasks in simulated environments.Source: §2.1 · pp. 3–4
• The embodied-intelligence evaluation uses high-level EmbodiedGPT tasks and Minecraft, VirtualHome, Meta-World, and Franks Kitchen benchmarks.The framework assesses agents’ ability to solve tasks requiring environmental interaction.Source: §2.1 · p. 4

2.2 Online Evaluation with LVLM Arena

• The arena randomly pairs anonymous models, supports single-round image- or text-based chats, and updates Elo ratings from votes for either model, a tie, or both being bad.Random pairing is used because the model hub is currently limited in size; multi-round chats are constrained by computational and memory demands.Source: §2.2 · p. 5
• The LVLM Arena supports open-world evaluation through user conversations and human judgments, addressing the challenge of evaluating open-ended model responses with quantitative metrics.Users chat with anonymous models about anything and judge pairwise battles, approximating real-world conditions.Source: Figure 2 · p. 4, §2.2 · p. 5

2.3 Zero-shot Evaluation

• Zero-shot evaluation measures generalization to new tasks without training, using task-specific prompts and question answering, prefix-based scores, multi-turn reasoning, or user studies.Multi-turn reasoning uses an LLM to generate sub-questions, an LVLM to answer them, and another LLM to assess answer quality; user studies anonymize and shuffle model outputs.Source: §2.3 · pp. 5–6, Figure 3 · p. 5

3.1 Results on Visual Perception

• mPLUG-Owl and LLaVA performed best on coarse-grained classification, while InstructBLIP showed strong fine-grained classification, counting, and identification but remained below supervised SOTA on classification.The paper links mPLUG-Owl and LLaVA’s coarse-task results to updating their LLMs with 158K instruction-following examples, and InstructBLIP’s fine-grained results to overfitting after fine-tuning on 1.6M VQA examples.Source: §3.1 · p. 6

3.2 Results on Visual Knowledge Acquisition

• BLIP2, InstructBLIP, and VPGTrans performed strongly across OCR, key information extraction, and image captioning, with InstructBLIP best on almost all tasks.The authors associate these results with large ViT-g/14 visual encoders and Q-Formers trained on massive image-text pairs, while attributing InstructBLIP’s results partly to overfitting from massive VQA fine-tuning.Source: §3.2 · p. 7, Table 3 · p. 7

3.3 Results on Visual Reasoning

• Instruction-tuned LVLMs other than InstructBLIP generally underperformed BLIP2 on visual reasoning, but surpassed it on SNLI-VE with multi-turn reasoning.The authors attribute weaker results to common words in instruction data influencing generated content in ways current metrics cannot evaluate; multi-turn reasoning offers a better evaluation scheme.Source: §3.3 · pp. 7–8, Table 4 · p. 7

3.4 Results on Visual Commonsense

• InstructBLIP performed best on ImageNetVC and VCR, and instruction-tuned models surpassed BLIP2 on both visual commonsense benchmarks.The authors link InstructBLIP’s performance to fine-tuning on 1.6M fine-grained VQA data, and note that VCR uses multi-turn reasoning to obtain its final answer.Source: §3.4 · p. 8, Table 5 · p. 8

3.5 Results on Object Hallucination

• InstructBLIP performed best on POPE hallucination evaluation, followed by BLIP2, while other instruction-tuned models tended to answer “Yes” and hallucinate objects absent from images.The evaluation used six yes-or-no questions per image across MSCOCO-Random, Popular, and Adversarial datasets; the paper reports that multi-turn reasoning can alleviate hallucination.Source: §3.5 · p. 8, Table 6 · p. 9

3.6 Results on Embodied Intelligence

• Embodied-planning results suggest that image-text alignment and visual instruction data matter: mPLUG-Owl outperformed Otter, while BLIP2’s lack of visual instruction tuning impaired plan quality.mPLUG-Owl had 204M image-text training pairs; the participant study rated plans on object recognition, spatial understanding, conciseness, reasonability, and executability.Source: §3.6 · pp. 8–9, Table 7 · p. 10

3.7 Results on Online Arena Evaluation

• In the arena, instruction-tuned models ranked among the top performers in open-world VQA, while InstructBLIP’s strong in-domain results contrasted with worse arena performance, indicating overfitting.The two data collections contained 21.8% and 22.0% “both are bad” votes, suggesting that current LVLMs still struggle with open-world visual questions.Source: §3.7 · p. 10

New Evaluation Metrics

• CIDEr can fail when LVLM answers differ in style from ground truth; sentence-embedding evaluation is more robust but limited, while GPT judges cannot see images and can be inaccurate.The paper notes that developing fast, accurate, generalized LVLM metrics remains an open problem.Source: §4 · p. 10
• LVLM Arena uses human-judged 1v1 battles for more accurate, realistic open-world evaluation, but reliable ratings require substantial human effort as the number of models grows.The paper proposes this approach because automated alternatives have shortcomings in LVLM evaluation.Source: §4 · p. 10

A Platform for LVLM Evaluation

• The evaluation platform organizes multiple tasks and datasets across six capabilities and lets users contribute datasets and models for one-click holistic assessment.The platform is intended to be updated regularly, with broader dataset and model support.Source: §4 · p. 10
• LVLM Arena offers free online inference for supported models, lets users vote for preferred models, and incorporates human feedback into Elo rankings.The rankings provide a view of model standing in open-world scenarios.Source: §4 · pp. 10–11
• Users can contribute LVLMs through the platform’s inference interface, while the project plans regular dataset updates and wider model support.The interface is intended to support a collaborative evaluation environment.Source: §4 · p. 10

Conclusion

• LVLM-eHub evaluates six capabilities across 16 tasks and 40+ text-related visual datasets, alongside an online arena for continually gathering human feedback.The arena provides an Elo ranking for LVLMs in open-world scenarios.Source: §4 · pp. 10–11

A.1 Overall Evaluation Results

• InstructBLIP achieves the best results in five capability categories but lags in embodied AI and the LVLM arena, consistent with overfitting to its 16M in-domain visual question-answering pairs.The arena tests open-world visual question answering, while embodied AI requires step-by-step planning from an image.Source: §A.1 · p. 18
• Moderate-data instruction-tuned models can hallucinate objects and answer “Yes” incorrectly, yielding higher recall and accuracy but lower precision than BLIP2 and InstructBLIP.LLaMA-Adapter V2 can produce accurate captions despite low CIDEr scores, and sentence-similarity measures indicate the captions are relatively accurate.Source: §A.1 · p. 18
• Under multi-turn reasoning evaluation, instruction-tuned models with moderate high-quality data outperform BLIP on SNLI-VE and VCR, and the framework can mitigate object hallucination.The paper reports supporting evidence for the mitigation effect in Fig. A.3.Source: §A.1 · p. 19

A.2 Model Details in LVLM-eHub

• BLIP2 [6] uses a two-stage Q-Former strategy with frozen ViT-g/14 and FlanT5-XL, while LLaVA [9] connects CLIP ViT-L/14 to LLaMA through an FC layer and fine-tunes on instructional data.BLIP2 pre-trains on 129M image-text pairs; LLaVA trains its FC layer on 595K image-text pairs and fine-tunes on 158K instructional vision-language examples.Source: §A.2 · p. 19
• LLaMA-Adapter V2 [7] tunes bias, scale, norm, and prompt parameters on captioning and instruction data, whereas MiniGPT-4 [10] connects its encoders with an FC layer and fine-tunes on 3.5K instructional examples.LLaMA-Adapter V2 uses 200M image-captioning, 158K visual-instruction, and 52K GPT-4-generated language-instruction examples; MiniGPT-4 first trains its FC layer on 5M image-text pairs.Source: §A.2 · p. 19
• mPLUG-Owl [11] bridges ViT-L/14 and LLaMA with a visual abstractor, while Otter [12] builds on OpenFlamingo and trains adaptation modules with frozen visual and text encoders.mPLUG-Owl fully fine-tunes the visual encoder and abstractor on 204M image-text pairs, then uses 158K LLaVA-Instruct data for LoRA; Otter trains 1.3B extra parameters on 158K instruction-following examples.Source: §A.2 · p. 19
• InstructBLIP [13] fine-tunes only the Q-Former across 13 visual question-answering datasets, while VPGTrans [8] transfers BLIP-2’s visual grounding from OPT6.7B to Vicuna7B.VPGTrans trains the Q-Former on 13.8M image-text pairs and further tunes the visual grounding and projector on MiniGPT-4’s 3.5K self-instruct examples.Source: §A.2 · p. 19

B.1 Details of Visual Perception

• Fine-grained classification uses dataset-specific prompts and accepts outputs containing the correct class name; object counting and object presence are scored as classification accuracy.ImageNet1K and CIFAR10 use the default prompt “The photo of the” and top-1 accuracy; Pets37 and Flowers102 use specific-category prompts and per-class accuracy. Object counting and presence tasks use MSCOCO and VCR1.0 [57], with prompts asking how many objects appear or whether an object exists.Source: §B.1 · pp. 19–20

B.2 Details of Visual Knowledge Acquisition

• OCR is evaluated on 12 text-image benchmarks using top-1 accuracy and the prompt “what is written in the image?”.The datasets include IIIT5K [31], ICDAR 2013 [32], ICDAR 2015 [33], Total-Text [34], CUTE80 [35], SVT [36], SVTP [37], COCO-Text [38], WordArt [39], CTW [40], HOST [41], and WOST [41].Source: §B.2 · p. 20
• KIE uses SROIE [42] and FUNSD [43], scoring entity-level F1 with prompts tailored to the information being extracted.For example, SROIE prompts ask for the company name or invoice address.Source: §B.2 · p. 20
• Image captioning is evaluated on NoCaps [44] and Flickr30K [45] using CIDEr and the prompt “what is described in the image?”.Source: §B.2 · p. 20

B.3 Details of Visual Reasoning

• VQA covers nine benchmarks, using MRR for Visdial [54] and top-1 accuracy for the remaining datasets; KGID uses ScienceQA [55] and VizWiz [56].Only image-containing ScienceQA samples are used.Source: §B.3 · p. 20
• For SNLI-VE [26], the evaluation randomly selects 500 dev samples and uses iterative Questioner–Answerer–Reasoner multi-turn reasoning because naive QA gives poor predictions.ChatGPT via the gpt-3.5-turbo API serves as Questioner and Reasoner; the eight LVLMs serve as Answerers and produce captions.Source: §B.3 · pp. 20–21

B.4 Details of Visual Commonsense

• ImageNetVC [15] is evaluated by scoring answer candidates from prefixed visual and text embeddings, averaging results across five similar prompts.The embeddings are input to a frozen pretrained LVLM, and the maximum-likelihood candidate is selected.Source: §B.4 · p. 21
• VCR evaluation randomly selects 500 validation samples and uses multi-turn reasoning because a naive QA prompt does not produce meaningful outputs.The task requires choosing the correct answer from four candidates.Source: §B.4 · p. 21

C.1 Sensitivity to Prompts.

• 19.66 to 48.13 was the performance range for VPGTrans on NoCaps image captioning across different prompts.Fig. A.2 shows examples generated from different prompts.Source: §C.1 · p. 21
• Careful prompt selection and design are important for achieving optimal performance in LVLM-based tasks.The authors report that LVLM models are highly sensitive to prompt choice.Source: §C.1 · p. 21

C.2 More Results on Object Hallucination

• Most LVLMs except BLIP2 and InstructBLIP generate image-inconsistent objects, but multi-turn reasoning evaluation can mitigate hallucination by helping models reorganize knowledge and produce correct answers [14].The authors argue that this makes it important to evaluate instruction-tuned LVLMs with an appropriate procedure.Source: §C.2 · pp. 21–22, Figure A.3 · p. 22

C.3 More Results on Embodied Tasks.

• All eight models recognized Minecraft’s floating boat, but only LLaMA-Adapter V2, InstructBLIP, and MiniGPT-4 used it to reach the opposite shore faster.The task asked models to plan a route for an agent to reach the opposite shore.Source: §C.3 · p. 22, Figure A.4 · p. 23
• In Franka Kitchen, all models except BLIP gave feasible plans; LLaVA recognized the kettle goal was already achieved, while mPLUG-Owl planned for that contingency.The task asked models to help move a kettle to the top-left burner and analyze whether the goal state had been achieved.Source: §C.3 · pp. 22–23, Figure A.5 · p. 23
• In Meta-World examples, models struggled with visual reasoning, with mPLUG-Owl mistaking the basketball in the robot’s hand and LLaVA, MiniGPT-4, and Otter mistaking the robotic arm for a human.The passage attributes some of these difficulties to low-resolution image input.Source: Figure A.6 · p. 23, §C.3 · p. 23

Minecraft

• BLIP identifies a Minecraft boat in water, while LLaVA and Otter add player positions and a goal of crossing to the other shore.LLaVA places the player on a beach looking at the boat; Otter describes a person on a raft near a small boat.Source: §C.3.1 · p. 24
• InstructBLIP invents a player’s island location and crossing goal, then adds a partially submerged boat, tree, rock, and spatial relationships.Its response also speculates about navigating obstacles and using the boat to reach the opposite shore.Source: §C.3.1 · p. 24
• LLaMA-Adapter V2 describes a wooden boat near shore and infers that it is the player’s primary transport for reaching the opposite shore.It also asserts a path to the opposite shore and a need to travel quickly.Source: §C.3.1 · p. 24
• MiniGPT-4 adds a cabin, mast, sail, rocky shore, sandy beach, and sun, then proposes paddles or oars and obstacle avoidance.The response presents these scene details and navigation ideas as part of its account of reaching the opposite shore.Source: §C.3.1 · p. 24
• mPLUG-Owl infers a person in a desert biome, a distant partly submerged boat, and scattered rocks and trees.It further speculates that the scene may be part of a desert-themed Minecraft world.Source: §C.3.1 · p. 24
• VPGTrans gives a simpler account of a wooden boat heading toward an island, with calm water and no visible obstacles.It also says the island appears close and reachable in a short time.Source: §C.3.1 · p. 24

C.3.2 Franka Kitchen

• InstructBLIP and LLaMA-Adapter V2 give multistep plans for locating and moving the kettle, while mPLUG-Owl makes its steps conditional on whether the kettle is already placed.mPLUG-Owl also describes grasping, navigating around objects, placing the kettle, and releasing it.Source: §C.3.2 · p. 25
• LLaVA says the kettle is already on the top-left burner and concludes that no further plan is needed.It also suggests confirming the kettle is stable and secure.Source: §C.3.2 · p. 25
• MiniGPT-4 describes a ladle and kettle on the counter, then proposes moving the kettle and using the ladle to pour hot water.Otter instead refers to a teapot and focuses on safe positioning and maintaining a firm grip.Source: §C.3.2 · p. 25
• VPGTrans proposes picking up and centering the kettle on the burner, then turning the burner on.Source: §C.3.2 · p. 25

• BLIP briefly describes a robot dunking a basketball in a pool.Source: §C.3.3 · p. 26
• InstructBLIP and LLaMA-Adapter V2 both provide multi-step plans for a robot dunk, including positioning, handling the ball, and putting it through the hoop.InstructBLIP's plan includes waiting for the ball to land and dribbling; LLaMA-Adapter V2 describes approaching and grabbing the ball before placing it in the hoop.Source: §C.3.3 · p. 26
• LLaVA describes a person holding a remote control and frames the scene as a simulation, while MiniGPT-4 describes a human player and multiple basketballs.MiniGPT-4 then proposes a plan for a robot to pick up and dunk a ball.Source: §C.3.3 · p. 26

Visual Reasoning Datasets

• VizWiz is a visual question-answering dataset whose answers were obtained by asking blind people, and it has 1,131 validation examples.Source: D Summary of Evaluation Datasets · p. 28

Dataset categories

• The dataset inventory also identifies Visual Commonsense and Object Hallucination categories.These appear as dataset category headings in the table.Source: D Summary of Evaluation Datasets · p. 28

Embodied Intelligence Datasets

• Embodied AI tasks use selected samples from Minecraft [59], VirtualHome [60], Meta-World [61], and Franka Kitchen [61].Source: D Summary of Evaluation Datasets · p. 28