PaperlayerView exact revision on arXiv

Source-linked AI summary

VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use

Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, Ludwig Schmidt

arXiv:2308.06595v4cs.CLcs.AIcs.CV

TL;DR

Real-world multimodal instruction following lacks a high-quality benchmark that reliably reflects how people interact with chatbots. VisIT-Bench evaluates 70 instruction families through 592 image-grounded queries, instruction-conditioned captions, human-verified references, and GPT-4-based response ranking. Human-verified references are preferred to model outputs, and LLaMA-Adapter-v2 wins against the reference in only 27.4% of pairwise comparisons.

  • Problem

    Existing vision-language datasets lack a high-quality benchmark that reliably replicates real-world interactions with multimodal chatbots, leaving model effectiveness across many tasks not well understood.

    Source: §5 · p. 12

  • Method

    VisIT-Bench covers 70 instruction families in 592 image-contextual queries, pairing instruction-conditioned captions and human-verified references with GPT-4 comparison of model responses.

    Source: §1 · pp. 2–3, §6 · p. 12

  • Results

    Human-verified references are preferred significantly more than multimodal-model outputs, and LLaMA-Adapter-v2 wins against the reference in only 27.4% of pairwise comparisons.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    VisIT-Bench offers broad evaluation of multimodal chatbot capabilities and an Elo-based ranking system that correlates with human judgments.

    Source: §6 · p. 12

    The benchmark covers single-turn image-text instances, not multi-turn interactions or modalities such as audio and video.

    Source: §7 · p. 13

Abstract

from arXiv · show

We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for evaluation of instruction-following vision-language models for real-world use. Our starting point is curating 70 'instruction families' that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and creative generation. Following curation, our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. These descriptions surface instruction-specific factors, e.g., for an instruction asking about the accessibility of a storefront for wheelchair users, the instruction-conditioned caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment. We quantify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-following model wins against the GPT-4 reference in just 27% of the comparison. VisIT-Bench is dynamic to participate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available at visit-bench.github.io.

1 Introduction

VisIT-Bench addresses limited evaluation of open-ended, real-world vision-language queries with a benchmark spanning diverse instruction families. Human comparisons reveal a gap between model outputs and references, while GPT-4 pairwise judging provides an automated evaluator.

1 Introduction

  • Existing vision-language benchmarks assess standard tasks, while performance on broader, open-ended queries resembling real-world use remains less understood.Evaluations of such queries typically rely on informal and qualitative approaches.Source: §1 · p. 2
  • VisIT-Bench contains 592 image-contextual instructions, each paired with an instruction-conditioned caption and a human-verified reference.Many instructions ask for open-ended generation, such as writing a poem or suggesting what to bring when visiting a place.Source: §1 · p. 2
  • The benchmark spans 70 wish-list instruction families, including 25 families adapted from existing tasks and 10 multi-image families.The tasks range from recognition to complex reasoning and are reformatted for open-ended chatbot responses.Source: §1 · p. 2
  • The authors note that promising applications may be outside the selected task set and do not advocate deploying models in every covered case.They release VisIT-Bench data, code, and automatic metrics to facilitate future evaluation.Source: §1 · pp. 2–3
  • LLaMA-Adapter-v2 wins against the human-verified reference in 27.4% of cases, and those references are preferred significantly more than multimodal-model outputs.The benchmark also reports relative Elo skill estimates and an absolute reference win rate.Source: §1 · pp. 2–3
  • GPT-4 pairwise judging aligns best with human preferences among the compared metrics, with 94% agreement when all five annotators agree.The comparison includes BLEU-4, ROUGE-L, METEOR, CIDEr, and BERTScore.Source: §1 · p. 3

2 VisIT-Bench: A Real-World Inspired VL Instruction-Following Benchmark

VisIT-Bench combines broad image-text task coverage with a collection process that turns family-matched instructions into caption-supported, human-verified reference examples.

2 VisIT-Bench: A Real-World Inspired VL Instruction-Following Benchmark

  • The corpus has 592 instances and 1,159 public images, spanning 45 newly assembled and 25 reformatted instruction families, including 10 multi-image families.The benchmark prioritizes family coverage over the number of instances per task.Source: §2 · p. 4

2.1 Data Collection

  • The authors initially curated 70 instruction families; for the 45 not derived from existing tasks, they assigned names and identified an exemplifying image-instruction pair with a sample response.Source: §2.1 · p. 4
  • Crowdworkers create new instructions within a family by using an example image-instruction pair as a seed task.The instructions are paired with images, with annotators directed to use Creative Commons-licensed images from Openverse.Source: §2.1 · pp. 4–5
  • Twenty-five existing vision-language task families were reformatted as chatbot-style instructions while retaining each task’s goal and original images.The reformatted tasks are integrated into the same collection process as newly assembled families.Source: §2.1 · p. 5
  • Instruction-conditioned captions describe image details targeted to the query and are designed to let a text-only reader carry it out.The captions support GPT-4 reference generation and text-only automatic evaluation.Source: §2.1 · p. 5
  • GPT-4 generates candidate responses from the instruction and caption, and human annotators verify them; cases judged incorrect are discarded.Annotators identify whether an error lies in caption detail or in GPT-4’s response.Source: §2.1 · pp. 5–6

2.2 Data Collection Annotation and Results

  • The collection steps were run on MTurk, where workers first passed a five-to-ten-task qualification test.Workers received an hourly wage of $18.Source: §2.2 · p. 6

3 VisIT-Bench Analysis

VisIT-Bench analysis examines whether instruction-conditioned captions matter, the skills its instructions require, and the objects represented in its images.

3.1 Are instruction-conditioned captions necessary?

  • In image-and-instruction tasks, instruction-conditioned captions yielded correct outputs in 91% of cases, versus a 31% success rate with BLIP2 captions (Table 2), highlighting their importance for accurate instruction-following.Source: §3.1 · p. 6, Figure 5 · p. 7

3.2 What skills are required for VisIT-Bench?

  • Instructions span common requests such as answering questions and writing stories, plus a long tail requiring comprehension, commonsense, cross-modal understanding, and complex reasoning.Examples range from identifying objects and finding ingredients to emotion identification and paper folding.Source: §3.2 · p. 7, Figure 6 · p. 7

3.3 What is contained in VisIT-Bench images?

  • The most common detected object was “person” at approximately 900 detections, while chair and car each had approximately 100.The analysis concludes that VisIT-Bench requires accounting for a broad range of scenes and objects.Source: §3.3 · p. 7

4 Experiments

The experiments compare vision-language chatbots with human-verified references using human preferences, then assess GPT-4-based automatic evaluation against human judgments.

4.1 Models

  • The evaluation covered publicly accessible vision-language chatbots, with model-specific input handling for VisualChatGPT, OpenFlamingo, and Otter.VisualChatGPT received images and instructions sequentially, while OpenFlamingo and Otter received them in an interleaved format.Source: §4.1 · p. 8

4.2 Human Evaluation

  • Elo 1223 for the human-verified reference corresponded to an estimated 69% win rate over LLaVA (Elo 1085); LlamaAdapter-v2 had the highest win rate versus the reference.The reference had a higher Elo rating than all alternatives, while the paper notes that the varied models and tasks make performance factors difficult to isolate.Source: §4.2 · p. 9, Table 3 · p. 9

4.3 Automatic Evaluation and Leaderboard

  • For pairs with 5/5 annotator agreement, GPT4-no-ref reconstructed majority human judgments 93% of the time, versus 80% for BERTScore, 78% for METEOR, and 70% for ROUGE-L.The reference-backed GPT-4 metric performed comparably but slightly worse than the reference-free version.Source: Figure 9 · p. 10, §4.3 · p. 11
  • Among the seven systems with human ratings, the automatic reference-free GPT-4-based metric produced the same ordering as human evaluation (ρ = 1.0, p < .01).Source: §4.3 · p. 11
  • GPT-4 may prefer responses close to its own response distribution, and human-judged win rates against references are higher overall than automatically judged rates.This is presented as one plausible explanation for the discrepancy.Source: §4.3 · p. 11
  • No model performed best or worst across all single-image instruction families, so category-level results expose different model strengths and weaknesses.The analysis covers instruction families drawn from real-world use cases and existing datasets.Source: Figure 8 · p. 9, §4.3 · p. 11

5 Related Work

Related work describes the growth of image-text and instruction-following models, while noting that multimodal instruction-following lacks high-quality benchmarks that reproduce real-world chatbot interactions.

Multimodal Models for Image-Text Understanding; Instruction Following

  • The field includes image-text models that generate textual outputs from images and text, alongside instruction-following models trained to generalize across textual instructions.The paper distinguishes tool-based approaches from single pretrained models fine-tuned on multimodal vision-language data.Source: §5 · p. 12

Multimodal Models for Image-Text Understanding

  • Recent image-text models include LLaVA, mPLUG-Owl, InstructBLIP, Flamingo, PandaGPT, and GPT-4.The paper notes that GPT-4 reports multimodal capabilities but its multimodal variant had not been released.Source: §5 · p. 12

Instruction Following; Benchmarks for Machine Learning

  • Despite progress on vision-language datasets, high-quality benchmarks for multimodal instruction-following that reflect real-world chatbot interactions are lacking.The paper says this makes progress difficult to assess and increases vulnerability to evaluation pitfalls.Source: §5 · p. 12

Benchmarks for Machine Learning

  • High-quality evaluation datasets have served to reassess and accelerate progress across machine learning tasks.The paper cites computer-vision and language benchmarks as examples.Source: §5 · p. 12

6 Conclusion

VisIT-Bench offers a dynamic, broad evaluation of multimodal chatbots, with 70 instruction families, human-verified references, and an Elo-based ranking system correlated with human judgments. The experiments reveal a gap between models and human performance, and the authors release the benchmark resources to support evaluation.

6 Conclusion

  • VisIT-Bench evaluates multimodal chatbot capabilities across 70 instruction families, spanning recognition through complex reasoning.Its collection process centers potential real-world use cases.Source: §6 · p. 12
  • The benchmark provides human-verified reference outputs and an Elo-based chatbot ranking that correlates with human judgments.Source: §6 · p. 12
  • Experiments reveal a gap between model and human performance.Source: §6 · p. 12
  • The authors release data, code, and automatic metrics and hope the benchmark will quantify multimodal AI progress and shortcomings.Source: §6 · pp. 12–13

7 Limitations

VisIT-Bench does not cover every vision-language task, and its current study is limited to single-turn image-text instances. The test set is withheld to mitigate data-contamination risk, while data licensing follows the terms of original image sources where applicable.

7 Limitations

  • The benchmark omits some vision-language tasks, uses single-turn image-text instances, and has relatively few examples per category.The authors identify multi-turn interactions and other modalities, including audio and video, as possible extensions.Source: §7 · p. 13
  • The GPT-4-based metric correlates with human judgment but shows some evidence of preferring GPT-4-based generations more strongly than humans do.The authors note this may advantage models trained by distilling from GPT-4 outputs.Source: §7 · p. 13

A License and Intended Use

  • The benchmark materials are CC BY 4.0, but pre-existing images remain subject to their original licensing terms.Source: A License and Intended Use · p. 21
  • The complete test set is withheld to mitigate contamination risk; researchers can submit predictions for evaluation and may request test data under a non-contamination policy.Source: A License and Intended Use · p. 21

A Dataset Analysis

Figure 10 counts COCO objects in VisIT-Bench images on a log scale; person is the most frequent, appearing 875 times.

A Dataset Analysis

  • 875 person instances make person the most frequent COCO object in the VisIT-Bench images.Source: Figure 10 · p. 21
  • Figure 10 reports object counts on a log scale.Source: Figure 10 · p. 21
  • Object detection for the figure was performed with Yolov5-Large [40].Source: Figure 10 · p. 21

B Interfaces for Collecting Human Annotations

The annotation interfaces collect image–instruction pairs, dense captions, model verification, and model-rating feedback. The verification examples include both successful instruction following and failure due to incorrect reasoning.

B Interfaces for Collecting Human Annotations

  • The interfaces collect the image and corresponding instruction, then a dense caption for each image–instruction pair.The caption is collected after the image and instruction are supplied.Source: Figure 11 · p. 22, Figure 12 · p. 22
  • Model verification interfaces show both a GPT-4 response that follows the instruction and one that fails through incorrect reasoning.Source: Figure 13 · p. 22, Figure 14 · p. 23
  • A separate interface collects feedback on model ratings.Source: Figure 15 · p. 23

C Existing Datasets incorporated in VisIT-Bench

The appendix describes the existing datasets incorporated into VisIT-Bench and the Elo rating approach used to rank instruction-following vision-language models.

C Existing Datasets incorporated in VisIT-Bench

D Elo Rating

  • Elo ratings rank instruction-following vision-language models and can accommodate a set that grows with field advances.The method estimates head-to-head win probability from the two models’ absolute Elo ratings.Source: D Elo Rating · p. 25
  • The implementation sets hyperparameters governing the weight assigned to each head-to-head win or loss and uses FastChat’s open Elo implementation for LLMs.Source: D Elo Rating · p. 25

E GPT-4 Pairwise Evaluation Prompts

The evaluation uses GPT-4 prompts to compare model responses in reference-free and reference-backed settings. An additional query handles malformed or refused judgments; unresolved cases are counted as ties.

E GPT-4 Pairwise Evaluation Prompts

Loading 2308.06595v4…