PaperlayerView exact revision on arXiv

Source-linked AI summary

ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories

Heming Xia, Qingxiu Dong, Lei Li, Jingjing Xu, Tianyu Liu, Ziwei Qin, Zhifang Sui

arXiv:2305.15028v2cs.CLcs.AIcs.CV

TL;DR

It remains unclear how well LLMs and VaLMs master visual commonsense, despite demand for visual knowledge in general-purpose language-model interfaces. The paper introduces ImageNetVC, a human-annotated benchmark for zero- and few-shot evaluation across 1,000 ImageNet categories. Evaluations report varying degrees of visual commonsense across models and identify factors associated with its acquisition and enhancement.

  • Problem

    Systematic evidence is lacking on how well widely used LLMs and their variants handle visual understanding, including shared visual concepts, and prior evaluations have textual-bias and model-coverage limitations.

    Source: §1 · p. 1

  • Method

    The authors construct ImageNetVC, a human-annotated dataset of visual-commonsense QA pairs across 1,000 ImageNet categories, and evaluate LLMs and VaLMs using textual and visual inputs.

    Source: §1 · pp. 1–2, §3.3 · p. 3, §4.1 · p. 5

  • Results

    ImageNetVC evaluations show varying degrees of visual commonsense across models; the authors report that it remains challenging for LLMs and identify emergent ability and in-context-learning effects.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The study provides a foundation for future work on visual-commonsense knowledge and its applications.

    Source: Limitations · p. 9

    The authors flag the high cost of region-based human annotation as a limitation for ImageNetVC.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recently, Large Language Models (LLMs) have been serving as general-purpose interfaces, posing a significant demand for comprehensive visual knowledge. However, it remains unclear how well current LLMs and their visually augmented counterparts (VaLMs) can master visual commonsense knowledge. To investigate this, we propose ImageNetVC, a human-annotated dataset specifically designed for zero- and few-shot visual commonsense evaluation across 1,000 ImageNet categories. Utilizing ImageNetVC, we benchmark the fundamental visual commonsense knowledge of both unimodal LLMs and VaLMs. Furthermore, we analyze the factors affecting the visual commonsense knowledge of large-scale models, providing insights into the development of language models enriched with visual commonsense knowledge. Our code and dataset are available at https://github.com/hemingkx/ImageNetVC.

1 Introduction

The paper introduces ImageNetVC to address limited systematic evaluation of visual commonsense in LLMs and VaLMs. Its evaluations report that the task remains challenging for LLMs and identify model-scale and in-context-learning findings.

1 Introduction

  • The paper frames visual commonsense as shared human knowledge about generic visual concepts, while noting that systematic evaluation across widely used LLMs and variants is lacking.Prior evaluation limitations include textual bias in mined attributes and limited coverage of model families.Source: §1 · p. 1
  • ImageNetVC contains 4,076 human-annotated QA pairs across 1,000 ImageNet categories and diverse visual attributes, and supports evaluation of LLMs and VaLMs with images.Human visual perception is used to identify shared attributes across relevant images, avoiding textual bias.Source: §1 · p. 2
  • Template-based datasets can inflate and destabilize visual-commonsense evaluation, while ImageNetVC provides evidence that the task remains challenging for LLMs.The authors also identify emergent visual commonsense and in-context learning effects.Source: §1 · p. 2
  • The authors suggest 1.3B may be a threshold for unimodal LLMs to exhibit emergent visual commonsense on the component.This is presented as a potential threshold, not a universal cutoff.Source: §1 · p. 2
  • In-context learning enhances visual-commonsense understanding for both LLMs and VaLMs, while reducing prompt-related variance and calibrating model confidence.Source: §1 · p. 2

2 Related Work

Prior visual-commonsense datasets often target specific attributes, while ViComTe addresses general evaluation using templates. ImageNetVC is introduced as a human-annotated QA dataset for evaluating both LLMs and VaLMs with multiple image sources.

2 Related Work

  • Text-only LLMs lack visual data during pretraining, whereas VaLMs integrate visual information into LLMs to enhance visual understanding.Source: §2 · p. 2
  • Earlier datasets address specific attributes, while ViComTe uses subject–object templates but relies on text alone and has data-quality and textual-bias concerns.Source: §2 · p. 2
  • ImageNetVC provides 4K natural-language QA pairs across visual attributes and supports LLM and VaLM evaluation with multiple image sources.Source: §2 · p. 2

3 IMAGENETVC

ImageNetVC is built through a multi-step crowdsourcing and review process, then evaluated against existing visual-commonsense data. The evaluation reports lower prompt variance than ViComTe and stronger human-assessed diversity, difficulty, and factuality.

3 IMAGENETVC

  • The dataset was assembled through three stages: annotator training, commonsense annotation, and cross-check examination.The paper presents this as a crowdsourcing pipeline beginning with ImageNet.Source: §3 · p. 2

3.1 Image Source

  • ImageNet was chosen as a diverse, representative source of commonly used real-life objects, with a unified image format intended to ease annotation.The study used the ImageNet ILSVRC 2012 subset, comprising 1.4 million images across 1,000 categories.Source: §3.1 · pp. 2–3

3.2 Prerequisite: Annotator Training

  • Annotators were recruited through Amazon Mechanical Turk, trained with guidelines and a quiz, and hired only if they scored above 95%.The authors report receiving over 500 applications from candidates with bachelor’s degrees or higher.Source: §3.2 · p. 3

3.3 Phase 1: Commonsense Annotation

  • Annotators formed category-level questions and answers from shared visual features across 50 randomly sampled images, rather than details of individual images.Features could describe an entire object or a specific region, and examples and rejection guidelines supported the task.Source: §3.3 · p. 3

3.4 Phase 2: Cross-Check Examination

  • Cross-checking first required unanimous approval from three category-level examiners, then majority approval from three examiners reviewing randomly assigned individual pairs.The second stage also classified accepted pairs into color, shape, material, component, or other.Source: §3.4 · pp. 3–4

3.5 Dataset Evaluation

  • ImageNetVC reduced prompt variance versus ViComTe; human raters found it matched or outperformed ViComTe on diversity and difficulty in >84% of cases and exceeded ChatGPT data in factuality in >81%.The comparisons were based on prompt distributions and human judgments across the named dimensions.Source: Figure 3 · p. 4, Figure 4 · p. 4, §3.5 · p. 4

3.4 Phase 2: Cross-Check Examination

  • The 60-day collection produced 4,076 QA items across 1,000 ImageNet categories and five tasks: color, shape, material, component, and others.The paper reports these as the final dataset’s size and task groupings.Source: §3.4 · p. 4, Table 2 · p. 4

3.5 Dataset Evaluation

  • GPT-Neo-1.3B scored 40% accuracy on ViComTe but 28% on ImageNetVC for color with a selected prompt; human assessment also favored ImageNetVC’s factual correctness over ChatGPT-generated data.The authors describe ImageNetVC as more challenging than ViComTe for region-based questions; Figure 4 reports human comparisons across diversity, difficulty, and factuality.Source: §3.5 · p. 4, Figure 4 · p. 4

4 Experiments

The experiments evaluate text-only LLMs and VaLMs on ImageNetVC in zero- and few-shot settings. Results show gains from in-context learning and visual augmentation, with improvements varying across subtasks.

4 Experiments

  • The experiments cover text-only GPT, OPT, LLaMA, Falcon, and Pythia models, plus the VaLMs Z-LaVI, BLIP-2, and MAGMA.The VaLMs are built mainly on frozen LLMs and use different mechanisms to integrate visual information.Source: §4 · p. 5

4.1.1 LLM Evaluation

  • For LLM evaluation, QA pairs become cloze prompts; few-shot examples are prepended, and answer-token scores are normalized to mitigate prompt-format and example bias.The answer is scored from its probability conditioned on preceding tokens, then calibrated following Zhao et al. (2021).Source: §4.1 · p. 5, §4.1.1 · p. 5

4.1.2 VaLM Evaluation

  • VaLM evaluation uses retrieved web or synthetic images ranked by CLIP; prefix-based models prepend visual embeddings, while ensemble models combine LLM and CLIP answer distributions.For prefix-based models, predictions across multiple images are averaged.Source: §4.1.2 · pp. 5–6

4.2 Experimental Details

  • For each QA pair, the researchers collect 100 web images and 100 synthetic images, rank all 200 with CLIP, and retain the top 10 as image sources.Synthetic images are generated with Stable Diffusion.Source: §4.2 · p. 6

4.3 Main Results

  • Falcon and LLaMA outperform OPT and GPT across subsets in both settings, except shape; LLaMA reaches 41% zero-shot color accuracy and Falcon reaches 76% few-shot component accuracy.LLaMA's color score exceeds GPT-J by 13%, while Falcon's component score is 14% above OPT.Source: Figure 5 · p. 6, §4.3 · p. 7
  • With ICL, LLaMA-7B averages 62% across five subtasks, a 12% improvement over zero-shot, and LLaMA's color-subset performance varies less across prompts.Figure 6 shows the performance distribution on the color subset.Source: Figure 5 · p. 6, §4.3 · p. 7, Figure 6 · p. 7
  • BLIP-2 improves over OPT by 17% on color and 15% on material, while BLIP-2 and MAGMA gain only 0.5% on shape and Z-LaVI drops.These results indicate that VaLM gains are limited on some subtasks.Source: Figure 5 · p. 6, §4.3 · p. 7
  • MAGMA with ICL improves consistently across subsets over itself and the few-shot results of its frozen LLM backbone.Z-LaVI and BLIP-2 support only zero-shot evaluation in these experiments.Source: Figure 5 · p. 6, §4.3 · p. 7

5 Analysis

The analysis finds that visual-commonsense performance varies with model scale and improves with in-context learning, while VaLM results depend on image source and number. It also notes a pretraining-corpus boundary on generalizing the LLM findings.

Model scale

  • Models above 1.3B show gradual component-task improvements; OPT-30B reaches 59% zero-shot and 66% few-shot average accuracy.Models below 1.3B perform near random guessing at approximately 50% accuracy.Source: §5 · pp. 7–8

Calibration

  • For LLaMA-7B on the component subset, ICL raises the confidence–accuracy correlation from r = 0.57 to r = 0.98.The analysis measures calibration by whether confidence reflects the probability that a prediction is correct.Source: §5 · p. 8, Figure 8 · p. 8

Image sources

  • BLIP-2 improves with extra image sources such as SEARCH, and CLIP-ranked images yield the best performance.The authors use ranked images as the default source for their main experiments and analysis.Source: §5 · p. 8

Scope

  • Because OPT and Pythia both use the Pile for pretraining, the authors caution that these findings may not generalize to other LLMs.Source: §5 · p. 8

Image number

  • Performance rises as top-ranked images increase from 1 to 10, with only marginal improvement beyond 10.Figure 9 reports average IMAGENETVC performance for BLIP-2 (OPT-2.7B) across image counts.Source: §5 · p. 8, Figure 9 · p. 8

6 Visual Commonsense in Other Models

The paper evaluates multimodal models and ChatGPT on IMAGENETVC through open-ended generation. ChatGPT with ICL has the highest average accuracy among the compared models, but remains below human performance.

Evaluation

  • The evaluation includes 12-in-1 and BLIP, both fine-tuned on VQAv2, and ChatGPT (gpt-3.5-turbo).Source: §6 · p. 8

Results

  • ChatGPT with ICL scores 75.8% average accuracy, compared with 93.5% for humans.The comparison reports Top-1 accuracy from open-ended generation.Source: §6 · p. 9, Table 4 · p. 9
  • The VQAv2-finetuned multimodal models perform strongly, especially on the color sub-task, despite BLIP’s 583M parameter scale.Source: §6 · p. 9

7 Conclusion

The paper introduces a human-annotated dataset for evaluating visual commonsense with textual and visual inputs, and uses it to study LLMs and VaLMs. The experiments describe variation across models and factors associated with acquiring or enhancing this knowledge.

Conclusion

  • IMAGENETVC is a human-annotated dataset for evaluating visual commonsense with textual and visual inputs.Source: §7 · p. 9
  • Experiments use IMAGENETVC to evaluate visual commonsense in unimodal LLMs and VaLMs.Source: §7 · p. 9
  • The results show varying degrees of visual commonsense knowledge across models and examine factors contributing to its acquisition and enhancement.Source: §7 · p. 9

Limitations

The appendices detail dataset annotation, answer-set construction, and qualitative examples, while the limitations bound the evaluation’s categories and settings. The annotation process uses human visual judgments and quality checks.

Limitations

  • IMAGENETVC covers 1,000 ImageNet categories; zero- and few-shot evaluation of text-pretrained models leaves fine-tuning effects unclear and may not capture real-world settings with prior knowledge.The authors also say that explaining how models acquire visual-commonsense knowledge remains challenging.Source: Limitations · p. 9

A.1 Annotation Guidelines

  • Annotators inspect 20–50 category images, identify a visible shared feature, and write a category-specific question and answer that accords with commonsense.Features may describe the whole object or a region; submissions must meet formatting and content rules and avoid duplicate or nonvisual questions.Source: §A.1 · p. 13

A.2 Annotation UI

  • The annotation interface combines task instructions, a prompt-refinement pipeline, and examples of rejected cases.Source: §A.2 · p. 13

A.3 Hierarchical Supervision

  • Quality control includes examiner cross-checks, warnings and removal for repeated rejections, and random checks by five authors.Source: §A.3 · p. 13

A.4 Payment Policy

  • Workers receive different rates for accepted and rejected samples, while cross-checkers are paid per sample; average hourly pay is approximately twice the local minimum wage.Phase 1 rates are $0.50 for accepted samples and $0.10 for rejected samples; Phase 2 cross-checking pays $0.30 per sample.Source: §A.4 · pp. 13–14

B.1 Details of the Others subset

  • The Others subset collects QA pairs outside color, shape, material, and component; its examples include length comparison (21%), relative size (20%), living environment (16%), and counting (12%).The authors describe this subset as more diverse and challenging.Source: §B.1 · p. 14, Figure 10 · p. 14

B.2 Answer Set

  • Because open-ended answers are less controllable, the main experiments use constrained decoding and divide Others into yes/no, number, and other-answer test sets.The paper provides the possible answer set for each subset.Source: §B.2 · p. 14, Table 5 · p. 14

B.3 More Qualitative Examples

  • Qualitative examples show BLIP-2 outperforming OPT-7B with visual input, while ChatGPT can favor common object attributes over region-specific properties.The authors hypothesize that this tendency may reflect frequent co-occurrence in pretraining text; even leading models struggle with some cases, including a flamingo’s beak-tip color.Source: §B.3 · p. 14, Table 6 · p. 15

C Experimental Details

The experimental details specify prompt, human-assessment, and implementation procedures for evaluating LLMs and VaLMs on ImageNetVC. VaLMs differ in implementation, so the paper does not make direct head-to-head comparisons among them.

C.1 Multiple Prompts

  • The evaluation uses prompts from Table 7 without tuning them for individual ImageNetVC subsets, while ViComTe data-quality evaluation uses its original prompts.The human assessment proceeds in multiple rounds based on the dataset with fewer test samples.Source: §C.1 · p. 15, §C.2 · p. 15

C.2 Details of Human Assessment

  • In human assessment, annotators compared random pairs of 32 instances on diversity, difficulty, and factuality; ImageNetVC matched or exceeded ViComTe on diversity in 86% of cases.The evaluation used 556 comparison pairs for ImageNetVC versus ViComTe and 510 for ImageNetVC versus ChatGPT-generated data.Source: §C.2 · pp. 15–16

C.4 Model Details

  • Table 8 reports mean Top-1 accuracy over five prompts for LLMs across model families and scales, while Table 10 reports VaLM mean accuracy over five prompts and marks performance improvements and drops.The VaLM comparison is with each model’s frozen LLM backbone.Source: Table 8 · p. 16, Table 10 · p. 17

C.3 Model Implementation

  • LLMs use the Hugging Face API, and VaLM evaluations use released checkpoints rather than task-specific fine-tuned models to assess zero- and few-shot capability.VaLM hyperparameters follow their original papers; experiments ran on six NVIDIA A40 GPUs.Source: §C.3 · p. 16

C.4 Model Details

  • Z-LaVI ensembles LLM and CLIP solutions, MAGMA adds adapter layers and visual prompts, and BLIP-2 uses a Querying Transformer; their differing implementations preclude direct VaLM comparisons.Table 9 lists extra parameters, visual encoder architectures, and pretraining image counts for the evaluated VaLMs.Source: §C.4 · pp. 16–17, Table 9 · p. 16

D Details of Main Experimental Results

This section directs readers to detailed main-result tables for LLMs and VaLMs. The supplied passages describe the tables’ scope and comparison setup, but do not state outcome values.

D Details of Main Experimental Results

  • Tables 8 and 10 provide the detailed main experimental results for LLMs and VaLMs, respectively.The supplied passages do not include the tables’ result values.Source: D Details of Main Experimental Results · p. 17
  • LLM results cover zero- and few-shot evaluation of OPT, GPT, Pythia, Falcon, and LLaMA across model scales, while VaLM results compare Z-LaVI, BLIP-2, and MAGMA with frozen LLM backbones.The supplied passages do not establish which models perform best.Source: D Details of Main Experimental Results · p. 17

E Evaluation Details of Other Models

The section describes evaluation procedures for multimodal and RLHF models, alongside the annotation interface and its checks for ImageNetVC prompts.

E Evaluation Details of Other Models

  • For multimodal models such as BLIP, evaluation uses open-ended generation for each QA-image pair, with the top 10 ranked images and majority prediction determining the final answer.The settings follow those used in VQAv2.Source: E Evaluation Details of Other Models · p. 17
  • ChatGPT is evaluated with constrained prompts, and its top-1 accuracy is computed automatically.The prompts are presented in Table 11.Source: E Evaluation Details of Other Models · p. 17
  • The annotation interface presents 20–50 category images and guides annotators to create simple QA pairs about common visual features, checking commonsense conformity and avoiding pre-written QAs.Annotations target visual features rather than commonsense or non-visual attributes.Source: Figure 11 · p. 18

Related papers

Loading 2305.15028v2…