PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?

Wenhao Wu, Huanjin Yao, Mengxi Zhang, Yuxin Song, Wanli Ouyang, Jingdong Wang

arXiv:2311.15732v2cs.CV

TL;DR

The paper evaluates GPT-4’s linguistic and visual capabilities in zero-shot visual recognition, a fundamental computer-vision task. Across 16 benchmarks spanning images, videos, and point clouds, GPT-generated descriptions improve average top-1 accuracy by 7%.

  • Problem

    How GPT-4’s linguistic and visual capabilities perform in zero-shot visual recognition remains to be evaluated.

    Source: §1 · p. 1

  • Method

    The study evaluates GPT-4’s language-generated category descriptions and visual recognition across 16 benchmarks covering images, videos, and point clouds.

    Source: §1 · pp. 1–2

  • Results

    7% average top-1 accuracy improvement across 16 datasets from GPT-4-generated descriptions, with gains across images, videos, and point clouds.

    Source: §4.3 · p. 5

  • Takeaways & Limitations

    The authors offer empirical evidence and experience intended to benefit future multimodal-model research.

    Source: §6 · p. 9

    The study covers fundamental visual-recognition tasks; other tasks, including object detection, remain to be quantitatively analyzed.

    Source: §6 · p. 9

Abstract

from arXiv · show

This paper does not present a novel method. Instead, it delves into an essential, yet must-know baseline in light of the latest advancements in Generative Artificial Intelligence (GenAI): the utilization of GPT-4 for visual understanding. Our study centers on the evaluation of GPT-4's linguistic and visual capabilities in zero-shot visual recognition tasks: Firstly, we explore the potential of its generated rich textual descriptions across various categories to enhance recognition performance without any training. Secondly, we evaluate GPT-4's visual proficiency in directly recognizing diverse visual content. We conducted extensive experiments to systematically evaluate GPT-4's performance across images, videos, and point clouds, using 16 benchmark datasets to measure top-1 and top-5 accuracy. Our findings show that GPT-4, enhanced with rich linguistic descriptions, significantly improves zero-shot recognition, offering an average top-1 accuracy increase of 7% across all datasets. GPT-4 excels in visual recognition, outshining OpenAI-CLIP's ViT-L and rivaling EVA-CLIP's ViT-E, particularly in video datasets HMDB-51 and UCF-101, where it leads by 22% and 9%, respectively. We hope this research contributes valuable data points and experience for future studies. We release our code at https://github.com/whwu95/GPT4Vis.

1. Introduction

The paper evaluates GPT-4’s linguistic and visual capabilities for zero-shot recognition across images, videos, and point clouds. It reports gains from richer descriptions and strong direct visual recognition against CLIP baselines.

1. Introduction

  • The study tests whether GPT-4’s detailed category descriptions improve zero-shot recognition by enriching CLIP’s category-name text representations.The descriptions aim to improve intra-class diversity and inter-class distinguishability.Source: §1 · p. 1
  • For direct visual recognition, GPT-4V receives visual content and candidate categories and ranks the categories by relevance.The paper refers to GPT-4 with visual capabilities as GPT-4V.Source: §1 · pp. 1–2
  • GPT-4-generated descriptions improve average top-1 accuracy by 7% across 16 datasets, while GPT-4’s visual recognition is nearly on par with EVA-CLIP ViT-E.GPT-4 surpasses ViT-E on UCF-101 and HMDB-51.Source: §1 · p. 2
  • The evaluation covers 16 benchmarks across images, videos, and point clouds, including texture, satellite, scene, facial-expression, fine-grained object, and action recognition.Point-cloud datasets can be projected onto multi-view depth maps for visual recognition.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors present their empirical evidence and experience as resources for future research.The introduction situates the evaluation after the emergence of GPT-4V and its API.Source: §1 · pp. 1–2

2. Related Works

Prior GPT-4V work explored diverse visual-language tasks, but much of it was qualitative or preliminary. This paper’s related-work discussion places its benchmark evaluation alongside earlier LLM-based approaches to enriching CLIP recognition.

2. Related Works

  • Most prior GPT-4V studies were qualitative initial explorations, leaving large-scale quantitative evaluation on established visual benchmarks as a gap.Earlier work included qualitative analysis of visual reasoning [41] and studies of tasks such as video understanding and OCR.Source: §2 · p. 2
  • Figure 2 contrasts using GPT-4-generated category descriptions with CLIP and prompting GPT-4V to rank candidate categories from visual content.The caption identifies the CLIP visual-language bridge as [27].Source: Figure 2 · p. 3
  • Earlier LLM-based approaches used GPT-3 to describe unseen classes [22] or ChatGPT to structure classes hierarchically, whereas this paper prompts GPT-4 for multi-sentence category descriptions.The paper examines this approach across image, video, and point-cloud recognition.Source: §2 · pp. 2–3

3. Methodology

The methodology converts video and point-cloud inputs into images, then evaluates two zero-shot routes: CLIP with GPT-4-generated descriptions and direct GPT-4V category ranking. The setup spans 16 datasets and reports top-1 and top-5 accuracy.

3.1. Data Processing

  • Videos use eight uniformly sampled frames, while point clouds are rendered as six frontal views at 30-degree intervals to create image inputs.The point-cloud rendering follows MVCNN.Source: §3.1 · p. 3, Figure 3 · p. 3

3.2. Exploration of Linguistic Capabilities

  • For the linguistic route, GPT-4 generates K category-description sentences, which CLIP encodes alongside the visual input for similarity-based category ranking.The method compares K text embeddings with the visual embedding, normalizes the similarity scores with Softmax, and averages them for each category.Source: §3.2 · p. 3
  • Table 1 provides statistics for the evaluated datasets.The supplied table caption does not specify individual statistics.Source: Table 1 · p. 4

3.3. Evaluation of Visual Capabilities

  • For direct visual evaluation, GPT-4V receives one or more visual samples and a category list, then ranks categories to produce top-5 predictions.Predictions are compared with ground truth to calculate top-1 and top-5 accuracy.Source: §3.3 · p. 4

4. Experiments

Across 16 benchmark datasets, GPT-4-generated descriptions improved CLIP zero-shot recognition, while GPT-4V’s direct visual performance varied across modalities and temporal demands.

4.1. Datasets

  • The evaluation covered 16 publicly available classification benchmarks across images, videos, and point clouds, using benchmark validation sets.CLIP zero-shot performance was included as a reference.Source: §4.1 · p. 4

4.2. Implementation Details

  • GPT-4 generated 20 descriptive sentences per category; GPT-4V received eight uniformly sampled video frames or six rendered point-cloud views.Figure 4 gives the British Shorthairs description example, while Figure 5 shows prompts for image, video, and point-cloud evaluation.Source: §4.2 · pp. 4–5, Figure 4 · p. 4, Figure 5 · p. 4

4.3. Gains from GPT Prompts

  • GPT-4 descriptions raised average top-1 accuracy by 7% across 16 datasets, with gains across images, videos, and point clouds.Where temporal modeling is crucial, CLIP’s frame averaging falls short and GPT prompts cannot compensate.Source: §4.3 · p. 5, Table 2 · p. 6

4.4. Zero-shot Visual Performance of GPT-4V

  • 83.7% vs. 74.8% on UCF101 and 63.2% vs. 41.5% on HMDB51: GPT-4V exceeded EVA ViT-E, but scored 4.6% top-1 on SSV1.GPT-4V’s average top-1 accuracy approached EVA ViT-E; it also performed on par with EVA ViT-E using six rendered point-cloud images and reached 68.7% vs. 31.0% on RAF-DB.Source: §4.4 · pp. 5, 7, Table 2 · p. 6

4.5. Ablation Studies on GPT Prompts

  • GPT Prompts outperformed the category-name baseline on 14 datasets; combining them with hand-crafted prompts improved 11, and 20 EuroSAT sentences became the default after gains plateaued.On EuroSAT, three sentences reached 42.9% versus 40.2% for category names, five improved further, and 20 added little; GPT Prompts were below or roughly on par with hand-crafted prompts in six datasets.Source: §4.5 · pp. 7–8, Table 4 · p. 8

5. Special Cases and Discussion on GPT-4V

The discussion records testing and prompting issues that can affect GPT-4V evaluations, alongside refusals and API cost. The authors recommend single-sample testing to improve result accuracy.

Batch Testing vs. Single Testing

  • Batch testing can misassign, duplicate, or omit predictions, and samples may interfere; the authors recommend processing one sample per API request.They initially used batches under a 100-request daily limit, then updated datasets with single testing after that limit was removed for tier 5 accounts.Source: §5 · p. 8

Predict categories beyond the given list

  • GPT-4V sometimes predicts categories outside the supplied list, so the authors excluded those predictions after prompt-based restrictions proved ineffective.Source: §5 · p. 8

Prediction based on image names

  • On DTD, top-1 accuracy was 98% with original filenames but 57% after anonymization, showing that filename hints could outweigh visual content.The authors hashed sample names to reduce reliance on filename cues.Source: §5 · p. 8

Safety system in GPT-4V

  • GPT-4V refused some inputs under its safety system, including examples from ImageNet-1K and Kinetics-400.Source: Figure 9 · p. 8, §5 · p. 9

GPT-4V API Costs

  • One GPT-4V API testing round across all datasets was estimated to cost about $4000.Source: §5 · p. 9

6. Conclusion and Limitation

The paper evaluates GPT-4’s linguistic and visual capabilities for zero-shot recognition across three modalities and 16 benchmarks. Its conclusions are limited to recognition, GPT-4 alone, and straightforward prompting.

Conclusion

  • The study evaluates GPT-4’s linguistic and visual capabilities in zero-shot visual recognition across images, videos, and point clouds, using 16 benchmarks.Source: §6 · p. 9

Limitations

  • The evaluation covers fundamental visual recognition, not other tasks such as object detection needed to assess broader visual-analysis capabilities.Source: §6 · p. 9
  • The study evaluates GPT-4 alone, leaving comparisons with other multimodal models for future work.Source: §6 · p. 9
  • The straightforward prompting approach may face adverse effects from the input tokens required by overly long category lists.Source: §6 · p. 9
Loading 2311.15732v2…