PaperlayerView exact revision on arXiv

Source-linked AI summary

The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)

Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, Lijuan Wang

arXiv:2309.17421v2cs.CVcs.CL

TL;DR

The report examines how GPT-4V processes multimodal inputs and performs across diverse tasks through curated qualitative probes. It finds broad multimodal capabilities, including arbitrary mixtures of inputs and impressive performance across many domains, motivating further research and applications.

  • Problem

    Existing LMM approaches provide limited evidence about broad multimodal capabilities, supported inputs, working modes, and effective prompting across domains.

    Source: §1.1 · p. 8

  • Method

    The report curates qualitative samples spanning diverse domains and tasks to probe GPT-4V’s inputs, working modes, capabilities, and prompting techniques.

    Source: §1.1 · p. 8

  • Results

    GPT-4V processes arbitrarily mixed images, sub-images, text, scene text, and visual pointers, while showing impressive human-level capabilities across many tested domains.

    Source: §1.1 · p. 8

  • Takeaways & Limitations

    The findings provide a reference for exploring additional GPT-4V uses, deepening understanding of LMMs, and guiding future research on more powerful multimodal systems.

    Source: §1.1 · p. 8, §11.1 · p. 156

    The report presents qualitative examples rather than comprehensive quantitative benchmarks, and some demonstrated capabilities may require carefully designed prompts and may not work consistently.

    Source: §1.2 · p. 9

Abstract

from arXiv · show

Large multimodal models (LMMs) extend large language models (LLMs) with multi-sensory skills, such as visual understanding, to achieve stronger generic intelligence. In this paper, we analyze the latest model, GPT-4V(ision), to deepen the understanding of LMMs. The analysis focuses on the intriguing tasks that GPT-4V can perform, containing test samples to probe the quality and genericity of GPT-4V's capabilities, its supported inputs and working modes, and the effective ways to prompt the model. In our approach to exploring GPT-4V, we curate and organize a collection of carefully designed qualitative samples spanning a variety of domains and tasks. Observations from these samples demonstrate that GPT-4V's unprecedented ability in processing arbitrarily interleaved multimodal inputs and the genericity of its capabilities together make GPT-4V a powerful multimodal generalist system. Furthermore, GPT-4V's unique capability of understanding visual markers drawn on input images can give rise to new human-computer interaction methods such as visual referring prompting. We conclude the report with in-depth discussions on the emerging application scenarios and the future research directions for GPT-4V-based systems. We hope that this preliminary exploration will inspire future research on the next-generation multimodal task formulation, new ways to exploit and enhance LMMs to solve real-world problems, and gaining better understanding of multimodal foundation models. Finally, we acknowledge that the model under our study is solely the product of OpenAI's innovative work, and they should be fully credited for its development. Please see the GPT-4V contributions paper for the authorship and credit attribution: https://cdn.openai.com/contributions/gpt-4v.pdf

1 Introduction

The introduction frames GPT-4V as a state-of-the-art vision-enabled LMM whose multimodal abilities remain insufficiently understood, and outlines a primarily qualitative exploration of its inputs, capabilities, prompting methods, and future applications. It also explains the report’s evaluation choices, sample-selection safeguards, default zero-shot setting, and organization around four guiding questions.

1.1 Motivation and Overview

  • GPT-4V extends state-of-the-art language modeling with vision, supporting arbitrary mixtures of images, sub-images, text, scene text, and visual pointers while enabling instruction following, chain-of-thought, and in-context learning.The report investigates these supported inputs and working modes because existing LMMs’ smaller model and data scales leave their emergent multimodal abilities unclear.Source: §1.1 · p. 8
  • The exploration samples many domains and tasks, where GPT-4V demonstrates impressive human-level capabilities, and introduces visual referring prompting by editing images with visual pointers or scene text.Visual referring prompting can be combined with image and text prompts to provide a nuanced interface for instructions and demonstrations.Source: §1.1 · p. 8

1.2 Our Approach in Exploring GPT-4V

  • Because existing benchmarks may be mismatched to LMM outputs or compromised by unknown GPT-4V pretraining data, the report emphasizes qualitative examples rather than comprehensive quantitative benchmarking.The selected examples preview potential capabilities and use cases, but some require carefully designed prompts and may not work consistently across samples.Source: §1.2 · p. 9
  • The sample design uses original queries, images unlikely to have appeared in training, rationale questions, and primarily zero-shot instructions to reduce memorization, leakage, and in-context-example effects.The report notes that few-shot examples can improve reliability but do not consistently create new capabilities, so zero-shot is the default presentation mode.Source: §1.2 · p. 9

1.3 How to Read this Report?

  • The report is organized around four questions covering GPT-4V’s inputs and working modes, capabilities across domains, effective prompting, and promising future directions.Later sections cover visual referring prompting, temporal and abstract reasoning, emotion understanding, novel use cases, and systems such as multimodal plugins, chains, self-reflection, self-consistency, and retrieval-augmented LMMs.Source: §1.3 · pp. 9–10

2 GPT-4V’s Input Modes

GPT-4V supports text-only, single image-text, and interleaved multimodal inputs. These modes span language and coding, vision-language tasks, and flexible multi-image reasoning and prompting.

2 GPT-4V’s Input Modes

  • GPT-4V supports text-only, single image-text, and interleaved image-text inputs, including cases with only text, one image, or multiple images.These input modes define the representative use cases discussed in the section.Source: §2 · p. 10

2.1 Text-only Inputs

  • With text-only inputs, GPT-4V functions as an effective unimodal language model for diverse language and coding tasks.The GPT-4 technical report provides comprehensive analysis and comparison with GPT-4 without vision.Source: §2.1 · p. 10

2.2 Single Image-text Pair

  • With a single image or image-text pair, GPT-4V performs vision and vision-language tasks including recognition, localization, captioning, and visual question answering.Text can serve as an instruction, such as requesting a description, or as a query, such as a visual question.Source: §2.2 · p. 10

2.3 Interleaved Image-text Inputs

  • Interleaved image-text inputs support visually centric, text-centric, and balanced mixtures, enabling multi-image extraction, cross-input association, and flexible applications.GPT-4V can aggregate receipt taxes, find menu prices, count beers, and calculate total costs; this mode also supports few-shot learning and test-time prompting.Source: §2.3 · p. 11, Figure 1 · p. 11

3 GPT-4V’s Working Modes and Prompting Techniques

GPT-4V supports flexible text, visual, and multimodal prompting modes that shape outputs, define new tasks, and adapt to varied inputs. Its demonstrations show that visual referring prompts and sufficient in-context examples can solve tasks that zero-shot or one-shot prompting fails to handle.

3.1 Following Text Instructions

  • Following text instructions lets GPT-4V customize outputs, perform challenging tasks, and adapt to unseen vision-language applications.Text instructions can specify output constraints or intermediate reasoning steps, enabling task definition without parameter updates.Source: §3.1 · p. 15
  • Constrained prompting preserves a requested JSON format even when GPT-4V makes extraction errors from driver’s licenses.The method is used to structure outputs for application scenarios despite imperfect information extraction.Source: §3.1 · p. 15
  • In apple counting, the prompt to count row by row and explicitly ensure correctness produces the correct per-row counts and total, unlike several weaker prompts.The basic prompt yields 12 instead of 11; step-by-step variants remain incorrect, while the expert, success-oriented prompt succeeds.Source: Figure 4 · p. 14, §3.1 · p. 15

3.2 Visual Pointing and Visual Referring Prompting

  • Visual referring prompting uses drawn pointers, handwriting, or other pixel edits to identify regions and specify tasks while retaining global image context.It supports grounded descriptions, associating pointed objects with scene-text indices, and answering questions placed near queried edges or angles.Source: §3.2 · pp. 15–16, Figure 6 · p. 17

3.3 Visual + Text Prompting

  • GPT-4V can process arbitrary mixtures of images, sub-images, texts, scene texts, and visual pointers as instructions, examples, or queries.This flexible integration avoids the stricter formatting and image-count constraints described for existing multimodal models.Source: §3.3 · p. 16, Figure 7 · p. 18
  • Multimodal example-grounded instructions attach task explanations to corresponding visual demonstrations, making examples more effective than text-only instructions or format-matched few-shot examples.In the illustrated pattern task, grounding “finding the pattern in the first column” onto circled patterns and sub-figures simplifies learning and improves performance.Source: §3.3 · p. 16

3.4 In-context Few-shot Learning

  • Two in-context examples enable accurate speed-meter and line-plot reasoning after zero-shot and one-shot attempts fail.GPT-4V predicts approximately 9 mph with two examples and correctly identifies 2022 for the June gas-price question; the report limits few-shot use to avoid leakage and leaves quantitative gains for future work.Source: §3.4 · p. 19, Figure 10 · p. 22, Figure 13 · p. 25

4 Vision-Language Capability

This section explores GPT-4V’s broad vision-language capabilities across image description, spatial and multimodal reasoning, visual information extraction, tables, charts, documents, languages, cultural contexts, and visual-to-code conversion. It also examines performance limitations arising from occlusion, clutter, prompting, missed details, complex equations, and exact figure replication.

4.1 Image Description on Diverse Domains

  • GPT-4V accurately describes diverse visual content, recognizing celebrities, landmarks, dishes, medical conditions, logos, scenes, and misleading-image contents.It can identify eight celebrities, describe a 2023 G7 Summit speech, recognize ingredients and garnishes, interpret medical images, and describe novel or partially occluded logos.Source: §4.1 · pp. 26–27

4.2 Object Localization, Counting, and Dense Captioning

  • GPT-4V understands spatial relationships and supports dense captioning, while object counting succeeds on simple scenes but errors under occlusion or clutter.It generates textual bounding-box coordinates, though localization is inaccurate in complex scenes and improves when backgrounds are simpler; one example counts 12 people instead of 11.Source: §4.2 · p. 38

4.3 Multimodal Knowledge and Commonsense

  • GPT-4V combines visual and textual information to explain memes, answer science questions, produce tutorial-style explanations, and infer actions or plausible scenarios from visual clues.Bounding boxes can serve as visual prompts for identifying people and actions, while specific roles such as “teacher” or “detective” elicit structured explanations and hypotheses.Source: §4.3 · p. 43

4.4 Scene Text, Table, Chart, and Document Reasoning

  • GPT-4V extracts and reasons over scene text, visual mathematics, charts, tables, and documents, including cross-page technical reports.It solves a right-triangle problem step by step, answers chart and table questions, reconstructs exam tables in Markdown, and captures multi-page reports’ main ideas while sometimes missing implementation details.Source: §4.4 · p. 49
  • GPT-4V understands tables, charts, flow charts, and documents, answering questions, describing processes, translating flow charts into Python, and summarizing technical reports.It correctly identifies table-based quantities, including 113 families earning more than Rs. 13000 and owning more than 2 cars, and extracts detailed document content.Source: Figure 35 · p. 52, Figure 36 · p. 53, Figure 37 · p. 54
  • GPT-4V analyzes research-paper documents by identifying their topic, methodology, dataset, features, results, and possible future extensions.For the example paper, it reports a dataset of 1196 computer science research papers and describes supervised learning based on visual features such as figures, equations, and references.Source: §4.4 · p. 57

4.5 Multilingual Multimodal Understanding

  • GPT-4V generates and understands multilingual image descriptions, recognizes and translates scene text, summarizes Catalan webpages in 20 languages, and captures cultural nuances.It follows prompts in languages including Chinese, French, Czech, and Spanish, produces requested output languages, and generates reasonable multilingual descriptions of wedding images [147].Source: §4.5 · p. 58

4.6 Coding Capability with Vision

  • GPT-4V converts handwritten equations and image tables into LaTeX or Markdown, but complex equations require decomposition into shorter components.It handles shorter equations effectively, whereas longer equations can fail unless broken down.Source: §4.6 · p. 63, Figure 45 · p. 63, Figure 46 · p. 64
  • GPT-4V writes Python, TikZ, and SVG code that roughly reproduces input figures, producing layouts that are editable but not exact matches.The generated code can be modified to meet specific needs, although the rendered output is not an exact reproduction.Source: §4.6 · p. 63, Figure 47 · p. 65

5 Interaction with Humans: Visual Referring Prompting

GPT-4V understands visual pointers, coordinates, and scene text, motivating visual referring prompting as a pixel-space interaction channel complementary to text prompts. Although coordinate-based pointing remains coarse, human- and model-generated visual pointers support grounded description, interpretation, and iterative visual reasoning.

5.2 Visual Referring Prompting

  • Visual referring prompting provides a natural human-computer interaction channel because humans and models can both generate and understand its visual pointers.The paper highlights applications spanning computer and mobile GUIs, documents, slides, and other interaction settings.Source: §5 · p. 66, Figure 51 · p. 71

5.1 Understand Pointing Inputs

  • GPT-4V understands overlaid circles, boxes, and hand drawings, enabling grounded descriptions that preserve both the selected region and global image context.This addresses limitations of conventional localized descriptions and cropped or masked dense captioning methods [138].Source: §5.1 · p. 66, Figure 48 · p. 67
  • GPT-4V understands numerical region coordinates without box-token finetuning, but overlaid visual pointers are more reliable and spatially precise in the tested prompts.The model may mention surrounding objects outside the specified region, unlike the intended coordinate-grounded description; prior methods used extra box-token finetuning.Source: §5.1 · p. 66, Figure 49 · p. 68

5.2 Visual Referring Prompting

  • Visual referring prompting edits image pixels with pointers or scene text, complementing text prompts through object indexing, image-grounded questions, document highlighting, and visual pattern specification.The method offers a more nuanced interaction channel for grounded visual dialogue and complex documents or charts.Source: §5.2 · p. 69, Figure 50 · p. 70

5.3 Generate Pointing Outputs

  • GPT-4V can generate coarse coordinate-based pointers for text- or image-referred objects, but struggles to produce closely fitted bounding boxes.Example-grounded instructions improve its understanding of coordinate definitions and the resulting pointing outputs.Source: §5.3 · p. 69, Figure 52 · p. 72
  • Even imperfect generated pointers support interpretation and grounded description, allowing GPT-4V to understand its own outputs and continue reasoning from them.This creates a closed-loop interaction process in which model-generated pointers can be inspected by humans or reused by GPT-4V.Source: §5.3 · p. 69, Figure 52 · p. 72
  • Iterative pointer generation lets GPT-4V decompose multi-hop visual questions, focus on different regions step by step, and combine the resulting information into an answer.The model can generate distinct visual markers for intermediate reasoning steps before collating the final response.Source: §5.3 · p. 69

6 Temporal and Video Understanding

GPT-4V demonstrates broad temporal and video understanding from selected image frames, including sequencing, anticipation, localization, reasoning, and grounded interpretation of social interactions. Visual referring prompting further enables focused temporal analysis of a specified person.

6.1 Multi-image Sequencing

  • GPT-4V interprets video-frame sequences by recognizing scenes, human actions, and pose variations to derive nuanced activity-level meaning.This capability extends beyond object and scene recognition to contextual interpretation of ongoing activities.Source: §6.1 · p. 73, Figure 53 · p. 74

6.2 Video Understanding

  • GPT-4V correctly orders shuffled long- and short-term sequences, anticipates future actions, and localizes events while reasoning about their causal outcomes.It orders sushi-making and door-action frames, anticipates soccer and sushi-preparation steps, identifies when a player strikes the ball, and infers whether the goalkeeper blocks it.Source: §6.2 · pp. 73–74, Figure 54 · p. 75, Figure 56 · p. 77

6.3 Visual Referring Prompting for Grounded Temporal Understanding

  • With a circled person as a visual reference, GPT-4V tracks that individual’s temporal activities and distinguishes interaction types such as friendly contact and violence.Visual referring prompting provides enhanced control over temporal video comprehension by grounding descriptions in a person of interest.Source: §6.3 · p. 79, Figure 58 · p. 79

7 Abstract Visual Reasoning and Intelligence Quotient Test

This section evaluates whether GPT-4V can derive semantics from abstract visual signals and perform human IQ-test tasks. The results show capabilities across tangrams, object-part association, Wechsler sub-tests, and Raven’s Progressive Matrices, with image decomposition improving reliability for RPMs.

7.1 Abstract Visual Stimuli

  • GPT-4V interprets abstract stimuli, including tangrams, ASCII art, and symbolic visual inputs, assigning plausible semantic meanings to depicted shapes.For example, it identifies a tangram configuration as a flying goose and offers alternative interpretations for other configurations.Source: §7.1 · p. 80, Figure 59 · p. 81

7.2 Discovery and Association of Parts and Objects

7.3 Wechsler Adult Intelligence Scale

  • GPT-4V shows promise on Wechsler Adult Intelligence Scale sub-tests involving text-only questions, symbolic visual inputs, natural images, and visual analogies.The examples include interpreting an analogy and selecting the best comparison for shoes.Source: §7.3 · p. 80, Figure 61 · p. 83

7.4 Raven’s Progressive Matrices

  • GPT-4V produces reasonable Raven’s Progressive Matrices answers from an entire question page, while decomposing it into interleaved text and sub-figures makes answers more reliable.The whole-page setup mimics human testing and avoids preprocessed text descriptions; the decomposed setup simplifies the task.Source: §7.4 · p. 80, Figure 62 · p. 84, Figure 63 · p. 85

8 Emotional Quotient Test

GPT-4V demonstrates broad emotional-quotient capabilities, from reading facial expressions and visual sentiment to judging aesthetics and generating emotion-conditioned descriptions. These abilities support more empathetic and effective human-facing communication.

8.1 Read Emotion from Facial Expressions

  • GPT-4V reliably identifies emotions from facial expressions and provides reasonable rationales based on observed visual cues.Figure 64 illustrates this facial-emotion understanding.Source: §8.1 · p. 86, Figure 64 · p. 86

8.2 Understand How Visual Content Arouses Emotions

  • GPT-4V interprets visual sentiments including contentment, anger, awe, and fear using both semantic content and image style.Figure 65 presents this capability, which is relevant to applications such as home robots.Source: §8.2 · p. 87, Figure 65 · p. 87
  • GPT-4V judges image aesthetics according to societal standards and norms, aligning with human subjective judgments.Figure 66 shows examples of this aesthetic assessment.Source: §8.2 · p. 88, Figure 66 · p. 88

8.3 Emotion Conditioned Output

  • GPT-4V generates text conditioned on perceived or desired emotions, making descriptions more horrifying or comforting as prompted.Figure 67 demonstrates the potential for emotion-aware human-robot communication.Source: §8.3 · p. 89, Figure 67 · p. 89

9 Emerging Application Highlights · 9.1 Spot the Difference · 9.2 Industry

GPT-4V enables out-of-the-box multimodal application scenarios and can be extended through external tools and plugins. Across visual comparison, manufacturing, safety inspection, and checkout tasks, performance is useful but depends on product familiarity, references, prompting, and task decomposition.

9 Emerging Application Highlights

  • GPT-4V’s out-of-the-box capabilities support high-value applications, while external tools and plugins expand its potential for innovative and collaborative use cases.The section emphasizes generic capabilities without meticulous task-specific fine-tuning.Source: §9 · p. 90

9.1 Spot the Difference

  • GPT-4V identifies differing regions across four image pairs, but its explanations are imperfect, motivating applications such as defect detection.It correctly identifies several differences in the first example but confuses hairband cuts with hair shade.Source: §9.1 · p. 90

9.2 Industry

  • GPT-4V detects common-product defects such as holes, stripped screw heads, and bumper dents, but hesitates on uncommon or visually variable products.For a tire with multiple defects, it reports minor wheel dirt while missing major rim damage requiring repair.Source: §9.2 · p. 90
  • Providing a defect-free reference image and refining the prompt enables GPT-4V to identify defects in all three single-image failure cases.This reference-based approach demonstrates a potential high-value manufacturing application.Source: §9.2 · p. 90, Figure 72 · p. 96
  • GPT-4V fails zero-shot PPE counting by reporting 8 helmet-wearing people and missing 3 without helmets, whereas cropped-person inputs enable correct violation counting.An external person detector separates detection from GPT-4V’s visual reasoning, illustrating tool use and divide-and-conquer.Source: §9.2 · pp. 90–91, Figure 73 · p. 97
  • GPT-4V misidentifies grocery items from a basket photograph, but catalog-image references allow it to correctly identify all five products for price retrieval.The reference-augmented result supports a prototype automated self-checkout workflow.Source: §9.2 · p. 91, Figure 74 · p. 98

9.3 Medical · 9.4 Auto Insurance

GPT-4V shows promise for radiology assistance and automated car-accident insurance workflows, but medical outputs require professional verification and insurance inputs can be incomplete or occluded.

9.3 Medical

  • GPT-4V generates radiology reports from diverse medical images, with accuracy evaluated by a medical professional.The investigation covers abdominal X-rays, knee MRI, hand/wrist X-rays, chest CT, and brain MRI.Source: §9.3 · p. 99, Figure 75 · p. 100
  • GPT-4V correctly identified studies and diagnoses in two examples, but missed a distal radial fracture in another hand/wrist X-ray.Its reports nevertheless maintained a high-quality format that could reduce clinicians’ drafting workload.Source: §9.3 · p. 99, Figure 76 · p. 101
  • GPT-4V mislocalized a chest-CT nodule and hallucinated measurements, underscoring the need for medical-professional review of generated reports.The report-review figures distinguish confirmed, incorrect, and hallucinated content.Source: §9.3 · p. 99, Figure 77 · p. 102
  • Interleaved image-text inputs let GPT-4V reference prior scans and diagnosis histories, supporting symptom-progression analysis across multiple chest X-rays.This capability was described as critical to medical professionals’ diagnosing processes, with examples using MIMIC data.Source: §9.3 · p. 99, Figure 78 · p. 103

9.4 Auto Insurance

  • Auto-insurance applications cover damage evaluation and insurance reporting, combining vehicle-damage assessment with recognition of make, model, and license plate.These categories target car-accident reporting workflows.Source: §9.4 · p. 104
  • GPT-4V accurately identified and localized damage in all four presented images, described each damage instance, and sometimes estimated repair costs.The evaluation used an expert-oriented damage-assessment prompt.Source: §9.4 · p. 104, Figure 79 · p. 105
  • GPT-4V attempted JSON extraction of vehicle details, but repair costs may be unavailable and license plates can be unreadable under occlusion.Real insurance reporting usually requires multiple vehicle angles, which are generally unavailable in public images; Figures 79–80 illustrate automation potential.Source: §9.4 · p. 104, Figure 80 · p. 106

9.5 Customized Captioner

GPT-4V can generate personalized captions that identify named family members for photo organization and enrich object-level dense captions by combining segmentation cut-outs with the original image as context.

Photo organization

  • GPT-4V generates customized captions naming identified family members, enabling precise photo organization and personalized image searches.Visual prompts paired with names let it identify people, a cat, and a dog, supporting searches such as family-member combinations or spatial relationships.Source: §9.5 · p. 107, Figure 81 · p. 108, Figure 82 · p. 109

Dense captioning w/ segmentation

  • GPT-4V produces intricate dense captions for segmented objects while incorporating relevant details from the full context image.For example, it mentions a snail on a frog’s head despite the snail being absent from the frog cut-out, and recognizes that a turtle is floating in water.Source: §9.5 · p. 107, Figure 83 · p. 110

9.6 Image Generation

GPT-4V supports image-generation workflows by evaluating prompt-image alignment and explaining mismatches, while also generating and iteratively rewriting prompts to improve image editing.

Evaluation of generated images

  • GPT-4V’s ratings track progressively improved RL-Diffusion generations, scoring “a parrot driving a car” examples 4 → 8 → 9.It scores an irrelevant dolphin image 1 and the most relevant image 9, aligning ratings with refinement.Source: §9.6 · p. 111, Figure 84 · p. 112
  • GPT-4V evaluates generated images by comparing visual content and rendered text against the requested prompt, including OCR-based checks of cake text.It recognizes variants such as “Azuze Research,” “ARAUIE,” and “Azure Azure” against “Azure Research.”Source: §9.6 · p. 111, Figure 85 · p. 113

Prompt generation for image editing

  • GPT-4V generates optimized editing prompts from an original image and textual requirements, tailoring instructions to the image’s characteristics.Figure 86 demonstrates this prompt-generation workflow for improving subsequent image editing.Source: §9.6 · p. 111, Figure 86 · p. 114
  • GPT-4V rewrites editing prompts using the original image, initial prompt, and edited result, enabling repeated refinement for greater control and creative freedom.Users can alternate prompt generation and rewriting until the edited image reaches a satisfying outcome.Source: §9.6 · p. 111, Figure 87 · p. 115

9.7 Embodied Agent

GPT-4V shows promising embodied-agent capabilities by interpreting appliance interfaces and planning actions in dynamic virtual environments. The examples also reveal that decomposed visual prompts can correct interface-recognition errors, while iterative action-feedback enables successful navigation to a fridge.

Embodied Agent

  • The embodied-agent examples position GPT-4V as a home robot that can read appliance menus and perform task-oriented navigation through a house.These applications illustrate a bridge between multimodal understanding of static inputs and physical interaction with dynamic environments.Source: §9.7 · p. 116

Operating machine

  • Presenting each coffee-machine button menu separately in one interleaved image-text prompt enabled GPT-4V to identify the precise “6 OZ coffee” button.This revised prompting strategy was designed to address the confusion caused by the full-menu presentation.Source: §9.7 · p. 116, Figure 89 · p. 118

Navigation

  • GPT-4V reached the fridge within three turns in a task-oriented virtual-house navigation scenario, ultimately predicting how to align with and open the fridge door.The agent received the previous turn as context while actions were manually executed between predictions.Source: §9.7 · p. 116

9.8 GUI Navigation

GPT-4V demonstrates broad GUI interaction capabilities across computer and smartphone navigation, including web browsing, online shopping, notification handling, and video understanding. Its predicted actions can complete multi-step tasks and support grounded interaction, although occasional navigation and localization errors remain.

Web browsing

  • GPT-4V predicts grounded computer actions from screenshots, goals, and available operations, enabling multi-turn GUI navigation with the potential for automation without a human in the loop.After each manually executed action, the resulting screenshot becomes the next input; mouse movements require detailed position predictions.Source: §9.8 · p. 121
  • GPT-4V completes a Mapo Tofu recipe search and printing task, accurately describing printed details, and reasonably reads two news articles despite a minor error returning to a prior search page.The recognized printout details include cooking time, ingredients, author, and the original recipe link.Source: §9.8 · p. 121, Figure 95 · p. 125, Figure 96 · p. 126

Online shopping

  • GPT-4V navigates a smartphone shopping workflow for an ergonomic keyboard by searching, filtering for a $50–$100 budget, selecting a result, viewing details, adding it to the cart, and proceeding to checkout.The workflow spans opening Amazon, applying the price filter, selecting the top result, and reaching checkout, with some inaccurate locations or actions shown in the figures.Source: §9.8 · p. 121, Figure 103 · p. 133, Figure 108 · p. 138

Notification understanding

  • GPT-4V interprets notifications and responds appropriately, such as suggesting Maps for a Seattle meeting proposal while also handling call and message notifications.The notification examples cover both phone and computer screens.Source: §9.8 · p. 121, Figure 112 · p. 142, Figure 113 · p. 143

Watching videos

  • GPT-4V describes short-form videos from temporally ordered screenshots, generating useful content summaries whether subtitles are present or absent and suggesting automatic transcript generation.The examples also contain some inaccurate video descriptions.Source: §9.8 · p. 121, Figure 115 · p. 145, Figure 116 · p. 146

10 LMM Powered Agents

The section outlines future directions for GPT-4V-powered agents, emphasizing multimodal plugins, multimodal chains, self-reflection, self-consistency, and retrieval augmentation. Human-generated examples illustrate how these methods could extend GPT-4V’s capabilities and enable richer reasoning, interaction, and specialized applications.

10.1 Multimodal Plugins

  • Multimodal plugins can provide GPT-4V with time-sensitive visual knowledge unavailable at training time, enabling accurate identification of an earthquake photograph’s location as İzmir, Turkey.Without Bing Image Search, GPT-4V fails to identify the location because the February 6, 2023 earthquake occurred after training; the plugin retrieves current information.Source: Figure 120 · p. 150, §10.1 · p. 152

10.2 Multimodal Chains

  • Multimodal chains let GPT-4V combine vision tools and ReAct-style reasoning to process plugin outputs jointly, enabling PPE counting through detection, cropping, and helmet classification.The example uses two thought-action-observation rounds: detecting eight people, cropping them, and determining the helmet status of each person.Source: Figure 121 · p. 151, §10.2 · p. 152

10.3 Self-Reflection

  • Self-reflection improves generated figures and text-to-image prompts by correcting visual discrepancies and revising omitted details, although the result may remain imperfect.In the figure example, the point count changes from 4 to 3 and a missing percentage is restored; in prompt generation, the dog’s breed is added.Source: §10.3 · p. 152, Figure 122 · p. 153, Figure 123 · p. 154

10.4 Self-Consistency

  • Self-consistency improves counting reliability by aggregating repeated GPT-4V outputs, including samples from multiple runs or rephrased instructions, with majority voting producing “4 boats.”The approach extends self-consistency from sampled outputs to final-answer aggregation, while broader exploration for multimodal models remains future work.Source: §10.4 · pp. 155–156, Figure 124 · p. 155

10.5 Retrieval-Augmented LMMs

  • Retrieval-augmented LMMs retrieve task-relevant, current, specialized, or user-customized information to improve applications such as grocery checkout and personalized captioning.Store databases can supply image-text-price triplets for correct checkout information, while family photo albums can support customized captioning.Source: §10.5 · p. 156

11 Conclusions

The report probes GPT-4V across diverse application scenarios, revealing remarkable but non-exhaustively documented capabilities and motivating future research on more capable LMMs. It proposes future directions including richer multimodal generation, additional sensory modalities, and learning from web and physical-world data.

11.1 Summary and Conclusions

  • The report finds that GPT-4V demonstrates remarkable capabilities across diverse application scenarios, while acknowledging that the exploration is not exhaustive.The report is intended as a reference for discovering additional uses, understanding LMMs, and building more powerful systems.Source: §11.1 · p. 156

11.2 Towards Future LMMs

  • The future-research discussion is motivated by extensively documented weaknesses and limitations of GPT models [24].The report presents its own perspective on future research directions after noting related discussions.Source: §11.2 · p. 156
  • Future LMMs should generate interleaved image-text content and incorporate modalities such as video, audio, and other sensor data.These extensions aim to support comprehensive multimodal content understanding and generation.Source: §11.2 · p. 156
  • More versatile LMMs could learn from online web content and real-world physical environments to enable continuous self-evolution.Current approaches predominantly rely on organized image-tag or image-text datasets.Source: §11.2 · p. 156

Related papers

Loading 2309.17421v2…