PaperlayerView exact revision on arXiv

Source-linked AI summary

A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise

Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui Lin, Deqiang Jiang, Di Yin, Peng Gao, Ke Li, Hongsheng Li, Xing Sun

arXiv:2312.12436v2cs.CVcs.AIcs.CLcs.MM

TL;DR

How Gemini Pro’s visual understanding compares with GPT-4V and open-source MLLMs across domains remains an open question. This preliminary evaluation compares Gemini Pro, GPT-4V, and Sphinx across visual tasks, finding task-specific strengths and weaknesses: Gemini identifies faces where GPT-4V declines, while all three struggle with low-resolution road signs.

  • Problem

    The paper examines the limited comparative evidence on Gemini’s visual understanding relative to GPT-4V and an open-source MLLM across visual domains.

    Source: §1.1 · p. 6

  • Method

    The authors evaluate Gemini Pro, GPT-4V, and Sphinx using qualitative samples across visual domains and the MME quantitative benchmark.

    Source: §1.1 · p. 6, §7.1 · p. 123

  • Results

    Gemini identifies faces that GPT-4V declines to identify, while all three models struggle to recognize small, low-resolution traffic signs precisely.

    Source: §4.1 · p. 89, §5.1 · p. 100

  • Takeaways & Limitations

    The examples show that general-purpose MLLMs cannot consistently produce satisfactory results on domain-specific medical images.

    Source: §5.3 · p. 100

    The authors identify shared weaknesses in Gemini and GPT-4V in spatial perception, OCR and abstract visual understanding, and reasoning self-consistency.

    Source: §7.2 · p. 123

Abstract

from arXiv · show

The surge of interest towards Multi-modal Large Language Models (MLLMs), e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both academia and industry. They endow Large Language Models (LLMs) with powerful capabilities in visual understanding, enabling them to tackle diverse multi-modal tasks. Very recently, Google released Gemini, its newest and most capable MLLM built from the ground up for multi-modality. In light of the superior reasoning capabilities, can Gemini challenge GPT-4V's leading position in multi-modal learning? In this paper, we present a preliminary exploration of Gemini Pro's visual understanding proficiency, which comprehensively covers four domains: fundamental perception, advanced cognition, challenging vision tasks, and various expert capacities. We compare Gemini Pro with the state-of-the-art GPT-4V to evaluate its upper limits, along with the latest open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and black-box systems. The qualitative samples indicate that, while GPT-4V and Gemini showcase different answering styles and preferences, they can exhibit comparable visual reasoning capabilities, and Sphinx still trails behind them concerning domain generalizability. Specifically, GPT-4V tends to elaborate detailed explanations and intermediate steps, and Gemini prefers to output a direct and concise answer. The quantitative evaluation on the popular MME benchmark also demonstrates the potential of Gemini to be a strong challenger to GPT-4V. Our early investigation of Gemini also observes some common issues of MLLMs, indicating that there still remains a considerable distance towards artificial general intelligence. Our project for tracking the progress of MLLM is released at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.

1 Introduction 6 · 1 Introduction

The paper explores Gemini’s visual understanding by comparing it with GPT-4V [59] and Sphinx [35] across four domains, using qualitative samples and Gemini’s MME results. It also describes prompting techniques and sample-collection choices for the exploration.

1.1 Motivation and Overview

  • The paper frames MLLMs as combining LLMs with inputs such as images, audio, and 3D data, and identifies Gemini as a challenger to GPT-4V [59].The authors motivate an early comparison because Gemini’s full potential had not yet been explored.Source: §1.1 · p. 6
  • The study compares Gemini with GPT-4V [59] to assess its capabilities and with Sphinx [35] to examine the gap between open- and closed-source models.The authors describe GPT-4V as representing the field’s highest standard and Sphinx as a state-of-the-art LLaMA-based MLLM.Source: §1.1 · p. 6
  • Fundamental perception covers object-, scene-, and knowledge-based abilities, while advanced cognition tests visual reasoning in text-rich tasks, abstract reasoning, science, emotion understanding, and games.Examples include object counting and captioning, table and chart reasoning, mathematics and physics, and Sudoku.Source: §1.1 · p. 6
  • Challenging vision tasks include detection, referring and localization tasks, and video temporal reasoning; expert-capacity tests cover medical imaging, defect detection, stock prediction, autonomous driving, and surveillance.These domains probe whether MLLMs can generalize as multipurpose vision systems and apply learned capabilities in professional contexts.Source: §1.1 · pp. 6–7
  • The paper evaluates Gemini quantitatively on MME [19], which covers 14 subtasks spanning perception and cognition.Source: §1.1 · p. 7

1.2.1 Prompt Technique

  • The authors use simple instructions for most tasks, visual markers or physical objects for referring prompts, CoT for complex reasoning, and few-shot examples when instructions alone are insufficient.They note that physical objects such as a finger or pen may be more practical than visual markers in real-time interaction.Source: §1.2.1 · p. 7

1.2.2 Sample Collection

  • To limit sample leakage, the authors write query text from scratch and seek Internet images timestamped after November 2023; they also collect samples at varied difficulty levels.Image sources include original drawings, offline photographs, Internet images, and existing datasets [10].Source: §1.2.2 · p. 7

2 Fundamental Perception

Fundamental perception—the accurate processing of visual input—supports higher-order tasks. This section examines object-centric perception, scene understanding, and knowledge applied to visual information.

2.1 Object-Centric Perception

  • Models struggle with left-right recognition; few-shot examples help only GPT-4V, while Sphinx matches the closed models on simple counts but all err in visually crowded images.Source: §2.1 · p. 8
  • All models can spot fine-grained image differences, but performance is unstable; when asked for five differences where only three exist, Gemini and GPT-4V fabricate answers.Source: §2.1 · p. 8
  • Gemini recognizes equal pear brightness despite an optical illusion that fools GPT-4V and Sphinx; GPT-4V also notes human-like similarities in another illusion.Source: §2.1 · p. 8

2.2 Scene-Level Perception

  • All models describe key image elements, but GPT-4V gives more detail and fewer hallucinations, particularly in cluttered scenes.Source: §2.2 · p. 8
  • Gemini integrates video frames into a cohesive scene description, correctly combining two tables and three plants; GPT-4V describes frames separately, while Sphinx misses the sequence.Source: §2.2 · pp. 8–9

2.3 Knowledge-based Perception

  • Sphinx matches Gemini and GPT-4V on social norms but is weaker on physical laws; Gemini and GPT-4V show relevant subject knowledge, while Sphinx is slightly inferior but sometimes correct.Source: §2.3 · p. 9
  • All three models understand depicted cultural elements and generally identify familiar world entities, but recent updates can produce inaccuracies and GPT-4V sometimes declines real-person questions.Source: §2.3 · p. 9

3 Advanced Cognition

This section evaluates MLLMs’ advanced cognition across text-rich visual reasoning, abstract reasoning, science problems, emotion understanding, and games. Results show task-dependent strengths alongside persistent limits in perception, inference, and grounded output.

3.1 Text-Rich Visual Reasoning

  • For text-rich visual reasoning, Gemini summarizes flowcharts, GPT-4V gives more detail but errs, and all three struggle with precise plot and table answers because of OCR.Both Gemini and GPT-4V also interpret hand-drawn prompts, although Gemini sometimes answers incorrectly.Source: §3.1 · p. 48, Figure 39 · p. 51, Figure 43 · p. 55
  • Gemini and GPT-4V outperform Sphinx on formula-to-LaTeX generation, including a complex printed formula, but website-to-HTML remains weak, with only Gemini building rough simple-site structures.All three models still misrecognize some minor formula characters or symbols.Source: §3.1 · p. 48, Figure 44 · p. 56, Figure 46 · p. 58

3.2 Abstract Visual Reasoning

  • GPT-4V leads on simple abstract composition, while nearly all models miss Raven’s Matrices and WAIS predictions, failing to infer relations despite recognizing individual elements.GPT-4V can provide detailed reasoning but may be misled by an incorrect intermediate step.Source: §3.2 · p. 48, Figure 48 · p. 60, Figure 50 · p. 62

3.3 Science Problem-Solving

  • Gemini and GPT-4V solve simple arithmetic and algebra and reason through harder math and physics, but visual recognition, calculations, and equations limit accuracy; chain-of-thought prompting can correct Sphinx.The models use external tools for difficult trigonometry and integral calculus, while physics reasoning draws on pretrained specialized knowledge.Source: §3.3 · p. 49, Figure 54 · p. 66, Figure 60 · p. 72

3.4 Emotion Understanding

  • All three handle facial expressions and infer emotions from scenes; GPT-4V is more analytical, Gemini gives concise judgments, and Sphinx matches peers on scene emotion.GPT-4V treats scene emotions as subjective, while Gemini tends to state an emotion aligned with mainstream perspectives.Source: §3.4 · p. 49, Figure 63 · p. 75, Figure 65 · p. 77
  • Gemini and GPT-4V can condition descriptions on requested emotions but hallucinate objects or sounds, whereas Sphinx avoids these errors.The reported hallucinations include nonexistent objects and sounds in the described scenes.Source: §3.4 · p. 49, Figure 70 · p. 82

3.5 Game Playing

  • Text input enables Gemini and GPT-4V to solve Sudoku, while Gemini slightly outperforms GPT-4V on crosswords; image OCR and blank-cell recognition remain barriers.Sphinx’s crossword output is unstable across the two examples.Source: §3.5 · p. 49, Figure 72 · p. 84, Figure 73 · p. 85
  • In Go, models produce some legal moves, but only GPT-4V finds the optimal move in one sample; Sphinx mistakes Go for Reversi, showing weak advanced play.The paper concludes that current MLLMs have not mastered advanced chess-playing capabilities.Source: §3.5 · p. 50, Figure 76 · p. 88

4 Vision Task

The section assesses whether MLLMs can handle challenging vision tasks beyond standard visual question answering, spanning image-level and temporal tasks. Results vary by task, with models showing both notable capabilities and recurring failures.

4.1 Image-Level Vision Task

  • In object detection [11] [47] and referring expression comprehension [37] [62], Gemini’s boxes are imprecise, GPT-4V avoids direct coordinates or gives imprecise boxes, and Sphinx gives more reliable detection and exact referring boxes.Sphinx’s detection still falls short of conventional object detectors.Source: §4.1 · p. 89, Figure 77 · p. 90, Figure 78 · p. 91
  • None of the models performs phrase localization [45] satisfactorily: Gemini and GPT-4V provide incorrect boxes, while Sphinx fails to follow the instruction.GPT-4V’s box for the zebra is nearly accurate; Gemini also has out-of-bound boxes.Source: §4.1 · p. 89, Figure 80 · p. 93
  • Gemini identifies and names all faces, GPT-4V declines on privacy grounds, and Sphinx localizes them nearly correctly but fails to recognize identities [31] [58].Gemini does so without an explicit bounding-box instruction.Source: §4.1 · p. 89, Figure 81 · p. 94

4.2 Temporal-Level Vision Task

  • In object tracking [52], Gemini and GPT-4V describe the target but give incorrect boxes in the next two frames, while Sphinx misunderstands the instructions.Source: §4.2 · p. 89, Figure 82 · p. 95
  • For video action recognition [18] [33] [17] [4], Gemini and GPT-4V recognize actions and provide detailed descriptions, while Sphinx is correct but less detailed.The evaluation uses five representative frames extracted from a video segment.Source: §4.2 · p. 89, Figure 83 · p. 96
  • For visual story generation, Gemini produces a coherent story aligned with the first two comics, while GPT-4V gives descriptions without a cohesive story and Sphinx diverges from the comics [46] [29].Source: §4.2 · p. 89, Figure 86 · p. 99

5 Expert Capacity

This section explores Gemini, GPT-4V, and Sphinx across specialized professional tasks, showing varied strengths alongside recurring limitations in fine-grained visual perception and generalization.

5.4 Economic Analysis

  • On two economic line-chart questions, Gemini gives correct answers, GPT-4V declines to answer clearly due to security risks, and Sphinx cannot understand the questions.The paper attributes Sphinx’s limitation to a lack of related training data.Source: §5.4 · p. 100

5.5 Surveillance and Security

  • In construction-site surveillance samples, Gemini detects missing helmets but hallucinates character locations, while GPT-4V identifies missing protective gear; both misrecognize details.The authors link these limitations to fine-grained visual perception.Source: §5.5 · p. 101

5.6 Remote Sensing Image Analysis

  • In remote-sensing images, Gemini and Sphinx count four airplanes where GPT-4V counts two; Gemini identifies Japan in another sample, while GPT-4V describes its details.GPT-4V’s description includes water bodies, roads, vegetation, residential patterns, and terrain.Source: §5.6 · p. 101

5.7 Robot Motion Planning

  • For phone and chair assembly, Gemini and GPT-4V provide detailed steps, GPT-4V appears more reasonable on battery installation order, and Sphinx fails on the phone task.Sphinx successfully handles chair assembly, indicating limited generalization across these examples.Source: §5.7 · p. 101

5.1 Autonomous Driving

  • All three models understand basic driving scenes and make appropriate decisions, but struggle to recognize small, low-resolution traffic and road signs.The authors call for finer-grained visual representation encoding in autonomous driving.Source: §5.1 · p. 100, Figure 87 · p. 102, Figure 88 · p. 103

5.2 Defect Detection

  • For obvious defects, all three models answer correctly; on harder samples, GPT-4V gives the standard thread-damage answer while Gemini and Sphinx err, and models differ on a small cup hole.Gemini calls the cup’s small damaged hole condensation, whereas GPT-4V and Sphinx report no abnormalities.Source: §5.2 · p. 100, Figure 91 · p. 106, Figure 92 · p. 107

5.3 Medical Diagnosis

  • On complex chest X-rays, MLLMs often judge that there are no symptoms, and specific prompting may be needed to prevent rejection of medical problems.The authors report this especially for the last two samples and give a prompt framing the report as capability assessment.Source: §5.3 · p. 100, Figure 94 · p. 109, Figure 95 · p. 110

Sec. 5.4 Economic Analysis

The section presents Gemini’s economic-analysis results, with Figure 98 highlighting the right answer in green.

Economic Analysis

  • Figure 98 presents the economic-analysis results and marks the right answer in green.Source: Figure 98 · p. 113

Sec. 5.4 Economic Analysis

The economic-analysis results are presented in Figure 99, with the correct answer highlighted in green.

Economic Analysis

  • Figure 99 presents the economic-analysis results and highlights the correct answer in green.Source: Figure 99 · p. 114

Sec. 5.5 Surveillance and Security

The section presents results on surveillance and security.

Surveillance and Security

  • Figure 100 marks right answers in green, wrong answers in red, and task incompetence in yellow.Source: Figure 100 · p. 115

Sec. 5.5 Surveillance and Security

Figure 101 presents results on surveillance and security, with correct answers highlighted in green and incorrect answers in red.

Surveillance and Security

  • Figure 101 presents surveillance and security results, marking correct answers green and incorrect answers red.The caption refers readers to Section 5.5 for detailed discussion.Source: Figure 101 · p. 116

Remote Sensing Image Analysis

The section presents visual results for remote sensing image analysis, but the supplied captions do not report specific outcomes or comparisons.

Remote Sensing Image Analysis

  • Figure 102 presents remote-sensing results and marks correct, incorrect, and incompetent responses with different colors.Green indicates correct answers, red incorrect answers, and yellow incompetence.Source: Figure 102 · p. 117
  • Figure 103 presents remote-sensing results, with green highlighting the right answer.Source: Figure 103 · p. 118

6 Quantitative Experiments

On the MME benchmark, Gemini has the highest overall perception-and-cognition score, GPT-4V leads cognition tasks, and Sphinx surpasses the other models on most perception tasks.

6.1 MME Benchmark

  • MME uses paired yes/no questions per image and reports question-level accuracy plus accuracy+ requiring both answers correct, with random baselines of 50% and 25%.Each task score sums both accuracy measures, and perception and cognition scores sum their respective subtasks.Source: §6.1 · p. 121
  • Perception samples come from public datasets, while cognition samples are manually photographed or generated with diffusion models.The perception tasks cover existence, count, position, color, OCR, poster, celebrity, scene, landmark, and artwork; cognition tasks cover commonsense reasoning, numerical calculation, text translation, and code reasoning.Source: §6.1 · p. 121

6.2 Results

  • Sphinx leads most perception tasks and beats Gemini and GPT-4V by 60 points on position; GPT-4V scores zero on celebrity recognition after refusing real-person information.The authors hypothesize that MME perception samples from public academic datasets align closely with Sphinx’s training distribution.Source: §6.2 · p. 121, Figure 106 · p. 122
  • GPT-4V dominates almost all cognition tasks, scoring 170.0 on code reasoning; Sphinx lags considerably on the cognition subtasks.The cognition tasks include commonsense reasoning, numerical calculation, text translation, and code reasoning.Source: §6.2 · p. 121, Figure 106 · p. 122
  • Gemini scores 1933.4 overall, ahead of GPT-4V at 1926.6 and Sphinx at 1870.2, with a more balanced performance across tasks.The results cover MME perception and cognition.Source: §6.2 · p. 121, Table 1 · p. 122

7 Conclusion

The report compares Gemini Pro, GPT-4V, and Sphinx across four visual-understanding domains using qualitative examples and the MME benchmark. It finds Gemini competitive with GPT-4V, identifies shared limitations and open-source gaps, and outlines future priorities for MLLMs.

7.1 Summary

  • The evaluation compares Gemini Pro, GPT-4V, and Sphinx across fundamental perception, advanced cognition, challenging vision tasks, and expert capacities.It collects qualitative samples across these domains and includes the MME quantitative benchmark.Source: §7.1 · p. 123

7.2 Gemini vs GPT-4V

  • Gemini achieves competitive answering accuracy with GPT-4V in most cases, though the models differ in response style and task-specific strengths.GPT-4V tends to provide detailed descriptions and step-by-step reasoning, while Gemini is more concise; GPT-4V is stronger on fine-grained perception in images with many visual elements, whereas Gemini shows broader knowledge on some vision and expert tasks.Source: §7.2 · p. 123
  • Both Gemini and GPT-4V struggle with spatial perception, OCR and abstract visual understanding, reasoning consistency, and robustness to prompt wording.They can misread chart characters, give reasoning that conflicts with a final answer, and produce opposite answers to differently framed versions of the same question.Source: §7.2 · p. 123

7.3 Gemini vs Sphinx

  • Sphinx matches Gemini and GPT-4V in some cases but does not produce consistently high-quality answers, illustrating a gap between open- and closed-source MLLMs.The report attributes its failures to limited training-data diversity in some domains and the reasoning ceiling of its initialized LLaMA-2-7B model relative to larger models.Source: §7.3 · p. 123

7.4 Future Directions

  • Future MLLM development should target visual representation encoding, multimodal alignment, and LLM reasoning capacity, while substantial distance to artificial general intelligence remains.The suggested priorities include fine-grained appearance and spatial-relation awareness, reducing hallucination and improving OCR, and strengthening quantitative processing and logical self-consistency.Source: §7.4 · p. 124
Loading 2312.12436v2…