PaperlayerView exact revision on arXiv

Source-linked AI summary

Lost in Translation: When GPT-4V(ision) Can't See Eye to Eye with Text. A Vision-Language-Consistency Analysis of VLLMs and Beyond

Xiang Zhang, Senyu Li, Zijun Wu, Ning Shi

arXiv:2310.12520v1cs.CLcs.CV

TL;DR

Prior evaluations of individual modalities do not capture how modalities interact or whether a model’s vision and language abilities are consistent. The paper introduces a pairwise consistency framework and six datasets covering translation-variant and translation-invariant vision-language tasks. On challenging mathematical and reasoning tasks, the model’s image-format accuracy differs by over 40% from text-format accuracy, with consistency scores as low as 58%.

  • Problem

    Prior evaluations of individual modalities do not capture how modalities interact or whether a model’s vision and language abilities are consistent.

    Source: §1 · p. 2

  • Method

    The paper introduces a pairwise consistency framework and six datasets covering translation-variant and translation-invariant vision-language tasks.

    Source: §1 · p. 2

  • Results

    On challenging mathematical and reasoning tasks, the model’s image-format accuracy differs by over 40% from text-format accuracy, with consistency scores as low as 58%.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    The findings highlight the need for more integrated system designs for multimodal models.

    Source: §7 · p. 8

    The authors hypothesize that GPT-4V lacks a highly consistent joint representation of vision and language, leading it to solve tasks separately by input format.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

Recent advancements in multimodal techniques open exciting possibilities for models excelling in diverse tasks involving text, audio, and image processing. Models like GPT-4V, blending computer vision and language modeling, excel in complex text and image tasks. Numerous prior research endeavors have diligently examined the performance of these Vision Large Language Models (VLLMs) across tasks like object detection, image captioning and others. However, these analyses often focus on evaluating the performance of each modality in isolation, lacking insights into their cross-modal interactions. Specifically, questions concerning whether these vision-language models execute vision and language tasks consistently or independently have remained unanswered. In this study, we draw inspiration from recent investigations into multilingualism and conduct a comprehensive analysis of model's cross-modal interactions. We introduce a systematic framework that quantifies the capability disparities between different modalities in the multi-modal setting and provide a set of datasets designed for these evaluations. Our findings reveal that models like GPT-4V tend to perform consistently modalities when the tasks are relatively simple. However, the trustworthiness of results derived from the vision modality diminishes as the tasks become more challenging. Expanding on our findings, we introduce "Vision Description Prompting," a method that effectively improves performance in challenging vision-related tasks.

1 Introduction

The introduction argues that evaluating vision and language separately misses how VLLM modalities interact, and presents a consistency framework, datasets, and prompting approach motivated by GPT-4V’s modality-dependent performance.

1 Introduction

  • Modality-by-modality evaluations cannot reveal cross-modal influence, while differences in training methods and data quality can contribute to disparities between vision and language abilities.The introduction notes that models may excel in visual tasks or language tasks depending on their training conditions.Source: §1 · pp. 1–2
  • The framework groups vision-language tasks as translation-variant or translation-invariant based on information integrity across modalities, and uses paired datasets and pairwise consistency to measure disparities.The authors also investigate internal reasoning in each modality with tailored prompts to study cross-modal representations and mutual influence.Source: §1 · p. 2
  • GPT-4V shows substantial vision-language inconsistency, with performance varying depending on whether tasks are conducted primarily in one modality or the other.The authors identify this disparity as motivation for Vision-Text Prompting (VDP), a potential remedy.Source: §1 · p. 2
  • The authors release six datasets for vision-language consistency testing and introduce Vision-Text Prompting (VDP) as a potential remedy for the observed disparity.They present these contributions as supporting evaluation of modality consistency beyond conventional metrics.Source: §1 · p. 2

2 A multilingualism view of Vision-Language models types.

The section uses bilingualism as an analogy for vision-language integration, contrasting shared representations with separate systems that rely on translation. This framing motivates examining cross-modal interaction rather than evaluating modalities only on individual tasks.

2 A multilingualism view of Vision-Language models types.

  • When multimodal systems are not trained properly, they can behave as independent modality-specific networks with limited cross-modal exchange and a cumbersome translation process.Their individual-modality performance may be reasonable, but the limited exchange can produce errors.Source: §2 · p. 3
  • Vision and language are framed as compound bilingualism (shared inner representation) versus non-compound bilingualism (distinct systems and inefficient translation).The analogy draws on multilingual-model research and linguistic accounts of bilingualism (Zhang et al., 2023b; D’Acierno, 1990; De Groot and Nas, 1991; Moradi, 2014; Zhang et al., 2023c; Jakobovits, 1968).Source: §2 · p. 3
  • Multilingual LLMs often translate tasks into a dominant language, but whether VLLMs behave similarly remains uncertain given their distinct training techniques and paired multimodal data.The discussion notes that compound multilingual LLMs are rare because large paired multilingual text corpora are scarce (Zhang et al., 2023b).Source: §2 · p. 3
  • The investigation focuses on vision-language compound-level interaction rather than performance on individual tasks, aiming to inform future research on model use.The stated aim is to assess how vision and language relate within LVLMs.Source: §2 · p. 3

3 Tasks categorization based on Translation-variability

The section classifies downstream tasks as Translation Invariant or Translation Variant according to whether essential task information survives translation between modalities. Invariance means translation leaves the solver’s output unchanged; TI tasks preserve task information, whereas TV tasks often lose core information.

3 Tasks categorization based on Translation-variability

  • The framework classifies downstream tasks as Translation Invariant (TI) or Translation Variant (TV) based on whether information essential for solving the task is retained across modalities.The categorization is inspired by multilingualism research (Zhang et al., 2023b).Source: §3 · p. 3
  • Under translation invariance, translating an input between modalities does not change the solver’s output.For a vision task, the framework considers image-to-text description translation and expects the multimodal system to produce the same result; it generalizes to other modalities.Source: §3 · pp. 3–4
  • TI tasks typically retain their information across modalities, while TV tasks often lose core information and are harder to solve in one modality than another.The section points to Figure 3 for examples of both task types.Source: §3 · p. 4

4 Method

The method evaluates vision–language consistency through paired instances of translation-invariant tasks, where equivalent information is presented as an image or text. It measures whether the model’s results agree across the two modalities.

4.1 A framework of evaluating cross-modality consistent

  • Pairwise evaluation focuses on translation-invariant (TI) tasks, where a well-trained model should produce the same result across modalities; translation-variant tasks may legitimately yield different outcomes.This addresses the limits of evaluating each modality independently and avoids attributing expected task differences to model inconsistency.Source: §4.1 · p. 4
  • For each TI task, the method pairs image- and text-based instances containing the same information, prompts them in separate sessions, and calculates pairwise accuracy.Pairwise accuracy is the proportion of instances for which both modality results are either correct or incorrect.Source: §4.1 · p. 4

5 Experiment

GPT-4V’s vision and language performance is relatively consistent on simpler tasks but diverges sharply on harder reasoning tasks, with text outperforming images. An image-text extraction ablation suggests that these gaps primarily reflect differences in internal reasoning rather than image recognition.

5.1.1 TI tasks dataset.

  • The evaluation introduces five Translation Invariant datasets, pairing image and text inputs for math questions, GSM8K math reasoning, WebQuestions knowledge access, CommonsenseQA common-sense reasoning, and generated state-machine reasoning.The state-machine datasets vary the number of states and steps; text transition rules are shuffled. The paper also describes the six-dataset suite as comprising five Translation Invariant tasks and one Translation Variant task.Source: §5.1 · p. 4, §5.1.1 · pp. 4–6

5.2 Results

  • On challenging Translation Invariant tasks, image and text accuracy differs by over 40%, with consistency as low as 58%; text performs significantly better despite equal information in the images.The reported tasks include math equation solving, math reasoning, and state-machine reasoning.Source: Table 1 · p. 6, §5.2 · p. 6
  • On simpler Knowledge Access and Common Sense Reasoning tasks, image and text accuracies are approximately 80–90%, with high cross-modal agreement and a slight text advantage.Source: Table 1 · p. 6, §5.2 · p. 6

5.1.2 TV task dataset.

  • The Translation Variant VQA dataset pairs CommonsenseQA questions with images replacing their text descriptions, while retaining the original text questions for comparison.The paper frames this as Translation Variant because text and images cannot always be fully converted into one another.Source: §5.1.2 · p. 6

5.2 Results

  • For Translation Variant VQA Common Sense Reasoning, text prompts perform significantly better than image prompts, even though images often contain more information.Source: §5.2 · p. 6

5.3 Ablation study on text extraction from image.

  • GPT-4V achieves 87% accuracy extracting math equation-solving questions from images, supporting the conclusion that modality performance gaps primarily reflect internal reasoning differences rather than recognition failures.The ablation tested extraction of the image-based questions, including their complex mathematical symbols.Source: §5.3 · p. 7

6 Inspired Vision-depicting-prompting (VDP)

Vision-depicting-prompting (VDP) converts image-based tasks into text descriptions before answering with both the description and original image, leveraging stronger text-based reasoning. On two math datasets, it improved image-input accuracy by over 30% and increased consistency relative to text-only prompting.

6 Inspired Vision-depicting-prompting (VDP)

  • VDP builds on the finding that GPT4V performs better when the same problem is presented in text, even when its information is fully recoverable from images.The method applies this observation to improve reasoning through image context.Source: §6 · p. 7

6.1 Prompting Details

  • For image-based tasks, VDP first prompts the model to extract and articulate a textual description of the task.This step aims to transform the image signal into a text signal.Source: §6.1 · p. 7
  • VDP then asks the model to answer using both the textual task description and the original image.This differs from requesting an answer based solely on the image.Source: §6.1 · p. 7

6.2 Experiment results for VDP

  • VDP improved image-input accuracy by over 30% on the Math Equation Solving and Math Reasoning datasets.The paper reports the outcomes in Table 2.Source: Table 2 · p. 8, §6.2 · p. 8

6.1 Prompting Details

  • Unlike prior multimodal reasoning work that augmented images with supplementary text (Lin et al., 2022; Hu et al., 2023), VDP extracts image text to engage language processing, not add information.This distinction is especially relevant to the TI task, where images already contain all information needed; converting them to text adds no insights, but textual signals can stimulate reasoning even when visual context is incomplete.Source: §6.1 · p. 8

6.2 Experiment results for VDP

  • VDP also substantially increased the consistency score compared with prompting using text alone.The passage gives no numerical value for this increase.Source: §6.2 · p. 8

7 Conclusion · Appendix

The study finds that GPT-4V reasons with relatively independent visual and textual representations and performs better in language modeling than in visual-context reasoning. It also introduces a prompting solution to address this disparity.

7 Conclusion

  • The study systematically analyzes cross-modal consistency in multimodal systems.Source: §7 · p. 8
  • GPT-4V performs better in language modeling than in reasoning within a visual context.The authors attribute this disparity to relatively independent internal representations of visual and textual reasoning.Source: §7 · p. 8
  • The findings suggest that GPT-4V maintains relatively independent internal representations for visual and textual reasoning.This conclusion is based on a specially designed dataset.Source: §7 · p. 8
  • The authors introduce a prompting solution to address the disparity between language modeling and visual-context reasoning.Source: §7 · p. 8
  • The findings offer insights into potential applications of multimodal systems and underscore the need for more integrated designs.Source: §7 · p. 8

A Math Equation Solving Dataset Samples

The section presents two math-equation dataset samples, each represented by an image and text; the supplied text for Sample 2 asks for a limit calculation.

B Math Reasoning Dataset

The Math Reasoning Dataset is illustrated with two samples, each presented as an image and text.

B Math Reasoning Dataset

C Knowledge Access Dataset

This section illustrates the Knowledge Access Dataset with image and text views for two samples.

D State Machine Dataset · E Common Sense Reasoning Dataset · F VQA Common Sense Reasoning Dataset

The sections present paired image and text examples for the State Machine, Common Sense Reasoning, and VQA Common Sense Reasoning datasets. The supplied captions identify samples but do not describe their contents.

D State Machine Dataset

  • The State Machine Dataset includes image examples labeled Sample 1 and Sample 2.The supplied figure captions identify the examples without describing what the images show.Source: Figure 12 · p. 16, Figure 13 · p. 17
  • The State Machine Dataset includes text examples labeled Sample 1 and Sample 2.The supplied table captions identify the examples without giving their text content.Source: Table 9 · p. 16, Table 10 · p. 17

E Common Sense Reasoning Dataset

  • The Common Sense Reasoning Dataset includes image examples labeled Sample 1 and Sample 2.The supplied figure captions identify the examples without describing what the images show.Source: Figure 14 · p. 18, Figure 15 · p. 19
  • The Common Sense Reasoning Dataset includes text examples labeled Sample 1 and Sample 2.The supplied table captions identify the examples without giving their text content.Source: Table 11 · p. 18, Table 12 · p. 19

F VQA Common Sense Reasoning Dataset

  • The VQA Dataset captions identify image examples labeled Sample 1 and Sample 2 in Figures 16 and 17, respectively.The captions describe each example only as an image.Source: Figure 16 · p. 20, Figure 17 · p. 21
  • The VQA Dataset captions identify Sample 1 and Sample 2 as text examples, but the supplied passages provide only their labels, not the examples’ actual text.Source: Table 13 · p. 20, Table 14 · p. 21
Loading 2310.12520v1…