PaperlayerView exact revision on arXiv

Source-linked AI summary

BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs

Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, Bingyi Kang

arXiv:2307.08581v1cs.CVcs.AI

TL;DR

Multimodal LLMs can describe inputs in detail but often lack fine-grained grounding between visual objects and other modalities. BuboGPT combines visual grounding with joint text, vision, and audio understanding through a grounding pipeline and two-stage training. The paper reports visual grounding during multimodal chat across arbitrary modality combinations, matched or unmatched.

  • Problem

    Multimodal LLMs often generate detailed descriptions without grounding them in fine-grained relations between visual objects and other modalities.

    Source: §1 · p. 1

  • Method

    BuboGPT combines a visual entity-grounding pipeline with two-stage training and multimodal instruction data for joint text, vision, and audio understanding.

    Source: §1 · pp. 1–2

  • Results

    BuboGPT demonstrates visual grounding during multimodal chat with arbitrary input combinations, whether matched or unmatched.

    Source: §6 · p. 6

  • Takeaways & Limitations

    BuboGPT jointly understands text, image, and audio while relating visual objects to other modalities through fine-grained grounding.

    Source: §6 · p. 6

    Grounding QA on specific objects remains limited because grounding results are linked to modalities through text conversations without extra training.

    Source: §7 · p. 6

Abstract

from arXiv · show

LLMs have demonstrated remarkable abilities at interacting with humans through language, especially with the usage of instruction-following data. Recent advancements in LLMs, such as MiniGPT-4, LLaVA, and X-LLM, further enlarge their abilities by incorporating multi-modal inputs, including image, video, and speech. Despite their effectiveness at generating precise and detailed language understanding of the given modality signal, these LLMs give up the ability to ground specific parts of inputs, thus only constructing a coarse-grained mapping. However, explicit and informative correspondence between text and other modalities will not only improve the user experience but also help to expand the application scenario of multi-modal LLMs. Therefore, we propose BuboGPT, a multi-modal LLM with visual grounding that can perform cross-modal interaction between vision, audio and language, providing fine-grained understanding of visual objects and other given modalities. As a result, BuboGPT is able to point out the specific location of an object in the image, when it is generating response or description for that object. Our contributions are two-fold: 1) An off-the-shelf visual grounding module based on SAM that extracts entities in a sentence and find corresponding masks in the image. 2) A two-stage training scheme and instruction dataset to endow joint text-image-audio understanding. Our experiments show that BuboGPT achieves impressive multi-modality understanding and visual grounding abilities during the interaction with human. It performs consistently well when provided by arbitrary modality combinations (either aligned or unaligned). Our code, model and dataset are available at https://bubo-gpt.github.io .

1 Introduction

Existing multimodal LLMs can produce detailed descriptions but often do not ground them in specific visual objects. BuboGPT combines visual grounding with joint text, vision, and audio understanding, using an off-the-shelf grounding pipeline and a two-stage training scheme with a multimodal instruction dataset.

1 Introduction

  • Prior multimodal LLMs often describe inputs without exposing fine-grained relations between visual objects and other modalities.The paper contrasts a detailed but black-box description with a teacher-bot that identifies which part of an image it refers to.Source: §1 · p. 1
  • BuboGPT combines visual grounding with joint text, vision, and audio understanding in a shared representation space aligned with pre-trained LLMs.Source: §1 · p. 1
  • Its grounding pipeline tags image content, finds corresponding masks or boxes, and uses LLM reasoning to match entities with descriptions.Language acts as the bridge between visual objects and other modalities.Source: §1 · p. 2
  • A two-stage training scheme and instruction dataset support arbitrary modality combinations, including matched and unmatched inputs.The dataset includes positive and negative image-audio pairs for semantic matching and reasoning.Source: §1 · p. 2

2 Related Work

Prior work connects pretrained LLMs to multimodal inputs through learned feature alignment or added cross-attention, but largely focuses on visual input or omits fine-grained visual-object relations to other modalities. BuboGPT also extends multimodal instruction-tuning data with fine-grained audio descriptions and negative image-audio pairs.

2 Related Work

  • Recent multimodal LLMs align modality features with pretrained LLMs or insert learnable cross-attention layers.Examples include Mini-GPT4 [2], X-LLM [3], Video-ChatGPT [10], LLaMA-Adapter [5], and Otter [7].Source: §2 · p. 2
  • Prior methods mainly focus on visual inputs or ignore fine-grained relations between visual objects and other modalities.Source: §2 · p. 3
  • Multimodal instruction-tuning approaches include a 62-task benchmark, curated image-text datasets, GPT-4-expanded COCO captions, and an in-context tuning dataset.Source: §2 · p. 3
  • BuboGPT adds fine-grained audio descriptions and negative image-audio pairs to multimodal instruction-tuning data.Source: §2 · p. 3

3 Methods

BuboGPT combines a shared text-vision-audio representation with a visual-grounding pipeline and two-stage multimodal LLM training. The grounding pipeline links detected image entities to the model’s text output, while instruction tuning includes varied modality combinations and negative pairs.

3.1 Visual Grounding Pipeline

  • RAM generates candidate image tags, Grounding DINO finds corresponding boxes, and SAM turns those boxes into fine-grained masks.Source: §3.1 · p. 3
  • GPT-4-based entity matching retrieves pairs linking grounded visual entities to the multimodal LLM’s text output.The visual entities carry grounding information in the form of boxes and masks.Source: §3.1 · p. 3

3.2 Multi-Modal LLM Training

  • BuboGPT aligns vision and audio encoders with Vicuna through modality-specific Q-formers and linear projections, keeping the encoders and LLM fixed during training.Source: §3.2 · p. 3
  • Stage one aligns modality features with the LLM word-embedding space using modality-text paired data, then stage two tunes on image-text, audio-text, and image-audio-text instructions.For vision, only the projection layer is trained while the BLIP-2 Q-former remains fixed; for audio, the Q-former and projection layer are trained.Source: §3.2 · pp. 3–4
  • Negative image-audio pairs teach the model to distinguish unrelated inputs rather than assume every image and audio sample is related.The prompt accommodates arbitrary modality combinations using vision and audio identifiers, modality tokens, and an instruction.Source: §3.2 · p. 4

4 Datasets

The dataset construction combines large image-text and audio-text sources with instruction data for image, audio, and cross-modal tasks. Audio captions are expanded into longer descriptions, and image-audio examples include both localized positive pairs and constructed negative pairs.

4.1 Pretraining Datasets

  • Visual pretraining uses 130 million image-text pairs from CC3M, CC12M, SBU, and LAION; audio pretraining uses WaveCaps with 403,050 clips.WaveCaps clips average 67.59 seconds, and captions average 7.8 words.Source: §4.1 · pp. 4–5

4.2.1 Image-Text Dataset

  • Visual instruction tuning combines 3,439 MiniGPT-4 image-text pairs with 158K LLaVA samples covering conversations, detailed descriptions, and complex reasoning.Source: §4.2.1 · p. 5

4.2.2 Audio-Text Dataset

  • Clotho’s five short captions per audio are assembled with GPT-4 into Clotho-Detail 4, a 3,938-item instruction dataset with 52.70-word average descriptions.The captions represent different annotators and possible scenes related to each audio clip.Source: §4.2.2 · p. 5

4.2.3 Audio-Image-Text Dataset

  • VGGSS supplies 5,158 positive audio-image-text triples in which audio relates to a specific image region.Source: §4.2.3 · p. 5
  • Negative triples pair randomly sampled audio and images with separate captions describing each modality independently.Source: §4.2.3 · p. 6

5 Experiment Results

The experiments report fine-grained visual grounding, informative audio descriptions, and sound localization with aligned audio-image inputs. Negative image-audio pairs help BuboGPT handle arbitrary, including irrelevant, combinations.

5 Experiment Results

  • BuboGPT accurately associates image words and phrases with regions across visual scenarios of different complexities.The experiments also report informative audio descriptions covering nearly all acoustic parts, including fragments too short for humans to notice.Source: §5 · p. 6
  • With matched audio-image inputs, BuboGPT describes both modalities and identifies the image object that emits the sound.It also predicts correctly when the audio changes while the image stays the same.Source: §5 · p. 6
  • Training only on well-aligned image-audio data leaves BuboGPT unable to reliably reject irrelevant pairs, producing non-factual responses.After adding negatively paired data, it can judge image-audio relevance and respond to arbitrary pairs.Source: §5 · p. 6

6 Conclusion

BuboGPT combines image, audio, and text understanding with finer-grained relations between visual objects and other modalities. The paper reports visual grounding during multimodal chat, including with arbitrary matched or unmatched input combinations.

6 Conclusion

  • BuboGPT performs joint understanding of image, audio, and text while exploring fine-grained relations between visual objects and modalities.The approach is described as a multimodal LLM in the conclusion.Source: §6 · p. 6
  • The paper reports visual grounding abilities during multimodal chat with arbitrary combinations of inputs, whether matched or unmatched.Source: §6 · p. 6
  • The conclusion identifies a high-quality instruction-tuning dataset as a contribution alongside the multimodal model.Source: §6 · p. 6

7 Limitations

The paper identifies inherited language hallucination and limited question answering about specific grounded objects as limitations. It suggests more high-quality data and trustworthy LLMs for the former, and fine-grained grounding data or spatial input for the latter.

7 Limitations

  • Because BuboGPT uses pre-trained Vicuna, it inherits the risk of generating nonexistent knowledge or non-factual information.The authors suggest more high-quality data and trustworthy LLMs as possible remedies.Source: §7 · p. 6
  • Question answering about specific objects remains limited because grounding results connect to modalities through text conversations without extra training.The paper suggests fine-grained visual grounding datasets and spatial location as extra input as possible improvements.Source: §7 · p. 6

Related papers

Loading 2307.08581v1…