PaperlayerView exact revision on arXiv

Source-linked AI summary

GroundingGPT:Language Enhanced Multi-modal Grounding Model

Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Van Tu Vu, Zhida Huang, Tao Wang

arXiv:2401.06071v5cs.CVcs.CL

TL;DR

Existing multimodal LLMs often miss fine-grained local information, and prior grounding work is primarily limited to a single modality. GroundingGPT combines multimodal grounding with coarse-to-fine training and stage-specific data; the authors report effectiveness across a range of multimodal benchmarks. The paper also identifies video/audio sampling and predominantly single-modal training data as limitations.

  • Problem

    Existing multimodal LLMs often neglect fine-grained local information, while prior fine-grained grounding approaches are primarily limited to a single modality.

    Source: §1 · p. 1

  • Method

    GroundingGPT is an end-to-end multimodal model trained with a three-stage coarse-to-fine strategy and diverse, stage-specific datasets.

    Source: §1 · pp. 1–2

  • Results

    Extensive experiments on a wide range of MLLM benchmarks demonstrate GroundingGPT’s generality and effectiveness across multimodal grounding and understanding tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors make the model, code, and dataset openly accessible to support further work.

    Source: §5 · p. 8

    Video and audio sampling can lose crucial information, especially for longer videos, and most training data uses single-modal inputs.

    Source: §6 · p. 9

Abstract

from arXiv · show

Multi-modal large language models have demonstrated impressive performance across various tasks in different modalities. However, existing multi-modal models primarily emphasize capturing global information within each modality while neglecting the importance of perceiving local information across modalities. Consequently, these models lack the ability to effectively understand the fine-grained details of input data, limiting their performance in tasks that require a more nuanced understanding. To address this limitation, there is a compelling need to develop models that enable fine-grained understanding across multiple modalities, thereby enhancing their applicability to a wide range of tasks. In this paper, we propose GroundingGPT, a language enhanced multi-modal grounding model. Beyond capturing global information like other multi-modal models, our proposed model excels at tasks demanding a detailed understanding of local information within the input. It demonstrates precise identification and localization of specific regions in images or moments in videos. To achieve this objective, we design a diversified dataset construction pipeline, resulting in a multi-modal, multi-granularity dataset for model training. The code, dataset, and demo of our model can be found at https: //github.com/lzw-lzw/GroundingGPT.

1 Introduction

Existing MLLMs often capture global information but miss fine-grained local details, while prior grounding approaches are primarily limited to one modality. GroundingGPT combines multimodal grounding with a three-stage training strategy and stage-specific data, and the authors report effectiveness across a range of benchmarks.

1 Introduction

  • Existing MLLMs often neglect local details, and prior fine-grained grounding approaches are primarily limited to a single modality.The authors identify this as a gap in multimodal grounding and understanding.Source: §1 · p. 1
  • GroundingGPT is presented as an end-to-end model for grounding and understanding image, video, and audio inputs.The authors claim it is the first model to achieve multimodal fine-grained understanding and grounding.Source: §1 · pp. 1–2
  • Its three-stage coarse-to-fine training combines high-level semantic learning with fine-grained detail using a diverse multimodal dataset.The stages align modality encoders with the LLM, train fine-grained information such as coordinates and timestamps, then refine responses through multi-granularity instruction tuning.Source: §1 · p. 2
  • Extensive experiments on a wide range of MLLM benchmarks demonstrate the model’s generality and effectiveness across multimodal grounding and understanding tasks.Source: §1 · p. 2

2 Related Work

Related work spans general multimodal LLMs and methods for fine-grained grounding. The reviewed approaches map modality features into LLMs or add mechanisms for local perception, but the cited grounding methods cover different modalities and capabilities.

Multi-modal Large Language Models (MLLMs)

  • Multimodal LLMs extend pretrained language models to multimodal inputs, but often fail to capture input details.Examples include image models that map encoder embeddings into LLM space and separate video and speech models.Source: §2 · p. 2

MLLMs For Grounding Task

  • Image-grounding methods incorporate coordinates or add region encoders, while VTimeLLM addresses fine-grained video moments and temporal reasoning.The cited approaches include KOSMOS-2 and Shikra for coordinates, and NExT-Chat, GlaMM, and Ferret for region encoders.Source: §2 · p. 2
  • BuboGPT supports cross-modal interaction among image, audio, and language for fine-grained understanding.Source: §2 · p. 2

3 Methods

GroundingGPT uses modality-specific encoders and adapters to connect image, video, and audio features to an LLM, while representing spatial and temporal grounding information as text. Its three training stages progress from semantic alignment to fine-grained grounding and multi-granularity instruction tuning, with stage-specific datasets.

3.1 Model Architecture

  • Separate modality encoders extract input features, and corresponding adapters map them into the LLM embedding space.The image branch uses a pretrained CLIP visual encoder and an MLP adapter before concatenating image embeddings with text embeddings.Source: §3.1 · p. 3, §3.1.1 · p. 3, Figure 1 · p. 3

3.1.2 Video Branch

  • The video branch uniformly samples frames, adds temporal position encoding, then uses a Video Q-former to produce a fixed-length video representation.Source: §3.1.2 · p. 3

3.1.3 Audio Branch

  • The audio branch samples two-second segments, adds temporal position encoding, and uses an audio Q-former to form a fixed-length representation.Source: §3.1.3 · p. 3

3.1.4 Spatial-temporal Representation

  • Image bounding boxes use four relative coordinates, while temporal segments use relative start and end timestamps, represented as text without vocabulary expansion.Coordinates are rounded to three decimal places; timestamps use two decimal places.Source: §3.1.4 · pp. 3–4

3.2 Coarse-to-Fine Training and Dataset

  • The three-stage coarse-to-fine strategy gives each stage a specific training objective and dataset construction pipeline.The first stage aligns pretrained modality encoders with the LLM embedding space while keeping the LLM and encoders frozen and training adapters.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.1 Multi-modal Pre-training

  • Multi-modal pretraining targets high-level semantic perception using public image, video, and audio datasets, with only modality adapters trained.The LLM and modality encoders remain frozen in this stage.Source: §3.2.1 · p. 4

3.2.2 Fine-grained Alignment Tuning

  • Fine-grained alignment tuning trains the LLM and adapters to understand coordinates and timestamps, using converted grounding and localization datasets.The datasets cover image visual grounding, video temporal grounding, and sound localization.Source: §3.2.2 · pp. 4–5

3.2.3 Multi-granularity Instruction Tuning

  • Multi-granularity instruction tuning uses constructed fine-grained and public instruction data to improve responses and multimodal interactions.The pipeline uses annotated in-context examples to guide GPT-3.5 in generating single- and multi-turn conversations; prior-stage data is also sampled during training.Source: §3.2.3 · p. 5

4 Experiments

Experiments evaluate GroundingGPT across image and video grounding and understanding, then examine qualitative examples and REC ablations.

4.1 Experimental Setup

  • Experiments use Vicuna-v1.5 for one epoch per stage, with 336 × 336 padded images, 64 sampled video frames, and three 2-second audio segments.Source: §4.1 · p. 5

4.2.1 Multi-modal Grounding

4.2.2 Multi-modal Understanding

  • GroundingGPT is state of the art on six image-understanding benchmarks, competitive on three, and outperforms baselines on three video-QA datasets.Source: §4.2.2 · p. 6, Table 4 · p. 7, Table 5 · p. 7
  • On POPE’s Adversarial subset, GroundingGPT-7B gains 14.07% accuracy and 8.18% F1 while decreasing the “Yes” metric by 27.60%.On VisWiz’s unanswerable subset, it improves from 67.8% to 84.0% versus LLaVA-1.5; the authors also report maintained or improved multimodal understanding and effective suppression of object hallucination.Source: §4.2.2 · p. 6, Table 6 · p. 7

4.2.3 Qualitative Results

  • Qualitative examples show fine-grained grounding while retaining comprehensive understanding of multimodal inputs.Source: §4.2.3 · p. 7, Figure 3 · p. 8

4.3 Ablation Study

  • Including fine-grained data in stage one reduces REC performance, which the authors attribute to limited early image understanding and possible training interference.Source: §4.3 · pp. 7–8, Table 7 · p. 8
  • An MLP adapter improves REC performance over a linear layer, and increasing LLM size also improves results.The authors attribute the adapter gain to preserving more image information and improving embedding mapping; larger models are attributed richer knowledge and stronger modeling.Source: §4.3 · p. 8, Table 8 · p. 8

5 Conclusion

GroundingGPT is presented as a unified end-to-end multimodal grounding model trained with a three-stage strategy and stage-specific datasets. The authors report strong performance across multimodal grounding and understanding benchmarks and make the model, code, and dataset accessible.

5 Conclusion

  • GroundingGPT uses a three-stage coarse-to-fine strategy with stage-specific training datasets.The authors report that experiments across a wide range of MLLM benchmarks confirm the model’s effectiveness and generality.Source: §5 · p. 8
  • The authors report remarkable performance on multimodal grounding and understanding tasks, supported by experiments across a wide range of MLLM benchmarks.Source: §5 · p. 8
  • The model, code, and dataset are made openly accessible to support further work.Source: §5 · p. 8

6 Limitations

The limitations concern sampled video and audio inputs, predominantly single-modal training data, and the model’s inability to produce segmentation masks or other finer-grained grounding outputs.

Sampling Strategy

  • Memory constraints require sampling video and audio, which can lose crucial information, especially in longer videos.The authors identify better modeling of longer videos and reduced information loss as future work.Source: §6 · p. 9

Cross-modal Inputs

  • Most training data consists of single-modal inputs, leaving simultaneous multimodal grounding as an area for future exploration.The authors propose investigating simultaneous spatial and temporal grounding on video and annotating such data.Source: §6 · p. 9

Grounding Ability

  • GroundingGPT cannot currently output finer-grained results such as segmentation masks.The authors plan to expand grounding tasks to support a broader range of requirements.Source: §6 · p. 9

A Implementation details

The appendix points to visual examples of grounding and understanding, and provides experiment-reproduction details and training hyperparameters.

A Implementation details

  • Figures 7 and 8 show multimodal grounding examples, while Figures 9–11 show multimodal understanding examples.Source: A Implementation details · p. 12
  • The appendix provides experiment configuration details for reproducing the model.It directs readers to Table 9 for hyperparameters covering all training stages.Source: A Implementation details · p. 12
  • Table 9 lists the hyperparameters for model training.Source: Table 9 · p. 12

B Training Dataset Source

The appendix catalogs datasets used to construct the training data across all three stages and notes that much of the source data requires formatting.

B Training Dataset Source

C Dataset Construction Templates

The first two training stages use task-specific instruction templates, with three example instructions shown for each task.

C Dataset Construction Templates

D Fine-grained Instruction-tuning Dataset Generation Prompts

GPT-3.5 generates instruction-tuning data using prompts for image descriptions, image conversations, and video grounding.

D Fine-grained Instruction-tuning Dataset Generation Prompts

E More Visualization

The appendix presents dataset-source and instruction-template information alongside examples of GroundingGPT’s grounding and understanding capabilities across modalities.

E More Visualization

  • Table 10 lists publicly available dataset sources used to construct the training data, while Table 11 describes instruction-template placeholders for image regions and video segments or events.The placeholders represent image-region coordinates, expressions corresponding to image regions, video time segments, and events to locate in videos.Source: Table 10 · p. 13, Table 11 · p. 14
  • Figures 4, 5, and 6 show the system messages and in-context examples used to generate detailed-description, conversation, and video-grounding datasets, respectively.Source: Figure 4 · p. 15, Figure 5 · p. 16, Figure 6 · p. 17
  • Appendix captions identify Figure 7 as a grounding example in image modality and Figure 8 in video and audio modalities; Figures 9–11 show understanding examples in image, video, and audio modalities, respectively.Source: Figure 7 · p. 18, Figure 8 · p. 19, Figure 9 · p. 20
Loading 2401.06071v5…