PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models

Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, Jianwei Yang

arXiv:2312.02949v1cs.CV

TL;DR

Existing LMMs typically separate visual grounding from chat, and available grounding data lacks the longer interactions needed for grounded visual chat. The paper creates 150K grounded visual-chat instances, connects a language model with a segmentation model, and introduces Grounding-Bench. LLaVA-Grounding outperforms other open-source LMMs on Grounding-Bench while remaining competitive on classic grounding benchmarks.

  • Problem

    Existing LMMs struggle to combine visual grounding with chat because grounded visual-chat data is scarce and prior systems lack pixel-wise grounding and referring.

  • Method

    The paper creates 150K grounded visual-chat instances, connects an LMM with OpenSeeD, and introduces Grounding-Bench for concurrent chat-and-grounding evaluation.

  • Results

    LLaVA-Grounding outperforms other open-source LMMs on Grounding-Bench and achieves competitive results on RefCOCO/+/g and Flickr30K Entities.

  • Takeaways & Limitations

    The model combines visual chat and grounding capabilities while supporting both object- and pixel-level grounding.

Abstract

from arXiv · show

With the recent significant advancements in large multi-modal models (LMMs), the importance of their grounding capability in visual chat is increasingly recognized. Despite recent efforts to enable LMMs to support grounding, their capabilities for grounding and chat are usually separate, and their chat performance drops dramatically when asked to ground. The problem is the lack of a dataset for grounded visual chat (GVC). Existing grounding datasets only contain short captions. To address this issue, we have created GVC data that allows for the combination of grounding and chat capabilities. To better evaluate the GVC capabilities, we have introduced a benchmark called Grounding-Bench. Additionally, we have proposed a model design that can support GVC and various types of visual prompts by connecting segmentation models with language models. Experimental results demonstrate that our model outperforms other LMMs on Grounding-Bench. Furthermore, our model achieves competitive performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K Entities. Our code will be released at https://github.com/UX-Decoder/LLaVA-Grounding .

1. Introduction

Existing LMMs struggle to combine fine-grained grounding with visual chat because grounded visual-chat data and suitable model designs are scarce. LLaVA-Grounding addresses these gaps with grounded-chat data, an integrated grounding architecture, and Grounding-Bench.

  • 1. Introduction: Earlier LMMs often separate grounding from chat, produce short captions when grounding, and lack pixel-wise grounding and referring capabilities.Their grounding outputs are commonly restricted to coordinates represented as text, while grounded-chat data remains scarce.
  • 1. Introduction: 150K grounded visual chat instances are annotated by matching GPT-4-generated chat noun phrases to human-labeled object-detection instances.The pipeline combines human-labeled instances with GPT-4 matching to reduce annotation noise.
  • 1. Introduction: The architecture connects language-model output features to OpenSeeD, enabling object- and pixel-level grounding while relieving the language model of localization.This design supports box and pixel-level grounding simultaneously.
  • 1. Introduction: Grounding-Bench evaluates chat and phrase grounding concurrently across conversation, detailed description, and complex reasoning contexts.It introduces grounded recall, grounded precision, and F1 metrics, using GPT-4 to assess semantic matching of free-form phrases.
  • 1. Introduction: Table 1 compares LMMs by their input referring and output grounding formats.The paper uses this comparison to position LLaVA-Grounding's supported prompt and output types relative to other LMMs.
  • 1. Introduction: LLaVA-Grounding combines grounded visual chat data, a model connecting an LMM with a grounding model, and Grounding-Bench for joint evaluation.The paper presents these contributions as responses to limitations in existing grounded visual chat systems.

2. Method

The method combines grounded visual-chat data, a multimodal architecture, and a three-stage training and benchmarking pipeline to support conversational responses with box- and pixel-level grounding.

  • 2.3. Network Architectures: The model supports object- and pixel-level grounding with mark, click, box, and scribble visual prompts.Visual prompts can be inserted through ⟨obj⟩ placeholders, and corresponding masks are provided for referred objects.
  • 2.3. Network Architectures: LLaVA-Grounding connects an LMM to OpenSeeD so language-model grounding features generate bounding boxes and masks.A Semantic-SAM prompt encoder converts image and visual prompts into projected visual tokens, while ⟨seg⟩-aligned hidden features are projected into grounding space.
  • 2.4. Training: Training proceeds through alignment pretraining, grounded-visual-chat instruction tuning, and extension to visual prompts.Stage 1 uses RefCOCO/+/g, COCO, Visual Genome, LLaVA instruction data, and Flickr30K Entities, including granularity alignment for the grounding model.

3. Experiments

Experiments evaluate LLaVA-Grounding on grounded visual chat, classic grounding benchmarks, visual prompts, qualitative capabilities, and ablations. The model outperforms open-source grounding methods on Grounding-Bench, performs strongly across REC, RES, and phrase grounding, and retains chat capability while supporting multiple prompt types.

  • 3.1. Experimental Settings: The experiments initialize a Vicuna-7B language model, OpenSeeD Tiny grounding model, and Semantic-SAM Tiny interactive encoder, with staged training of grounding, prompt, projection, and language components.The experimental plan covers Grounding-Bench, REC/RES, visual prompts, visualizations, and ablations.
  • 3.2. Grounding-Bench: LLaVA-Grounding outperforms all open-source methods in grounded response and chat scores on Grounding-Bench, except chat-only CogVLM-Chat and LLaVA.GPT4-V achieves the best grounded detailed-description performance with SoM, while LLaVA-Grounding leads the open-source methods.
  • 3.3. Traditional Grounding Benchmarks: LLaVA-Grounding is the only evaluated LMM that performs effectively on both REC and RES, surpassing all LMMs on RES and phrase grounding.It also outperforms all LMMs except the larger CogVLM-Grounding model on REC.
  • 3.4. Visual Prompts: Boxes yield higher referring accuracy than clicks in COCO object classification, while the model supports visual prompts including marks, clicks, and boxes.The mark-prompt experiment evaluates phrase grounding on Flickr30K using overlaid ground-truth masks and center-positioned marks.
  • 3.5. Visualizations: Visual comparisons show that LLaVA-Grounding generates longer descriptions, detects more objects, maintains high precision, and grounds open-set concepts such as dragons.The model also supports visual chat with boxes or clicks and can use marks after fine-tuning with mark data.
  • 3.6. Ablation Studies: Grounding-model detachment slightly improves chat performance but significantly compromises grounding, showing that the grounding model is important for grounded visual chat quality.Using 50 grounding queries is sufficient for both REC and RES and achieves optimal performance.

4. Related Work

Related work spans multimodal instruction tuning, fine-grained vision-language understanding, visual grounding models, and grounding-capable LMM architectures. The paper positions its dataset and benchmark as addressing the trade-off between grounding and chat abilities.

  • Multimodal Models: LMM research has improved instruction following through visual instruction-tuning data, while other models target fine-grained understanding using language-guided tokens or bounding-box features.Examples include LLaVA, MiniGPT-4, VisionLLM, GPT4RoI, and PVIT.
  • Visual Grounding: Visual grounding methods locate image objects from text by aligning language with image regions, with grounded pretraining improving localization and visual-language understanding.Representative approaches include MDETR, GLIP, GLIPv2, and Grounding-DINO.
  • Grounding LMM Architectures: Grounding LMMs fall into coordinate-generating models, models using separate grounding modules, and models feeding language-model outputs into grounding models.LISA represents the third category but focuses primarily on segmentation rather than chat interactions.
  • Grounded Visual Chat: The paper’s dataset and benchmark assess the compositional combination of grounding and chat interactions rather than emphasizing only one capability.This addresses the trade-off between grounding and chat abilities found in previous work.

5. Conclusion

LLaVA-Grounding combines visual chat and grounding through a new dataset, integrated architecture, and Grounding-Bench evaluation. It outperforms other open-source language models on chat and grounding tasks while remaining effective on traditional grounding benchmarks, though its semantic scope is limited.

  • LLaVA-Grounding consistently outperforms other open-source LM models in both chat and grounding tasks on Grounding-Bench.
  • The paper contributes a grounded visual chat dataset, an end-to-end architecture integrating grounding with a language model, and Grounding-Bench covering chat and grounding.
  • LLaVA-Grounding also excels on traditional grounding benchmarks, demonstrating versatility beyond Grounding-Bench.
  • The paper identifies limited semantic scope as a limitation and proposes extending the dataset and labeling methods to open-vocabulary settings.

Supplementary Material

Figure 4 presents the LLaVA-Grounding network architecture for supporting visual prompts and distinguishes frozen from trainable components.

  • Figure 4 depicts LLaVA-Grounding’s network architecture for supporting visual prompts.
  • Snowflake symbols mark frozen components, while fire symbols mark trainable components.

A. More details about visual prompt

Visual prompt support adds clicks, boxes, and scribbles by encoding prompts into the language space and replacing a special token embedding. Training uses GPT-4-generated matched data, while RefCOCO supplements instance discrimination.

  • Visual prompts include clicks, boxes, and scribbles, supported by training only the visual prompt encoder to align prompt features with the language space.
  • A pretrained Semantic-SAM encoder maps each visual prompt through a projection layer, replacing the ⟨obj⟩ token’s input embedding with the resulting feature.
  • GPT-4 matches noun phrases or question entities to ground-truth instances, enabling sampling of varied visual prompts from the annotated instances.
  • The visual-prompt data labels part of 150K LLaVA instruction-tuning examples and retains high-quality visual chat because GPT-4 generated them.
  • RefCOCO supplements the data because discussions of only a few main objects are insufficient for distinguishing different instances.

B. Implementation details

The supplementary material provides experiment configurations and hyperparameters for both training stages.

  • Table 15 reports the hyperparameters used for stage 1 and stage 2.

C. Instruction lists for different data format.

This section presents instruction lists used to construct instruction-following data from different data formats.

  • Instruction lists are used to construct instruction-following data across different data formats.
  • The lists provide format-specific instructions for preparing the corresponding data.
  • Their purpose is data construction rather than evaluation or model comparison.

D. More visualizations

This section provides additional visualizations, including instruction templates and examples of grounded detailed descriptions generated by LLaVA-G and other LMMs. The examples contrast LLaVA-G's detailed scene descriptions with shorter descriptions from comparison models.

  • More visualizations: Tables 20 and 21 compare grounded detailed descriptions generated by LLaVA-G and other grounding LMMs using each model's recommended prompt.
  • More visualizations: Tables 16–19 list instructions and templates for referring expressions, brief image descriptions, grounding captions, and region captions.
  • More visualizations: LLaVA-G describes a fantasy scene with people, horses, and two dragons while specifying their relative positions.
  • More visualizations: Comparison outputs are shorter, including CogVLM-Grounding's brief horse-riding description and models' concise pizza descriptions.
  • More visualizations: LLaVA-G gives a detailed dining-table description covering three pizzas, bowls, toppings, and their approximate locations.
Loading 2312.02949v1…