PaperlayerView exact revision on arXiv

Source-linked AI summary

GLaMM: Pixel Grounding Large Multimodal Model

Hanoona Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Erix Xing, Ming-Hsuan Yang, Fahad S. Khan

arXiv:2311.03356v3cs.CVcs.AI

TL;DR

Existing multimodal models either produce ungrounded responses, rely on user-specified regions, or lack dense pixel-level grounding and conversational breadth. GLaMM addresses this with grounded responses interleaved with segmentation masks, a new GCG benchmark, and the large-scale GranD dataset; it improves over baselines on GCG and transfers across multiple vision-language tasks.

  • Problem

    Existing models lack the combination of pixel-level grounding, multi-object understanding, and coherent conversational responses needed for visually grounded interaction.

  • Method

    GLaMM generates natural-language responses interleaved with object masks, accepts textual or visual prompts, and is trained using the GCG task and densely annotated GranD dataset.

  • Results

    GLaMM shows improved performance over baseline methods on GCG and is quantitatively evaluated across six benchmarks spanning grounding, captioning, and conversational tasks.

  • Takeaways & Limitations

    GCG unifies referring segmentation, captioning, phrase grounding, and vision-language conversation within a single densely grounded evaluation setting.

  • Takeaways & Limitations

    GranD’s automated dense annotations contain noise, while cleaner labels could improve pretrained representations at substantially higher annotation cost.

Abstract

from arXiv · show

Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations.

1. Introduction

GLaMM addresses the lack of pixel-level grounding and standardized benchmarks by generating conversational responses interleaved with segmentation masks and introducing the GCG task and GranD dataset. The model supports textual and visual prompts for multimodal interaction while unifying several previously isolated vision-language tasks.

  • Motivation: Existing multimodal and referring-segmentation models lack detailed pixel-level grounding, support only limited object grounding, or cannot sustain natural coherent conversations.These limitations restrict applications requiring detailed visual understanding and interactive multimodal reasoning.
  • Grounded Conversation Generation: GCG requires natural-language responses interleaved with object segmentation masks, unifying referring expression segmentation, captioning, phrase grounding, and vision-language conversations.The task was introduced to address the lack of benchmarks for visually grounded conversations.
  • Grounding-anything Dataset: GranD contains 7.5M unique concepts anchored in 810M regions, each with a segmentation mask, using an automated annotation pipeline with dedicated verification steps.The pipeline reduces the laborious manual effort required for large-scale region annotations.
  • GLaMM: GLaMM generates natural-language responses seamlessly integrated with object segmentation masks and accepts both textual and visual prompts for multimodal interaction.Visual prompts can specify regions of interest, enabling interaction at different textual and visual granularities.
  • Evaluation: The proposed GCG evaluation protocol measures grounded conversation generation while unifying multiple isolated tasks and filling a gap in standardized evaluation.The protocol is introduced alongside the GCG task to assess model efficacy in this setting.

2. Related Work

Prior LMMs established image-text alignment and instruction tuning but lacked region-specific understanding, while newer models support region-level interaction through location bins, bounding boxes, or RoI-aligned features. GLaMM extends this landscape with pixel-level grounding, multi-turn conversations, flexible image/region inputs, and a large-scale instance-level grounded dataset.

  • Early LMMs: Early LMMs such as BLIP-2, LLaVA, InstructBLIP, and MiniGPT-4 align image-text features before instruction tuning but lack region-specific understanding.Other representative models include Otter, mPLUG-Owl, LLaMa-Adapter, Video-ChatGPT, and InternGPT.
  • Region-aware LMMs: Recent region-aware LMMs include Kosmos-2, Shikra, GPT4RoI, VisionLLM, Ferret, and All-Seeing.Several methods use location bins and bounding boxes, while GPT4RoI additionally uses spatial boxes and RoI-aligned features with region-text training pairs.
  • Landscape categorization: The LMM landscape separates textual-response models from region-capable models using external vision modules or relying exclusively on LMMs for region understanding.This categorization is presented alongside a comparison emphasizing region-level input, multi-region handling, and grounded outputs.
  • GLaMM’s distinction: GLaMM distinctly offers pixel-level grounding, multi-turn conversations, and operation on both input images and specific regions.The work also provides a large-scale instance-level grounded visual understanding dataset supporting generalization across multiple vision-language tasks.

3. Method

GLaMM combines global, region, and pixel-level visual processing with language generation to produce responses interleaved with object segmentation masks. The method introduces Grounded Conversation Generation (GCG) and a densely annotated GranD dataset to support and evaluate this setting.

  • GLaMM Overview: GLaMM generates natural-language responses seamlessly integrated with corresponding object segmentation masks for visually grounded conversation.This addresses limitations including ungrounded text, single-object grounding, user-specified regions, and absent dense pixel-level grounding.
  • Architecture: Five components—global image encoder, region encoder, LLM, grounding image encoder, and pixel decoder—support scene-, region-, and pixel-level grounding.The architecture accepts textual and optional image-level or region-level visual prompts, enabling interaction at multiple granularities.
  • Scene- and Region-Level Understanding: The global encoder and Vicuna-based LLM provide holistic scene understanding, while the region encoder uses hierarchical CLIP features and RoIAlign to interpret user-specified regions.The region encoder produces a 14x14 feature map and projects region features into language space through the shared vision-to-language layer.
  • Pixel-Level Grounding: The grounding image encoder and SAM-like pixel decoder generate binary segmentation masks activated by specialized <SEG> tokens and language-to-prompt projections.The model transforms embeddings corresponding to <SEG> tokens into the decoder’s feature space before producing masks M, with M_i ∈ {0, 1}.
  • Grounded Conversation Generation: GCG requires image-level captions whose phrases are directly tied to corresponding segmentation masks, producing detailed responses with interleaved text and masks.The method introduces a benchmarking protocol for this novel visually grounded conversation setting, while GranD’s stage-wise pipeline automatically creates densely annotated captions for GCG training.

4. Data Annotation Pipeline

The GranD annotation pipeline builds dense, multi-level scene understanding from object localization through relationships, scene graphs, and extended context. Applied to 11M images, it produces a large-scale grounded dataset, while GranDf adds higher-quality manually and automatically derived GCG training data.

  • GranD Annotation Pipeline: The four-level pipeline progresses from object labels, masks, attributes, and depth to relationships, hierarchical scene graphs, dense captions, and enriched context.Level-3 uses the scene graph to query an LLM for dense captions, while Level-4 extracts landmark, historical, interaction, and predictive information.
  • GranD Annotation Pipeline: Level-1 retains boxes detected by at least two models after class-agnostic NMS and IoU-based filtering, then generates region attributes.This procedure combines multiple state-of-the-art object detectors to reduce false positives before attribute generation.
  • GranD Dataset: 810M regions and 7.5M unique concepts result from annotating 11M diverse, high-resolution, privacy-compliant SAM images.The dataset also contains 84M referring expressions, 22M grounded short captions, and 11M densely grounded captions.
  • GranDf Dataset: GranDf contains 214K image-grounded text pairs, including 2.5K validation and 5K test samples, to provide higher-quality fine-tuning data.It combines a manually annotated subset with data repurposed from existing open-source datasets.
  • GranDf Dataset: A manual benchmark set contributes around 1000 focused samples for training and 1000 for evaluation by refining referring expressions to match SAM ground-truth masks.The annotations use GranD’s automatic outputs as a base and target the GCG task.

5. Experiments

GLaMM is evaluated across six benchmarks spanning grounded conversation, segmentation, captioning, question answering, and phrase grounding. The reported experiments include GranD/GranDf-trained GCG, region-level captioning with competitive fine-tuned and zero-shot performance, and image-level captioning.

  • Evaluation Scope: GLaMM is quantitatively evaluated on six benchmarks, including GCG, referring-expression segmentation, region- and image-level captioning, conversational question answering, and phrase grounding.The first four benchmarks are presented in the main text, while conversational question answering and phrase grounding are discussed in Appendix B.
  • Grounded Conversation Generation: For GCG, GLaMM is pretrained on GranD and then fine-tuned on the GranDf dataset.
  • Qualitative Results: Qualitative examples demonstrate GLaMM’s region understanding, pixel-level groundings, and conversational abilities across downstream tasks.The examples cover region-level understanding, referring-expression segmentation, and image-level captioning.
  • Region-Level Captioning: On region-level captioning, GLaMM is evaluated on Visual Genome and refCOCOg using METEOR and CIDEr, improving over GRiT and GPT4RoI after fine-tuning.It also demonstrates robust zero-shot performance, highlighting the contribution of GranD’s region-text pairs.

6. Conclusion · A. Additional Implementation Details

GLaMM generates natural-language responses intertwined with object segmentation masks, while introducing Grounded Conversation Generation, its evaluation protocol, and the GranD dataset to support research on visually grounded conversations.

  • 6. Conclusion: GLaMM is introduced as the first model capable of intertwining natural-language responses with object segmentation masks.This design enables responses that are visually grounded at the object level.
  • 6. Conclusion: Intertwined responses and segmentation masks enable enhanced multimodal user interactions.
  • 6. Conclusion: The work introduces Grounded Conversation Generation as a novel task for visually grounded conversations.The task addresses the lack of standardized benchmarks in this setting.
  • 6. Conclusion: A comprehensive evaluation protocol is established for the new Grounded Conversation Generation task.
  • 6. Conclusion: The authors create GranD, the Grounding-anything Dataset, to facilitate research and model development.The passage characterizes GranD as large-scale and densely annotated.
  • A. Additional Implementation Details: The supplied passage for Additional Implementation Details contains no implementation-specific claims.

A.1. Evaluation Metrics … A.3. Finetuning on Downstream Tasks

The appendix defines mask recall through jointly thresholded mask overlap and textual similarity, then details GLaMM’s architecture, training losses, GranD pretraining, and downstream-task finetuning. The model uses a 7B Vicuna backbone with frozen global encoders, finetuned grounding components, and LoRA adaptation of the language model.

  • A.1. Evaluation Metrics: Mask recall matches predicted and ground-truth masks one-to-one, then counts a pair as a true positive only when IoU and BERT similarity both exceed 0.5.The metric first computes IoU for assigned mask pairs before applying the textual similarity criterion.
  • A.2. Model Architecture and Training: GLaMM uses a 7B-parameter Vicuna LLM, with its region encoder, grounding image encoder, and pixel decoder designs motivated by GPT4RoI and LISA.The V-L and L-P layers use two-layer GELU MLPs following LLaVA-v1.5.
  • A.2. Model Architecture and Training: Training combines autoregressive cross-entropy for text with a linear combination of per-pixel binary cross-entropy and DICE losses for segmentation.The global and grounding image encoders remain frozen; the region encoder, projection layers, and pixel decoder are fully finetuned, while the LLM uses LoRA with α = 8.
  • A.2.1 Pretraining on GranD: During GranD pretraining, GLaMM jointly learns referring expression segmentation, region-level captioning, image-level captioning, and grounded conversation generation.Pretraining uses batch size 160 for 35K iterations, initializes from GPT4RoI, and applies LORA-8; the resulting model is called GLaMM (ZS).
  • A.3. Finetuning on Downstream Tasks: GLaMM is finetuned on GranDf for grounded conversation generation using batch size 160 for 5K iterations.GranDf combines multiple open-source datasets repurposed for GCG using GPT4.
  • A.3. Finetuning on Downstream Tasks: For referring expression segmentation, finetuning uses refCOCO, refCOCO+, and refCOCOg, while region-level captioning uses refCOCOg and Visual Genome.The referring-segmentation model is denoted GLaMM (FT) in Table 4.
  • A.3. Finetuning on Downstream Tasks: Image-level captioning uses LLaVA-Instruct-150K, LLaVA-bench uses LLaVA-Instruct-80K, and pretraining and finetuning run on eight NVIDIA A100-40GB GPUs.These dataset assignments are specified for the respective downstream tasks.

A.4. Automated Dataset Annotation Pipeline · A.4.1 LLM Prompts and In-context Learning

The GranD construction pipeline uses hierarchical, fully automated visual annotation with diverse open-source models, SAM-based mask generation, and LLM feedback. It also uses structured scene-graph prompts and in-context examples to produce dense captions, extra scene context, and landmark categories.

  • A.4. Automated Dataset Annotation Pipeline: Level-1 combines models for image tagging, object localization, attribute generation, and depth estimation, while Level-2 adds scene descriptions, landmark categorization, phrase extraction, and phrase grounding.Named models include Tag2Text, RAM, Co-DETR, EVAv02, OWL-ViT, POMP, GRiT, GPT4RoI, MiDAS, BLIP-2, LLaVA-v1.5, SpaCy, and MDETR.
  • A.4. Automated Dataset Annotation Pipeline: The pipeline organizes visual annotation into multiple hierarchical levels to construct the GranD dataset.The supplied passage introduces a fully automated, multilevel design for dataset construction.
  • A.4. Automated Dataset Annotation Pipeline: Most regions use SAM annotations by matching detected labeled regions against SAM’s class-agnostic regions.Unmatched regions are processed by running SAM with a bounding-box query to obtain masks.
  • A.4. Automated Dataset Annotation Pipeline: The open-source pipeline uses an LLM chain-of-thought feedback loop without human feedback, enabling scalable generation of dense noisy labels for more images.The passage connects this scalability to pretraining a larger LMM.
  • A.4.1 LLM Prompts and In-context Learning: LLaVA-v1.5-13B assigns landmark categories to each image using the primary and fine-grained categories summarized in Table 7.The supplied passage identifies LLaVA-v1.5-13B as the landmark-categorization model.
  • A.4.1 LLM Prompts and In-context Learning: For dense captioning, objects, attributes, and relationships are arranged hierarchically to form a visual scene graph.The scene graph is the structured representation used for the captioning prompt.
  • A.4.1 LLM Prompts and In-context Learning: Vicuna-v1.5-13B is also queried to generate additional context about the visual scene.The corresponding prompt is shown in Figure 6(b).

B. Additional Downstream Tasks … C.5. Conditional Image Generation

GLaMM extends beyond grounded conversation generation to phrase grounding and conversational question answering, while qualitative results demonstrate dense captioning, multi-object segmentation, region understanding, and conditional image generation.

  • B.1. Phrase Grounding: GLaMM is adapted for phrase grounding by treating GCG answers as questions and caption segments containing groundings as phrases to localize pixel-level groundings.Phrases are enclosed within <p> and </p> tokens during training.
  • B.2. Conversational Style Question Answering: On LLaVA-Bench, GLaMM is evaluated on conversation question-answering, detailed descriptions, and complex reasoning against previous open-source models.The reported discussion notes that GLaMM’s performance is on par with LLaVA-1.5.
  • C. Additional Qualitative Results: Additional qualitative examples are provided to further illustrate GLaMM’s capacity.This section introduces the subsequent qualitative demonstrations.
  • C.1. Grounded Conversation Generation (GCG): On GranDf, GLaMM produces dense captions together with dense pixel-level groundings of the captions.These results are shown qualitatively in Fig. 7.
  • C.2. Referring Segmentation: GLaMM understands natural-language queries and segments the corresponding objects, including multiple objects through multi-round conversations.The qualitative results are presented in Fig. 8.
  • C.3. Region-level Captioning: For region-level understanding, GLaMM generates detailed descriptions of user-specified image regions.The qualitative results are shown in Fig. 9.
  • C.4. Image-level Captioning: For image-level captioning, GLaMM generates dense captions for images.The qualitative results are shown in Fig. 10.
  • C.5. Conditional Image Generation: For conditional image generation, GLaMM produces a segmentation mask from the user query and passes it with the prompt to Stable Diffusion XL inpainting.Stable Diffusion then generates the final output from the mask and user prompt.

C.6. Conversations

GLaMM supports multi-purpose multimodal conversations by accepting text and/or region prompts and producing responses with text and/or segmentation masks. This flexibility, including multimodal input-output behavior, emerges mainly from GranD pretraining despite not being explicitly trained for such scenarios.

  • Multimodal conversations: GLaMM engages in multi-purpose conversations using textual and/or regional prompts.The model is described as a generic conversational model that accepts prompts in the form of text and/or region.
  • Multimodal conversations: GLaMM can interleave natural-language responses with corresponding pixel-level segmentation masks.Qualitative grounded-conversation results show detailed responses whose phrases are grounded using pixel-level masks.
  • Multimodal conversations: GLaMM’s multimodal interactions include multi-turn dialogue, detailed descriptions, region-specific inquiries, and grounded conversations.These interactions are presented as demonstrating adaptability in intricate visual-language exchanges while retaining LLM reasoning capabilities.
  • Multimodal conversations: GLaMM supports multimodal inputs and multimodal outputs within a single conversation.Figure 13 illustrates the model processing multimodal inputs and responding with multimodal outputs.
  • Multimodal conversations: This behavior emerges mainly from GranD pretraining, although GLaMM was not explicitly trained for such scenarios.GranD presents images to the LMM in different contexts, supporting the observed multimodal conversational behavior.

D. Dataset Visualization

This section presents additional samples from the GranD and GranDf datasets to illustrate the functionalities they offer, with visualizations provided in Figures 15 and 14.

  • Dataset samples: Additional GranD dataset samples are provided to clarify the dataset’s offered functionalities.The section directs readers to Fig. 15 and Fig. 14 for these examples.
  • Dataset samples: Additional GranDf dataset samples are likewise included to support understanding of its functionalities.The visual examples are referenced in Fig. 15 and Fig. 14.

E. Limitations and Future Work · F. Ethics and Societal Impact

The authors identify noise in GranD’s automated dense labelings and propose cleaner, cost-effective annotation and expanded modalities as future directions. They report privacy protections and no known strong biases, while urging responsible use of GranD and GLaMM.

  • E. Limitations and Future Work: GranD’s large-scale automated pipeline produces dense labelings that remain somewhat noisy.These labelings are important for pretraining.
  • E. Limitations and Future Work: A cleaner, higher-quality dataset could further improve pretrained representations.The authors note that achieving this would require significantly higher annotation cost.
  • E. Limitations and Future Work: Future work could develop a cost-effective annotation pipeline to reduce noise in dense labeling.This direction aims to improve annotation quality without incurring the full cost of highly clean data collection.
  • E. Limitations and Future Work: The GLaMM framework could be expanded to support video and 3D modalities.The paper identifies multimodal expansion as an additional research direction.
  • F. Ethics and Societal Impact: GranD uses SAM images with de-identified personal information.Faces and license plates are obscured in the dataset.
  • F. Ethics and Societal Impact: The authors report no strong biases or discrimination in GranD to the best of their knowledge.This statement is qualified by the authors’ stated level of knowledge.
  • F. Ethics and Societal Impact: The authors urge responsible use of GranD and GLaMM.They frame responsible use as a way to promote research progress while safeguarding privacy.
Loading 2311.03356v3…