PaperlayerView exact revision on arXiv

Source-linked AI summary

Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models

Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, Xiaojuan Qi

arXiv:2404.13013v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Groma addresses the localization gap in MLLMs by embedding localization into image tokenization and using region tokens for user-specified inputs and grounded outputs. It reports stronger performance on referring and grounding benchmarks, while its current implementation does not support free-form region inputs or pixel-level grounding.

  • Problem

    Current MLLMs typically fall short of localization and cannot ground their understanding in visual context.

    Source: §1 · p. 2

  • Method

    Groma discovers regions with a proposer, encodes them as location-anchored region tokens, and integrates those tokens into instructions and responses; it also trains on grounded chat data generated with GPT-4V.

    Source: §1 · pp. 2–3

  • Results

    Groma outperforms comparable MLLMs on established referring and grounding benchmarks and surpasses alternatives by over 10% AR on the adapted LVIS grounding benchmark.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Embedding localization into image tokenization offers a grounded MLLM approach that avoids external localization modules and unifies referring with visual grounding.

    Source: §6 · p. 14

    The current implementation does not support free-form region inputs or pixel-level grounding.

    Source: §6 · p. 14

Abstract

from arXiv · show

We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded and fine-grained visual perception ability. Beyond holistic image understanding, Groma is adept at region-level tasks such as region captioning and visual grounding. Such capabilities are built upon a localized visual tokenization mechanism, where an image input is decomposed into regions of interest and subsequently encoded into region tokens. By integrating region tokens into user instructions and model responses, we seamlessly enable Groma to understand user-specified region inputs and ground its textual output to images. Besides, to enhance the grounded chat ability of Groma, we curate a visually grounded instruction dataset by leveraging the powerful GPT-4V and visual prompting techniques. Compared with MLLMs that rely on the language model or external module for localization, Groma consistently demonstrates superior performances in standard referring and grounding benchmarks, highlighting the advantages of embedding localization into image tokenization. Project page: https://groma-mllm.github.io/.

1 Introduction

Current MLLMs excel at image-level reasoning but typically lack localization, limiting their ability to ground understanding in visual context. Groma embeds localization in image tokenization, represents regions as tokens, and reports stronger results on referring and grounding benchmarks.

1 Introduction

  • Current MLLMs typically lack localization, limiting their ability to ground understanding in visual context for applications such as robotics, autonomous driving, and augmented reality.Image captioning and visual question answering are examples of tasks where MLLMs demonstrate advanced understanding and reasoning.Source: §1 · p. 2
  • Coordinate-output LLMs struggle with high-resolution inputs and dense prediction, while external localization modules add latency by processing the image separately.The paper frames these as limitations of two existing approaches to localization.Source: §1 · p. 2
  • Groma decouples localization from recognition and uses the visual tokenizer for localization rather than relying on an LLM or an external module.The design draws inspiration from open-vocabulary detection and treats localization as a perceptual task.Source: §1 · p. 2
  • Region tokens are anchored to image locations, enabling grounded text without requiring the LLM to regress coordinates; user-specified boxes can also be encoded as tokens for referential dialogue.Groma combines region tokenization with standard image tokenization.Source: §1 · p. 2
  • High-resolution images can be used for tokenization while downsampled image tokens go to the LLM, and region tokens unify referring inputs with grounded outputs.The paper contrasts this with separate designs for grounding outputs and referring inputs.Source: §1 · p. 3
  • Training combines datasets with region-level annotations spanning objects, relationships, and detailed descriptions with Groma Instruct, a grounded chat dataset made using visual and textual prompts and GPT-4V.The authors describe Groma Instruct as the first grounded chat dataset constructed with both visual and textual prompts.Source: §1 · p. 3
  • Groma outperforms comparable MLLMs on established referring and grounding benchmarks and exceeds alternatives by over 10% AR on the adapted LVIS grounding benchmark.The experiments also report strong image-level understanding and reasoning on conversational VQA.Source: §1 · p. 3

2 Related Work

Prior MLLMs mainly support image-level tasks, while later work adds region-level understanding through several approaches to representing regions and producing localization. Groma shares this focus but proposes a different perspective on localization.

Image-level MLLMs.

  • Image-level MLLMs support tasks such as captioning and visual question answering, while region-level MLLMs aim for more granular visual-language interaction.Common architectures connect a visual encoder to an LLM through a vision-language connector.Source: Image-level MLLMs · p. 4
  • Prior region-level approaches represent boxes with quantized or numeric coordinates, pool features within regions, use a visual sampler, or add off-the-shelf models for pixel-level grounding.Examples include Kosmos-2 [38] and Shikra [7] for box representations, GPT4RoI [63] and RegionGPT for pooling, Ferret [57] for free-form inputs, and LISA [25] for segmentation.Source: Region-level MLLMs · p. 4
  • Groma shares prior work's focus on region-level understanding and grounding but distinguishes itself by proposing a new perspective on improving MLLM localization.The passage describes grounded conversation generation as an advance in work including GLaMM [42] and LLaVA-Ground [61].Source: Region-level MLLMs · p. 4

3 Method

Groma combines scene-level image tokens, proposed regions, region-level tokens, and a language model in one architecture. Its region proposer discovers candidate regions, and its region encoder converts boxes into semantically aligned tokens.

3.1 Model Architecture

  • Groma's architecture comprises an image encoder for scene-level tokenization, a region proposer, a region encoder, and a large language model.The model also integrates global image tokens and local region tokens to support referring and grounding.Source: §3.1 · p. 4, Figure 3 · p. 5

Image Encoder.

  • The image encoder uses DINOv2 [37] at 448×448 resolution and merges neighboring patch tokens in 2D to reduce the sequence passed to the language model.The paper selects DINOv2 [37] for compatibility with high-resolution inputs and fine-grained localization features; the unmerged sequence would contain 1024 tokens.Source: Image Encoder · p. 5

Region Proposer.

  • A class-agnostic Deformable DETR [67] head proposes 300 regions per image, using a binary classifier to score localization quality before NMS and objectness filtering.Its input feature pyramid is built from the image encoder's last four layers.Source: Region Proposer · p. 5

Region Encoder.

  • The region encoder uses multiscale ROIAlign on a feature pyramid from the image encoder's last three layers to turn user boxes or proposed boxes into region tokens.The authors say these tokens are semantically aligned with their regions, unlike numeric positions [7] or discrete location tokens [6] [38].Source: Region Encoder · p. 5

3.1 Model Architecture

  • Groma uses pretrained Vicuna 7B as its language model and an MLP to project image and region tokens into the language model's feature space.The projection follows LLaVA v1.5 [31].Source: Region Encoder · p. 6

3.2 Input and Output Formatting

Groma represents regions with tokens that can enter user instructions and model responses. Proxy tokens let the model refer to regions in grounded outputs, while user-specified regions are encoded in the same way as proposed regions.

3.2 Input and Output Formatting

  • Groma unifies referring inputs and grounded outputs through region tokens despite their different task formulations.The same format supports user-specified regions as inputs and visually grounded answers as outputs.Source: §3.2 · p. 6

Grounding Output.

  • Each region token is anchored to the location of its proposal, allowing the language model to ground text by referring to that token.Because region tokens are continuous embeddings, the model introduces proxy tokens such as <r1> to make them addressable in text.Source: Grounding Output · p. 6
  • In grounded output, proxy tokens identify referenced regions alongside phrases describing image content.The example uses region references to connect phrases about a dog, frisbee, and fallen man to specific image regions.Source: Grounding Output · p. 6
  • The output format uses <p> and </p> around grounded phrases, <roi> and </roi> around referenced regions, and [grounding] to prompt a grounded response.Image and region placeholders are replaced by visual tokens before the input is passed to the LLM.Source: Grounding Output · p. 6

Referring Input.

  • For a user-specified region, Groma encodes the box into a region token and assigns it a proxy token, like a region proposal.The resulting token can be inserted into the instruction to support referential dialogue.Source: Referring Input · p. 6
  • The referring-input prompt can ask the model to identify a specified region, such as <r10>.The example's answer describes the region as a cat sleeping on a wooden bench.Source: Referring Input · p. 6

3.3 Model Training

Groma training proceeds through detection pretraining, vision-language alignment pretraining, and instruction finetuning. Dedicated detection pretraining supports localization, while later stages align image- and region-level features with language and improve conversation ability.

3.3 Model Training

  • Training has three stages: detection pretraining, alignment pretraining, and instruction finetuning, with datasets enumerated by stage.The stages target localization, image- and region-level vision-language alignment, and conversation capability, respectively.Source: §3.3 · p. 7, Table 1 · p. 7

Detection Pretraining.

  • Detection pretraining uses COCO, Objects365, OpenImages, and V3Det with box supervision, omitting category information while keeping the image encoder frozen.A two-million-image SA1B subset adds box annotations spanning object instances, parts, and background regions.Source: Detection Pretraining · p. 7

Alignment Pretraining.

  • Alignment pretraining uses captioning, referring-expression, region-captioning, and grounded-caption datasets, then finetunes on selected modules while freezing others.Instruction finetuning unfreezes the language model and adds LLaVA Instruct, ShareGPT-4V, and Groma Instruct for conversation and grounding.Source: Alignment Pretraining · p. 7, Instruction Finetuning · p. 8

Discussions.

  • Dedicated detection pretraining gives Groma localization ability and lets it learn from millions of box annotations without involving the language model.The authors describe this scale of box pretraining as computationally prohibitive for classic MLLMs.Source: Discussions · p. 8

4 GPT4V-assisted Grounded Conversation Generation

Prior visual-dialogue data often lack fine-grained region details, and free-form dialogue data are insufficient for long-form grounded responses. Groma Instruct uses GPT-4V and prompted Visual Genome images to generate 30k grounded conversations, with region markers and text context helping structure the inputs.

4 GPT4V-assisted Grounded Conversation Generation

  • Free-form visual dialogues derived from coarse image descriptions lack precise region detail, and are insufficient for long-form grounded responses.Grounded responses differ in format from normal responses, making generalization to long-form conversations challenging.Source: §4 · p. 8
  • Groma Instruct contains 30k visually grounded conversations generated with GPT-4V from Visual Genome images selected for dense region annotations.The dataset is intended for instruction finetuning.Source: §4 · p. 8
  • After removing highly overlapping boxes, the authors generally retain 3–10 regions per image and mark each region with a bright number for GPT-4V.The numeric markers let GPT-4V refer to particular image regions by number.Source: §4 · p. 8
  • GPT-4V also receives region descriptions, image descriptions, and image-based Q&A from COCO and Visual Genome; this context helps reduce hallucinations and resolve prompt ambiguities.The textual context is optional, but the authors report it as useful.Source: §4 · pp. 8–9
  • The detailed-description portion of LLaVA Instruct is replaced with ShareGPT-4V because it contains severe hallucinations [8].Source: §4 · p. 8
  • Manually designed grounded-chat examples serve as GPT-4V context, and post-processing removes conversations that fail the predefined format.The authors use in-context learning to generate conversations in a uniform format.Source: §4 · p. 9

5 Experiments

Groma is evaluated on grounding, referring, and image-based conversation, with qualitative examples and design and training ablations. Experiments use DINOv2-L/14 and Vicuna-7B v1.5.

5.2 Grounding Benchmark Results

  • Groma outperforms similarly sized generalist models across RefCOCO, RefCOCO+, and RefCOCOg metrics, and averages above Qwen-VL despite its stronger tokenizer and 10× more grounding data.The comparison reports accuracy at an IoU threshold of 0.5.Source: §5.2 · pp. 10–11, Table 2 · p. 10
  • On LVIS-Ground, Groma leads grounded MLLMs especially on AR@0.75, while all MLLMs struggle with small-object AR@s.LVIS-Ground covers 4,299 images, 1,203 object categories, and an average of 3.7 target objects per image.Source: §5.2 · p. 11, Table 3 · p. 11

5.3 Referring Benchmark Results

  • Without task-specific finetuning, Groma matches or improves on GLaMM6 for region captioning on RefCOCOg and Visual Genome.The evaluation reports METEOR and CIDEr scores.Source: §5.3 · p. 11, Table 4 · p. 12

5.4 Conversational VQA Benchmark Results

  • On LLaVA Bench (COCO), Groma surpasses LLaVA and is competitive among grounded MLLMs, especially on detailed image descriptions.The benchmark covers conversation, detailed description, and complex reasoning; the authors speculate DINOv2 features may contribute to weaker conversation and reasoning performance.Source: §5.4 · p. 12, Table 5 · p. 12

5.5 Qualitative Results

  • In a challenging image with multiple occluded instances, Groma shows the highest recall and fewest hallucinations among compared grounded MLLMs.The paper also presents grounded chat and referential-dialogue examples with long-form, logically rich answers, which it attributes mainly to Groma Instruct finetuning.Source: §5.5 · p. 12, Figure 5 · p. 13, Figure 6 · p. 13

5.6 Ablation

  • With a DDETR detection head and the same resolution, DINOv2 outperforms CLIP by 6.5 AP; at 448×448, DINOv2 reaches 43.6 AP.The authors use this comparison to support their choice of DINOv2 as Groma’s image encoder.Source: §5.6 · p. 13, Table 6 · p. 14

Frozen LLM.

  • With the LLM frozen, Groma is on par with GPT4ROI for referring (148.0 vs. 145.2) and comparable to Ferret for grounding (84.02% vs. 83.91%).Referring ability is measured by CIDEr on Visual Genome and grounding by average accuracy on REC benchmarks.Source: Frozen LLM · pp. 13–14, Table 7 · p. 14

Token Merge.

  • Concatenating every four image tokens into one for LLM input causes less than 0.1% average accuracy drop on REC benchmarks.The reported ablation assesses image-token downsampling on REC benchmarks.Source: Token Merge · p. 14, Table 8 · p. 14

6 Limitations and Conclusions

Groma embeds localization into image tokenization through a perception-then-understand design, without external modules, and reports success in referring and visual grounding. The current implementation does not support free-form region inputs or pixel-level grounding.

6 Limitations and Conclusions

  • Groma reports superior performance in localized perception, including referring and visual grounding tasks.The approach embeds localization in image tokenization and uses a perception-then-understand design without external modules.Source: §6 · p. 14
  • Embedding localization into image tokenization is presented as a way to overcome LLM location decoders’ resolution bottleneck and unify referring with visual grounding.The paper describes localization as separate from high-level understanding and reasoning.Source: §6 · p. 14
  • The current implementation does not support free-form region inputs or pixel-level grounding.The paper names a visual-sampler region encoder and a mask region proposer as directions for future work.Source: §6 · p. 14

A Task-Specified Instruction Templates

The appendix provides instruction templates for converting task-specific datasets into instruction-following formats, including converting COCO detection data into multiple-object grounding data.

A Task-Specified Instruction Templates

B LVIS-Ground Benchmark

LVIS-Ground adapts LVIS validation data for grounding evaluation by querying one included object class at a time and proposes AS-MANY-Protocol to evaluate recall across multiple ground-truth boxes.

B LVIS-Ground Benchmark

  • LVIS-Ground queries one included object class at a time and samples at most five validation images per category to save time and maintain class balance.Some categories have fewer than five samples in the original LVIS validation set.Source: B LVIS-Ground Benchmark · p. 20
  • ANY-Protocol can miss whether all instances are found, while MERGED-BOXES-Protocol ignores individual-box atomicity for instance-level evaluation.Source: B LVIS-Ground Benchmark · pp. 20–21
  • AS-MANY-Protocol measures recall over the top-k predicted boxes, where k equals the number of ground-truth boxes, averaging over 10 IoU thresholds from 0.5 to 0.95.Average recall is the primary LVIS-Ground metric.Source: B LVIS-Ground Benchmark · p. 21

C More Implementation Details

The implementation appendix reports stage-wise training times, notes that subsets of some large-scale datasets are sampled, and defines the abbreviations used in Table 10.

C More Implementation Details

Loading 2404.13013v1…