PaperlayerView exact revision on arXiv

Source-linked AI summary

Ferret: Refer and Ground Anything Anywhere at Any Granularity

Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, Yinfei Yang

arXiv:2310.07704v1cs.CVcs.CL

TL;DR

Existing vision-language models largely treat referring and grounding separately, despite their shared spatial-semantic knowledge and the need to handle versatile region shapes. Ferret unifies both capabilities through hybrid region representations, a spatial-aware visual sampler, and the 1.1M-sample GRIT dataset. It achieves superior conventional task performance, outperforms existing MLLMs by 20.4% on average on Ferret-Bench, and alleviates object hallucination.

  • Problem

    Existing works mostly learn referring and grounding individually, while versatile free-form regions are difficult to represent efficiently and clearly.

  • Method

    Ferret combines discrete coordinates and continuous visual features, extracted with a spatial-aware visual sampler, and trains on the 1.1M-sample GRIT instruction-tuning dataset.

  • Results

    20.4% on average: Ferret outperforms the best existing MLLM on Ferret-Bench while achieving superior conventional referring and grounding performance.

  • Takeaways & Limitations

    Ferret supports open-vocabulary spatial understanding and localization across region-based and conversational tasks, with reduced object hallucination.

  • Takeaways & Limitations

    Ferret may produce harmful and counterfactual responses and currently outputs bounding boxes rather than segmentation masks.

Abstract

from arXiv · show

We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions. To unify referring and grounding in the LLM paradigm, Ferret employs a novel and powerful hybrid region representation that integrates discrete coordinates and continuous features jointly to represent a region in the image. To extract the continuous features of versatile regions, we propose a spatial-aware visual sampler, adept at handling varying sparsity across different shapes. Consequently, Ferret can accept diverse region inputs, such as points, bounding boxes, and free-form shapes. To bolster the desired capability of Ferret, we curate GRIT, a comprehensive refer-and-ground instruction tuning dataset including 1.1M samples that contain rich hierarchical spatial knowledge, with 95K hard negative data to promote model robustness. The resulting model not only achieves superior performance in classical referring and grounding tasks, but also greatly outperforms existing MLLMs in region-based and localization-demanded multimodal chatting. Our evaluations also reveal a significantly improved capability of describing image details and a remarkable alleviation in object hallucination. Code and data will be available at https://github.com/apple/ml-ferret

1 INTRODUCTION

Ferret addresses spatial understanding by unifying referring and grounding for diverse image regions within an MLLM. It combines hybrid region representations with the GRIT instruction-tuning dataset and reports strong conventional and conversational performance.

  • Referring interprets specified regions, while grounding localizes regions from semantic descriptions; both require spatial-semantic alignment.
  • Ferret combines discrete region coordinates with continuous visual features from a spatial-aware visual sampler to represent regions of varying shapes.The sampler extracts features from region masks while accounting for differing sparsity across shapes.
  • GRIT provides 1.1M instruction-tuning samples spanning hierarchical spatial knowledge, referring, grounding, mixed location-text inputs and outputs, and spatial-aware negative data.The dataset includes 34K conversations collected with ChatGPT/GPT-4 and additional data converted from existing vision-language tasks.
  • Ferret-Bench covers Referring Description, Referring Reasoning, and Grounding in Conversation, where Ferret outperforms the best existing MLLM by 20.4% on average.The model also demonstrates alleviated object hallucinations.
  • The paper contributes Ferret, GRIT, and Ferret-Bench for fine-grained referring and grounding, instruction tuning, robustness, and joint spatial-semantic evaluation.The stated contributions include reducing object hallucination.

2 RELATED WORK

Related work spans multimodal language modeling, visual instruction tuning, and efforts to integrate fine-grained referring, grounding, and vision-language understanding. Ferret is positioned within these directions while introducing a distinct LLM-based approach and evaluation benchmark.

  • Early multimodal language models primarily emphasized large-scale image-text pre-training.
  • Recent work increasingly uses pre-trained LLMs for visual instruction tuning, while related models also support image retrieval and generation.
  • MLLMs for referring and grounding: Kosmos-2 and Shikra closely resemble Ferret by enabling fine-grained image comprehension and open-world referring and grounding.
  • MLLMs for referring and grounding: Ferret-Bench is introduced to support future research and strengthen evaluation benchmarks for referring and grounding.
  • MLLMs for referring and grounding: Ferret is reported to outperform previous works and significantly mitigate object hallucination, with a side-by-side comparison in Table 1.
  • Unifying grounding and VL understanding: Unlike prior approaches that unify vision-language understanding and grounding through discrete bounding-box tokens, Ferret is built upon LLMs and combines LLM capabilities with grounding.

3 METHOD

Ferret unifies spatial referring and grounding through a hybrid region representation that combines coordinates with continuous visual features. Its spatial-aware sampler processes irregular regions, while the model architecture jointly handles image, text, and region features.

  • 3.1 HYBRID REGION REPRESENTATION: The hybrid representation supports points, bounding boxes, and free-form shapes, including regions whose similar bounding boxes would otherwise cause referring ambiguity.Points use coordinates plus a regional feature, while boxes and free-form shapes use bounding-box coordinates plus a regional feature.
  • 3.1 HYBRID REGION REPRESENTATION: Ferret represents regions with quantized coordinates and continuous visual features extracted from a binary region mask and image feature map.The continuous feature is produced by the spatial-aware visual sampler s(M, Z).
  • 3.2 MODEL ARCHITECTURE: Ferret combines an image encoder, spatial-aware visual sampler, and LLM to jointly model image, text, and region features.The image encoder is the only component whose parameters are not trainable.
  • 3.2 MODEL ARCHITECTURE: The spatial-aware sampler handles irregular regions by sampling positive mask points, gathering nearest-neighbor features, and pooling them into regional representations.Farthest point sampling provides coverage, while local features incorporate relative point positions.
  • 3.2 MODEL ARCHITECTURE: The sampler reduces point count while producing denser features that incorporate local neighbor information and relative positions.The described configuration uses N=512, r=4, k=24, two cascaded blocks, and outputs 32 points.
  • 3.2 MODEL ARCHITECTURE: For grounding, Ferret generates box coordinates immediately after corresponding regions or nouns in its textual response.This format is intended to make the model learn which image entities are groundable and where they are located.

4 GRIT: GROUND-AND-REFER INSTRUCTION-TUNING DATASET

GRIT is a 1.1M-sample instruction-tuning dataset designed to provide hierarchical spatial knowledge for refer-and-ground learning. It combines converted public data, GPT-assisted dialogues, and spatial negative samples.

  • 4 GRIT: GROUND-AND-REFER INSTRUCTION-TUNING DATASET: GRIT contains around 1.1M multimodal dialogues from converted public datasets, GPT-generated instruction data, and spatial negative mining.These are the dataset’s three stated data sources.
  • 4.1 HIERARCHY: The dataset organizes spatial understanding by four granularities: individual objects, object relationships, region descriptions, and region-based complex reasoning.This hierarchy is paired with multiple task formats.
  • 4.1 HIERARCHY: GRIT uses Region-in Text-out, Text-in Region-out, and combined Text-Region formats to cover referring, grounding, and mixed interactions.Combined sequences may place text and regions in the input, output, or both.
  • 4.1 HIERARCHY: Public datasets are converted into instruction-following templates, including object, relationship, region-caption, and visual-grounding data.Visual Genome, Object365, RefCOCOs, and Flickr30k-Entities contribute object-level data; relationship and region-caption data provide 177k examples.
  • 4.1 HIERARCHY: Table 2 provides an in-context example for constructing GPT-assisted refer-and-ground instruction tuning data.The cited material identifies the table as an example rather than specifying its individual rows or fields.
  • 4.2 GPT-ASSISTED VISUAL INSTRUCTION DATA GENERATION: A novel ChatGPT/GPT-4-assisted dataset adds region-based complex reasoning in the combined Text-Region format.The broader dialogue-generation process uses textual scene descriptions and human-annotated dialogues as few-shot demonstrations.
  • 4.2 GPT-ASSISTED VISUAL INSTRUCTION DATA GENERATION: GPT-assisted data emphasizes spatial knowledge by adding physical relationships, region captions, and coordinates to scene descriptions and dialogues.Existing LLaVA instruction data are also pseudo-grounded by localizing nouns with GLIPv2 and appending bounding boxes.
  • 4.3 SPATIAL NEGATIVE MINING: Spatial negative mining targets hallucination in detailed-region queries through image-conditioned and semantics-conditioned category localization.The dataset maintains equilibrium between positive and negative samples, collecting 95k examples.

5 EXPERIMENTS

Ferret is evaluated across conventional referring and grounding benchmarks, grounded captioning, multimodal chatting, ablations, hallucination, and comparison with GPT-4V. Results show strong spatial performance, mutual benefits between referring and grounding, improved detailed description, and reduced object hallucination.

  • Evaluation scope: Ferret’s experiments cover conventional referring and grounding, multimodal chatting, ablations, object hallucination, and GPT-4V comparison.The evaluation includes both standard benchmarks and newer region-based conversational tasks.
  • Referring evaluation: 2667 LVIS objects are used to evaluate referring object classification across point, box, and free-form-shape inputs, with nearby same-image categories as negatives.The setup covers over 1000 object categories and replaces locations with three referring types.
  • Grounded captioning: Ferret achieves state-of-the-art grounded captioning on Flickr30k Entities, evaluated with captioning metrics and grounding scores F1all and F1loc.The task requires generating captions and grounding all generated noun phrases to image regions.
  • Ablation studies: Adding grounding data improves referring performance, and adding referring data improves grounding performance.This mutual benefit is reported in the ablation study.
  • Ablation studies: The spatial-aware visual sampler outperforms the SEEM sampler in all three referring tasks, while larger language-model backbones generally improve performance.The SEEM comparison averages features from sampled points, whereas Ferret uses its spatial-aware sampler.
  • Object hallucination: Ferret’s object-hallucination performance is comparable to Shikra and far surpasses recent popular MLLMs on POPE.The paper attributes this result to fine-grained spatial knowledge and negative mining.
  • GPT-4V comparison: Compared with GPT-4V, Ferret is more precise for small-region referring and bounding-box grounding, while GPT-4V is stronger in general question answering and commonsense.Ferret accurately identifies most traffic lights in cluttered CAPTCHA scenes, whereas GPT-4V can provide richer commonsense explanations.

6 CONCLUSION

Ferret is presented as an MLLM for referring to and grounding image regions in free-form shapes, while supporting groundable text descriptions. The conclusion also identifies harmful or counterfactual responses as a limitation and proposes future segmentation-mask output.

  • Ferret refers to image regions in any free-form shape and grounds text that it deems groundable.
  • GRIT was curated for Ferret’s training, while Ferret-Bench was curated for evaluation.
  • Ferret may produce harmful and counterfactual responses, like most MLLMs.
  • Future work plans to add segmentation-mask output alongside bounding boxes, inspired by LISA.

A.1 TASK TEMPLATES FOR PUBLIC DATASETS

This appendix section documents task templates for converting public datasets into instruction-following format. Table 11 lists three examples for each task for simplicity.

  • Task templates convert public datasets such as Visual Genome into instruction-following format.
  • Table 11 provides three example templates for each task.

A.2 DETAILS ON SPATIAL NEGATIVE MINING

The appendix describes spatial negative-sample mining for image-conditioned and semantics-conditioned category localization. For semantics-conditioned negatives, ChatGPT or GPT-4 generates analogous entities to the original labels.

  • Negative-sample mining targets Image-conditioned Category Localization and Semantics-conditioned Category Localization.
  • For semantics-conditioned negatives, ChatGPT or GPT-4 generates entities analogous to the original class, attribute, or quantity.
  • The same template converts both original data types into object-hallucination tasks.

A.3 EXAMPLES FOR GENERATING REFER-AND-GROUND DATASETS

This appendix provides prompt examples for generating refer-and-ground datasets from visual conversation and reasoning data. The examples are organized across Tables 13–16, while Table 12 presents a prompt for spatial negative sets.

  • Prompts and in-context examples are provided for multiple-round visual conversation data.
  • Prompts and in-context examples are also provided for one-round reasoning data.
  • Table 12 presents a prompt for generating spatial negative sets.

B EXAMPLES AND PROMPTS FOR FERRET-BENCH

Ferret-Bench examples use GPT-4-generated region-based questions to evaluate referring description, referring reasoning, and grounding in conversation. The appendix documents prompts and in-context examples for constructing refer-and-ground instruction-tuning data.

  • GPT-4 generates region-based questions for referring description, referring reasoning, and grounding in conversation.
  • Tables 13 and 15 provide prompts for conversation and reasoning responses in refer-and-ground instruction tuning.
  • Tables 14 and 16 show in-context examples used to construct GPT-assisted refer-and-ground instruction-tuning data.

C MORE VISUALIZATION

The appendix visualizes Ferret across referring, grounding, captioning, hallucination, and conversational tasks. It also presents qualitative examples showing how models respond to region-based descriptions, reasoning, and conversational grounding.

  • More Visualization: Tables 19–21 provide qualitative comparisons for referring description, referring reasoning, and grounding in conversation across multimodal models.
  • More Visualization: Referring object classification on LVIS evaluates regions specified by point, box, or segmentation inputs.
  • More Visualization: A Ferret-Bench example describes a man on a skateboard while relating him to a handrail, bench, tree, and buildings.
  • More Visualization: Phrase grounding and referring expression comprehension evaluate localization of objects described by expressions or given entities.
  • More Visualization: Grounded captioning requires generating an image caption and grounding every generated noun phrase to image regions.
  • More Visualization: POPE evaluates object hallucination by asking whether an object exists and requiring a yes/no response.
Loading 2310.07704v1…