PaperlayerView exact revision on arXiv

Source-linked AI summary

Ferret: Refer and Ground Anything Anywhere at Any Granularity

Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, Yinfei Yang

arXiv:2310.07704v1cs.CVcs.CL

TL;DR

Existing systems often train referring and grounding separately despite their shared spatial-semantic knowledge. Ferret combines hybrid region representations and a spatial-aware sampler with GRIT, and on Ferret-Bench outperforms the best existing MLLMs by 20.4% on average.

  • Problem

    Existing works mostly learn referring and grounding individually, despite both requiring alignment between spatial information and semantics.

    Source: §1 · p. 2

  • Method

    Ferret unifies referring and grounding with hybrid discrete-coordinate and continuous-feature region representations, a spatial-aware sampler, and the 1.1M-sample GRIT dataset.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    20.4% on average: Ferret outperforms the best existing MLLMs on Ferret-Bench and reduces object hallucination.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Ferret-Bench evaluates conversational refer-and-ground tasks that jointly require semantics, knowledge, and reasoning.

    Source: §1 · p. 3

    Ferret refrains from relying on VQA because the authors observed that VQA answers tend to be concise and may restrict LLM conversational capabilities.

    Source: §5.6 · p. 12

Abstract

from arXiv · show

We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions. To unify referring and grounding in the LLM paradigm, Ferret employs a novel and powerful hybrid region representation that integrates discrete coordinates and continuous features jointly to represent a region in the image. To extract the continuous features of versatile regions, we propose a spatial-aware visual sampler, adept at handling varying sparsity across different shapes. Consequently, Ferret can accept diverse region inputs, such as points, bounding boxes, and free-form shapes. To bolster the desired capability of Ferret, we curate GRIT, a comprehensive refer-and-ground instruction tuning dataset including 1.1M samples that contain rich hierarchical spatial knowledge, with 95K hard negative data to promote model robustness. The resulting model not only achieves superior performance in classical referring and grounding tasks, but also greatly outperforms existing MLLMs in region-based and localization-demanded multimodal chatting. Our evaluations also reveal a significantly improved capability of describing image details and a remarkable alleviation in object hallucination. Code and data will be available at https://github.com/apple/ml-ferret

1 INTRODUCTION

Ferret unifies referring and grounding in an MLLM using a flexible hybrid representation of image regions, trained with instruction-tuning data for open-vocabulary, robust interaction. The paper reports strong performance on conventional tasks and a 20.4% average advantage over existing MLLMs on Ferret-Bench.

1 INTRODUCTION

  • Referring interprets a specified region, while grounding localizes a region from a semantic description; both depend on spatial-semantic alignment, though prior work typically trains them separately.Source: §1 · p. 2
  • Ferret combines natural-language discrete coordinates with continuous region features from a spatial-aware sampler, supporting points, boxes, and free-form shapes in inputs and generating grounded coordinates with text.The sampler accounts for varying sparsity across shapes; the hybrid representation combines discrete coordinates and continuous visual features.Source: §1 · p. 2
  • GRIT contains 1.1M instruction-tuning samples spanning hierarchical spatial knowledge and referring/grounding formats, plus 34K GPT-assisted conversations and mined spatial negatives for robustness.Its formats include location-in/text-out, text-in/location-out, and inputs and outputs that mix location with text.Source: §1 · p. 2
  • 20.4% on average, Ferret outperforms the best existing MLLMs on Ferret-Bench’s referring-description, referring-reasoning, and grounding-in-conversation tasks, while alleviating object hallucination.The paper also reports superior performance on conventional referring and grounding tasks.Source: §1 · pp. 2–3

2 RELATED WORK

The related work traces MLLMs from image-text pretraining toward instruction-tuned and task-extended systems, then positions Ferret against referring-and-grounding models and unified vision-language systems.

Multimodal large language models (MLLMs)

MLLMs for referring and grounding

  • Unlike prior MLLMs limited to boxes (and points in Shikra), Ferret supports free-form referring shapes, including sketches, scribbles, and polygons.The work also introduces a refer-and-ground instruction-tuning dataset and Ferret-Bench; it reports superior performance to previous works and reduced object hallucination.Source: §2 · pp. 3–4

Unifying grounding and VL understanding

3 METHOD

Ferret represents regions by combining discrete coordinates with continuous visual features, enabling point, box, and free-form inputs. Its architecture samples region features for joint image-text-region modeling and emits box coordinates to ground regions in responses.

3.1 HYBRID REGION REPRESENTATION

  • Free-form shapes such as scribbles, polygons, and masks are difficult to describe with coordinates alone, while similar bounding boxes can ambiguously refer to distinct objects.Figure 2 illustrates that hybrid region representation can distinguish objects with almost identical bounding boxes.Source: §3.1 · p. 4, Figure 2 · p. 4
  • Ferret combines quantized discrete coordinates with continuous visual features from a binary region mask and image feature map to represent points, boxes, and free-form shapes.A point is represented by its coordinates and a feature for a fixed-radius circle; boxes and free-form shapes use their bounding coordinates and a region feature.Source: §3.1 · p. 4, §3.2 · p. 4

3.2 MODEL ARCHITECTURE

  • Ferret combines a visual encoder, spatial-aware sampler, and LLM; its input places region coordinates and a continuous-feature placeholder alongside region names in text.The image encoder is CLIP-ViT-L/14, and the LLM is Vicuna; sampled region features replace the placeholder after projection to the LLM embedding dimension.Source: §3.2 · pp. 4–6, Figure 3 · p. 5
  • The spatial-aware sampler processes irregular regions by sampling positive mask points, selecting spatially distributed points with farthest point sampling, gathering nearest neighbors, and max-pooling their features.The method cascades two blocks; with N=512, r=4, and k=24, it outputs 32 points with features.Source: §3.2 · p. 5
  • For grounding, Ferret generates box coordinates after the corresponding regions or nouns in its text response.This output format is intended to support learning which image content can be grounded and where it is located.Source: §3.2 · p. 6

4 GRIT: GROUND-AND-REFER INSTRUCTION-TUNING DATASET

GRIT is a 1.1M-dialogue instruction-tuning dataset combining converted public data, GPT-assisted dialogues, and spatial negative samples. It covers multiple levels of spatial granularity and input/output formats.

4 GRIT: GROUND-AND-REFER INSTRUCTION-TUNING DATASET

  • GRIT contains around 1.1M multimodal dialogues from converted public datasets, GPT-assisted instruction data, and spatial negative mining for robustness.These are the dataset’s three main data sources.Source: §4 · p. 6

4.1 HIERARCHY

  • GRIT covers individual objects, object relationships, region descriptions, and region-based complex reasoning through three region/text input-output formats.The formats are Region-in Text-out, Text-in Region-out, and combined Text-Region data.Source: §4.1 · p. 6
  • Object-level data total 678k, combining template-converted public datasets; SAM masks support free-form object regions, alongside text-in region-out grounding data.Sources include Visual Genome (Krishna et al., 2017), Object365, RefCOCOs (Yu et al., 2016; Lin et al., 2014; Nagaraja et al., 2016), and Flickr30k-Entities (Plummer et al., 2015).Source: §4.1 · pp. 6–7
  • Visual Genome relationship and region-caption data contribute 177k Region-in Text-out examples, with SAM masks also extracted for relationship data.The relationship data address object relations, while region captions describe specific regions (Krishna et al., 2017).Source: §4.1 · p. 7

4.2 GPT-ASSISTED VISUAL INSTRUCTION DATA GENERATION

  • GPT-assisted collection produced 34k region-focused dialogues by adding coordinates to scene descriptions and examples, then refining first-round generations with GPT-4.ChatGPT generated the first round; GPT-4 refined dialogues using context averaging 10% of the first-round input length.Source: §4.2 · pp. 7–8
  • GLIPv2 localization of groundable nouns in LLaVA-158k dialogues adds corresponding bounding boxes, creating pseudo-grounded instruction data for Ferret.This extends existing instruction-tuning data with spatial grounding.Source: §4.2 · p. 8

4.3 SPATIAL NEGATIVE MINING

  • Spatial negative mining yields 95k balanced examples using absent object categories or analogous category, attribute, and quantity alternatives to teach localization and object absence.The two methods use Object365 with randomly selected unseen classes and Flickr30k with ChatGPT/GPT-4-selected alternatives.Source: §4.3 · p. 8

5 EXPERIMENTS

The experiments evaluate Ferret on referring, grounding, multimodal chatting, ablations, object hallucination, and a comparison with GPT-4V, reporting strong results across these settings.

5 EXPERIMENTS

  • Ferret uses CLIP-ViT-L/14@336p, Vicuna, and an LLaVA-initialized projection, trains for three epochs on GRIT, and generalizes to relationships and events.Training uses a learning rate of 2e-5 and batch size of 128; the paper attributes its versatility to LLM compositional capabilities.Source: §5 · p. 8

5.1 INPUT REFERRING

  • Ferret significantly outperforms previous models (Peng et al., 2023; Chen et al., 2023b) on referring object classification and handles point, box, and free-form-shape references.The evaluation uses 2,667 LVIS validation objects and creates binary-choice questions with a nearby negative category.Source: §5.1 · p. 9, Table 3 · p. 9

5.2 OUTPUT GROUNDING

  • Ferret performs strongly on visual grounding, is comparable to specialized fine-tuning approaches, and achieves state-of-the-art grounded captioning.Grounded captioning evaluates text with captioning metrics and word-to-region alignment with F1all and F1loc; F1loc scores grounding only for correctly predicted object words.Source: Table 4 · p. 9, Table 5 · p. 10, §5.2 · p. 10

5.3 FERRET-BENCH: MULTIMODAL CHATTING WITH REFERRING AND GROUNDING

  • Ferret outperforms comparison models across LLaVA-Bench and Ferret-Bench, raising detailed-description performance from 68.3 to 80.9 and excelling at the three new referring-and-grounding tasks.Ferret-Bench evaluates referring description, referring reasoning, and grounding in conversation; GPT-4 rates responses against its own answers and ground-truth scene descriptions.Source: §5.3 · p. 11, Table 7 · p. 12

5.4 ABLATION

  • Ablations show that grounding data improves referring and referring data improves grounding, while Ferret’s spatial-aware visual sampler beats the SEEM sampler on all three referring tasks.A larger language-model backbone generally helps referring and grounding performance.Source: §5.4 · p. 12, Table 8 · p. 12, Table 9 · p. 12

5.5 OBJECT HALLUCINATION

5.6 FERRET v.s. GPT-4V(ISION): A QUICK GLANCE AT REFERRING & GROUNDING

  • Compared with Ferret, GPT-4V is less precise on small-region referring and struggles to localize small objects in cluttered scenes, while showing stronger commonsense and general question answering.Ferret accurately identifies most traffic lights in a cluttered scene, and its precise bounding boxes suit applications requiring pinpoint localization of small regions.Source: Figure 6 · p. 13, §5.6 · pp. 13–14

6 CONCLUSION

Ferret is a multimodal large language model for referring to free-form image regions and grounding text the model deems groundable. The conclusion also presents GRIT and Ferret-Bench and identifies harmful or counterfactual responses as a limitation.

6 CONCLUSION

  • Ferret can refer to image regions in any free-form shape and automatically ground text it deems groundable.The paper describes Ferret as a multimodal large language model adept at referring and grounding.Source: §6 · p. 14
  • The authors curated GRIT for training and Ferret-Bench for evaluation.Source: §6 · p. 14
  • Ferret may produce harmful or counterfactual responses; future work, inspired by LISA (Lai et al., 2023), plans to add segmentation-mask output alongside bounding boxes.Source: §6 · p. 14

A DETAILS OF DATASET · A.1 TASK TEMPLATES FOR PUBLIC DATASETS

Ferret converts public datasets such as Visual Genome into instruction-following format using task templates; Table 11 provides three examples per task for simplicity.

A.1 TASK TEMPLATES FOR PUBLIC DATASETS

  • Ferret uses carefully designed task templates to convert public datasets such as Visual Genome into instruction-following format, with Table 11 listing three examples per task for simplicity.Source: §A.1 · p. 19, Table 11 · p. 19

A.2 DETAILS ON SPATIAL NEGATIVE MINING

Spatial negative mining covers image-conditioned and semantics-conditioned category localization, using analogous negative entities generated from caption-derived entities.

A.2 DETAILS ON SPATIAL NEGATIVE MINING

  • The two mined aspects are image-conditioned and semantics-conditioned category localization, and both use the same template for object-hallucination data in Table 11.Source: §A.2 · p. 19
  • For semantics-conditioned negatives, ChatGPT/GPT-4 generates entities analogous to the original class, attribute, or quantity, such as “man” versus “woman.”Source: §A.2 · p. 19
  • The generation prompt includes entities extracted from five captions for one image; its exact template appears in Table 12.Source: §A.2 · p. 19

A.3 EXAMPLES FOR GENERATING REFER-AND-GROUND DATASETS

The section provides example prompts for generating refer-and-ground data with ChatGPT/GPT-4, covering both multiple-round visual conversation and one-round reasoning data.

A.3 EXAMPLES FOR GENERATING REFER-AND-GROUND DATASETS

  • Tables 13 and 14 show the prompt and in-context example for generating multiple-round visual conversation data.Source: §A.3 · p. 19
  • Tables 15 and 16 show the prompt and in-context example for generating one-round reasoning data.Source: §A.3 · p. 19

B EXAMPLES AND PROMPTS FOR FERRET-BENCH

The appendix describes GPT-4-generated region-based questions for evaluating referring and grounding, and provides examples of prompts and responses for refer-and-ground instruction tuning.

B EXAMPLES AND PROMPTS FOR FERRET-BENCH

  • GPT-4 generates region-based evaluation questions for Referring Description, Referring Reasoning, and Grounding in Conversation.The section highlights the prompt for generating a referring-description response and an example GPT-4 answer, in Tables 17 and 18.Source: B EXAMPLES AND PROMPTS FOR FERRET-BENCH · p. 20
  • Conversation-response prompts for refer-and-ground instruction tuning follow the practice of LLaVA (Liu et al., 2023b).Source: Table 13 · p. 21
  • Tables 14 and 16 show in-context examples used to construct GPT-assisted refer-and-ground instruction tuning, with other prompt types available in the codebase.Source: Table 14 · p. 22, Table 16 · p. 24
  • Reasoning-response prompts for refer-and-ground instruction tuning also follow the practice of LLaVA (Liu et al., 2023b).Source: Table 15 · p. 23

C MORE VISUALIZATION

This section points to additional visualizations of Ferret predictions across classification, grounding, captioning, hallucination evaluation, and Ferret-Bench tasks. The supplied captions identify task scopes but do not report result values.

C MORE VISUALIZATION

Related papers

Loading 2310.07704v1…