PaperlayerView exact revision on arXiv

Source-linked AI summary

Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models

Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, Yinfei Yang

arXiv:2404.07973v1cs.CV

TL;DR

Grounding-capable MLLMs face limits from fixed, low-resolution visual encoders, constraining detailed visual understanding. Ferret-v2 combines any-resolution processing, multi-granularity encoding, and three-stage training; it reports superiority over existing works across several task families.

  • Problem

    Grounding-capable MLLMs are limited by fixed, low-resolution visual encoders, while global images and local patches provide different levels of visual detail.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Method

    Ferret-v2 combines any-resolution grounding and referring, CLIP and DINOv2 visual encoding at different granularities, and three-stage training with dense alignment before instruction tuning.

    Source: §1 · p. 2

  • Results

    Ferret-v2 demonstrates superiority over existing works across referring and grounding, visual question answering, and modern MLLM benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Ferret-v2 is an upgrade to Ferret for processing and understanding images at higher resolution and finer detail.

    Source: §6 · p. 9

    Like most MLLMs, Ferret-v2 may produce harmful and counterfactual responses.

    Source: §6 · p. 9

Abstract

from arXiv · show

While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks. In this work, we unveil Ferret-v2, a significant upgrade to Ferret, with three key designs. (1) Any resolution grounding and referring: A flexible approach that effortlessly handles higher image resolution, improving the model's ability to process and understand images in greater detail. (2) Multi-granularity visual encoding: By integrating the additional DINOv2 encoder, the model learns better and diverse underlying contexts for global and fine-grained visual information. (3) A three-stage training paradigm: Besides image-caption alignment, an additional stage is proposed for high-resolution dense alignment before the final instruction tuning. Experiments show that Ferret-v2 provides substantial improvements over Ferret and other state-of-the-art methods, thanks to its high-resolution scaling and fine-grained visual processing.

1 Introduction

Ferret-v2 addresses the resolution and fine-grained understanding limits of grounding-capable MLLMs with higher-resolution scaling, multi-granularity visual encoding, and a three-stage training recipe. Experiments report improvements over existing works across referring, grounding, visual question answering, and modern MLLM benchmarks.

1 Introduction

  • Grounding-capable MLLMs still face fine-grained comprehension limits from low-resolution encoders, while task-specific upscaling approaches can be complex and underperform on traditional MLLM benchmarks.The motivating question is how to improve detailed vision tasks without compromising global reasoning.Source: §1 · p. 2
  • Any-resolution scaling outperforms direct upsampling in harnessing image details while retaining pre-training knowledge for efficient scaling.The comparison covers visual-detail analysis, resolution-critical OCR, and reasoning benchmarks.Source: §1 · p. 2
  • Its three-stage training adds high-resolution dense alignment before final visual instruction tuning.The stages proceed from low-resolution image-caption alignment to dense alignment of local objects with detailed semantics, then instruction tuning.Source: §1 · p. 2
  • Ferret-v2 combines low-resolution CLIP encoding with DINOv2 encoding of high-resolution sub-patches to represent global and fine-grained visual contexts.Separate MLP projectors are used for the two vision encoders.Source: §1 · p. 2
  • Ferret-v2 demonstrates superiority over existing works across referring and grounding, visual question answering, and modern MLLM benchmarks.The paper attributes its upgrade to higher-resolution processing and finer visual detail.Source: §1 · p. 3

2 Background

Background work extends language-model capabilities to visual understanding, but coarse-level models often lack region-level outputs. Region-level approaches add grounding and spatial inputs, yet low-resolution encoders remain a limitation.

2 Background

3 Methods

Ferret-v2 extends Ferret’s regional referring and grounding framework to address fixed-resolution limitations through higher-resolution scaling, multi-granularity visual encoding, and three-stage training. The method preserves Ferret’s hybrid region representation while adding local visual detail and dense alignment.

3.1 A Revisit of Ferret

  • Ferret represents regions with discrete coordinates and continuous region features, then generates box coordinates after corresponding regions or nouns to ground them.Its hybrid representation supports points, boxes, and free-form shapes.Source: §3.1 · p. 3

3.2 Analysis of Higher Resolution Scaling

  • “Any resolution” outperformed direct upsampling and vanilla Ferret across the evaluated tasks, while updating the encoder helped this method but not consistently direct upsampling.The authors hypothesize direct upsampling forces the vision transformer to adapt to longer token sequences that differ from its pretraining data; patch-based processing keeps local patches at a similar token length.Source: §3.2 · pp. 4–5

3.3 Model Architecture

  • Ferret-v2 encodes global images with CLIP and local patches with DINOv2, using separate projectors to distinguish global semantics from fine-grained visual information.Local patch features are merged in their original spatial arrangement, then combined with the flattened global-image features as visual tokens for the LLM.Source: §3.3 · p. 5
  • For any-resolution referring, Ferret-v2 fuses upsampled global features with spatially arranged local-patch features before sampling continuous region features; grounding retains Ferret’s coordinate-output design.The fused high-resolution feature map supplies both strong semantics and local awareness to the spatial-aware visual sampler.Source: §3.3 · p. 6

3.4 Training Paradigm

  • Stage I aligns image captions with the LLM; Stage II adds high-resolution dense alignment by matching local objects to detailed semantics through dense referring and detection.LVIS samples average around 10 object locations, compared with mostly one or two in instruction-tuning referring and grounding datasets.Source: §3.4 · pp. 6–7
  • Stage III makes encoders, projectors, samplers, and the LLM trainable for instruction tuning, using GRIT plus VQA and OCR data and strategies for data unification and task generalization.The prompt “Include the coordinates for each mentioned object.” clarifies task requirements for tasks that do not otherwise require referring or grounding.Source: §3.4 · p. 7

4 Experiments

Ferret-v2 performs competitively across referring, grounding, and broader MLLM benchmarks. Its results include gains over prior models on regional tasks and on-par performance with recent models across 10 broader benchmarks.

4.1 Referring and Grounding Tasks

  • Ferret-v2 outperforms previous models on LVIS and SPHINX on the more challenging SA-refer set, which contains 700+ manually verified in-the-wild samples.The authors link the referring results primarily to high-resolution scaling.Source: §4.1 · pp. 7–8
  • Ferret-v2 significantly improves over Ferret on high-resolution grounding benchmarks and outperforms most state-of-the-art models, with its 7B model comparable to CogVLM-Grounding.The evaluation covers REC on RefCOCO, RefCOCO+, and RefCOCOg, plus phrase grounding on Flickr30k Entities; Table 3 reports Acc@0.5.Source: §4.1 · p. 8, Table 3 · p. 8
  • Ferret-v2 surpasses the compared models across all Ferret-Bench task types, demonstrating strong spatial understanding and commonsense reasoning.Ferret-Bench evaluates models’ ability to refer to, describe, and reason about specific image regions.Source: §4.1 · p. 8

4.2 Modern MLLM Benchmarks

  • Ferret-v2 achieves on-par performance with latest state-of-the-art models across 10 benchmarks, excelling particularly on VQAv2, GQA, and POPE.The authors attribute this broader evaluation to adding pseudo-labeled VQA and OCR datasets and a special prompt.Source: §4.2 · p. 8, Table 4 · p. 9

5 Ablation Studies

The ablations evaluate referring, grounding, OCR, and reasoning. They report benefits from any-resolution processing, multi-granularity visual encoding, and intermediate Stage II pre-training.

Ablation Studies

  • The ablations primarily assess performance across referring, grounding, OCR, and reasoning.These are the dimensions used for the studies reported in this section.Source: §5 · p. 9

Any Resolution Grounding and Referring

  • Any-resolution processing markedly improves tasks requiring high-resolution detail and referring precision, while modestly enhancing grounding.The authors report referring gains on both LVIS and SA and describe referring and grounding as mutually beneficial within their framework.Source: §5 · p. 9

Multi-Granularity Visual Encoding and Stage-II Pre-training

  • Multi-granularity visual encoding significantly improves referring and grounding, and adding intermediate Stage II pre-training improves every evaluated metric.The DINOv2 encoder processes high-resolution patches; its projector is initialized from Stage I CLIP weights and fine-tuned in Stage III.Source: §5 · p. 9, Table 6 · p. 9

6 Conclusions

Ferret-v2 upgrades Ferret with any-resolution referring and grounding, multi-granularity visual encoding, and a three-stage training pipeline. The authors report stronger processing of high-resolution images and finer details, while noting that the model may produce harmful and counterfactual responses.

Conclusions

  • Ferret-v2 combines any-resolution referring and grounding, multi-granularity visual encoding, and a three-stage training pipeline.The conclusion presents these as the model’s main upgrades over vanilla Ferret.Source: §6 · p. 9
  • The upgrades support processing and understanding images at higher resolution and finer detail.Source: §6 · p. 9
  • Ferret-v2, like most MLLMs, may produce harmful and counterfactual responses.Source: §6 · p. 9
Loading 2404.07973v1…