PaperlayerView exact revision on arXiv

Source-linked AI summary

Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks

Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, Lu Yuan

arXiv:2311.06242v1cs.CV

TL;DR

Unified vision models face challenges in representing diverse tasks across spatial hierarchy and semantic granularity. Florence-2 combines large-scale annotated data with a prompt-based multitask model, reporting strong zero-shot performance and state-of-the-art results on selected benchmarks.

  • Problem

    A unified representation for diverse vision tasks must address spatial hierarchy and semantic granularity, while comprehensive data and unified architectures remain limited.

    Source: §1 · pp. 1–2

  • Method

    Florence-2 pairs FLD-5B, an iteratively refined dataset of 5.4B annotations on 126M images, with a seq2seq model that uses text prompts for diverse tasks.

    Source: §1 · p. 2

  • Results

    Florence-2 reports state-of-the-art zero-shot results in captioning, visual grounding, and referring expression comprehension, and state-of-the-art fine-tuned results on RefCOCO/+/g.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that Florence-2's pretrained universal representation substantially enhances a range of downstream visual tasks.

    Source: §8 · p. 15

Abstract

from arXiv · show

We introduce Florence-2, a novel vision foundation model with a unified, prompt-based representation for a variety of computer vision and vision-language tasks. While existing large vision models excel in transfer learning, they struggle to perform a diversity of tasks with simple instructions, a capability that implies handling the complexity of various spatial hierarchy and semantic granularity. Florence-2 was designed to take text-prompt as task instructions and generate desirable results in text forms, whether it be captioning, object detection, grounding or segmentation. This multi-task learning setup demands large-scale, high-quality annotated data. To this end, we co-developed FLD-5B that consists of 5.4 billion comprehensive visual annotations on 126 million images, using an iterative strategy of automated image annotation and model refinement. We adopted a sequence-to-sequence structure to train Florence-2 to perform versatile and comprehensive vision tasks. Extensive evaluations on numerous tasks demonstrated Florence-2 to be a strong vision foundation model contender with unprecedented zero-shot and fine-tuning capabilities.

1. Introduction

Florence-2 uses a unified, prompt-based representation and large-scale visual annotations to support diverse vision tasks, addressing challenges in annotation coverage and task-specific model design.

Motivation

  • Vision foundation models must handle spatial detail from image-level concepts to pixels, semantic granularity from captions to nuanced descriptions, comprehensive annotations, and diverse tasks without task-specific architectures.Existing human-labeled datasets target specialized applications, while task-specific models may require substantial fine-tuning even to adapt to new tasks.Source: §1 · pp. 1–2

Related work

  • Florence [95] unified spatial, temporal, and multimodal pretraining, but its first version relied on noisy text-image pairs, task-specific adapters, and large task-specific datasets.Those requirements left gaps in addressing the challenges of comprehensive visual annotations and unified task handling.Source: §1 · p. 2

Data

  • FLD-5B contains 5.4B annotations across 126M images, generated by collaborating specialized models and iteratively refined and filtered with trained foundation models.The collaborating models reach consensus, drawing on the wisdom-of-crowds approach.Source: §1 · p. 2

Model

  • Florence-2 uses a seq2seq architecture [17] [66] [76] that standardizes annotations as text and uses prompts to activate tasks within one model, without task-specific architectural changes.Its image encoder and multimodality encoder-decoder support tasks such as object detection, captioning, and grounding with shared parameters and a consistent loss function.Source: §1 · p. 2

Results

  • After fine-tuning with public human-annotated data, compact Florence-2 competes with larger specialist models and establishes new state-of-the-art results on RefCOCO/+/g benchmarks.The pre-trained Florence-2 backbone enhances downstream object detection, instance segmentation, and semantic segmentation, surpassing supervised and self-supervised models. Compared to ImageNet-pretrained models, it improves training efficiency by 4× and achieves gains of 6.9, 5.5, and 5.9 points using Mask-RCNN, DINO, and UperNet, respectively.Source: §1 · p. 2

2. Rethinking Vision Model Pre-training

Florence-2’s pre-training reframes vision foundation models around overcoming the limitations of single-task paradigms and integrating visual and textual semantics. Its multitask framework combines image-level, region/pixel-level, and fine-grained visual-semantic objectives to learn a representation spanning spatial detail and semantic granularity.

2. Rethinking Vision Model Pre-training

  • Supervised pre-training excels at recognition but lacks adaptability; self-supervision may overemphasize attributes [8], while weak supervision yields only image-level understanding [64].The section motivates pre-training strategies that overcome these single-task limitations and integrate textual and visual semantics.Source: §2 · pp. 2–3
  • Image understanding requires global semantics, local details, and spatial relationships, so the approach uses diverse annotations to capture these nuances and bridge vision and language.Source: §2 · p. 3

2.1. Comprehensive Multitask Learning

  • The multitask objectives cover image-level understanding, region/pixel-level recognition, and fine-grained visual-semantic alignment between text phrases and image regions.Examples include captioning and visual question answering; detection, segmentation, and referring expression comprehension; and locating regions corresponding to objects, attributes, or relations.Source: §2.1 · p. 3
  • Combining the three objectives enables the model to handle varied spatial details and semantic granularity, moving beyond surface recognition toward a universal vision-understanding representation.Source: §2.1 · p. 3

3. Model

Florence-2 uses a prompt-based sequence-to-sequence framework to perform varied vision tasks with a unified architecture, generating text-form responses from images and task instructions. Visual and prompt embeddings are processed together by an encoder-decoder transformer.

Model overview

  • Florence-2 translates an image and task-specific prompt into a response, unifying varied vision tasks under a common language-modeling objective.The model uses a vision encoder and a multimodal encoder-decoder to generate responses in text form.Source: §3 · p. 4

Task formulation

  • Task prompts and responses may be plain text or region-specific, with plain text kept in the sequence-to-sequence format.Source: §3 · p. 4
  • Region tasks use 1,000 quantized-coordinate location tokens, represented as boxes for detection and dense captioning, quad boxes for text, or polygons for referring segmentation.Box coordinates mark top-left and bottom-right corners; quad-box corners proceed clockwise from top-left, as do polygon vertices.Source: §3 · p. 4

Vision encoder

  • DaViT [20] encodes an image into flattened visual token embeddings.The embeddings have shape Nv × Dv, representing the number and dimensionality of vision tokens.Source: §3 · p. 4

Multi-modality encoder decoder

  • A linear projection and LayerNorm align visual tokens with prompt embeddings, which are concatenated as input to a standard encoder-decoder transformer.Prompt embeddings are obtained using the extended language tokenizer and word embedding layer.Source: §3 · p. 4

Optimization objective

  • The model trains all tasks with standard language-modeling cross-entropy loss on the target sequence.Source: §3 · p. 4

4. Data Engine

Florence-2’s data engine builds FLD-5B, a large multitask dataset, through specialist annotation, filtering, and iterative refinement. Its procedures cover image-level text, region-text pairs, and text-phrase-region triplets at varying levels of detail.

4.1. Image Collection

  • The image corpus combines five datasets sourced from image classification, object detection, and image captioning: ImageNet-22k, Object 365, Open Images, Conceptual Captions, and LAION [68].Filtering yields 126 million images in total.Source: §4.1 · pp. 4–5

4. Data Engine

  • 126M images in FLD-5B carry 500M text annotations, 1.3B text-region annotations, and 3.6B text-phrase-region annotations.The dataset contains over 5B annotations in total.Source: §4 · p. 4, Figure 3 · p. 5

4.2. Data Annotation

  • The workflow uses specialist models for initial annotations, filters errors and irrelevant annotations, then iteratively refines the data.The general workflow covers text, region-text pairs, and text-phrase-region triplets; existing labels can be merged with synthetic labels when available.Source: §4.2 · p. 5, Figure 3 · p. 5
  • Text filtering removes captions with excessive objects and retains sufficient action and object complexity; box filtering uses confidence thresholds and non-maximum suppression.A SpaCy-based parser extracts objects, attributes, and actions from text annotations.Source: §4.2 · p. 5
  • Iterative training improves noisy annotations, while tasks omitted initially for sparse data use a fine-tuned pretrained model as a specialist to annotate all 126 million images.The fine-tuned model performed better than one trained from scratch on the same sparse data.Source: §4.2 · pp. 5–6

4.3. Annotation-specific Variations

  • Text annotations range from brief single-sentence captions to detailed and more detailed multi-sentence descriptions; costly LLM- or LMM-generated examples fine-tune a detailed-description specialist.Brief captions are generated by a Florence-2 specialist, with iterative refinement used to reduce noise.Source: §4.3 · p. 6
  • Region-text pairs describe visual objects and text regions with bounding boxes, while triplets link image descriptions and noun phrases to boxes and segmentation masks.OCR labels text regions; DINO [97] detects visual objects, and Grounding DINO [50] plus SAM [32] generate triplet boxes and masks, with filtering for confidence and relevance.Source: §4.3 · p. 6

5. Dataset

FLD-5B combines 126 million images with billions of annotations across text, regions, and phrases. Its analyses show that annotation detail expands semantic content, while its spatial annotations capture concepts at different scales and locations.

5.1. Overview

  • FLD-5B contains 126M images, more than 500M text annotations, 1.3B region-text annotations, and 3.6B text-phrase-region annotations across multiple levels of spatial and semantic granularity.The dataset has more annotations in total and per image than prior datasets commonly used to train vision foundation models.Source: §5.1 · p. 6, Table 1 · p. 7

5.2. Data Analysis

  • Detailed and more detailed text annotations contain 4x and 9x as many tokens as brief text, respectively, which is similar to COCO captions [13].The longer annotations provide richer information for comprehensive visual understanding.Source: §5.2 · pp. 6–7
  • FLD-5B has around 1.3B region-text annotations, over 30x the scale of academic object detection datasets such as OpenImages and Object 365.Each image averages around 5 regions; region briefs average 2.55 tokens versus 7.95 for typical brief text.Source: §5.2 · p. 7
  • FLD-5B has more than 3.6B phrase-region pairs: brief text averages 4.27 pairs, while detailed and more detailed text each have more than 10.The higher pair counts indicate that richer text covers more objects and their corresponding phrases.Source: §5.2 · p. 7
  • Average action counts rise 7× and 15× in detailed and more detailed text versus brief text; object and action complexity also increases, while proper-noun counts rise relatively little.Semantic elements are identified using POS tags and dependency parsing; complexity measures each token’s degree in the dependency tree.Source: §5.2 · p. 8, Table 3 · p. 8
  • Region-text pairs contain more small boxes, while text-phrase-region triplets have a uniform box-size distribution; both span wide aspect ratios, and both show center bias, stronger for triplets.The box-size difference reflects object-detector origins for region-text pairs and grounding-model alignment to phrases for triplets; box centers are more uniformly distributed for region-text pairs.Source: §5.2 · p. 8

6. Experiments

Florence-2 is evaluated as a single generalist vision model through zero-shot tasks, supervised transfer, downstream fine-tuning, and pre-training ablations. The experiments test its performance across task levels and how model, data, and training choices affect that performance.

6.2. Zero-shot Evaluation Across Tasks

  • Zero-shot, Florence-2-L reaches 135.6 CIDEr on COCO captioning and improves grounding and referring-expression results over Kosmos-2, while achieving 35.8% mIOU on RefCOCO segmentation.It uses less than 1% of Flamingo’s parameters for captioning; the cited comparisons are on the stated benchmarks and tasks.Source: §6.2 · p. 9, Table 4 · p. 10

6.3. Generalist Model with Public Supervised Data

  • On RefCOCO, Florence-2-L exceeds PolyFormer by 3.0 Accuracy@0.5 for REC and 3.54 mIOU for RES, and exceeds UNINEXT by 0.8 Accuracy@0.5.The generalist model is fine-tuned with public datasets spanning image-, region-, and pixel-level tasks.Source: §6.3 · p. 9, Table 6 · p. 11
  • On COCO Caption karpathy test, Florence-2-L scores 140.0 CIDEr, exceeding Flamingo’s 138.1 despite Flamingo’s 80B parameters.Florence-2-L achieves this competitive performance without LLMs.Source: §6.3 · p. 9, Table 5 · p. 10

6.4. Downstream Tasks Fine-tuning

  • Compared with supervised ImageNet-1k pre-training, Florence-2 pre-training achieves 4× efficiency and improves COCO detection by 6.9 AP with Mask R-CNN and 5.5 AP with DINO.The cited efficiency comparison covers COCO detection and segmentation and ADE20K semantic segmentation.Source: Figure 6 · p. 11, §6.4 · p. 11, Table 8 · p. 12
  • On ADE20K semantic segmentation, Florence-2 pre-training beats BEiT-pre-trained ViT-B by 1.3 points with single-scale and 1.4 with multi-scale testing.With the same DaViT-B backbone, it improves by 4.9 points and achieves 4× efficiency versus the ImageNet-1k-pre-trained counterpart.Source: §6.4 · pp. 11–12, Figure 6 · p. 11, Table 8 · p. 12
  • For COCO detection, Florence-2 pre-training improves Mask R-CNN AP by 0.7 over the previous best base model and DINO AP by 4.2 over ViT-B.The Mask R-CNN comparison uses a 1× (12-epoch) schedule versus 3× (36 epochs) for ConvNeXt v2-B.Source: §6.4 · p. 11, Table 7 · p. 12

6.5. Ablation Studies

  • Across four transfer tasks, the Image-Region-Pixel model performs strongly: it reaches 133.4 CIDEr on COCO caption, 28.3 on detection, 78.1 recall@1 on Flickr30k grounding, and 31.6 mIoU on RefCOCO segmentation.It matches or exceeds the Image-Region model on these tasks except that its caption result is slightly lower (134.6 CIDEr).Source: §6.5 · p. 12, Figure 7 · p. 13
  • Florence-2-L (771M parameters) clearly outperforms Florence-2-B (232M) on zero-shot downstream tasks; increasing pre-training data generally improves zero-shot performance, except RES is slightly better at 1.2M than 12M.Specialist models generate most annotations, reducing the time and cost associated with manual annotation.Source: §6.5 · p. 13

7. Related Works

The related work places Florence-2 among unified sequence-to-sequence vision-language models, distinguishing it by its large-scale comprehensive annotation data. It contrasts that approach with vision-language models and datasets that provide narrower task coverage or noisy, limited annotation types.

7.1. Vision-Language Foundation Models

  • Prior vision-language models [64] [95] use contrastive alignment [58] for zero-shot transfer and image classification, with task-specific heads extending models such as [95] to object detection.They are trained on large-scale image-text data and have achieved state-of-the-art performance in downstream tasks such as object detection.Source: §7.1 · p. 14
  • Autoregressive decoder approaches [2] [45] fuse modalities through token concatenation in GIT, attentional pooling in CoCa, or resampling and cross-attention in Flamingo [2].Flamingo freezes its pre-trained vision encoder and text decoder; GIT uses a causal attention mask, while CoCa uses learnable queries to select task-specific vision representations.Source: §7.1 · p. 14
  • Unified sequence-to-sequence work extends captioning to detection and segmentation, using special tokens for structures such as bounding boxes and potentially shared weights across tasks.Florence-2 follows this encoder-decoder approach to model dense visual information, using its comprehensive annotations rather than combining sparse annotated data.Source: §7.1 · pp. 14–15

7.2. Vision Datasets

  • Comprehensive datasets such as MS-COCO [13] [48] and Visual Genome [36] combine annotation perspectives but remain limited in size; this work covers text, region-text pairs, and text-phrase-region triplets with reduced human involvement.The field has shifted from single-perspective datasets toward comprehensive annotations that capture spatial and semantic granularity and enable interactions across annotations.Source: §7.2 · p. 15
  • Web-scale datasets often derive noisy, limited-type text labels from metadata, while pseudo-labeling [32] [45] scales annotations; this pipeline adds specialist-model annotations and iteratively refines and completes labels.Web data sources include query-derived categories [75], alt-text [64], and interleaved text [2] [41]; iterative pseudo-labeling can improve quality without significant diversity loss.Source: §7.2 · p. 15

8. Conclusion

Florence-2 is pretrained through unified multitask learning on FLD-5B, a dataset of 126M images and 5B annotations. It demonstrates zero-shot capabilities across visual tasks, while its universal representation contributes to downstream-task improvements.

8. Conclusion

  • FLD-5B contains 126M images paired with 5B comprehensive annotations collected by the Florence data engine.The project constructed the dataset to support Florence-2's development.Source: §8 · p. 15
  • The Florence Project aims to develop a foundational vision model spanning diverse perceptual capabilities, including spatial hierarchy and semantic granularity.Florence-2 is pretrained on FLD-5B through unified comprehensive multitask learning.Source: §8 · p. 15
  • Florence-2 demonstrates zero-shot capabilities across captioning, object detection, visual grounding, referring segmentation, and other visual tasks, while its universal representation enhances downstream tasks.The conclusion describes substantial contributions to the enhancement of a multitude of downstream tasks.Source: §8 · p. 15

A. Supported Tasks and Annotations in Florence-2

Table 13 catalogs the tasks supported by Florence-2 and the annotations used for its pretraining.

A. Supported Tasks and Annotations in Florence-2

  • Table 13 lists Florence-2’s supported tasks and pretraining annotations.Source: Table 13 · p. 20

B. Supervised Data Collection for Generalist Model Fine-tuning … E.6. Region to segmentation

The appendices document the data and model configuration used for generalist fine-tuning, show FLD-5B annotation examples, and illustrate Florence-2 predictions across captioning, grounding, detection, OCR, and segmentation tasks.

B. Supervised Data Collection for Generalist Model Fine-tuning

  • Table 14 covers dataset collection for fine-tuning one generalist model, with an asterisk marking FLD-5B annotations that merge original annotations with the authors’ annotations.Source: Table 14 · p. 20

C. Model Configuration

  • Table 15 presents model configurations for different model sizes.Source: Table 15 · p. 20

D. More Examples of Annotations in FLD-5B

E.1. Detailed Image Caption

  • Detailed image captioning uses the prompt “Describe with a paragraph what is shown in the image,” and prediction results are illustrated.Source: §E.1 · p. 23, Figure 10 · p. 23

E.2. Visual Grounding

E.3. Dense Region Caption

E.4. Open Vocabulary Detection

E.5. OCR; E.6. Region to segmentation

  • The qualitative results illustrate OCR with region prediction and region-to-segmentation outputs prompted by a request for a polygon mask.The polygon-mask prompt appears in both supplied region-to-segmentation passages.Source: Figure 15 · p. 28, Figure 16 · p. 29, §E.6 · p. 29

F. Comparision with LMMs on Detailed Caption Task

Figures 17–20 compare LMMs on the detailed caption task, with Figures 18–20 continuing the comparison.

F. Comparision with LMMs on Detailed Caption Task

G. Comarison with Kosmos-2 on Detailed Caption and Grounding Tasks

Figures 21–22 present a systematic comparison of Florence-2 and Kosmos-2 on detailed caption and grounding tasks, with both models generating caption and grounding results.

G. Comarison with Kosmos-2 on Detailed Caption and Grounding Tasks

  • The comparison covers detailed caption and grounding tasks for Florence-2 and Kosmos-2.The Kosmos-2 results are from the Hugging Face Space identified in the figure captions.Source: Figure 21 · p. 34, Figure 22 · p. 35
  • Both models generate detailed caption and grounding results in the comparison.Figure 22 continues the comparison shown in Figure 21.Source: Figure 21 · p. 34, Figure 22 · p. 35

Related papers

Loading 2311.06242v1…