Source-linked AI summary
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, Lu Yuan
TL;DR
Vision models have struggled to perform diverse tasks from simple instructions while handling spatial hierarchy and semantic granularity. Florence-2 addresses this with a unified prompt-based seq2seq model trained on FLD-5B, and reports strong zero-shot, fine-tuning, and downstream capabilities.
Problem
Computer vision lacks comprehensive annotations and a unified architecture spanning spatial hierarchy and semantic granularity across diverse tasks.
Method
Florence-2 uses a unified prompt-based sequence-to-sequence architecture trained with multitask learning on FLD-5B's extensive visual annotations.
Results
Florence-2 demonstrates broad zero-shot capabilities, competes with larger specialist models after fine-tuning, and reports 4× greater training efficiency than ImageNet-pretrained models in specified downstream settings.
Takeaways & Limitations
A single set of weights provides a universal representation applicable across captioning, detection, grounding, segmentation, and other visual tasks.
Abstract
from arXiv · showhide
We introduce Florence-2, a novel vision foundation model with a unified, prompt-based representation for a variety of computer vision and vision-language tasks. While existing large vision models excel in transfer learning, they struggle to perform a diversity of tasks with simple instructions, a capability that implies handling the complexity of various spatial hierarchy and semantic granularity. Florence-2 was designed to take text-prompt as task instructions and generate desirable results in text forms, whether it be captioning, object detection, grounding or segmentation. This multi-task learning setup demands large-scale, high-quality annotated data. To this end, we co-developed FLD-5B that consists of 5.4 billion comprehensive visual annotations on 126 million images, using an iterative strategy of automated image annotation and model refinement. We adopted a sequence-to-sequence structure to train Florence-2 to perform versatile and comprehensive vision tasks. Extensive evaluations on numerous tasks demonstrated Florence-2 to be a strong vision foundation model contender with unprecedented zero-shot and fine-tuning capabilities.
1. Introduction
Florence-2 addresses the need for a unified vision representation that handles spatial hierarchy and semantic granularity across diverse tasks. It combines extensive automated annotations with a prompt-based, unified architecture and reports strong zero-shot, fine-tuned, and downstream results.
- Motivation: Computer vision needs representations spanning image-level concepts, fine-grained pixels, high-level captions, and nuanced descriptions.
- Motivation: Comprehensive visual annotations are scarce, while existing datasets and models are commonly tailored to specialized applications or task-specific designs.
- Data and approach: FLD-5B provides 5.4B annotations across 126M images through autonomous specialist annotation, filtering, and iterative refinement.
- Data and approach: Florence-2 uses a seq2seq architecture that standardizes annotations as textual outputs and activates diverse tasks through textual prompts without task-specific architectural modifications.
- Results: Florence-2 achieves new state-of-the-art zero-shot results for COCO captioning, Flick30k visual grounding, and RefCOCO/+/g referring expression comprehension.
- Results: Fine-tuned Florence-2 competes with larger specialist models and establishes new state-of-the-art results on RefCOCO/+/g benchmarks.
- Results: 4× training-efficiency improvement and gains of 6.9, 5.5, and 5.9 points are reported against ImageNet-pretrained models across specified downstream frameworks.
2. Rethinking Vision Model Pre-training
The paper reframes vision pre-training around a universal representation that must capture multiple levels of visual detail and semantic understanding. Florence-2 combines image-level, region/pixel-level, and fine-grained visual-semantic objectives in one multitask framework.
- Pre-training challenge: Existing supervised, self-supervised, and weakly supervised paradigms are limited by single-task learning frameworks.
- Representation requirements: Image understanding requires global semantics, local details, and spatial relationships between objects and entities in context.
- Learning objectives: Image-level tasks capture high-level semantics through classification, captioning, and visual question answering.
- Learning objectives: Region/pixel-level tasks localize objects and entities while capturing relationships with their spatial context.
- Learning objectives: Fine-grained visual-semantic alignment locates image regions corresponding to textual phrases about objects, attributes, or relations.
- Learning objectives: Combining the three objectives teaches the model to handle different spatial details and levels of semantic understanding.
3. Model
Florence-2 formulates diverse vision tasks as prompted sequence-to-sequence translation under a shared architecture and language-modeling objective. Text and quantized location tokens allow the model to generate textual or region-based outputs with one parameter set.
- Architecture: Florence-2 combines a vision encoder with a multimodality encoder-decoder and processes images together with task prompts.
- Task formulation: Each task is formulated as translating an input image and task-specific prompt into a corresponding text or region response.
- Output representation: Plain-text prompts and answers remain in the unified sequence-to-sequence format.
- Output representation: Region-specific tasks use tokenizer location tokens representing quantized coordinates in 1,000 bins.
- Output representation: Boxes, quadrilateral boxes, and polygons encode detection, text recognition, and referring segmentation regions respectively.
- Design choice: Extending the tokenizer with location tokens removes the need for task-specific heads and supports a unified learning format.
- Optimization: The model applies standard language modeling with cross-entropy loss to all tasks using the image-prompt input and target response.
4. Data Engine
FLD-5B is a large multitask dataset built to address the scarcity of comprehensive visual annotations. Its data engine combines specialist-model annotation, filtering, and iterative refinement across text, region-text, and text-phrase-region data.
- Dataset: FLD-5B contains 126M images, 500M text annotations, 1.3B text-region annotations, and 3.6B text-phrase-region annotations.
- Image collection: The image collection combines five datasets sourced from image classification, object detection, and image captioning tasks.
- Data engine: The annotation workflow has three phases: specialist-model annotation, filtering, and iterative data refinement.
- Initial annotation: Specialist models generate synthetic labels, which can be merged with pre-existing annotations to improve coverage and diversity.
- Initial annotation: Tasks lacking sufficient data for robust specialists are omitted initially and generated later during iterative refinement.
- Filtering: Filtering removes noisy text and region annotations using parsing, confidence thresholds, and non-maximum suppression.
- Refinement: Iterative refinement improves training-data quality by updating annotations with multitask-model predictions.
- Annotation types: Text annotations span brief, detailed, and more detailed descriptions, while region-text pairs describe semantic regions at varying granularities.
5. Dataset
FLD-5B is a large, multi-level dataset built for comprehensive visual understanding, combining image-level text, region-text pairs, and text-phrase-region triplets. Its annotations vary in semantic detail and spatial coverage, providing statistics on scale, granularity, and annotation distributions.
- Dataset overview: FLD-5B contains 126M images with more than 500M text annotations, 1.3B region-text annotations, and 3.6B text-phrase-region annotations.Each image is annotated with text, region-text pairs, and text-phrase-region triplets across diverse granularities.
- Dataset overview: The dataset provides more annotations overall and per image than prior foundation-model training datasets, spanning multiple spatial and semantic granularity levels.These properties support more diverse and comprehensive visual understanding tasks.
- Semantic coverage: Detailed and more detailed text contain 4x and 9x as many tokens as brief text, respectively.The longer annotations provide richer information for comprehensive visual understanding.
- Spatial coverage: FLD-5B includes around 1.3B region-text annotations, averaging around 5 regions per image.These annotations capture localized visual concepts through region-text pairs.
- Spatial coverage: The dataset contains more than 3.6B phrase-region pairs, with 4.27 pairs for brief text and more than 10 for detailed and more detailed text.Richer text annotations therefore cover more objects and their corresponding textual phrases.
- Semantic coverage: Average semantic elements increase with annotation detail, while average actions rise 7× for detailed and 15× for more detailed text relative to brief text.The analysis categorizes semantic elements such as objects, attributes, actions, and proper nouns using dependency parsing.
6. Experiments
Florence-2 performs competitively across image-, region-, and pixel-level tasks in zero-shot evaluation and downstream transfer. Ablations show that its multilevel pretraining and trainable vision backbone are especially important for region- and pixel-level performance.
- Zero-shot Evaluation Across Tasks: Florence-2-L achieves strong zero-shot results across captioning, grounding, referring expression comprehension, and referring segmentation.It reaches 135.6 CIDEr on COCO captioning, improves Flickr30k Recall@1 by 5.7 over Kosmos-2, and attains 35.8% mIOU on RefCOCO RES.
- Ablation Studies: The Image-Region-Pixel Model performs competitively across four downstream tasks while outperforming the Image-level Model on object detection and grounding.It achieves 28.3 versus 0.1 on COCO detection and 78.1 versus 62.0 Recall@1 on Flickr30k grounding.
- Data Scaling: Larger pretraining datasets generally improve zero-shot performance across captioning, detection, grounding, and referring segmentation, except for a slight RES reversal.The 1.2M-data setting slightly exceeds the 12M setting on RES.
- Training Configuration: Freezing vision encoders leaves image-level tasks largely unaffected but sharply reduces region- and pixel-level performance.COCO detection AP drops from 19.7 to 6.9 when the vision encoders are frozen.
- Training Configuration: Language pretraining weights help text-oriented tasks slightly but provide little benefit for vision-focused tasks.COCO detection gains only 0.1, while RefCOCO RES drops by 2.91 mIOU.
7. Related Works
Related work develops vision-language pretraining, unified sequence-to-sequence modeling, comprehensive annotation, and scalable datasets. Florence-2 differs by combining a shared encoder-decoder design with large-scale comprehensive annotations.
- Vision-Language Pretraining: Vision-language models align image and text representations through contrastive learning and support zero-shot transfer to alignment and classification tasks.Some models extend this setup to downstream tasks such as object detection.
- Multimodal Decoders: Multimodal decoder models predict text autoregressively while using different mechanisms to fuse vision and language embeddings.Examples include concatenated tokens, attentional poolers, and cross-attention designs.
- Unified Vision Tasks: Unified sequence-to-sequence approaches formulate tasks such as detection and segmentation with shared architectures and specialized tokens.Bounding boxes can be represented using customized tokens beyond pure text.
- Florence-2 Positioning: Florence-2 uses a shared encoder-decoder design with large-scale comprehensive annotation data rather than combining existing sparse annotations.This is the stated distinction from related unified sequence-to-sequence approaches.
- Comprehensive Annotations: Comprehensive datasets combine multiple annotation perspectives to support richer spatial and semantic understanding.MS-COCO and Visual Genome are cited as examples.
- Scalable Annotations: Vision datasets have scaled from thousands to billions of examples, often using web images and noisy metadata-derived annotations.Recent foundation models use massive quantities of such data.
8. Conclusion
Florence-2 is presented as a universal vision backbone trained on comprehensive annotations to support diverse perceptual tasks. Its reported zero-shot capabilities and downstream-transfer results span image, region, and pixel levels.
- Conclusion: FLD-5B contains 126M images paired with 5B comprehensive annotations collected by the Florence data engine.Florence-2 is pretrained on this dataset through comprehensive multitask learning in a unified framework.
- Conclusion: Florence-2 exhibits zero-shot capabilities across captioning, object detection, visual grounding, and referring segmentation.The conclusion also reports contributions to multiple downstream tasks from its universal representation.
A. Supported Tasks and Annotations in Florence-2
The supplied table caption identifies the tasks and annotations supported during Florence-2 pretraining.
- Supported Tasks and Annotations in Florence-2: Table 13 summarizes the tasks and annotations used for Florence-2 pretraining.
B. Supervised Data Collection for Generalist Model Fine-tuning
The section presents supervised-data examples and task-specific prediction outputs spanning detailed captioning, visual grounding, object detection, OCR, and segmentation. It also identifies datasets and model configurations used for downstream evaluation.
- Supervised data collection: The fine-tuning evaluation draws on a collection of datasets, including annotations from FLD-5B merged with original annotations.The supplied table caption states that the collection supports fine-tuning one single generalist model for downstream-task evaluation.
- Model configurations: The section includes model configurations covering different model sizes.
- Detailed image captioning: Detailed captioning examples range from paragraph descriptions to concise descriptions of illustrated images.One example describes a gin-and-tonic illustration with ingredients and background details, while another gives a shorter image description.
- Prediction results: The figure set reports prediction results for detailed captioning, visual grounding, dense region captioning, open-vocabulary detection, OCR with region prediction, and region-to-segmentation.
- Visual grounding: Visual grounding examples locate phrases in captions, including multiple numbered monsters and food items within images.The supplied examples identify five cartoon monsters and fourteen food-related regions using numbered references.
- Detection, OCR, and segmentation: Additional prediction tasks cover open-vocabulary object detection, OCR with region prediction, and region-to-segmentation.The examples include locating Chewbacca and asking for polygon masks of regions.
F. Comparision with LMMs on Detailed Caption Task
This section compares Florence-2 with large multimodal models on the detailed caption task. The supplied passages identify a sequence of comparison figures but do not state their outcomes.
- Detailed caption comparison: Figures 17–20 present comparisons with LMMs on the detailed caption task.
- Detailed caption comparison: The comparison spans one main figure and three continuation figures.
- Detailed caption comparison: The supplied captions identify the comparison task but do not report which model performs better.
G. Comarison with Kosmos-2 on Detailed Caption and Grounding Tasks
This section provides a systematic comparison with Kosmos-2 on detailed captioning and grounding tasks. The models generate both detailed captions and grounding results, with Kosmos-2 results sourced from a Hugging Face space.
- Captioning and grounding comparison: Figures 21–22 systematically compare Florence-2 with Kosmos-2 on detailed caption and grounding tasks.
- Captioning and grounding comparison: Both models generate detailed captions together with grounding results.
- Comparison source: The Kosmos-2 results come from the specified Hugging Face space.