Source-linked AI summary
Florence: A New Foundation Model for Computer Vision
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, Pengchuan Zhang
TL;DR
Computer vision foundation models need broad representations and transferability across diverse real-world tasks, but existing image-text models mainly support shared image-text representations for classification and retrieval. Florence addresses this gap with a Web-scale, unified visual-language model and adapters spanning objects, videos, and multiple modalities, achieving state-of-the-art results across a wide range of benchmarks. Its scope remains bounded by the model’s smaller size and preliminary zero-shot performance that leaves a gap to supervised learning.
Problem
Computer vision foundation models need to generalize across diverse tasks with minimal customization, while existing image-text models mainly support cross-modal representations for classification and retrieval.
Method
Florence learns unified visual-language representations from noisy Web-scale data and adapts them across space, time, modality, and transfer settings.
Results
Florence achieves new state-of-the-art results across a wide range of vision and vision-language benchmarks, including 83.74 top-1 ImageNet-1K zero-shot accuracy and 87.8 Kinetics-600 top-1 accuracy.
Takeaways & Limitations
Florence demonstrates a general-purpose vision system extending transferability across space, time, and modality.
Takeaways & Limitations
Florence is smaller than several existing billion-scale models, and its preliminary zero-shot classification and object detection results leave a performance gap to supervised learning.
Abstract
from arXiv · showhide
Automated visual understanding of our diverse and open world demands computer vision models to generalize well with minimal customization for specific tasks, similar to human vision. Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications. While existing vision foundation models such as CLIP, ALIGN, and Wu Dao 2.0 focus mainly on mapping images and textual representations to a cross-modal shared representation, we introduce a new computer vision foundation model, Florence, to expand the representations from coarse (scene) to fine (object), from static (images) to dynamic (videos), and from RGB to multiple modalities (caption, depth). By incorporating universal visual-language representations from Web-scale image-text data, our Florence model can be easily adapted for various computer vision tasks, such as classification, retrieval, object detection, VQA, image caption, video retrieval and action recognition. Moreover, Florence demonstrates outstanding performance in many types of transfer learning: fully sampled fine-tuning, linear probing, few-shot transfer and zero-shot transfer for novel images and objects. All of these properties are critical for our vision foundation model to serve general purpose vision tasks. Florence achieves new state-of-the-art results in majority of 44 representative benchmarks, e.g., ImageNet-1K zero-shot classification with top-1 accuracy of 83.74 and the top-5 accuracy of 97.18, 62.4 mAP on COCO fine tuning, 80.36 on VQA, and 87.8 on Kinetics-600.
1. Introduction
Florence is proposed as a computer vision foundation model that broadens visual representations across space, time, and modality for diverse downstream tasks. It combines large-scale Web data, unified learning, Transformer architectures, and task adapters, achieving strong transfer and benchmark performance.
- Motivation and scope: Florence frames computer vision foundation models as adaptable systems spanning scene-to-object space, images-to-videos time, and RGB-to-multiple-modality representations.The paper defines transferability through zero-shot, few-shot, and fine-tuning with minimal customization.
- System design: Its ecosystem combines data curation, model pretraining, task adaptations, and scalable training infrastructure.Adapters extend representations across space, time, and modality, while infrastructure techniques reduce memory consumption and improve throughput.
- Data and learning: Florence is trained on 900 million image-text pairs curated from publicly available Web images using unified image-text contrastive learning.The objective addresses noisy, free-form Web text for more effective learning.
- Model pretraining: Florence uses two-tower image and language encoders with hierarchical Vision Transformers for image representation learning.The selected architectures retain Transformer self-attention benefits while modeling image scale variation with linear computational complexity.
- Results: 83.74 top-1 and 97.18 top-5 accuracy were achieved on ImageNet-1K zero-shot classification, alongside wins on 9 of 12 classification tasks.The reported results also include 62.4 mAP on COCO object detection and broad retrieval and vision-language benchmark performance.
- Results: 86.5/87.8 top-1 accuracy was obtained on Kinetics-400/Kinetics-600 video action recognition, extending Florence’s transfer results to dynamic visual understanding.The paper also reports zero-shot text-to-video retrieval at 37.6 R@1 on MSR-VTT.
2. Approach
Florence combines large-scale image-text pretraining with adapters that extend visual representations across object-level space, vision-language tasks, and video. Its approach also includes scalable training techniques and infrastructure for efficient adaptation and memory use.
- Data curation: 900 million image-text pairs are curated from approximately 3 billion Internet images using selection and post-filtering for relevance, quality, and legal and ethical constraints.
- Unified image-text contrastive learning: UniCL treats image-text pairs sharing an identical language description as positives, addressing duplicate captions in Web-scale data.The method maps each pair to an image, description, and language label identifying the description’s unique hash key.
- Pretraining architecture: Florence uses a 893M-parameter two-tower model with a 12-layer language transformer and hierarchical CoSwin-H image encoder.Two linear projection layers align image and language feature dimensions.
- Object-level visual representation learning: Dynamic Head extends CoSwin representations from coarse scenes to fine-grained objects for dense prediction tasks such as object detection.It applies level-wise, spatial-wise, and channel-wise attention to multiscale feature pyramids.
- Fine-grained V+L representation learning: METER adapts Florence for fine-grained vision-language representation learning by fusing CoSwin visual features and RoBERTa language features with co-attention.The adaptation uses image-text matching and masked-language modeling losses before downstream fine-tuning.
- Video adaptation: Video CoSwin adapts CoSwin to video by replacing 2D tokenization, patch merging, and shifted windows with 3D counterparts.Pretrained 2D convolutional weights are duplicated along the temporal dimension and divided by the temporal kernel size.
3. Experiments
Florence is evaluated for zero-shot classification using image-text matching, class-name prompts, and cosine-similarity ranking across ImageNet-1K and downstream datasets.
- Zero-shot learning here predicts classes defined through descriptive text and tests semantic matching between images and text snippets.
- Zero-shot classification selects the most probable class by ranking cosine similarities between image and candidate text embeddings.Florence encodes the image and all dataset class names, then predicts the top-1 or top-5 classes.
- The evaluation covers ImageNet-1K and 11 downstream datasets from a standard classification benchmark suite.Birdsnap is excluded because 20% of its provided image URLs are invalid.
- 83.74% top-1 and 97.18% top-5 accuracy were achieved on ImageNet-1K zero-shot classification.The evaluation follows prompt templates, engineering, and ensembling used by CLIP.
3.2. Linear Probe in Classification
Linear probing evaluates Florence’s representation quality by freezing the image encoder and fine-tuning only appended linear layers on downstream datasets.
- The linear-probe protocol freezes the image encoder and fine-tunes only appended linear layers.The implementation follows the setting used by CLIP for linear evaluation.
3.3. ImageNet-1K Fine-tune Evaluation
Florence supports continual fine-tuning on ImageNet-1K without architectural changes, using task-specific data and the same pre-training loss.
- Florence supports continual fine-tuning on target classification tasks without changing or adding architecture components.Task-specific training continues with the same pre-training loss.
- The ImageNet-1K evaluation uses class names encoded by Florence’s text encoder and the same prompt templates as prior work.
- ImageNet-1K fine-tuning compares Florence with BiT-L-ResNet152x4, ALIGN-Efficient-L2, ViT-G/14, and CoAtNet-7 using model scale, data scale, and Top-1/Top-5 accuracy.
- Florence outperforms BiT and ALIGN in Top-1 and Top-5 accuracy, while remaining slightly below the state-of-the-art result reported in the evaluation.
3.4. Few-shot Cross-domain Classification
Few-shot cross-domain classification measures adaptation to target tasks whose domains differ from typical consumer photographs, using a linear adapter head on Florence’s image encoder.
- The Cross-Domain Few-Shot Learning benchmark tests adaptation across domains with varying dissimilarity from consumer photographs.Its datasets include plant disease, satellite, and dermatology imagery among other domains.
- The adaptation appends a single linear layer to Florence’s CoSwin image encoder.Training uses episodes with data augmentation and dropout between the encoder and classifier head.
- 74.8 average performance was achieved by Florence versus 73.9 for the CD-FSL Challenge 2020 winner.
- Florence achieves higher results than the challenge winner while using a single model and no transductive test-data processing.
3.5. Image-Text Retrieval
Florence supports zero-shot and fine-tuned image-text retrieval on Flickr30K and MSCOCO, matching or exceeding prior results while requiring relatively little fine-tuning.
- Florence matches or outperforms all prior zero-shot retrieval results on Flickr30K and MSCOCO.It ranks image-text candidates using cosine similarity between image and language encoder embeddings.
- Florence’s fine-tuned retrieval results are superior to all previous results on Flickr30K and MSCOCO.
- Fine-tuning retrieval uses roughly 6% and 8% of ALIGN’s fine-tuning epochs on Flickr30K and MSCOCO, respectively.
3.6. Object Detection and Zero-shot Transfer
Florence extends visual representations to object-level detection and transfers to novel detection tasks without task-specific training, including settings with very limited labeled data.
- Object Detection: Florence’s object-level adaptation supports fine-tuned detection across sparse boxes, dense boxes, and object attributes.
- Object Detection: Florence establishes new results on COCO, Object365, and Visual Genome object-detection benchmarks.Evaluation uses mAP for COCO and Object365, and AP50 for Visual Genome.
- Zero-shot Transfer: For zero-shot detection, Florence freezes the CoSwin backbone, pre-trains Dynamic Head for object proposals, and applies zero-shot classification to each proposal.
- Zero-shot Transfer: Florence zero-shot transfers to an open-set benchmark spanning 11 diverse object-detection datasets.The benchmark includes fine-grained fish and chess detection, drone-view detection, and thermal object detection.
- Zero-shot Transfer: Florence outperforms 5-shot fine-tuning on 7/11 tasks and surpasses full-set fine-tuning on the Packages dataset.The Packages training set contains only 26 images.
3.7. V+L Representation Learning
Florence’s vision-language representation is adapted to visual question answering, where it achieves new state-of-the-art performance using less pretraining data than SimVLM.
- Representation Learning: Florence uses vision-language pretraining on 14 million images with 20 million associated captions.The pretraining datasets include MSCOCO, Conceptual Captions, CC12M, SBU, and Visual Genome.
- VQA: On VQA, Florence fine-tunes pooled representations through an MLP to classify answers over 3,129 classes.VQA asks questions based on image context.
- VQA: Florence achieves new state-of-the-art VQA performance compared with existing methods.
- VQA: Florence uses 900M image-text pairs for image-encoder pretraining and 20M examples for VLP, versus SimVLM’s 1.8B image-text pairs, while achieving better results.
3.8. Zero-Shot Text-to-Video Retrieval
Florence adapts its image-text pretrained encoder for zero-shot text-to-video retrieval by expanding image representations to video inputs, achieving a large-margin R@1 advantage over existing methods.
- Video Adaptation: Florence expands 2D patch and positional embeddings to 3D so its encoder can process video inputs.
- Evaluation: Zero-shot text-to-video retrieval is evaluated on the MSR-VTT 1K-A test set using standard recall metrics.
- Results: Florence and CLIP outperform all state-of-the-art methods by a large margin on the R@1 metric.
- Results: The comparison includes methods using video-language pretraining data such as HowTo100M, AudioSet, ImageNet, CC, WebVid-2M, and WIT400M.
3.9. Video Action Recognition
Florence is fine-tuned for video action recognition on Kinetics-400 and Kinetics-600, using a regularized training setup. It outperforms prior state of the art on both datasets.
- Florence is fine-tuned for action recognition on Kinetics-400 and Kinetics-600 at 384 × 384 resolution for 30 epochs.The setup uses label smoothing, RandAugment, a learning rate of 0.0002, and drop path rate 0.5 to reduce over-fitting.
- A publicly available CLIP checkpoint is used for comparison.
- 1.1% and 1.5% higher performance than the state of the art is achieved on Kinetics-400 and Kinetics-600, respectively.
4. Conclusion and Future Work
Florence is presented as a general-purpose vision foundation model that transfers across space, time, and modality while achieving strong benchmark performance. Future work will broaden its task coverage and improve preliminary zero-shot capabilities.
- Conclusion: Florence extends to different tasks across space, time, and modality with strong transferability.
- Conclusion: Florence achieves new state-of-the-art results on a wide range of vision benchmarks.
- Future Work: Future work will add depth and flow estimation, tracking, and additional vision-language tasks.
- Future Work: Preliminary zero-shot classification and object detection results leave a performance gap to supervised learning.