PaperlayerView exact revision on arXiv

Source-linked AI summary

FLAVA: A Foundational Language And Vision Alignment Model

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, Douwe Kiela

arXiv:2112.04482v3cs.CVcs.CL

TL;DR

Vision-and-language models often specialize in contrastive alignment or multimodal fusion and cover only selected modalities or tasks. FLAVA combines alignment and fusion objectives in a shared transformer, pretrained on paired and unpaired data, and performs well across vision, language, and vision-and-language tasks. Across 35 tasks, it achieves better or competitive performance while using a corpus several orders of magnitude smaller than similar recent models.

  • Problem

    Existing models often target limited modalities or tasks and do not jointly provide strong performance across vision, language, and vision-and-language domains.

  • Method

    FLAVA uses image, text, and multimodal transformer encoders with joint pretraining that combines alignment and fusion objectives over paired and unpaired data.

  • Results

    FLAVA performs well across vision, language, and vision-and-language tasks, with better or competitive performance across 35 tasks despite training on a much smaller public-data corpus.

  • Takeaways & Limitations

    FLAVA supports generalized, open models that perform across a wide variety of multimodal tasks using publicly available datasets.

  • Takeaways & Limitations

    The public datasets used to train FLAVA contain biases that may affect the models, and further study is needed to identify and reduce harmful biases.

Abstract

from arXiv · show

State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal (with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising direction would be to use a single holistic universal model, as a "foundation", that targets all modalities at once -- a true vision and language foundation model should be good at vision tasks, language tasks, and cross- and multi-modal vision and language tasks. We introduce FLAVA as such a model and demonstrate impressive performance on a wide range of 35 tasks spanning these target modalities.

1. Introduction

Existing vision-and-language models typically emphasize either contrastive alignment or multimodal fusion, while often covering only selected modalities or tasks. FLAVA addresses this gap with a common transformer model pretrained on paired and unpaired data for vision, language, and multimodal tasks.

  • Motivation: Contrastive models transfer natural-language supervision effectively to visual representations but are difficult to use for tasks requiring simultaneous image and text processing.They also depend on large corpora whose details and accessibility can be problematic.
  • Motivation: Fusion-based vision-and-language transformers support shared cross-modal attention, but their unimodal vision or language performance is often overlooked.
  • Motivation: A vision-and-language foundation model should perform well on vision, language, and vision-and-language problems at the same time.
  • Motivation: Combining modalities in one universal architecture may improve sample efficiency and produce richer representations.
  • FLAVA: FLAVA jointly pretrains on paired and unpaired image and text data using alignment and fusion objectives under a common transformer architecture.It targets visual recognition, language understanding, and multimodal reasoning across a broad task scope.
  • Evaluation: FLAVA is evaluated across multiple modalities using comparisons of recent models spanning cross-modal and multimodal vision-and-language settings.

2. Background

Vision-and-language research has advanced through self-supervised transformer pretraining, but models generally do not perform broadly across vision, language, and vision-and-language domains. FLAVA combines dual-encoder and fusion approaches while using both paired and unpaired data.

  • Research gap: Foundational models that work well across multiple domains and modalities at once remain an open need despite progress from self-supervised transformer pretraining.
  • Research gap: Recent models typically focus on one domain, combine a unimodal domain with vision-and-language, or cover all domains only for selected tasks.
  • Model families: Vision-and-language systems commonly use either dual encoders with shallow interaction or fusion encoders with self-attention spanning modalities.
  • Model families: Dual encoders perform well for unimodal and retrieval tasks, whereas their limited fusion can hurt visual reasoning and question answering.
  • Model families: Fusion encoders differ in whether they use one transformer for early unconstrained fusion or modality-specific layers with selective cross-attention.
  • FLAVA: FLAVA combines dual and fusion encoders in one shared-trunk model pretrained on public paired data and designed to use unpaired unimodal data.This supports unimodal, retrieval, cross-modal, and multimodal tasks.

3. FLAVA: A Foundational Language And Vision Alignment Model

FLAVA is a single transformer-based model designed to support unimodal vision and language understanding alongside multimodal reasoning. It combines separate image and text encoders with a multimodal fusion encoder and trains them using objectives on both paired and unpaired data.

  • 3. FLAVA: A Foundational Language And Vision Alignment Model: The model combines dual-encoder alignment and fusion-encoder objectives within one architecture for vision, language, and vision-and-language tasks.Its multimodal objectives include global contrastive learning, masked multimodal modeling, and image-text matching.
  • 3.1. The model architecture: FLAVA uses image, text, and multimodal transformer encoders to produce unimodal representations and fuse modalities for multimodal reasoning.The image and text encoders process their modalities separately, while the multimodal encoder integrates their projected hidden states through cross-attention.
  • 3.3. Unimodal pretraining objectives: FLAVA pretrains on both paired image-text data and unpaired images or text to incorporate unimodal and multimodal information.Unimodal data uses masked image modeling or masked language modeling, while paired data uses multimodal losses.
  • 3.2. Multimodal pretraining objectives: Global contrastive learning aligns matched image-text representations while separating unmatched pairs in a shared embedding space.The objective projects image and text classification-token representations, normalizes them, and compares them with dot products.
  • 3.2. Multimodal pretraining objectives: Masked multimodal modeling masks image patches and text tokens jointly, extending multimodal masked language modeling to the image side.The paper reports improvements over and beyond contrastive pretraining, especially on multimodal tasks such as VQA.
  • 3.3. Unimodal pretraining objectives: Joint training alternates among unimodal image, unimodal text, and paired image-text datasets, applying the corresponding MIM, MLM, or multimodal losses.The model can be initialized from separately pretrained image and text encoders, while the multimodal encoder is initialized randomly.

4. Experiments

FLAVA is evaluated across vision, language, and multimodal tasks against ablations and prior models. Full pretraining performs best among FLAVA settings, while comparisons show strong public-data performance and trade-offs across modalities.

  • Evaluation setup: 35 tasks span 22 vision tasks, 8 GLUE language tasks, and multimodal VQA, visual entailment, hateful memes, and image-text retrieval.
  • Ablations: Full FLAVA pretraining achieves the highest average performance across vision, language, and multimodal tasks among the evaluated ablations.The comparison includes unimodal MIM and MLM, contrastive-only, multimodal-only, and initialization variants.
  • Ablations: Adding MMM and ITM objectives to contrastive pretraining improves multimodal average score by +2.86%, NLP average score by +9%, and vision average score by +0.3%.
  • Ablations: Joint unimodal and multimodal pretraining raises the NLP average score from 74.22 to 75.55, although the overall macro average decreases slightly in that setting.The authors attribute the NLP improvement to additional CCNews and BookCorpus text used with MLM, while noting harder optimization when tasks are mixed.
  • Comparison to prior models: FLAVA largely outperforms prior multimodal approaches pretrained on public data across language and multimodal tasks and approaches BERT on several GLUE tasks.
  • Comparison to prior models: Compared with CLIP trained on 400M image-text pairs, FLAVA uses ∼6x less data, performs better on language and multimodal tasks, and is slightly worse on some vision-only tasks.

5. Conclusion

FLAVA performs well across vision, language, and vision-and-language tasks, evaluated across 35 tasks. It uses publicly available data and supports the development of generalized, open models, while public-dataset biases remain a limitation.

  • 5. Conclusion: FLAVA performs well across vision, language, and vision-and-language modalities.The conclusion identifies all three target modalities as evaluation domains.
  • 5. Conclusion: 35 tasks were used to analyze FLAVA’s performance across a wide variety of tasks.
  • 5. Conclusion: Publicly available training datasets make FLAVA several orders of magnitude smaller in training data than similar recent models while achieving better or competitive performance.
  • 5. Conclusion: Public datasets improve reproducibility and diversity, but their biases may affect the models and require further study.

A. Hyperparameters and details of FLAVA

This section summarizes FLAVA’s hyperparameters, dataset sampling setup, and training-stability choices. The setup includes multimodal paired data alongside unimodal image and text data.

  • A. Hyperparameters and details of FLAVA: Joint pretraining samples PMD, ImageNet-1k, and CCNews & BookCorpus as multimodal, unimodal image, and unimodal text data, respectively.
  • A. Hyperparameters and details of FLAVA: Table A.1 summarizes the hyperparameters used in the FLAVA model.
  • A. Hyperparameters and details of FLAVA: Large batch size, large weight decay, and long warmup stabilize training and prevent divergence under a large learning rate.
  • A. Hyperparameters and details of FLAVA: Hyperparameters were selected by monitoring the learning curve and zero-shot ImageNet accuracy from the image-text contrastive loss.

B.1. Pretraining details

FLAVA pretrains separate text and image encoders before jointly pretraining on unimodal and multimodal datasets. The joint schedule uses dataset sampling probabilities and ImageNet zero-shot accuracy for checkpoint selection.

  • B.1. Pretraining details: The text encoder is pretrained with masked language modeling on CCNews and BookCorpus for 125K iterations.The setup uses batch size 2048 and learning rate 5e-4, following RoBERTa-base hyperparameters.
  • B.1. Pretraining details: The image encoder is pretrained on ImageNet-1k using either BEiT masked image modeling or DINO training protocols.
  • B.1. Pretraining details: Full FLAVA pretraining jointly samples unimodal and multimodal datasets according to the probabilities listed in Table A.2.
  • B.1. Pretraining details: The PMD dataset is sampled for 150K iterations in the ablations, with checkpoints selected using ImageNet-1k zero-shot accuracy.

B.2. Vision, language and multimodal evaluation

FLAVA is evaluated across vision, language, and multimodal tasks using linear probing, fine-tuning, and zero-shot retrieval. Multimodal evaluations use a classifier over the multimodal encoder, while CLIP uses concatenated unimodal features.

  • B.2. Vision, language and multimodal evaluation: Vision evaluation extracts final image-encoder features and trains a logistic-regression classifier for linear probing.
  • B.2. Vision, language and multimodal evaluation: VQAv2, SNLI-VE, and Hateful Memes are fine-tuned with a 2-layer classifier head over the multimodal encoder representation.
  • B.2. Vision, language and multimodal evaluation: CLIP is evaluated on the same multimodal datasets by concatenating image and text features before applying the classifier head.
  • B.2. Vision, language and multimodal evaluation: GLUE fine-tuning follows FairSeq RoBERTa repository hyperparameters without additional sweeps.
  • B.2. Vision, language and multimodal evaluation: Zero-shot image-text retrieval uses cosine similarity between image and text features as matching scores.

C. Additional ablations and analyses

The analyses examine how pretrained encoders and data affect FLAVA, including weaknesses on image-based text understanding and VQAv2 performance. They also compare full pretraining with additional settings and ablations.

  • Unimodal-pretrained vision encoders: Initializing FLAVA with BEiT or DINO vision encoders significantly improves performance across all tested tasks.The comparison covers FLAVAC, FLAVAMM, and full FLAVA pretraining settings.
  • Observations on SST and VQA: FLAVA performs poorly on SST because its PMD pretraining data contains insufficient scene-text information for learning image-based text reading.The PMD-pretrained CLIP variant shows a similar SST weakness.
  • Additional ablations: Table C.1 compares full FLAVA pretraining with alternative settings and additional ablations, distinguishing overall best results from the best results on public data.Underlining marks the overall best result, while bold marks the best result on PMD and unimodal data.
  • Observations on SST and VQA: 72.49 accuracy on VQAv2 is below state-of-the-art performance but is considered reasonable given FLAVA’s pretraining-data scale.The authors expect more pretraining data to benefit FLAVA.

D. Architectural differences between FLAVA and CLIP encoders

This section compares FLAVA’s image and text encoders with the original CLIP under matched PMD training settings. FLAVA and CLIP use transformer encoders but differ in tokenization, vocabulary, and hidden size.

  • Encoder comparison: FLAVA and CLIP use transformers as image and text encoders in comparable model variants.The comparison includes FLAVAC-local contrastive and CLIP-ViT-B/16 configurations.
  • Text encoding: FLAVA uses BERT’s 30,522-token vocabulary, whereas CLIP uses a 49,152-token vocabulary.The models also use different tokenizers: BERT’s tokenizer for FLAVA and lowercased byte-pair encoding for CLIP.
  • Architectural dimensions: FLAVA uses a hidden size of 768 instead of CLIP’s 512 in the compared encoder designs.The cited architectural comparison describes this difference alongside their vocabulary and tokenizer choices.
Loading 2112.04482v3…