PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks

Xinsong Zhang, Yan Zeng, Jipeng Zhang, Hang Li

arXiv:2301.05065v2cs.CVcs.AI

TL;DR

Existing foundation models perform best in one task type, leaving open whether one model can excel across language, vision, and vision-language tasks. X-FM combines three encoders with training techniques for text, images, and image-text pairs; the authors report it outperforms other general models and matches or surpasses task-specific models.

  • Problem

    Existing foundation models perform best in one task type, and it remains challenging to build one general model that performs best across language, vision, and vision-language tasks.

    Source: §1 · p. 1

  • Method

    X-FM uses language, vision, and fusion encoders with training techniques that stop vision-language gradients from reaching the language encoder and use vision-language training to guide vision learning.

    Source: §1 · p. 2

  • Results

    Across 23 language, vision, and vision-language tasks, X-FM outperforms other general foundation models and is better than or comparable to task-specific state-of-the-art models.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that X-FM can learn language, vision, and vision-language representations together while achieving better or competitive performance on unimodal and multimodal tasks.

    Source: §1 · p. 3, §6 · p. 9

    The project consumed over 5 A100 GPU years, and the experiments tested only base models.

    Source: §5 · p. 8

Abstract

from arXiv · show

Foundation models or pre-trained models have substantially improved the performance of various language, vision, and vision-language understanding tasks. However, existing foundation models can only perform the best in one type of tasks, namely language, vision, or vision-language. It is still an open question whether it is possible to construct a foundation model performing the best for all the understanding tasks, which we call a general foundation model. In this paper, we propose a new general foundation model, X-FM (the X-Foundation Model). X-FM has one language encoder, one vision encoder, and one fusion encoder, as well as a new training method. The training method includes two new techniques for learning X-FM from text, image, and image-text pair data. One is to stop gradients from the vision-language training when learning the language encoder. The other is to leverage the vision-language training to guide the learning of the vision encoder. Extensive experiments on benchmark datasets show that X-FM can significantly outperform existing general foundation models and perform better than or comparable to existing foundation models specifically for language, vision, or vision-language understanding. Code and pre-trained models are released at https://github.com/zhangxinsong-nlp/XFM.

1 Introduction

Foundation models have advanced language, vision, and vision-language tasks, but existing general models struggle to match modality-specific leaders. X-FM combines three encoders with training techniques that yield strong reported performance across these task types.

1 Introduction

  • Existing general models had not matched the strongest models for individual task types, and joint training is difficult because the modalities differ substantially.The paper frames building one model that performs best across language, vision, and vision-language tasks as an open challenge.Source: §1 · pp. 1, 3
  • Across 23 language, vision, and vision-language tasks, X-FM outperforms other general foundation models and is better than or comparable to task-specific state-of-the-art models.The comparison is reported in Table 1.Source: Table 1 · p. 2, §1 · p. 3
  • X-FM uses modular language, vision, and fusion encoders, trained with text, image, and image-text data for downstream tasks across the three modalities.The language and vision encoders follow BERT and ViT implementations, while the fusion encoder adds cross-attention sub-layers.Source: §1 · p. 2
  • Stopping fusion-to-language gradients leaves the language encoder unaffected by image-text fusion training while focusing fusion training on language-vision alignment.The activation flow from the language encoder to the fusion encoder remains unchanged.Source: §1 · p. 2
  • Vision-language training guides the vision encoder through masked image modeling, using predicted representations and target representations at masked and [CLS] positions.The vision encoder supplies target representations because it is also trained during vision-language learning.Source: §1 · pp. 2–3
  • The authors report that the stop-gradient technique maintains text-understanding capability while enhancing multimodal understanding, and that MIM guidance can enhance vision and multimodal understanding.X-FM is presented as achieving better or competitive performance on both unimodal and multimodal tasks.Source: §1 · p. 3

2 Related Work

Prior work established strong modality-specific foundation models and proposed several approaches to unified language, vision, and vision-language systems. The paper identifies the remaining challenge as achieving the best performance across all task types with one model.

2 Related Work

  • Transformer-based language, vision, and vision-language pretraining had each advanced performance on their respective understanding tasks.The passage lists prior work in all three areas.Source: §2 · p. 3
  • Earlier general models used shared representations, masked unimodal and multimodal objectives, sequence-to-sequence training, or unified perception architectures.Examples include UNIMO, FLAVA, OFA, SimVLM, DaVinci, and Uni-Perceiver.Source: §2 · p. 3
  • Few prior general foundation models had demonstrated best performance across all tasks, motivating comparisons on multiple dimensions.The paper notes that earlier general models had not matched the best individual-task models.Source: §2 · p. 3
  • The paper considers base-scale public-dataset comparisons, while several recent models exceed 1B parameters and use in-house datasets with over 900M image-text pairs.The cited examples include CoCa, BEiT-3, Flamingo, and PaLI.Source: §2 · p. 3
  • X-FM’s pretraining uses text for language MLM, images for vision MIM, and image-text pairs for fusion objectives and additional vision MIM.The figure also shows fusion-to-language gradients stopped for BBP, ITM, and IMLM.Source: Figure 1 · p. 4

3 Method

X-FM trains modular language, vision, and fusion encoders from text, images, and image-text pairs. Its training separates language updates from fusion gradients while using vision-language learning to guide vision-side masked-image modeling.

3.1 Model Architecture and Training Process

  • X-FM assigns language, vision, and fusion encoders to language, vision, and vision-language tasks, training them with text, image, and image-text-pair data.MLM and ITC train the language encoder; MIM and ITC train the vision encoder; ITM, IMLM, and BBP train the fusion encoder.Source: §3.1 · p. 4
  • Stopping fusion-to-language gradients leaves the language encoder trained by MLM and ITC while fusion training focuses on cross-modal alignment.Activations still flow from the language encoder into the fusion encoder.Source: §3.1 · p. 4
  • Vision-language-guided MIM uses semantic targets from the vision encoder; the authors say it avoids external tokenizers and mutually enhances vision and fusion learning.MIM compares predicted and target representations at masked and [CLS] positions using MSE, with no gradients from the targets.Source: §3.1 · pp. 4–5

3.2 Pre-training Objectives

  • MLM recovers masked text with cross-entropy, while ITC learns image-text alignment using in-batch matched and unmatched pairs.The text-only MLM trains the language encoder; ITC uses image-text pairs.Source: §3.2 · p. 5
  • Fusion training uses ITM for pair matching, IMLM to recover masked text from image-text pairs, and BBP to align visual concepts using box losses; the objective sums the losses.BBP minimizes generalized intersection-over-union and L1 losses against ground-truth boxes.Source: §3.2 · pp. 5–6

4 Experiments

Experiments compare X-FM with general and task-specific foundation models across language, vision, and vision-language tasks. Results favor X-FM overall, while ablations report gains from both proposed training designs and their combination.

4.1 Implementation Details

  • Experiments use public in-domain and out-of-domain image-text datasets, object and region annotations, and unimodal text and image data.Base-model pretraining runs for 200K steps; the more-data setting runs for 400K steps.Source: §4.1 · pp. 6–7

4.2 Comparison with Foundation Models

  • X-FMbase outperforms previous general foundation models across almost all tasks by a large margin and is better than or comparable to task-specific foundation models.The comparisons cover language, vision, and vision-language tasks.Source: §4.2 · p. 7
  • X-FMbase wins 11 of 18 image-classification tasks in linear evaluation: 7 for CLIP, 2 for FLAVA, and 2 for DaVinci.Table 4 reports linear-evaluation performance over 18 datasets.Source: §4.2 · p. 7, Table 4 · p. 8

4.3 Comparison with multi-modal Models

  • X-FM demonstrates superiority on five downstream vision-language tasks, and scaled-data X-FMbase is consistently better than previous vision-language models in most cases.On retrieval, X-FMbase outperforms CLIP, ALIGN, and Florence with fewer parameters and much less training data, and is much better than X2-VLM on zero-shot retrieval.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 Ablation Study

  • Both stopping gradients and vision-language-guided MIM bring improvements, and their combination further improves all three downstream task groups.The ablations also report that separated language representations help language and multimodal performance, while vision-language-guided MIM helps vision and vision-language learning.Source: §4.4 · pp. 7–8
  • X-FMbase improves average performance over naive multi-task learning by 0.9% on language, 1.7% on vision, and 1.6% on vision-language tasks.The same-data comparisons also find X-FM better than foundation models designed specifically for language, vision, and vision-language tasks.Source: §4.4 · p. 8

5 Limitations and Potential Risks

The project required substantial computing resources despite evaluating only base models, and the authors leave scalability for future investigation. The image-text training material also has a North American or Western European bias.

5 Limitations and Potential Risks

  • Training the project consumed over 5 A100 GPU-years despite testing only base models.The authors identify sparse attention and the lottery ticket hypothesis as possible efficiency avenues.Source: §5 · p. 8
  • Table 6 marks models over 1B parameters in grey because they are not directly comparable with the other models.Source: Table 6 · p. 9
  • The authors did not test models with at least 1.9B parameters or pre-train large models on large-scale datasets, leaving scalability for future work.Source: §5 · p. 9
  • Training image-text pairs derived mostly from English lexical databases and image queries results in North American or Western European source-material bias.Source: §5 · p. 9

6 Conclusion

The conclusion presents X-FM as an approach to building one foundation model for language, vision, and vision-language understanding. It reports large-margin gains over general foundation models and performance better than or comparable to task-specific models.

6 Conclusion

  • X-FM addresses the challenge of building a general foundation model that performs best across language, vision, and vision-language understanding tasks.Source: §6 · p. 9
  • The proposed training method uses two techniques to learn rich language, vision, and vision-language representations at the same time.Source: §6 · p. 9
  • X-FM outperforms other general foundation models by a large margin and is better than or comparable to task-specific foundation models.Source: §6 · p. 9

A Comparison of Foundation Models

The comparison groups prior foundation models by their task coverage and performance, contrasting them with X-FM's claim to perform best across language, vision, and vision-language understanding.

A Comparison of Foundation Models

B Details of Pre-training Datasets

The pre-training data combine in-domain and out-of-domain image-text datasets with object and region annotations and unimodal corpora. A larger-data setting adds CC-12M and LAION pairs, with filtering applied to the LAION data.

B Details of Pre-training Datasets

C Details of Downstream Tasks

The downstream evaluation spans language, vision, and vision-language benchmarks, using different evaluation settings across tasks. Table 7 labels the task domains as vision, cross-modal retrieval, multimodal fusion, and language.

C Details of Downstream Tasks

  • Evaluation covers eight GLUE language tasks, eight vision tasks, and four multimodal tasks, with fine-tuning used for most tasks.Image-text retrieval is evaluated zero-shot and fine-tuned; ImageNet is evaluated with linear evaluation and fine-tuning; other vision tasks use linear evaluation.Source: C Details of Downstream Tasks · p. 15
  • The language benchmark uses MNLI, CoLA, MRPC, QQP, SST-2, QNLI, RTE, and STS-B, with a linear classifier or regression head on the [CLS] state.Inputs are fed to the language encoder.Source: C Details of Downstream Tasks · p. 15
  • Table 7 distinguishes vision, cross-modal retrieval, multimodal fusion, and language tasks when comparing foundation models.Its caption also marks partially targeted modalities and notes that models above 1B parameters are not directly comparable.Source: Table 7 · p. 16
  • Vision linear evaluation freezes the pretrained backbone and appends an MLP head, using ImageNet, Food101, CIFAR10, CIFAR100, DTD, Pets, and Flowers102.The experiments include both linear evaluation and fine-tuning.Source: C Details of Downstream Tasks · pp. 15–16
  • Image-text retrieval on MSCOCO and Flickr30K encodes images and texts separately, then uses the fusion encoder to rerank top-k candidates.Both datasets use the Karpathy split.Source: C Details of Downstream Tasks · p. 16
  • VQA uses a Transformer decoder over fusion-module outputs, while NLVR2 predicts from concatenated [CLS] features for two image-text pairs and RefCOCO+ predicts bounding boxes.The final X-FMbase VQA result uses 768×768 images; its ablations use 480×480 images.Source: C Details of Downstream Tasks · p. 16

D Details of hyper parameters

X-FMbase pretraining uses three 12-layer encoders initialized from RoBERTa and BEiTv2, with batches combining image-text pairs, images, and text-only sentences. Fine-tuning uses AdamW and task-specific settings, with image position parameters interpolated when resolution changes.

D Details of hyper parameters

  • Pretraining uses 12-layer language, vision, and fusion encoders, initialized with RoBERTa and BEiTv2 for the language and vision encoders, respectively.The hidden size is 768, intermediate size 3072, and maximum input length 128.Source: D Details of hyper parameters · p. 17, Table 10 · p. 17
  • Each pretraining batch contains 3,072 image-text pairs, 3,072 images, and 8,192 text-only sentences, with images center-cropped to 224×224.The learning rate is 1e-4, with a 2,500-step warm-up followed by linear decay; weight decay is 0.01.Source: D Details of hyper parameters · p. 17, Table 10 · p. 17
  • Fine-tuning uses AdamW with learning rates in {1e-5, 2e-5, 5e-5}; image position parameters are linearly interpolated when pretraining and fine-tuning resolutions differ.Downstream image training uses random resized crops and horizontal flips, with settings tabulated by task category.Source: D Details of hyper parameters · p. 17, Table 11 · p. 17, Table 12 · p. 17
Loading 2301.05065v2…