PaperlayerView exact revision on arXiv

Source-linked AI summary

mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video

Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, Guohai Xu, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou

arXiv:2302.00402v1cs.CVcs.CLcs.MM

TL;DR

Single-network multimodal foundation models struggle to balance modality collaboration with modality entanglement across many modalities and tasks. mPLUG-2 uses shared universal and modality-specific modules, achieving strong results across over 30 tasks, including 48.0 top-1 accuracy on MSRVTT video QA.

  • Problem

    Single-network multimodal foundation models struggle to balance modality collaboration with modality entanglement across many modalities and downstream tasks.

  • Method

    mPLUG-2 combines shared universal modules for modality collaboration with modality-specific modules for disentangling modalities across uni-modal and cross-modal tasks.

  • Results

    mPLUG-2 achieves state-of-the-art or competitive performance across over 30 tasks, including 48.0 top-1 accuracy and 80.3 CIDEr on MSRVTT video QA and captioning.

  • Takeaways & Limitations

    The modular paradigm supports broad text, image, and video task coverage and can be extended to additional tasks by selecting and adding modules.

Abstract

from arXiv · show

Recent years have witnessed a big convergence of language, vision, and multi-modal pretraining. In this work, we present mPLUG-2, a new unified paradigm with modularized design for multi-modal pretraining, which can benefit from modality collaboration while addressing the problem of modality entanglement. In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG-2 introduces a multi-module composition network by sharing common universal modules for modality collaboration and disentangling different modality modules to deal with modality entanglement. It is flexible to select different modules for different understanding and generation tasks across all modalities including text, image, and video. Empirical study shows that mPLUG-2 achieves state-of-the-art or competitive results on a broad range of over 30 downstream tasks, spanning multi-modal tasks of image-text and video-text understanding and generation, and uni-modal tasks of text-only, image-only, and video-only understanding. Notably, mPLUG-2 shows new state-of-the-art results of 48.0 top-1 accuracy and 80.3 CIDEr on the challenging MSRVTT video QA and video caption tasks with a far smaller model size and data scale. It also demonstrates strong zero-shot transferability on vision-language and video-language tasks. Code and models will be released in https://github.com/alibaba/AliceMind.

1. Introduction

mPLUG-2 proposes a modularized multi-modal foundation-model paradigm that balances modality collaboration with modality-specific handling of modality entanglement. It supports flexible task-specific module selection and achieves state-of-the-art or competitive performance across over 30 uni-modal and cross-modal benchmarks.

  • Sharing one network across modalities can cause modality entanglement, allowing different modalities and tasks to interfere across many downstream tasks.
  • mPLUG-2 uses shared functional modules for modality collaboration and modality-specific modules for modality entanglement, jointly training them across modalities.
  • The unified dual-vision encoder shares spatial Transformer modeling between images and videos while adding local temporal modeling for video tasks.
  • 48.0 top-1 accuracy and 80.3 CIDEr are reported as new state-of-the-art results on MSRVTT video QA and video caption tasks.The model is evaluated on over 30 uni-modal and cross-modal understanding and generation benchmarks, achieving state-of-the-art or competitive results.

2. Related Work

Related work spans vision-only and language-only foundation models, as well as unified multimodal pretraining paradigms using contrastive, masked modeling, or generative objectives.

  • Vision-only Foundation Models: Vision-only foundation models evolved from ConvNets toward vision transformers, whose greater capacity supports strong performance across downstream tasks.Scaling approaches include naive ViT architectures trained on large-scale datasets such as JFT-3B and extensions such as SwinV2-G.
  • Language-only Foundation Models: Language-only foundation models build on BERT-style pretraining, while BART uses a denoising encoder-decoder architecture for text generation and comprehension.The passage contrasts BERT-series natural-language understanding methods with BART’s encoder-decoder formulation.
  • Multimodal Task Unification: Unified foundation models combine modalities through contrastive, masked data modeling, or generative pretraining objectives.Florence uses contrastive objectives; BEiT-3 uses masked data modeling; SimVLM, OFA, and CoCa use generative pretraining.

3. Method

mPLUG-2 uses modular encoders, shared universal layers, fusion, and a shared decoder to support modality collaboration while preserving modality-specific representations across understanding and generation tasks. The modules are jointly trained with language, cross-modal, and instruction-based generation objectives.

  • Modular Architecture: mPLUG-2 combines dual-vision, text-encoder, universal-layers, multi-modal-fusion, and shared-decoder modules for uni-modal and cross-modal tasks.Image/video and text are encoded separately before shared and task-specific processing.
  • Universal Layers: Universal layers project visual and linguistic representations into a shared semantic space for alignment while preserving each modality’s original representation.Their outputs support uni-modal discrimination and feed an additional fusion module for cross-modal tasks.
  • Universal Layers: The universal module reduces visual inputs to k tokens, aligns them with text through shared self-attention, and combines common with original representations through cross-attention.This produces text-aware visual and visual-aware text representations across repeated universal layers.
  • Generation and Fusion: The shared Transformer decoder accepts arbitrary uni-modal or multi-modal inputs, enabling generation tasks such as image captioning and visual question answering.The fusion module captures vision-language interaction using Transformer blocks with cross-attention layers.
  • Training Objectives: mPLUG-2 jointly trains its modules with three objectives: masked language modeling, cross-modal matching losses, and instruction-based language modeling.The cross-modal losses comprise vision-language matching and vision-language contrastive learning, while instructions distinguish tasks and modalities.

4. Experiment

mPLUG-2 is evaluated across more than 30 multimodal, language-only, and vision-only benchmarks, achieving competitive or state-of-the-art results across retrieval, question answering, captioning, grounding, classification, and action recognition. Its results also demonstrate data efficiency, including strong performance with substantially less pretraining data than several larger foundation models.

  • Overall Evaluation: mPLUG-2 is evaluated on over 30 vision-language, language-only, and vision-only benchmarks spanning understanding and generation tasks.Evaluations include multimodal retrieval, question answering, captioning, text classification, summarization, image classification, and video action recognition.
  • Video Question Answering: mPLUG-2 outperforms existing foundation models by a large margin on MSRVTT-QA and TGIF-FrameQA, while matching GIT2 on MSVD-QA with significantly less pretraining data.The video question-answering results use open-vocabulary generation.
  • Video Captioning: 4.4% on CIDEr and 3.0% on BLEU@4 are the gains over VideoCoCa and GIT2 on MSRVTT Caption, despite less pretraining data.mPLUG-2 also achieves comparable performance on MSVD video captioning.
  • Image-Text Retrieval: 17M data enables mPLUG-2 to achieve comparable or better image-text retrieval performance than state-of-the-art methods using substantially larger pretraining datasets.Florence uses 0.9B data and BLIP uses 129M data for pretraining.
  • Visual Question Answering: 0.95% on test-dev and 0.77% on test-std are mPLUG-2’s gains over Florence on visual question answering, using 89.11% less pretraining data.These results support efficient multimodal representation learning.

5. Conclusion

mPLUG-2 presents a unified, modularized paradigm for multimodal foundation models that combines modality collaboration with modality-specific disentanglement and performs strongly across more than 30 text, image, and video tasks.

  • mPLUG-2 presents a new unified paradigm with a modularized design for building multimodal foundation models.
  • Its module-based network shares universal modules for modality collaboration while disentangling modality-specific modules to address modality entanglement.
  • Over 30 tasks across text, image, and video modalities show that mPLUG-2 achieves strong performance.

A. More Results · A.1. Detection and Segmentation

mPLUG-2 is evaluated on COCO for object detection and instance segmentation across 80 common categories. It outperforms typical ResNet-based state-of-the-art methods, including DETR and Pix2seq, although a performance gap remains relative to computer-vision-optimized foundation models.

  • A.1. Detection and Segmentation: mPLUG-2’s object detection performance is evaluated on the COCO dataset.COCO is widely used for object-level detection and segmentation.
  • A.1. Detection and Segmentation: mPLUG-2’s instance segmentation performance is also evaluated on COCO.The evaluation covers object-level detection and segmentation.
  • A.1. Detection and Segmentation: COCO evaluation spans 80 common object categories.
  • A.1. Detection and Segmentation: mPLUG-2 outperforms typical state-of-the-art ResNet-based detection methods.The cited examples are DETR and Pix2seq.
  • A.1. Detection and Segmentation: DETR is cited as one ResNet-based detection baseline surpassed by mPLUG-2.DETR is identified as Carion et al. (2020).
  • A.1. Detection and Segmentation: Pix2seq is cited as another ResNet-based detection baseline surpassed by mPLUG-2.Pix2seq is identified as Chen et al. (2021).
  • A.1. Detection and Segmentation: A performance gap remains between mPLUG-2 and foundation models optimized for computer vision.

A.2. Zero-Shot Transferability · A.3. Visual Grounding

mPLUG-2 demonstrates strong zero-shot transferability, achieving leading results in text-to-video retrieval and video question answering. Visualizations show fine-grained vision-language alignment alongside failures caused by ambiguity, blur, and spatial confusion.

  • A.2. Zero-Shot Transferability: 5.1 points of R@1 improvement on MSRVTT illustrates mPLUG-2’s large margin over previous text-to-video retrieval methods.mPLUG-2 obtains state-of-the-art results on MSRVTT, DiDeMo, and LSMDC.
  • A.2. Zero-Shot Transferability: mPLUG-2 achieves state-of-the-art text-to-video retrieval results across the MSRVTT, DiDeMo, and LSMDC datasets.These results support the model’s excellent zero-shot transferability.
  • A.3. Visual Grounding: Figure 5 denotes the ground-truth box in green and the predicted bounding box in red for visual grounding visualization.The figure provides the visual encoding used to compare grounding outputs with ground truth.
  • A.2. Zero-Shot Transferability: mPLUG-2 achieves the best zero-shot performance on both the MSRVTT-QA and MSVD-QA datasets.The result is attributed to the universal module and instructional-based learning.
  • A.2. Zero-Shot Transferability: The video question answering results demonstrate strong zero-shot transferability under the help of the universal module and instructional-based learning.The evaluation results are summarized in Table 20.
  • A.3. Visual Grounding: mPLUG-2 understands various visual concepts and their relationships while making fine-grained alignment between vision and language.These capabilities are illustrated in the first row of the visual grounding cases.
  • A.3. Visual Grounding: Visual grounding includes failure cases involving the ambiguous “trunk,” the blurred “donut,” and failure to realize left and right.The cases are presented in the second row of the visualization.

B. Implementation Details … B.2.2. LANGUAGE ONLY TASKS

mPLUG-2 is implemented with BERT and CLIP-ViT encoders, with specified image/video preprocessing and contrastive-learning settings. Downstream configurations cover vision-only and language-only tasks using task-specific training schedules, optimization parameters, and augmentation or decoding procedures.

  • B. Implementation Details: mPLUG-2 uses PyTorch with a Wikipedia- and BookCorpus-pretrained BERT text encoder and a CLIP-ViT visual encoder pretrained on 400M noisy image-text pairs.For mPLUG-2Base, the vision encoder is ViT-B/16 and the text encoder is BERT-Base.
  • B.1. Pre-training: Pre-training randomly crops images and video frames to 224 × 224, sparsely samples 4 ordered video frames, and uses a 65,536 queue with momentum coefficient 0.995 for vision-text contrastive learning.The queue size is 65,536 and the momentum coefficient is 0.995.
  • B.2.1. VISION ONLY TASKS: Video action recognition trains on Kinetics-710 for 40 epochs, then fine-tunes separately on Kinetics-400, Kinetics-600, and Kinetics-700 for 5 epochs.Kinetics-710 combines Kinetics-400, Kinetics-600, and Kinetics-700 after removing validation- and test-set videos.
  • B.2.1. VISION ONLY TASKS: Video action recognition uses base learning rates of 1e-5 for mPLUG-2Base and 5e-6 for mPLUG-2, with batch sizes of 256 and 128 respectively.The same learning rates and batch sizes are used during individual Kinetics fine-tuning.
  • B.2.1. VISION ONLY TASKS: Image classification fine-tunes for 30 epochs with learning rate 6e-5 and batch size 4096, using RandomCrop, HorizontalFlip, RandAug, and RandErase augmentation.These transformations are applied for data augmentation.
  • B.2.1. VISION ONLY TASKS: Object detection and segmentation follow EVA’s training settings but omit pre-training on Object365 before fine-tuning on MSCOCO.The stated difference from EVA is not pre-training mPLUG-2 on Object365.
  • B.2.2. LANGUAGE ONLY TASKS: Natural language understanding selects the best development-set performance by tuning epochs among 5, 7, 10, learning rates from 3e-5 to 1e-4, and batch sizes among 32, 64, 128.The candidate learning rates are 3e-5, 5e-5, 6e-5, 7e-5, and 1e-4.
  • B.2.2. LANGUAGE ONLY TASKS: Natural language generation fine-tunes for 50,000 steps with learning rate 3e-5 and batch size 256, using beam size 5 and maximum generation length 512.The decoding configuration uses beam size 5 and maximum generation length 512.

B.2.3. VIDEO-TEXT MULTI-MODAL TASKS

Video-text downstream tasks resize frames to 224 × 224 and use task-specific frame sampling, training settings, and augmentation. Video captioning additionally uses a prefix prompt and staged CIDEr optimization.

  • Shared video processing: 224 × 224 frames are used for all video-language downstream tasks, with 12 frames for text-to-video retrieval and 16 for video question answering and captioning during fine-tuning.Inference uses uniform sampling; training augmentation applies RandomCrop with minimum ratio 0.5 and HorizontalFlip with 0.5 probability.
  • Text-to-Video Retrieval: 10 epochs, learning rate 2e-5, and batch size 192 train mPLUG-2Base and mPLUG-2 for MSRVTT, DiDeMo, and LSMDC text-to-video retrieval.Both model variants are trained on each task’s training set.
  • Video Question Answering: 10 epochs, learning rate 2e-5, and batch size 128 train mPLUG-2Base and mPLUG-2 for MSRVTT-QA, MSVD-QA, and TGIF-FrameQA video question answering.Both model variants are trained on the corresponding training sets.
  • Video Captioning: Video captioning uses the prompt “What does the video describe?” and trains with cross-entropy for 10 epochs before 5 extra CIDEr-optimization epochs.Both stages use learning rate 2e-5 then 1e-6, with batch sizes 128 then 16, respectively.

B.2.4. IMAGE-TEXT MULTI-MODAL TASKS

The image-text tasks use task-specific image resolutions and standardized augmentation, training schedules, and inference procedures. These settings cover retrieval, visual question answering, captioning, and visual grounding.

  • Implementation: 336/576/384/336 image-frame resolutions are used for retrieval, VQA, captioning, and grounding, respectively, with ResizedCrop, HorizontalFlip, and center-crop inference.ResizedCrop uses a minimum ratio of 0.5, HorizontalFlip has 0.5 probability, and inference uses center crop.
  • Image-Text Retrieval: 8 epochs, learning rate 1e-5, and batch size 512 are used to train mPLUG-2 on MSCOCO/Flickr30K image-text retrieval.Training uses the MSCOCO/Flickr30K training sets.
  • Visual Question Answering: 8 epochs, learning rate 3e-5, and batch size 512 are used to train mPLUG-2Base on the VQA dataset.
  • Image Captioning: 5 epochs, learning rate 1e-5, and batch size 256 are used for COCO caption fine-tuning with a prefix prompt, followed by beam search with beam size 5.The model is evaluated on the COCO Caption Karpathy validation split and reused directly for Nocaps validation prediction.
  • Visual Grounding: 120 epochs at learning rate 2e-5 followed by 30 epochs at learning rate 2e-6 are used for RefCOCO-series visual grounding fine-tuning.Query length is limited to 20 for RefCOCO and 40 for RefCOCOg.

B.3. Dataset Description

The evaluation spans multimodal video and image tasks, as well as text-only, video-only, and image-only benchmarks. The datasets include retrieval, question answering, captioning, language understanding and generation, action recognition, and image classification settings.

  • Video Question Answering: Video question answering is evaluated on MSRVTT-QA, MSVD-QA, and TGIF-FrameQA, including open-ended and single-frame-answerable questions.MSRVTT-QA contains 243K questions over 10K videos, MSVD-QA contains 47K questions over 2K videos, and TGIF-FrameQA has 35K training and 14K test questions.
  • Video Captioning: Video captioning uses MSRVTT and MSVD, with MSRVTT retaining the retrieval-task split and MSVD divided into 1,200 train, 100 validation, and 670 test videos.MSRVTT contains 10K videos with 20 captions per video, while MSVD contains 1970 YouTube short video clips.
  • Visual and Image-Text Tasks: Image tasks include VQA 2.0, COCO and Flickr30K image-text retrieval, and COCO image captioning, using the widely used Karpathy split for retrieval.VQA 2.0 contains 83k/41k/81k training/validation/test images, while COCO and Flickr30K contain over 330k and 31k images, respectively.
  • Uni-Modal Recognition: Video action recognition uses Kinetics 400/600/700, and image classification uses ImageNet-1K.Kinetics videos are YouTube clips of around 10 seconds labeled with one action class, while ImageNet-1K contains 1.28M training and 50K validation images from 1,000 classes.
Loading 2302.00402v1…