Source-linked AI summary
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, Lu Yuan
TL;DR
Existing vision-language foundation models do not establish a unified approach that benefits both image and video tasks bidirectionally. OmniVL addresses this with a universal architecture, unified contrastive learning over multiple data types, and decoupled spatial-temporal pretraining, achieving state-of-the-art or competitive results across a wide range of downstream tasks.
Problem
Existing foundation-model approaches do not establish one model that jointly supports image-language and video-language tasks while benefiting both image and video tasks.
Method
OmniVL uses a unified transformer-based visual encoder, UniVLC over image-text, video-text, image-label, and video-label data, and decoupled pretraining across spatial and temporal dimensions.
Results
OmniVL achieves state-of-the-art or competitive results across a wide range of downstream tasks; with ViT-Base-scale pretraining, it reports 82.1/64.8 R@1 on COCO image-text retrieval.
Takeaways & Limitations
The model provides one architecture spanning visual-only, cross-modal alignment, and multimodal understanding and generation tasks.
Takeaways & Limitations
OmniVL lacks the commonsense reasoning needed for some visual-language interaction tasks and requires better support for zero-shot visual question answering and few-shot customization.
Abstract
from arXiv · showhide
This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture. It adopts a unified transformer-based visual encoder for both image and video inputs, and thus can perform joint image-language and video-language pretraining. We demonstrate, for the first time, such a paradigm benefits both image and video tasks, as opposed to the conventional one-directional transfer (e.g., use image-language to help video-language). To this end, we propose a decoupled joint pretraining of image-language and video-language to effectively decompose the vision-language modeling into spatial and temporal dimensions and obtain performance boost on both image and video tasks. Moreover, we introduce a novel unified vision-language contrastive (UniVLC) loss to leverage image-text, video-text, image-label (e.g., image classification), video-label (e.g., video action recognition) data together, so that both supervised and noisily supervised pretraining data are utilized as much as possible. Without incurring extra task-specific adaptors, OmniVL can simultaneously support visual only tasks (e.g., image classification, video action recognition), cross-modal alignment tasks (e.g., image/video-text retrieval), and multi-modal understanding and generation tasks (e.g., image/video question answering, captioning). We evaluate OmniVL on a wide range of downstream tasks and achieve state-of-the-art or competitive results with similar model size and data scale.
1 Introduction
OmniVL is a unified foundation model for image-language and video-language pretraining and downstream visual, alignment, and multimodal tasks. It combines a unified visual architecture, broader data and loss design, and decoupled pretraining to support both modalities.
- OmniVL supports visual-only, cross-modal alignment, and multimodal understanding and generation tasks within one foundation model.Examples include image classification, video action recognition, image/video-text retrieval, question answering, and captioning.
- A unified transformer-based visual encoder processes image and video inputs, with video-specific 3D patch tokenization and temporal attention blocks.The architecture also includes a text encoder and two visual-grounded decoders for multiple tasks.
- UniVLC combines image-text, video-text, image-label, and video-label data to use supervised and noisily supervised pretraining sources together.The design seeks both discriminative representations from labeled data and broader concepts from webly crawled vision-language data.
- With a ViT-Base model and approximately 14M image-text plus 2.5M video-text examples, OmniVL reports state-of-the-art results across several image and video tasks.Reported figures include 82.1/64.8 R@1 on COCO image-text retrieval, 39.8 BLEU@4 on COCO captioning, 47.8 R@1 on MSRVTT text-to-video retrieval, and 51.9% accuracy on MSVD video question answering.
2 Related Work
Prior work spans unimodal vision pretraining, image- or video-focused vision-language pretraining, and broader vision foundation models. These approaches target generic representations but differ in supported modalities and task scope.
- Vision-only pretraining uses supervised or self-supervised objectives but does not support cross-modal or multimodal tasks.Classical supervised approaches use ImageNet, while self-supervised methods include contrastive learning and BERT-style pretraining.
- Vision-language pretraining learns generic multimodal representations for tasks such as image captioning, image-text retrieval, and video question answering.Existing approaches are categorized partly by input-data modality and targeted downstream tasks.
- Vision foundation models such as CLIP and ALIGN use web-scale noisy image-text pairs for cross-modal alignment and zero-shot classification, while Florence broadens coverage into space-time-modality.
3 Methodology
OmniVL uses one encoder-decoder architecture to process images, videos, and text across alignment, generation, and visual classification tasks. Its decoupled pretraining combines image/video-label-text data with three learning objectives.
- Overall Framework: OmniVL combines a unified visual encoder, text encoder, and two visual-grounded decoders for alignment and open-ended generation.The visual encoder processes both images and videos, while the decoders support semantic alignment and text generation.
- Unified Visual Encoder: The visual encoder tokenizes images and videos with separate 2D/3D patch tokenizers and adds spatial and temporal positional encodings.Its transformer uses decoupled spatial-temporal attention to model visual inputs.
- Pre-training Objectives: OmniVL jointly optimizes unified contrastive, vision-language matching, and autoregressive language-modeling objectives.The overall loss combines the three objectives with weighting hyper-parameters λ1, λ2, and λ3, set to 1 by default.
- Unified Contrastive Learning: UniVLC places image/video-label data and web-crawled image/video-text data in a shared visual-label-text space for contrastive pretraining.Manually annotated labels are associated with prompt-generated language descriptions, while memory banks provide additional contrastive vectors.
- Pretraining Corpus: The corpus includes 14M image-text examples, 2.5M WebVid videos, and both human-annotated and webly collected data.The supplied corpus description identifies COCO, Visual Genome, CC3M, CC12M, and SBU captions for image-text data.
- Pretraining Paradigm: Decoupled joint pretraining first learns from image-label-text data, then jointly trains on image-label-text and video-label-text data.The design separates spatial and temporal modeling and is intended to reduce video-pretraining cost while enabling bidirectional image-video transfer.
4 Experiments
OmniVL is evaluated across visual-only, cross-modal alignment, and multimodal understanding and generation tasks. Results show strong performance across image and video tasks, while ablations favor decoupled joint pretraining and UniVLC.
- Cross-modal Alignment Tasks: OmniVL surpasses BLIP by 1.8% on COCO and 0.8% on Flickr30K in average recall@1 for image-text retrieval using 14M image-text pairs.The model remains competitive with Florence despite Florence using larger-scale data and model.
- Cross-modal Alignment Tasks: 42.0% and 40.6% recall@1 are obtained on MSRVTT and DiDeMo, respectively, under zero-shot text-to-video retrieval.Fine-tuning further improves performance, indicating discriminative multimodal representations.
- Multimodal Understanding and Generation Tasks: 107.5 CIDEr on NoCaps and 133.9 CIDEr on COCO are reported for image captioning, with comparable or better performance than some methods using more pretraining data.The reported comparisons cover both datasets and include SimVLM as a larger-data reference.
- Multimodal Understanding and Generation Tasks: OmniVL outperforms most existing methods across all reported metrics on YouCook2 video captioning and surpasses QA-specific and pretraining methods on MSRVTT-QA and MSVD-QA.Unlike methods using an extra pretrained S3D backbone for offline video features, OmniVL is fine-tuned end-to-end.
- Ablation Study: Decoupled joint pretraining improves both image and video tasks, whereas joint pretraining from scratch and naive image-to-video combinations show weaker performance on image tasks.The authors hypothesize that image-only initialization first learns spatial representations before joint training incrementally learns temporal dynamics.
- Ablation Study: UniVLC provides larger gains on visual-only tasks than on vision-language tasks, suggesting improved discriminative visual representations for transfer learning.The ablation excludes visual-label data when replacing UniVLC with vanilla contrastive loss.
5 Conclusion and Discussion of Broader Impact
OmniVL unifies image-language and video-language modeling, supporting visual-only, cross-modal, and multimodal tasks while combining multiple data types. The paper also identifies limitations in commonsense reasoning and responsible deployment.
- Conclusion: OmniVL unifies image-language and video-language modeling and supports visual-only, cross-modal alignment, and multimodal understanding and generation tasks.Its unified contrastive loss combines image-text, image-label, video-text, and video-label data, while decoupled pretraining separates spatial and temporal modeling.
- Limitations: The model still lacks the commonsense reasoning capability required by some visual-language interaction tasks, including visual and video question answering.
- Broader Impact: Web-crawled pretraining data may contain toxic language or bias, and model outputs are not easy to control explicitly.The paper therefore calls for careful attention to responsible deployment.
Checklist
The checklist reports that the paper describes its contributions, limitations, and potential negative societal impacts, while noting several reproducibility and asset-documentation omissions.
- Claims and impact: The authors report that the paper accurately reflects its contributions and scope, discusses limitations, and addresses potential negative societal impacts.
- Reproducibility: The paper reports that code, data, and reproduction instructions were not included because the code would be released after acceptance.
- Evaluation reporting: The authors report no error bars from multiple runs, while stating that their method is stable.
- Assets and data: The checklist reports that existing assets were cited, but their licenses were not mentioned and no new assets were included.
A Specification for the Visual-grounded Alignment / Generation Decoder
OmniVL uses visual-grounded alignment and generation decoders to process multimodal information and generate language, with separate arrangements for visual question answering.
- Decoder architecture: The visual-grounded alignment decoder enables deep multimodal interaction through cross-attention blocks.
- Decoder architecture: The visual-grounded generation decoder generates natural language conditioned on visual input.
- Initialization: Both visual-grounded decoders are initialized with BERT-base, which contains 12 transformer layers.
- Question answering: For visual question answering, image or video features are fused with question text before the generation decoder predicts answers.The unified visual encoder processes the visual input, and the alignment decoder combines its output with question features.
C.1 Image-Language Tasks
The image-language downstream-task setup specifies input resizing and augmentation, with dataset-specific end-to-end fine-tuning configurations listed in Table 11.
- Image-language task settings: Images are resized to 384 × 384 for image-text retrieval and image captioning.
- Image-language task settings: Images are resized to 480 × 480 for visual question answering, following prior work.
- Fine-tuning configuration: RandomAugment is used for data augmentation, and Table 11 reports the default fine-tuning settings for each dataset.
C.2 Video-Language Tasks
Video-language tasks use task-specific frame sampling during fine-tuning and uniform sampling during inference, with all frames resized to 384 × 384.
- Video frames are resized to 384 × 384 for all video-language downstream tasks.
- During fine-tuning, OmniVL samples 8, 16, or 24 frames for text-to-video retrieval, video question answering, or video captioning, respectively.
- Inference uses uniform frame sampling, and video-language fine-tuning adopts RandomAugment for data augmentation.
D More Comparison Results on Vision-language Tasks for Different Pretraining Paradigms
The paper compares image-only, video-only, joint-from-scratch, img2vid, and decoupled joint pretraining across vision-language tasks. Decoupled joint pretraining achieves consistently better results on all downstream tasks.
- The comparison covers image-only, video-only, joint-from-scratch, and decoupled pretraining paradigms across various vision-language downstream tasks.
- The img2vid strategy initializes with image-only pretraining and then performs video-only pretraining.
- Decoupled joint pretraining achieves consistently better results on all downstream tasks in the comparison.
- Table 13 reports comparison results for different paradigms, while Table 14 lists the pretraining data used for those paradigms.
E Image/Video Captioning Examples
OmniVL generates image and video captions presented as natural and abundant, adapting image-caption specificity to the richness of visual content. Figures 5 and 6 provide example outputs.
- The paper presents image-captioning and video-captioning examples generated by OmniVL in Figures 5 and 6.
- For relatively simple images, OmniVL generates relatively general captions.The paper identifies lines 2 and 3 as examples.
- For images with richer content, OmniVL generates more fine-grained descriptions.
- The generated image and video captions are characterized as natural and abundant.