Source-linked AI summary
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, Jing Liu
TL;DR
Existing video-text models have focused primarily on vision and text, while audio and subtitles remain underused for video comprehension. The paper introduces VAST-27M and trains VAST to connect vision, audio, subtitles, and text, with evaluations showing effectiveness across diverse cross-modality tasks and benchmarks.
Problem
Video-text research has advanced mainly around vision and text, despite audio and subtitles providing valuable information for comprehensive video understanding.
Method
The authors construct VAST-27M with automatically generated vision, audio, and integrated omni-modality captions, then train VAST with three modality-focused pretraining objectives.
Results
VAST outperforms existing state-of-the-art methods across diverse vision-text, audio-text, and multi-modal video-text retrieval, captioning, and QA benchmarks.
Takeaways & Limitations
VAST supports unified perception and processing of four modalities across a wide range of downstream tasks, with text-to-modality correlations explored during pretraining.
Takeaways & Limitations
The method requires more diverse and larger-scale omni-modality corpora, and integrating an LLM remains necessary to further enhance generalization capabilities.
Abstract
from arXiv · showhide
Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including Vision, Audio, and Subtitle, and Text by exploring an automatically generated large-scale omni-modality video caption dataset called VAST-27M. Specifically, we first collect 27 million open-domain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off-the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new state-of-the-art results on various cross-modality benchmarks. Code, model and dataset will be released at https://github.com/TXH-mercury/VAST.
1 Introduction
Video-text learning supports captioning, retrieval, and question answering, but existing models underuse audio and subtitles. VAST addresses the missing omni-modality corpus and trains a foundation model over vision, audio, subtitles, and text.
- Video-text cross-modality learning supports video captioning, text-to-video retrieval, and video question answering.
- Audio and subtitles provide information about speech, conversations, news, and instructional procedures that complements visual content.
- Existing corpora lack captions jointly corresponding to vision, audio, and subtitles, while manual annotation is prohibitively expensive.
- VAST-27M is generated through separate vision and audio captioners followed by Vicuna-13b integration with subtitles and prompts.
- VAST processes vision, audio, subtitles, and text and supports retrieval, captioning, and question answering across multiple modality combinations.
- VAST is reported to outperform state-of-the-art methods across diverse cross-modality benchmarks.
2 Related Work
Prior pretraining corpora rely mainly on subtitles or visual captions, while audio-text resources remain small and multimodal methods often learn weak or indirect modality-text relations. VAST-27M is introduced as a large-scale automatically generated omni-modality corpus.
- Cross-Modality Pretraining Corpus: HowTo100M, YT-Temporal-180M, and HD-VILA_100M use subtitles, whereas WebVid datasets use alt-texts as video captions.
- Audio-Text Pretraining Corpus: Human-labeled audio-caption datasets contain fewer than 50,000 clips, limiting their suitability for large-scale audio-text pretraining.
- Cross-Modality Pretraining Corpus: VAST-27M is presented as the first large-scale omni-modality video-caption corpus with automatically generated subtitle, vision, audio, audiovisual, and omni-modality captions.
- Multi-Modality Learning: Earlier multimodal models incorporate subtitles or audio, but their modality-text correlations are often implicit, weak, or focused on audio-subtitle relations rather than audio-text.
3 Dataset
VAST-27M is built from selected open-domain clips using trained vision and audio captioners, then an LLM integrates their outputs with subtitles into omni-modality captions. The dataset spans broad categories and provides captions at multiple modalities and lengths.
- Vision Captioner Training: The vision captioner is trained on large-scale image-text corpora and fine-tuned with manually labeled image and video caption datasets.
- Audio Captioner Training: The audio captioner is trained on VALOR-1M and WavCaps without second-stage fine-tuning to avoid overfitting to narrow audio concepts.
- Clip Selection: 27M clips are selected by restricting duration to 5–30 seconds, requiring vision, audio, and subtitles, and sampling evenly across source videos.
- Omni-Modality Caption Generation: The pipeline combines multiple vision captions, audio captions, and a speech subtitle into one integrated sentence rather than simply concatenating them.
- Omni-Modality Caption Generation: Vicuna-13b integrates generated vision and audio captions with raw subtitles and instructional prompts to produce omni-modality captions.
- Dataset Statistics: VAST-27M contains 27M clips across 15+ categories, with average vision, audio, and omni-modality caption lengths of 12.5, 7.2, and 32.4.
4 Approach
VAST uses separate vision, audio, and text encoders with cross-attention to process multimodal inputs, and trains them with contrastive, matching, and generation objectives. It also models modality groupings to accommodate missing modalities in downstream tasks.
- Architecture: VAST combines ViT, BEATs, and BERT encoders for images, video, audio, subtitles, and captions.The text encoder performs multimodal encoding and decoding through cross-attention layers.
- Pretraining Objectives: OM-VCC aligns global omni-modality video and caption representations in a shared semantic space using contrastive learning.Representations are concatenated, projected through linear layers, normalized, and regularized by their feature distance.
- Pretraining Objectives: OM-VCM trains the model to determine whether an omni-modality video-caption pair is matched.Caption tokens attend through cross-attention to concatenated unpooled vision, audio, and subtitle features.
- Pretraining Objectives: OM-VCG uses conditional causal masked language modeling to generate omni-modality captions from video features.Sixty percent of caption tokens are masked, and causal attention prevents information leakage during reconstruction.
- Loss and Modality Grouping: VAST assigns equal weights to its three omni-modality losses and models V-T, A-T, VA-T, VS-T, and VAS-T relations.The grouping strategy addresses missing modalities between pretraining and downstream adaptation.
- Evaluation: Table 3 evaluates VAST across vision-text, audio-text, and multimodal video-text benchmarks using Recall@1, CIDEr, and Acc.The table reports 22 new state-of-the-art results and distinguishes vision-only from multimodal-track comparisons.
5 Experiments
VAST is evaluated across vision-text, audio-text, and multi-modal video-text tasks, where it generally surpasses prior methods and benefits from omni-modality pretraining and modality grouping.
- Comparison to State-of-the-Art Models: VAST establishes new state-of-the-art results on TGIF-QA and MSVD-QA and achieves five new state-of-the-art results on audio-text benchmarks.
- Comparison to State-of-the-Art Models: VAST surpasses GIT2 by 2.1, 67.6, 5.0, and 8.0 CIDEr points on MSRVTT, YouCook2, VATEX, and TVC while using 22.5% of its parameters and 3.4% of its training data.
- Corpus Quality: Models trained with VAST-27M vision captions achieve the best results on all six vision-text benchmarks, while multi-modal captions introduce noise when only vision is used.
- Corpus Quality: Omni-modality pretraining with VAST-27M improves performance across all seven evaluated benchmarks and exceeds VALOR-1M on five benchmarks.
- Ablation Study: Modality grouping improves generalization across task types by exploring text correlations with every modality group during pretraining.
6 Conclusion, Broader Impact and Limitation
The paper introduces VAST-27M and VAST for omni-modality video understanding, reporting broad cross-modality benchmark performance while acknowledging data and generalization limitations.
- VAST-27M provides automatically generated vision, audio, and integrated omni-modality captions, while VAST connects video modalities with text for retrieval, captioning, and question answering.
- VAST surpasses existing state-of-the-art methods across public vision-text, audio-text, and multi-modal video-text benchmarks.
- The authors identify a need for more diverse and larger-scale omni-modality corpora, further LLM integration for generalization, and attention to inherited dataset and model biases.
A.1 Pretraining Settings
The appendix specifies VAST’s pretraining mixture, caption replacement strategy, objectives, and configuration terminology.
- VAST pretraining configurations include corpus-specific training steps, dataset mix ratios, and objectives such as retrieval and captioning.
A.2 Downstream Datasets Descriptions
The appendix describes the downstream benchmarks used to evaluate VAST across video, audio, audiovisual, image-text, retrieval, captioning, and question-answering settings.
- The evaluation covers MSRVTT, VATEX, YouCook2, VALOR-32K, MSVD, LSMDC, DiDeMo, ActivityNet Caption, TGIF, MUSIC-AVQA, TVC, Clotho, AudioCaps, MSCOCO, Flickr30K, and VQAv2.
- MSRVTT supports text-to-video retrieval, video captioning, and video QA, using the 1K-A split for retrieval and the standard split for captioning and QA.
- VATEX supports text-to-video retrieval and video captioning on 41,250 video clips with 825,000 sentence-level descriptions.
- YouCook2 contains 14K clips from 2K instructional cooking videos, with retrieval and captioning evaluated using official splits.
- VALOR-32K is an audiovisual benchmark of 32K audible ten-second clips, each annotated with a caption describing visual and audio content.
- ActivityNet Caption contains 20K long-form videos and 100K captions for text-to-video retrieval and video QA, while TGIF-Frame provides open-ended video QA.
- MUSIC-AVQA contains more than 45K audiovisual question-answer pairs across 33 question templates spanning modalities and question types.
- TVC contains 108K video moments and 262K paired captions, with subtitles available as additional input for video captioning.
A.4 Detailed Comparisons to State-of-the-Art Methods
VAST achieves strong results across video retrieval, QA, captioning, audio-text, and image-text benchmarks. Its reported gains extend beyond vision-oriented tasks to audio- and subtitle-oriented settings.
- Text-to-Video Retrieval: 5.1, 1.6, 3.7, and 6.1 points improve previous SOTA on MSRVTT, DiDeMo, ActivityNet, and VATEX text-to-video retrieval.
- Text-to-Video Retrieval: 6.8 and 16.7 points surpass previous methods on VALOR-32K and YouCook2, covering audio-oriented and subtitle-oriented retrieval.
- Video QA: VAST achieves new SOTA performance on all five evaluated video QA benchmarks and surpasses VALOR by 1.8 points on MUSIC-AVQA.
- Video Captioning: VAST achieves new state-of-the-art CIDEr scores on all five video captioning benchmarks and processes subtitle-oriented benchmarks such as YouCook2 and TVC.
- Text-to-Audio Retrieval and Audio Captioning: 7.6, 5.4, and 9.8 R@1 points improve previous SOTA on three text-to-audio retrieval benchmarks.
- Image-Text Benchmarks: 27.0 SPICE on COCO captioning surpasses previous methods, while image QA also exceeds GIT.
B.1 Word cloud distribution
The section presents visual summaries and examples of VAST-27M, including word-cloud distributions, an instructional-prompt ablation, and additional samples.
- Word cloud distribution: Figure 3 presents Top-200 word-cloud maps for vision, audio, omni-modality captions, and raw subtitles in VAST-27M.
- Word cloud distribution: Figure 4 presents an ablation study of the instructional prompt used for omni-modality video caption generation.
- Word cloud distribution: Figure 5 provides additional VAST-27M samples.