PaperlayerView exact revision on arXiv

Source-linked AI summary

VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, Jing Liu

arXiv:2305.18500v2cs.CVcs.AIcs.CLcs.LGeess.AS

TL;DR

Contemporary video-text models have focused on vision and text, with audio and subtitles receiving less attention despite their value for video comprehension. The paper builds VAST-27M and trains VAST to connect these modalities with text; VAST outperforms prior models across diverse cross-modality benchmarks, including by 67.6 CIDEr points on YouCook2 captioning.

  • Problem

    Audio and subtitles provide valuable information for video comprehension, motivating their integration alongside vision and text in omni-modality models.

    Source: §1 · p. 2

  • Method

    The authors build VAST-27M with automatically generated vision and audio captions plus LLM-integrated captions using vision, audio, and subtitles, then train VAST to process these modalities with text.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    VAST outperforms prior vision-only and multimodal models across benchmark types, exceeding GIT2 by 67.6 CIDEr points on YouCook2 captioning.

    Source: §5.2 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    The authors report good generalization across task types when pretraining explores correlations between text and every modality group.

    Source: §5.4 · p. 10

    The authors identify a need for more diverse and larger-scale omni-modality corpora beyond VAST-27M.

    Source: §6 · p. 10

Abstract

from arXiv · show

Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including Vision, Audio, and Subtitle, and Text by exploring an automatically generated large-scale omni-modality video caption dataset called VAST-27M. Specifically, we first collect 27 million open-domain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off-the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new state-of-the-art results on various cross-modality benchmarks. Code, model and dataset will be released at https://github.com/TXH-mercury/VAST.

1 Introduction

The paper addresses the omission of audio and subtitles from video-text pretraining by introducing an automatically generated omni-modality caption corpus and a foundation model that processes vision, audio, subtitles, and text.

1 Introduction

  • Audio can add context that complements visual information, while speech and subtitles convey conversations, news, and instructional procedures relevant to video understanding.The paper argues these modalities should be jointly modeled with vision and text.Source: §1 · pp. 1–2
  • Existing video-text corpora lack the needed omni-modality captions: they use raw subtitles, vision-only alt-texts, or limited-scale audiovisual captions [7], while manual annotation is prohibitively costly.The required captions must correspond simultaneously to vision, audio, and subtitles.Source: §1 · p. 2
  • VAST-27M contains 27 million video clips, each paired with 11 captions, generated by combining trained vision and audio captioners with a zero-shot Vicuna-13b LLM using subtitles and prompts.The captions comprise five vision captions, five audio captions, and one omni-modality caption; the paper reports that VAST-27M surpasses existing video-text, audio-text, and audiovisual-text corpora by a significant margin.Source: §1 · p. 2
  • VAST is presented as the first foundation model to process vision, audio, subtitles, and text and support retrieval, captioning, and QA across vision-text, audio-text, and multimodal video-text tasks.It uses three single-modality encoders and a text encoder with cross-attention fusion, and is trained with OM-VCC, OM-VCM, and OM-VCG objectives.Source: §1 · p. 2, Table 1 · p. 3
  • The authors report that VAST outperforms state-of-the-art models across a diverse range of cross-modality benchmarks.They introduce VAST-27M as the first large-scale automatically generated omni-modality video caption dataset and VAST as the first vision-audio-subtitle-text foundation model.Source: §1 · p. 3

2 Related Work

Prior video- and audio-text corpora have limitations in modality coverage, scale, caption quality, or alignment, while VAST-27M and VAST are presented as addressing these gaps through automated omni-modality captions and four-modality learning.

2.1 Cross-Modality Pretraining Corpus

  • VAST-27M is presented as the first high-quality, large-scale video-caption corpus whose captions cover vision, audio, and subtitles through a cost-effective automated pipeline.Earlier subtitle captions can be weakly related to visuals, alt-texts can be noisy and omit other modalities, and VALOR-1M’s manual audiovisual captions omit subtitles and are costly to scale.Source: §2.1 · p. 3, Table 2 · p. 4
  • VAST-27M is nearly two orders of magnitude larger than LAION-Audio-630K or WavCaps, and generates audio captions with a trained high-quality caption model.Human-labeled audio-caption datasets contain fewer than 50,000 clips, while WavCaps uses ChatGPT to filter and rewrite raw descriptions, making caption quality dependent on those descriptions.Source: §2.1 · p. 3

2.2 Multi-Modality Learning

  • VAST is presented as the first omni-modality foundation model to perceive vision, audio, and subtitles and connect them to captions, addressing limitations in prior subtitle- and audio-aware methods.It uses an LLM to distill crucial subtitle information into language descriptions; earlier subtitle methods rely on masked or next-utterance prediction of raw subtitles, while several audio methods focus on audio-subtitle relations or environmental sounds.Source: §2.2 · p. 4

3 Dataset

VAST-27M is built from selected HD_VILA_100M clips, captioned through separate vision and audio models and an LLM that combines their outputs with subtitles. It contains 27M clips across 15+ categories, with vision, audio, and omni-modality captions averaging 12.5, 7.2, and 32.4 words, respectively.

3.1 Data Collection of VAST-27M

  • The vision captioner is trained on large image-text corpora, then fine-tuned on manually labeled image and video captions to describe static objects and dynamic actions.Training draws on CC4M, CC12M, 100M randomly selected LAION-400M pairs [29], and datasets including MSCOCO, VATEX, MSRVTT, and MSVD, inspired by BLIP [28].Source: §3.1 · p. 4
  • The audio captioner is trained on VALOR-1M and WavCaps without second-stage fine-tuning, which the authors avoid because limited downstream data could cause overfitting.VALOR-1M is human-annotated, providing some assurance of generated-caption accuracy.Source: §3.1 · p. 4
  • VAST-27M selects clips evenly across HD_VILA_100M’s 3.3M long-form videos, retaining only 5–30-second clips that include vision, audio, and subtitles.The selection limits downloading, storage, and computation costs.Source: §3.1 · p. 4
  • For each clip, vision and audio captioners generate five captions each with Top-K sampling at K=10; Vicuna-13b then integrates three randomly selected captions of each type with subtitles and prompts.The LLM produces natural-style omni-modality captions describing visual, audio, and subtitle content.Source: §3.1 · p. 5

3.2 Statistics of VAST-27M

  • 27M clips from 15+ categories make up VAST-27M, whose vision, audio, and omni-modality captions average 12.5, 7.2, and 32.4 words, respectively.The dataset is sampled from HD_VILA_100M and contains more captions in both modalities and quantity than other commonly used cross-modality training corpora.Source: §3.2 · p. 5

4 Approach

VAST combines vision, audio, and text encoders in an end-to-end framework, then trains omni-modality video-caption alignment and generation while modeling multiple modality combinations. Its modality-grouping objective addresses missing modalities in downstream use.

4.1 Basic Framework

  • VAST’s end-to-end Transformer combines ViT, BEATs, and BERT encoders to process images, videos, audio, subtitles, and captions.Audio is split into 10-second clips and represented as 64-dimensional log Mel filterbank spectrograms; text encoding also supports cross-attention for multimodal encoding and decoding.Source: §4.1 · pp. 5–6

4.2 Pretraining Objectives

  • The three pretraining losses align video and captions contrastively, classify matched pairs using cross-attention and hard negatives, and generate captions by reconstructing masked tokens.For generation, 60% of caption tokens are masked, with causal self-attention and video-conditioned cross-attention; the three losses are equally weighted.Source: §4.2 · p. 6

4.3 Modality Grouping

  • VAST uniformly models V-T, A-T, VA-T, VS-T, and VAS-T relations to address missing modalities at downstream adaptation, following VALOR [7].Vision and audio captions support V-T and A-T, respectively; omni-modality captions support VA-T, VS-T, and VAS-T.Source: §4.3 · p. 6
  • VAST achieves 22 new state-of-the-art results across retrieval, captioning, and QA benchmarks, evaluated with Recall@1, CIDEr, and Acc, respectively.The table distinguishes vision-only and multimodal state-of-the-art methods for benchmarks with audio or subtitle tracks.Source: Table 3 · p. 7

5 Experiments

Experiments compare VAST with prior models and training corpora across vision-text, audio-text, and multimodal video-text tasks, and examine modality choices in pretraining and finetuning.

5.1 Implementation Details

  • VAST uses 1.3B parameters and is trained for 200K steps on a five-corpus mixture, sampling one corpus per step.Training uses PyTorch on 64 V100 GPUs, encoders initialized from EVAClip-ViT-G, BEATs, and BERT-B, a 1e-4 initial learning rate with linear decay, and batch size 1024; ablations use frozen CLIP-ViT-B. Retrieval uses VCC then VCM to rerank the top 50 candidates, captioning uses beam size 3, and QA is open-ended generation.Source: §5.1 · p. 7

5.3 Comparison to Open-Source Cross-Modality Training Corpus

  • Vision-caption-trained VAST-27M model (h) achieves the best results on all six visual-only MSVD and MSRVTT benchmarks, while subtitle-as-caption model (f) performs worst.Model (g), trained on VAST-27M, beats other corpora but trails (h), as multimodal captions add noise when only visual content is used; models were trained for 50K steps.Source: Table 4 · p. 8, §5.3 · p. 8
  • Audio-text model (d) surpasses the baseline and models trained on VALOR-1M and WavCaps on all four Clothov2 and AudioCaps benchmarks.Because VALOR-1M and WavCaps tended to overfit, their training iterations were adjusted to obtain their best results rather than held equal.Source: Table 5 · p. 8, §5.3 · p. 9

5.2 Comparison to State-of-the-Art Models

  • VAST surpasses BEiT-3 and BLIP-2 on Flickr30K text-to-image retrieval under finetuning and zero-shot settings, respectively, sets new state-of-the-art results on TGIF-QA and MSVD-QA, and achieves five new audio-text state-of-the-art results.The paper attributes the audio-text gains to abundant generated captions in VAST-27M.Source: §5.2 · p. 8
  • VAST beats GIT2 by 2.1 CIDEr on MSRVTT, 67.6 on YouCook2, 5.0 on VATEX, and 8.0 on TVC captioning, using 22.5% of its parameters and 3.4% of its training data.VAST also outperforms prior vision-only and multimodal state-of-the-art models across subtitle-oriented, audio-oriented, and event-oriented video-text benchmarks, and exceeds VALOR on 11 benchmarks.Source: §5.2 · p. 8

5.3 Comparison to Open-Source Cross-Modality Training Corpus

  • Omni-modality pretraining on VAST-27M improves all seven benchmarks over the baseline and beats VALOR-1M on five MSRVTT and YouCook2 benchmarks, but trails it on VALOR-32K.Replacing LLM-integrated omni-modality captions with simple concatenations lowers model (c)'s results on every benchmark relative to model (d); the VALOR-32K gap is attributed to matching video distribution and caption style in VALOR-1M.Source: Table 6 · p. 9, §5.3 · p. 9

5.4 Ablation Study

  • Adding audio and subtitles improves the vision-only baseline, and omni-modality pretraining further improves most benchmarks; model (i) outperforms model (f) on all benchmarks.Models (h) and (f) underperform models (g) and vision baseline (e) on VALOR-32K, where most videos lack subtitles despite subtitles being present in VAST-27M; modality grouping in model (j) generalizes across task types by exploring text correlations with each modality group during pretraining.Source: Table 7 · p. 9, §5.4 · pp. 9–10

6 Conclusion, Broader Impact and Limitation · Appendix · A More about VAST Foundation Model

The paper presents VAST-27M and VAST for connecting video vision, audio, subtitles, and text, and reports that VAST surpasses existing state-of-the-art methods on public cross-modality benchmarks. It also identifies potential applications and limitations, including corpus scale, reliance on LLM integration, and inherited biases.

6 Conclusion, Broader Impact and Limitation

  • VAST surpasses existing state-of-the-art methods on public cross-modality benchmarks spanning vision-text, audio-text, and multimodal video-text retrieval, captioning, and question answering.The paper reports effectiveness across this range of tasks.Source: §6 · p. 10
  • VAST-27M pairs each video clip with automatically generated vision and audio captions and an LLM-generated caption integrating vision, audio, and subtitles.The corpus is intended to advance research in multimodality pretraining.Source: §6 · p. 10
  • The authors identify entertainment, education, security, transportation, and healthcare as domains that could benefit from advances in video understanding and generation.They suggest omni-modality foundation models such as VAST-27M and VAST could contribute to progress and practical applications in these areas.Source: §6 · p. 10
  • The authors call for more diverse and larger-scale omni-modality corpora beyond VAST-27M.This is identified as a limitation despite VAST’s broad downstream-task support.Source: §6 · p. 10
  • VAST requires LLM integration to further enhance its generalization capabilities.The authors identify this as a limitation of the model’s current downstream-task support.Source: §6 · p. 10
  • VAST-27M and VAST may inherit biases from LLMs used in data collection and open-source cross-modality corpora used to train the video and audio captioners.The authors note that both dataset creation and captioner training may transmit biases from their sources.Source: §6 · p. 10

A.1 Pretraining Settings

VAST’s pretraining settings specify its training corpora, data mix, and objectives; the authors report 205K training steps and expect further training to increase model capabilities.

A.1 Pretraining Settings

  • VAST was trained for 205K steps, and the authors state that more training may further increase its capabilities.The paper reports that the model already performs well on various downstream tasks.Source: §A.1 · p. 17
  • The authors replaced the original CC12M and LAION captions with new captions generated by their trained vision captioner.Source: §A.1 · p. 17
  • Table 8 presents VAST’s training corpora, dataset mix ratios, and training objectives; it defines ‘ret’ as VCC plus VCM and ‘cap’ as VCG.The table caption says different modality groups are separated by ‘%’.Source: Table 8 · p. 17, §A.1 · p. 17

A.2 Downstream Datasets Descriptions · A.3 Finetuning Settings

The appendix describes downstream benchmarks spanning video, audiovisual, audio, image, and visual-question-answering tasks, and points to tables listing dataset splits and finetuning hyperparameters.

A.2 Downstream Datasets Descriptions

  • Table 9 provides the train, validation, and test splits for the downstream benchmarks.The dataset descriptions also specify benchmark-specific official or cited split protocols.Source: §A.2 · p. 17, Table 9 · p. 18
  • VAST is evaluated across retrieval, captioning, and QA benchmarks including MSRVTT, VATEX, YouCook2, and audiovisual VALOR-32K [7].MSRVTT uses the 1K-A retrieval split and standard captioning/QA splits; VATEX uses its official captioning split and HGR retrieval protocol; YouCook2 and VALOR-32K use official splits.Source: §A.2 · p. 17
  • The evaluation also covers MSVD video QA, LSMDC retrieval, DiDeMo paragraph-to-video retrieval, and ActivityNet Caption retrieval and QA.DiDeMo’s four ordered sentences are concatenated for retrieval; official splits are used except that MSVD QA follows Xu et al. and ActivityNet Caption QA follows Yu et al..Source: §A.2 · pp. 17–18
  • Additional video benchmarks include TGIF-frame QA, MUSIC-AVQA with over 45K question-answer pairs, and TVC captioning with optional subtitle input.TGIF-frame uses the official split; MUSIC-AVQA spans 33 question templates and modalities; TVC’s official captioning split is used.Source: §A.2 · p. 18
  • Audio evaluation uses Clotho and AudioCaps for text-to-audio retrieval and audio captioning, while MSCOCO and Flickr30K support image tasks and VQAv2 image QA.Clotho v1/v2 contain 4,981/6,974 audio clips, with v2 enlarging only training; MSCOCO and Flickr30K use Karpathy splits, while VQAv2 uses its official split.Source: §A.2 · pp. 18–19

A.3 Finetuning Settings

  • Table 10 presents VAST’s benchmark-specific finetuning settings, including learning rate, batch size, epochs, objectives, resolution, and sampled video-frame or audio-clip counts.The appendix notes that most listed hyperparameters were not precisely tuned.Source: §A.3 · p. 19, Table 10 · p. 19

A.4 Detailed Comparisons to State-of-the-Art Methods

VAST reports improved or state-of-the-art performance across video retrieval, video QA, video captioning, audio tasks, and image-text benchmarks. The comparisons highlight its results across modalities and its use of a unified architecture.

Text-to-Video Retrieval

  • 5.1, 1.6, 3.7, and 6.1 points are VAST’s gains over previous SOTA on MSRVTT, DiDeMo, ActivityNet, and VATEX, respectively.VAST also beats VALOR-L [7] by 6.8 points on VALOR-32K and MELTR [55] by 16.7 points on YouCook2; its zero-shot R@1 scores are 49.3 and 55.5, exceeding previous SOTA by 7.3 and 6.9 points, respectively.Source: §A.4 · p. 19, Table 11 · p. 20, Table 12 · p. 20

Video QA

  • VAST achieves new SOTA performance on all five open-ended video QA benchmarks, outperforming GIT [19], MaMMUT [49], Flamingo [92], and CoCa [93].On MUSIC-AVQA, VAST surpasses VALOR by 1.8 points.Source: §A.4 · p. 19, Table 13 · p. 21

Video Captioning

  • VAST achieves new state-of-the-art CIDEr scores on all five multi-modal video captioning benchmarks, with evident margins over prior methods.Compared with GIT [19], VAST uses 22.5% of its parameters and 3.4% of its training data; compared with VALOR [19], it handles subtitle-oriented benchmarks YouCook2 and TVC and achieves better results through joint modeling of text and video omni-modalities.Source: §A.4 · p. 20, Table 14 · p. 21

Text-to-Audio Retrieval and Audio Captioning

  • 7.6, 5.4, and 9.8 R@1 points are VAST’s gains over previous SOTA on three text-to-audio retrieval benchmarks, respectively.VAST achieves new SOTA on Clotho V1 and V2 audio captioning and comparable performance to WavCaps [18] on AudioCaps, using one unified architecture without benchmark-specific optimization.Source: Table 15 · p. 22, Table 16 · p. 22, §A.4 · p. 22

Image-Text Benchmarks

  • VAST shows strong image-text benchmark capabilities despite being designed for omni-modality video-language tasks.It outperforms BLIP-2 [46] and BEiT-3 [47] on Flicker30K R@1 and MSCOCO R@5 and R@10; it scores 27.0 SPICE on COCO captioning, surpassing OFA and GIT2 [19], and performs better than GIT [19] on open-ended image QA.Source: Table 17 · p. 22, §A.4 · pp. 22–23

B More about VAST-27M Dataset … B.3 More Examples

The appendix presents a word-cloud distribution for VAST-27M’s caption and subtitle types, an instructional-prompt ablation, and additional dataset samples.

B.1 Word cloud distribution

  • Figure 3 shows a top-200 word-cloud map for vision, audio, and omni-modality captions, alongside raw subtitles.Source: Figure 3 · p. 24

B.2 Prompts for Omni-Modality Caption Generation

  • Figure 4 presents an ablation study of the instructional prompt used to generate omni-modality video captions.Source: Figure 4 · p. 24

B.3 More Examples

Related papers

Loading 2305.18500v2…