PaperlayerView exact revision on arXiv

Source-linked AI summary

VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

Jing Liu, Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang

arXiv:2304.08345v2cs.LGcs.CLcs.CVcs.MMeess.AS

TL;DR

Vision-language pretraining does not capture complementary audio information and existing datasets often lack suitable tri-modality supervision. VALOR jointly models vision, audio, and language with dedicated alignment and captioning tasks, using the human-annotated VALOR-1M dataset. It achieves new state-of-the-art results across retrieval, captioning, and question-answering benchmarks.

  • Problem

    Existing vision-language datasets and models do not adequately support tri-modality pretraining with diverse, objectively described audio-visual content.

  • Method

    VALOR uses separate vision, audio, and language encoders, a multimodal decoder, and MGA and MGC to learn shared alignment and conditional generation.

  • Results

    VALOR achieves new state-of-the-art performances across public retrieval, captioning, and question-answering benchmarks, including vision-, audio-, and audiovisual-language tasks.

  • Takeaways & Limitations

    The model and datasets provide a unified basis for evaluating and developing vision-audio-language understanding and generation.

  • Takeaways & Limitations

    The authors plan to scale VALOR-1M with unsupervised pseudo-caption generation and add vision and audio generation modeling.

Abstract

from arXiv · show

In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale high-quality tri-modality dataset named VALOR-1M, which contains 1M audiable videos with human annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and be generalized to various downstream tasks (e.g., retrieval, captioning and question answering), with different input modalities (e.g., vision-language, audio-language and audiovisual-language). VALOR achieves new state-of-the-art performances on series of public cross-modality benchmarks. Code and data are available at project page https://casia-iva-group.github.io/projects/VALOR.

1 INTRODUCTION

VALOR addresses the limits of vision-language pretraining by jointly modeling vision, audio, and language for multimodal understanding and generation. It introduces unified pretraining tasks and the VALOR-1M dataset, achieving strong results across cross-modality benchmarks.

  • Motivation: Vision-language modeling alone is insufficient because audio can provide semantic information complementary to visual content.The paper gives the example of a police siren revealing events outside a room that video frames cannot show.
  • Model: VALOR uses three single-modality encoders and a multimodal decoder to support conditional text generation across vision, audio, and language.The model is designed as an end-to-end framework for tri-modality understanding and generation.
  • Pretraining: MGA aligns vision-language, audio-language, and audiovisual-language groups in a shared space, while MGC trains multimodal conditional text generation.Together, the tasks target discriminative and generative capabilities with different modality inputs.
  • Datasets: VALOR-1M contains one million audiable videos with human-annotated audiovisual captions, and VALOR-32K adds audiovisual retrieval and captioning benchmarks.The captions describe audio and visual contents simultaneously.
  • Results: 3.8%, 6.2%, 12.7%, 0.6%, 10.4% (R@1) improvements are reported on MSRVTT, DiDeMo, ActivityNet, LSMDC, and VATEX text-to-video retrieval.The paper also reports gains on video question answering and text-to-audio retrieval, plus a comparison against GIT2 on VATEX captioning.
  • Results: Pretrained on VALOR-1M and public vision-language datasets, VALOR achieves new state-of-the-art performances across cross-modality benchmarks.The reported coverage includes retrieval, captioning, and question answering tasks.

2 RELATED WORK

Prior multimodal pretraining largely centers on vision and language, with datasets and models often limited by caption quality, missing audio, or task scope. VALOR extends this landscape with unified tri-modality modeling and broader task coverage.

  • Cross-Modality Datasets: Early vision-language datasets such as HowTo100M scale through ASR transcriptions, while WebVid uses alt-text captions paired with 2.5M videos.ASR-based datasets contain speech-derived text, whereas alt-text provides a different captioning source.
  • Cross-Modality Datasets: ASR captions may contain recognition errors, subjective speech, weak visual descriptions, and temporal misalignment with video content.These limitations constrain the quality of large-scale video-text supervision.
  • Cross-Modality Datasets: Existing image-language and several video-language datasets lack audio, while audio-containing datasets may offer mainly human speech and ASR transcriptions rather than objective descriptions.This leaves them unsuitable for full vision-audio-language pretraining.
  • Pretraining Frameworks: Vision-language frameworks include dual encoders for efficient retrieval and fusion encoders for finer-grained captioning or visual question answering.The distinction is based on how deeply the modalities are fused.
  • Unified Modeling: Unified multi-task methods remove task-specific heads, but prior approaches remain constrained to the vision-language domain.They use sequence-to-sequence, contrastive, or masked-language-modeling formulations for multiple tasks.
  • Beyond Vision-Language: VALOR extends multimodal video learning beyond discriminative tasks by supporting discriminative, contrastive, and generative objectives in one framework.Its unified architecture and pretraining tasks enable this broader task coverage.

3 VALOR DATASET FOR AUDIOVISUAL-LANGUAGE PRETRAINING

VALOR addresses the lack of strongly correlated vision-audio-language data by constructing annotated audiovisual datasets for tri-modality pretraining and evaluation. The datasets combine diverse audiovisual videos with captions describing both modalities and introduce retrieval and captioning benchmarks.

  • Dataset motivation: VALOR annotates public audiovisual data to create a vision-audio-language correlated dataset for tri-modality model pretraining and benchmarking.The dataset is designed to overcome the weak correspondence between audio concepts and captions in ASR- and alt-text-based datasets.
  • AudioVisual Data Collection: The source videos come from AudioSet, providing audiovisual tracks with high quality and diversity for dataset construction.VALOR-1M uses AudioSet’s unbalanced training set, while VALOR-32K uses its balanced training and evaluation sets.
  • Annotation: VALOR uses paid labeling with a three-step interactive procedure covering annotator training, first-stage annotation, and second-stage annotation.VALOR-1M descriptions are checked by three annotators and re-annotated when more than one considers them unsatisfactory.
  • Benchmark: VALOR-32K expands audiovisual-language evaluation beyond question answering with audiovisual retrieval and audiovisual captioning tasks.AVC generates audiovisual captions, while AVR retrieves the best-matching video for a given audiovisual query.
  • Dataset characteristics: VALOR-1M contains open-domain audiable videos with manually annotated audiovisual captions describing audio and visual contents simultaneously.The dataset emphasizes rich audio concepts and strong vision-language and audio-language correlations at large scale.
  • Quantitative Comparison: Audio concept density measures detected audio concepts relative to total caption words, using a 759-concept vocabulary derived from AudioSet’s ontology.Captions are normalized and searched for audio concepts before computing the metric.
  • Quantitative Comparison: VALOR captions have higher audio concept density and longer average lengths than comparison datasets, including 16.4 words for VALOR-1M and 19.8 for VALOR-32K.For comparison, WebVid-2.5M averages 14.2 words and CC3M averages 10.3.

4 VALOR MODEL

VALOR combines separate vision, audio, and text encoders with a multimodal decoder, training them through modality-grouped alignment and captioning objectives. This design supports retrieval, captioning, and question answering across vision-language, audio-language, and audiovisual-language inputs.

  • Architecture: VALOR uses separate text, vision, and audio encoders plus a multimodal decoder for conditional text generation.The decoder partly shares parameters with the text encoder and attends to video, audio, or concatenated audiovisual features.
  • Pretraining Tasks: MGA aligns text with vision, audio, and audiovisual inputs using bidirectional contrastive learning over matching and non-matching pairs.Fine-grained similarities are computed between text tokens and video frames or audio clips in a shared normalized semantic space.
  • Pretraining Tasks: The modality grouping strategy jointly covers text-vision, text-audio, and text-audiovisual task settings.MGA and MGC are optimized simultaneously, with a tunable hyperparameter controlling their loss ratio.
  • Pretraining Tasks: MGC trains causal masked language modeling to generate text conditioned on vision, audio, or audiovisual features.Its total loss averages the three modality-group captioning losses, with audiovisual features formed by concatenating visual and audio features.
  • Downstream Adaptation: VALOR adapts its objectives to retrieval, captioning, and question answering across audiovisual, visual, and audio inputs.Retrieval uses MGA, while captioning and question answering use MGC; answers are generated from the whole vocabulary rather than a fixed top-k set.

5 EXPERIMENTS

The experiments introduce the evaluation settings and implementation context for comparing VALOR. The section then proceeds to present ablations and visualizations of the model’s predictions.

  • Experiment Setup: The experiments section begins by defining pretraining datasets, downstream benchmarks, and implementation details.
  • Evaluation: The paper compares VALOR with existing methods before presenting detailed ablation studies and prediction visualizations.

5.1 Experiment Settings

VALOR is pretrained and evaluated across video-, audio-, audiovisual-, and image-language settings using multiple public datasets and the proposed VALOR-1M dataset.

  • Pretraining Datasets: VALOR-1M contains one million open-domain audiable videos with manually annotated audiovisual captions.
  • Pretraining Datasets: Pretraining uses WebVid-2.5M, CC14M, HD VILA 10M, and VALOR-1M.
  • Evaluation Tasks: Retrieval evaluation covers nine datasets spanning video retrieval, audio retrieval, and audiovisual retrieval, using R@K metrics.
  • Evaluation Tasks: Captioning evaluation covers seven datasets across video, audio, and audiovisual captioning, reporting BLEU4, METEOR, ROUGE-L, CIDEr, and SPICE.
  • Evaluation Tasks: Open-ended question answering is evaluated on six video and audiovisual datasets using accuracy.
  • Implementation Details: VALORB and VALORL differ in training data, batch size, iterations, and vision encoder, while sharing text and audio encoders.

5.2 Comparison to State-of-the-arts

VALOR achieves strong results across video-language, audio-language, image-language, captioning, and question-answering benchmarks, often outperforming substantially larger systems.

  • Video-Language Benchmarks: VALORL surpasses previous state-of-the-art video retrieval results by 3.8%, 6.2%, 12.7%, 0.6%, and 10.4% R@1 on five benchmarks.
  • Video-Language Benchmarks: VALORB outperforms all Group-A methods on four video-captioning benchmarks.
  • Video-Language Benchmarks: VALORL outperforms GITL and GIT on most captioning metrics and surpasses GIT2 on VATEX with 11.6% of its parameters and 0.26% of its data.
  • Video-Language Benchmarks: VALORL achieves new state-of-the-art results on four video QA benchmarks, exceeding prior results by 3.8%, 3.4%, 5.1%, and 12.5%.
  • Video-Language Benchmarks: VALORL and VALORB improve MUSIC-AVQA over baseline by 10.3% and 7.1%, respectively.
  • Audio-Language Benchmarks: VALOR exceeds prior audio-retrieval state-of-the-art results by 38.9% on ClothoV1 and 13.0% on AudioCaps R@1.
  • Image-Language Benchmarks: VALOR achieves comparable or better performance than selected larger models on COCO retrieval, captioning, and VQAv2 benchmarks.

5.3 Ablation Study

Ablations show benefits from combining modalities, VALOR pretraining, modality grouping, and audiovisual alignment, while joint MGA–MGC training produces task-dependent trade-offs.

  • Vision-Audio-Language Cross-Modality Learning: Combining vision and audio consistently improves performance across nine benchmarks under both pretrained and non-pretrained settings.
  • Vision-Audio-Language Cross-Modality Learning: Adding audio improves AVR and MSRVTT VR performance by 26.0% and 14.9%, respectively.
  • Modality Grouping: Modality grouping is evaluated in both MGA and MGC, with M6 selected as the default setting based on the ablation results.
  • Audiovisual Fusion: MGA ablations compare fine-grained and coarse-grained audiovisual alignment, as well as feature-fusion and score-fusion strategies.
  • Audiovisual Fusion: MGC ablations compare multimodal decoder attention mechanisms for predicting masked tokens conditioned on visual and audio features.
  • Combination of MGA and MGC: Joint MGA and MGC training decreases AVR from 55.6 to 53.8 but increases AVC from 49.6 to 50.3; parameter sharing improves AVR by 0.9 points.
  • Pretraining Dataset Comparison: Models pretrained on VALOR-1M outperform models pretrained on WebVid-2.5M, CC3M, and HD VILA 10M across all three MSRVTT benchmarks.
  • Architecture Comparison: Using a more powerful vision encoder improves all four retrieval benchmarks when BERTB is the text encoder.

5.4 Visualizations

Qualitative visualizations compare VALOR with audiovisual task-specific methods and show its ability to use both visual and audio concepts in retrieval and captioning.

  • Visualization Results: VALOR is qualitatively compared with AVLNet for audiovisual retrieval and SMPFF for audiovisual captioning on VALOR-32K.
  • Visualization Results: Figure 8 presents prediction visualizations for different models on VALOR-32K, with audio playable for the examples.
  • Visualization Results: For retrieval, VALOR ranks the ground-truth video above candidates that share visual content but lack query-relevant sounds.
  • Visualization Results: For captioning, VALOR recognizes visual and audio concepts across all three examples, whereas SMPFF sometimes misrecognizes or ignores audio.

6 CONCLUSION

VALOR unifies vision, audio, and language pretraining through two tasks and introduces datasets for tri-modality research. It achieves new state-of-the-art results across diverse downstream tasks, while future work targets dataset scaling and broader generation modeling.

  • VALOR models tri-modality understanding and generation through Multimodal Grouping Alignment and Multimodal Grouping Captioning.
  • VALOR-1M provides a strongly correlated vision-audio-language dataset, while VALOR-32K supports audiovisual-language retrieval and captioning benchmarks.
  • VALOR achieves new state-of-the-art performances on downstream vision, audio, and audiovisual retrieval, captioning, and question answering tasks.
  • Future work will scale VALOR-1M through unsupervised audiovisual-caption generation and filtering, and add vision and audio generation modeling.
Loading 2304.08345v2…