Source-linked AI summary
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, Li Yuan
TL;DR
Existing video-language pretraining is difficult to extend beyond vision and language, especially with directly aligned data for additional modalities. LanguageBind uses language as the semantic binding space and trains other modality encoders with contrastive learning, supported by the five-modal VIDAL-10M dataset. The method and dataset show effectiveness across video, audio, depth, and infrared benchmarks, including zero-shot and indirect cross-modal evaluation.
Problem
Current VL pretraining frameworks are often limited to vision and language, while multimodal datasets with direct semantic alignment across five or more modalities are needed.
Method
LanguageBind freezes a pretrained language encoder and uses contrastive learning to align other modality encoders to the language space within a shared embedding space.
Results
LanguageBind achieves superior performance across 15 benchmarks covering video, audio, depth, and infrared, and exceeds VideoCoca by 8.3% on MSR-VTT retrieval.
Takeaways & Limitations
LanguageBind supports multimodal semantic alignment and reports indirect alignment and complementary capabilities among diverse modalities.
Abstract
from arXiv · showhide
The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N>=3) beyond vision and language. We thus propose LanguageBind, taking the language as the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pretraining, then train encoders for other modalities with contrastive learning. As a result, all modalities are mapped to a shared feature space, implementing multi-modal semantic alignment. While LanguageBind ensures that we can extend VL modalities to N modalities, we also need a high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M. In our VIDAL-10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. LanguageBind has achieved superior performance on a wide range of 15 benchmarks covering video, audio, depth, and infrared. Moreover, multiple experiments have provided evidence for the effectiveness of LanguageBind in achieving indirect alignment and complementarity among diverse modalities. Code address: https://github.com/PKU-YuanGroup/LanguageBind
1 INTRODUCTION
LanguageBind addresses the difficulty of extending video-language pretraining beyond vision and language by using language as a shared semantic anchor. It introduces the five-modal VIDAL-10M dataset and reports strong performance across downstream modality-understanding benchmarks.
- Current VL frameworks primarily support vision and language, while indirect alignment through images can degrade performance.
- LanguageBind removes the need for images as intermediaries, facilitating expansion to additional modalities in downstream tasks.
- LanguageBind directly aligns every non-language modality to a shared language embedding space through contrastive learning.The language encoder is frozen during pretraining, while other modality encoders are trained for alignment.
- VIDAL-10M contains 10 million aligned data pairs spanning video-language, infrared-language, depth-language, and audio-language modalities.The dataset uses short videos with complete stories and applies multi-view text generation and enhancement.
- Extensive experiments validate the dataset and approach for video and other modality-understanding tasks.
2 RELATED WORK
Prior multimodal pretraining largely grew from vision-language resources, while datasets for infrared and depth remain difficult to scale and align. This motivates accessible datasets with direct semantic alignment across many modalities.
- CLIP established large-scale image-text alignment that supports zero-shot classification and image-text retrieval, while later methods extend alignment to video and audio.
- Existing multimodal datasets often use truncated video segments, and specialized infrared and depth data remain difficult to acquire and annotate.
- VIDAL-10M is presented as the first accessible multimodal dataset containing aligned video-language, infrared-language, depth-language, and audio-language pairs.
- The field needs multimodal datasets with directly semantically aligned data pairs, especially when five or more modalities are involved.
3 METHOD
LanguageBind aligns multiple modalities through a shared language-centered embedding space. Its encoders use token masking, LoRA fine-tuning, and contrastive learning while the language encoder remains frozen.
- LanguageBind comprises multi-modal encoders, a language encoder, and multi-modal joint learning for cross-modal semantic alignment.
- Multi-modal encoders: The non-language encoders use OpenCLIP-large vision transformers; depth and infrared are replicated across three channels, while audio is represented as padded spectrograms.
- Patch masking: Patch masking converts modality inputs into patches, retains a selected subset, and adds learnable positional embeddings to the visible tokens.
- LoRA fine-tuning: LoRA fine-tuning freezes the original weight matrix W0 and learns an additional matrix BA with matching input and output dimensions.
- Modality extending: The modality-extending pipeline tokenizes each modality, initializes encoders from OpenCLIP, trains with masking and LoRA, and keeps the language encoder frozen.
- Multi-modal joint learning: Contrastive learning increases similarity between paired modality-text data and decreases similarity between unpaired data, binding each modality to language.
4 THE VIDAL-10M DATASET
VIDAL-10M is a five-modal dataset built through search-term-driven collection, filtering, modality generation, and multi-view text enhancement. It combines video, infrared, depth, and audio with aligned language data.
- VIDAL-10M contains 3 million video-language, 3 million infrared-language, 3 million depth-language, and 1 million audio-language pairs.
- Dataset construction: The dataset construction has three stages: visual search-term database creation, video and audio collection with filtering, and modality generation with enhancement.
- Search and collection: Search terms are drawn from labels and captions across visual datasets, then used to collect diverse videos from YouTube Shorts and audio-text pairs from relevant platforms.
- Video and audio collection and filtering: Video filtering removes samples with titles shorter than two words or lacking hashtags, excludes irrelevant tags, and limits duration to 20 seconds.
- Multi-view text generation and enhancement: Multi-view text combines titles, hashtags, keyframe captions, video captions, and enhanced captions generated with OFA, mPLUG-owl, and ChatGPT.
- Infrared and depth modality generation: Advanced generative models are used to construct large-scale infrared and depth data because specialized equipment and human effort make such datasets limited.
5 EXPERIMENTS AND RESULTS
LanguageBind is evaluated across video-language retrieval, multimodal zero-shot classification and retrieval, joint retrieval, emergent alignment, and training-design ablations. Results show strong performance across modalities, while LoRA offers an efficiency–performance trade-off.
- Zero-shot video-text retrieval: LanguageBind outperforms prior video-language methods across four zero-shot retrieval datasets, including gains of 1.9% on MSR-VTT and 8.8% on MSVD over InterVideo.It also exceeds VideoCoca by 8.3% and OmniVL by 8.0% on MSR-VTT.
- Zero-shot X-language classification: LanguageBind surpasses ImageBind by 14.0% on Kinetics-400 video classification, 23.8% on LLVIP infrared classification, and 11.1% on NYU-D depth classification.It also outperforms OpenCLIP on all three reported infrared datasets and by 19.7% on NYU-D.
- Zero-shot X-language classification: LanguageBind exceeds ImageBind by 10.1% on Audioset, 1.1% on VGGSound, and 23.9% on ESC-50 in zero-shot audio classification.The reported results use mAP for Audioset audio-only and top-1 accuracy for the other datasets.
- Zero-shot audio-language retrieval: LanguageBind outperforms prior methods on Clotho and Audiocaps audio-language retrieval, with reported gains over ImageBind of 6.1% and 5.5%, respectively.On Clotho it also exceeds AVFIC by 9.1% and VALOR by 3.7%.
- Joint and emergent retrieval: Adding audio embeddings improves MSR-VTT video-language retrieval from 41.4 to 42.0, demonstrating complementary information in the joint space.The section reports analogous trends for other modalities and evaluates R@1 or accuracy depending on the dataset.
- Training loss and architecture: LoRA reduces time and memory costs relative to full tuning while outperforming full tuning on LLVIP, FLIRv1, and Clotho.The ablations select one training epoch for cost–accuracy balance and identify batch size 1,024 as optimal.
6 CONCLUSION
The paper concludes that LanguageBind aligns language with multiple modalities through contrastive learning, while VIDAL-10M supplies large-scale language-aligned training pairs. Experiments support multimodal alignment, indirect cross-modal alignment, complementarity, and dataset effectiveness.
- LanguageBind: LanguageBind uses contrastive learning to align language semantically with all other modalities in a shared embedding space.The approach is presented as a language-based semantic alignment method for multimodal pretraining.
- VIDAL-10M: VIDAL-10M contains 10 million aligned VL, IL, DL, and AL pairs and is presented as the first large-scale video dataset with depth and infrared modalities.The dataset is intended to improve modal integrity during pretraining.
- Evidence: Extensive experiments demonstrate zero-shot cross-language understanding, indirect alignment between modalities, complementary capabilities, and the effectiveness of LanguageBind and VIDAL-10M.
REPRODUCIBILITY STATEMENT
The reproducibility statement points readers to detailed descriptions of the LanguageBind architecture, VIDAL-10M construction, training settings, and downstream task datasets.
- LanguageBind approach: The paper documents the multimodal encoder architecture, language encoder, and multimodal joint-learning methodology.
- VIDAL-10M construction: The dataset construction details cover search-term acquisition, video and audio collection and filtering, infrared and depth generation, and multiview text enhancement.
- Settings and release: Training hyperparameters and downstream task dataset setups are described in the appendices, and the authors promise to release VIDAL-10M upon publication.
A STATISTICS OF VIDAL-10M DATASET
VIDAL-10M is built from diverse short videos and enriched with multiple textual views, while its statistics examine category, duration, frame-rate, aspect-ratio, and resolution distributions.
- Text descriptions: Textual descriptions combine hashtags, titles, keyframe captions, and video captions, with captions generated and refined through OFA, mPLUG-owl, and ChatGPT.
- Data collection: The dataset uses search terms from visual datasets to collect diverse YouTube Shorts videos across categories such as sports, animals, and nature.
- Dataset statistics: The dataset includes stop-word statistics as part of its reported dataset analysis.
- Categories and duration: The dataset covers 15 categories across three duration grades: 1-7, 8-14, and 15-20 seconds.
- Video properties: Figures report the distributions of video FPS and aspect ratio, and separately summarize video height, width, and resolution.
B PRETRAINING DETAILS
The section presents training configurations for LanguageBind across video, depth, infrared, and audio modalities, alongside video-retrieval settings under different tuning and data regimes.
- Video-language training adds temporal attention before each spatial attention in LanguageBind.
- Depth-language training initializes from OpenCLIP with a frozen language encoder.
- Depth zero-shot templates replace “photo” with “depth photo,” while infrared templates retain “photo” because no improvement was observed.
- Audio-language training increases the training epoch and LoRA rank, and replaces “photo” with “sound” in zero-shot templates.
- Video-retrieval results are reported for LanguageBind with LoRA or full tuning, using large or huge models and 3M or 10M video-text pairs.
C DOWNSTREAM DATASETS
The paper evaluates LanguageBind across video, infrared, depth, and audio datasets using retrieval and classification tasks, while also examining text-source effects and dataset-use terms.
- Video-language evaluation uses MSR-VTT and MSVD for video-text retrieval.MSR-VTT reports on the 1K-A test subset, while MSVD reports on 670 test samples.
- Infrared-language evaluation uses LLVIP for binary person-versus-background classification and FLIR v1 for thermal-image tasks.
- Depth-language evaluation uses NYU-v2 Depth-only with depth constrained to 10 meters and reorganized into 10 scene categories.
- Audio-language evaluation uses ESC-50 for zero-shot classification and Clotho for text-to-audio retrieval.Clotho audios each have five corresponding captions.
- Different text sources are not universally suitable across tasks; ChatGPT-enhanced captions help video and depth, while OFA text is advantageous for infrared.Table 16 reports text-to-video R@1 for retrieval and top-1 accuracy for other datasets.
E.2 SCALING THE SIZE OF DATASET
Increasing the pretraining data amount improves zero-shot video recognition performance, with consistent trends in both retrieval directions.
- 0.9% and 1.6% are the performance gains of 3M ChatGPT-enhanced text over 500k and 100k data, respectively, on MSRVTT zero-shot retrieval R@1.
- Increasing data amount leads to significant improvement in MSRVTT zero-shot retrieval recognition performance.
- Video-to-text and text-to-video retrieval show consistent trends as data size expands.