Source-linked AI summary
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, Boqing Gong
TL;DR
Transformers need ways to learn from abundant unlabeled visual data rather than relying on costly supervised pre-training. VATT learns multimodal representations from raw video, audio, and text with convolution-free Transformers, and reports strong downstream results, including new records and competitive image-classification performance.
Problem
Large-scale supervised Transformer training excludes abundant unlabeled visual data and requires costly, time-consuming labeled-data collection.
Method
VATT self-supervises convolution-free Transformers on raw video frames, audio waveforms, and text transcripts using multimodal learning, with separate or shared modality backbones and DropToken.
Results
VATT reports new records for video action recognition and audio event classification, plus competitive image classification and video retrieval performance without supervised pre-training.
Takeaways & Limitations
The results suggest that Transformers can learn semantic video, audio, and text representations from multimodal self-supervision, including with one backbone shared across modalities.
Abstract
from arXiv · showhide
We present a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, our Video-Audio-Text Transformer (VATT) takes raw signals as inputs and extracts multimodal representations that are rich enough to benefit a variety of downstream tasks. We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval. Furthermore, we study a modality-agnostic, single-backbone Transformer by sharing weights among the three modalities. We show that the convolution-free VATT outperforms state-of-the-art ConvNet-based architectures in the downstream tasks. Especially, VATT's vision Transformer achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600, 72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding supervised pre-training. Transferring to image classification leads to 78.7% top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer from scratch, showing the generalizability of our model despite the domain gap between videos and images. VATT's audio Transformer also sets a new record on waveform-based audio event recognition by achieving the mAP of 39.4% on AudioSet without any supervised pre-training. VATT's source code is publicly available.
1 Introduction
VATT addresses the challenge of training convolution-free Transformers from unlabeled multimodal data by learning representations from raw video, audio, and text. It reports strong downstream performance, supports shared weights across modalities, and reduces training complexity with DropToken.
- Large-scale supervised Transformer training excludes abundant unlabeled visual data and requires costly, time-consuming collection of labeled examples.
- VATT pre-trains three Transformers from raw RGB frames, audio waveforms, and speech transcripts using a self-supervised multimodal strategy with minimal human-curated labels.
- 78.7% top-1 accuracy on ImageNet is comparable to ViT’s 79.9%, despite VATT transferring from videos to images without supervised pre-training.
- VATT’s results establish competitive or record performance across video action recognition, audio event classification, image classification, and video retrieval.
- A modality-agnostic Transformer sharing weights across video, audio, and text is on par with three modality-specific Transformers of slightly smaller sizes.
- DropToken randomly removes video and audio tokens during training, reducing training complexity with a minor performance reduction while enabling high-resolution inputs.
2 Related work
Related work establishes Transformers as increasingly general architectures and multimodal video as a source of self-supervision. VATT combines convolution-free Transformers with multimodal contrastive learning for raw video, audio, and text inputs.
- Convolution-free Transformers have achieved competitive image-recognition performance and have since been applied to several vision tasks.
- Self-supervised visual learning has used manually designed pretext tasks, instance discrimination, contrastive learning, and clustering on unlabeled images.
- Multimodal video supports self-supervision through audio correspondence prediction, cross-modality clustering, evolving losses, and contrastive learning across video, audio, and text.
- VATT combines convolution-free Transformers with multimodal contrastive learning.
3 Approach
VATT is a convolution-free multimodal Transformer framework that tokenizes raw video, audio, and text inputs, optionally shares one backbone across modalities, and trains representations with hierarchical common-space contrastive learning. DropToken reduces training complexity by randomly shortening video and audio token sequences while preserving high-resolution inputs.
- Architecture: The architecture supports either separate modality-specific backbones or a single Transformer backbone shared across video, audio, and text.In both settings, modality outputs are mapped into common spaces for contrastive comparison.
- Tokenization and Positional Encoding: VATT uses separate modality-specific tokenization layers and positional encodings to convert raw video, waveform audio, and text into Transformer input sequences.Video is divided into spatiotemporal patches, audio into waveform segments, and text words into projected embeddings.
- DropToken: DropToken randomly removes video and audio tokens during training, reducing Transformer computation that scales quadratically with sequence length.The strategy enables high-resolution inputs while introducing only a minor reduction in end-Transformer performance.
- The Transformer Architecture: A learnable aggregation token produces a representation for the entire input sequence, which is used for classification and common-space mapping.The Transformer uses standard self-attention, GeLU activations, and pre-layer normalization; the text model instead uses a learnable relative attention bias in its first layer.
- Common Space Projection: Hierarchical projections map video and audio outputs into a video-audio space, then map video and text into a video-text space for cosine-similarity comparisons.The hierarchy reflects different semantic granularities across modalities.
- Multimodal Contrastive Learning: VATT trains end-to-end with NCE for video-audio alignment and MIL-NCE for video-text alignment using matching and non-matching temporal locations.The overall objective combines the two losses with λ, which balances their contributions, and optimizes the batch-average loss by back-propagation.
4 Experiments
VATT is evaluated across video, audio, image, and text-to-video tasks using multimodal self-supervised pre-training, including modality-specific and shared-backbone configurations. The experiments show strong transfer across modalities, competitive zero-shot retrieval, separable representations, and modality-dependent activation patterns.
- Experimental setup: VATT is pre-trained on video-audio-text triplets and video-audio pairs, then evaluated across 4 downstream task families using 10 datasets.The tasks cover action recognition, audio event classification, zero-shot text-to-video retrieval, and ImageNet classification.
- Video action recognition: VATT achieves higher action-recognition accuracy than previous works on Kinetics-400, Kinetics-600, and Moments in Time without human-curated labels.On Kinetics-700, VATT reaches 72.7% top-1 accuracy versus 72.4% for the cited state of the art.
- Shared backbone: The modality-agnostic backbone is on par with modality-specific VATT-Base for video action recognition and AudioSet event classification.The shared backbone uses weights across video, audio, and text modalities while retaining separate tokenization and projection layers.
- Audio event classification: VATT’s audio Transformer consistently outperforms existing CNN-based models across AudioSet mAP, AUC, and d-prime while operating directly on raw waveforms.The model does not use handcrafted audio features.
- Text-to-video retrieval: Zero-shot text-to-video retrieval is comparable to MMV despite using half its pre-training epochs and batch size, while results remain sensitive to both factors.The evaluation uses YouCook2 and MSR-VTT and ranks videos by video-text similarity.
- Representation analysis: Fine-tuned VATT features separate classes better than a vision Transformer trained from scratch, with no clear difference between modality-specific and modality-agnostic features.The comparison uses t-SNE visualizations of Kinetics-400 representations.
5 Conclusion and Discussion
VATT uses self-supervised multimodal Transformers to learn representations across video, audio, and text, including a shared backbone across modalities. The study reports strong downstream performance while identifying dependencies on meaningful multimodal correspondences and representative data.
- Table 6 compares Kinetics400 top-1 accuracy using high-resolution inputs with DropToken against low-resolution inputs.
- VATT presents a self-supervised multimodal representation learning framework based on Transformers.
- Transformers learn semantic video, audio, and text representations even when one model is shared across modalities.
- VATT reports new records for video action recognition and audio event classification, with competitive image classification and video retrieval.
- DropToken reduces pre-training complexity for video and audio while having minor impact on model generalization.
- The approach depends on meaningful multimodal correspondences, while speech transcripts can be noisy and sparse.
A Appendix
The appendix provides additional dataset and experimental details, alongside evaluation comparisons and input-parameter ablations.
- Appendix A.1 contains more detailed explanations about the datasets.
- Appendix A.2 describes the experimental setup for pre-training and downstream tasks.
- Appendix A.4 reports linear evaluation results compared to state-of-the-art methods.
- Appendix A.5 presents an ablation study on the input parameters.
A.1.1 Pre-training
VATT is pre-trained on large-scale video-audio-text data from HowTo100M and AudioSet, using a policy-compliant subset of HowTo100M.
- VATT uses HowTo100M and AudioSet for pre-training.
- HowTo100M contains 1.2M unique videos with audio and narration scripts, producing 136M video-audio-text triplets.
- HowTo100M narration scripts are extracted from speech audio using an off-the-shelf ASR system.
- The pre-training data uses a policy-compliant HowTo100M subset with almost 1M unique videos and fewer than 100M clips.
A.1.2 Downstream
VATT is evaluated across video action recognition, audio event classification, zero-shot text-to-video retrieval, and image classification using diverse datasets and task-specific protocols.
- Video action recognition: Video action recognition is evaluated on UCF101, HMDB51, Kinetics-400, Kinetics-600, and Moments in Time.
- Video action recognition: For small UCF101 and HMDB51 datasets, the vision backbone is frozen and a linear classifier is trained on its outputs.
- Audio event classification: Audio event classification uses ESC50 and AudioSet, with linear evaluation on ESC50 and fine-tuning on AudioSet.
- Zero-shot video retrieval: Zero-shot text-to-video retrieval is evaluated on YouCook2 and MSR-VTT using Recall at 10.
- Image classification: The learned vision Transformer is fine-tuned on ImageNet without modifying its architecture or tokenization pipeline.
A.2.1 Inputs
VATT preprocesses synchronized raw video, audio, and text inputs with modality-specific sampling, normalization, tokenization, and sequence limits.
- Video inputs: Video frames are sampled and augmented through temporally consistent crops, resizing, horizontal flips, and color transformations.The pipeline samples 32 frames at 10 fps, crops them to 224 × 224, and applies spatial and color augmentation.
- Audio inputs: Audio waveforms are synchronized with video at 48kHz and normalized alongside video inputs to [-1, 1].Video uses 4 × 16 × 16 patches, while raw waveforms use patch size 128.
- Text inputs: Text is encoded with one-hot vectors using a vocabulary of 216 and limited to a maximum sequence length of 16 words.Sequences longer than 16 words are clipped, while shorter ones are padded.
A.2.2 Network setup in VATT
VATT uses modality-specific or shared Transformer backbones, contrastive objectives, and task-specific evaluation procedures. Experiments examine frozen-backbone transfer and the effects of input and patch choices.
- Network variants: VATT uses separate modality-specific Transformer variants and a Medium modality-agnostic backbone, with architecture details summarized in Table 7.The modality-specific configurations include BBS and MBS variants, while VATT-MA uses the Medium model.
- Pre-training objectives: The contrastive objectives use normalized projections into common spaces with dimensions 512 and 256, temperature τ = 0.07, and loss weight λ = 1.These hyperparameters follow previously established practice and may be improved by further tuning.
- Pre-training: VATT is pretrained from scratch for 500k steps with Adam, a batch size of 2048, warmup, and cosine learning-rate annealing on 256 TPUs.The models are trained for 3 days using TensorFlow 2.4.
- Linear evaluation: Frozen-backbone linear evaluation shows VATT does not outperform the best CNN counterparts, though it achieves comparable results to other baselines.The authors suggest the features may be less linearly separable, partly because the contrastive head includes nonlinear projections.
- Video input ablations: Smaller frame crops with more sampled frames hurt video results, while changing these choices does not significantly affect audio classification.Going beyond video patches of 4 × 16 × 16 along temporal or spatial dimensions is not helpful, and smaller patches substantially increase wall-clock time.
- Audio input ablations: Audio patch size 128 gives the best waveform-based results, while spectrogram inputs provide no conclusive improvement over raw waveforms.The spectrogram experiment supports learning semantic representations directly from raw audio.