Source-linked AI summary
VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, William Yang Wang
TL;DR
Video-and-language research lacked a large, multilingual video-description dataset with broad coverage and translation pairs. VATEX supplies such a dataset and introduces unified multilingual captioning and video-guided machine translation; experiments report efficient, improved multilingual captioning and effective use of video context for translation.
Problem
Existing large-scale video-captioning datasets are mostly monolingual, limiting multilingual video-language research and leaving the role of video context in language tasks underexamined.
Method
VATEX provides over 41,250 videos, 825,000 English and Chinese captions, and around 206K parallel sentences, with unified captioning and video-guided translation tasks.
Results
Unified multilingual captioning is more efficient and effective than monolingual models, while temporal-attention video features substantially improve translation performance.
Takeaways & Limitations
VATEX supports multilingual video understanding and research on using spatiotemporal video context to align source and target languages.
Abstract
from arXiv · showhide
We present a new large-scale multilingual video description dataset, VATEX, which contains over 41,250 videos and 825,000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSR-VTT dataset, VATEX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on VATEX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the VATEX dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using VATEX for other video-and-language research.
1. Introduction
VATEX addresses the limits of monolingual video-captioning datasets by providing a large multilingual benchmark and introducing multilingual captioning and video-guided translation tasks. Experiments support compact unified captioning and video context for translation.
- Motivation: Existing large-scale video-captioning datasets are mostly monolingual, while multilingual captioning requires high-quality, diverse captions spanning varied videos.The paper motivates multilingual video captioning as important for people who do not speak English.
- Dataset: VATEX contains over 41,250 unique videos, 825,000 captions, 600 human activities, and 10 English plus 10 Chinese captions per video.Its captions are described as unique, lexically rich, and diverse.
- Tasks: VATEX supports multilingual video captioning, training a unified model to generate descriptions in multiple languages such as English and Chinese.The task examines how sharing architectural components across languages affects efficiency and effectiveness.
- Tasks: Around 206K English-Chinese parallel sentences support video-guided machine translation using video as additional spatiotemporal context.The task is designed to reduce language ambiguity and improve alignment between source and target languages.
- Results: Video-guided translation performs much better when temporal attention dynamically selects relevant spatiotemporal features for word mapping.Simply incorporating average or LSTM video features provides only marginal improvements.
2. Related Work
Related work spans video-description datasets and multimodal language understanding. VATEX is positioned as a large multilingual benchmark that supports richer evaluation and multilingual multimodal research.
- Video Description Datasets: Video-description datasets cover domains including cooking, movies, social media, and human activities, with single-sentence and multi-sentence generation as major task families.Other variants include video title and video story generation.
- Video Description Datasets: VATEX provides 20 captions per video, supporting consideration of human variation and more human-consistent evaluations.The dataset targets single-sentence generation and emphasizes broad video and language coverage.
- Video Description Datasets: VATEX contains English and Chinese descriptions at scale and translation pairs, unlike MSVD, enabling large-scale multilingual multimodal research.The related-work discussion contrasts VATEX with datasets lacking translation pairs or comparable multilingual scale.
- Multimodal Machine Translation: Multimodal machine translation supplements a source sentence with information from another modality, with prior work focusing mainly on images.VATEX extends this research direction to video-guided translation.
3. VATEX Dataset
VATEX is a large, multilingual video-description dataset built from Kinetics-600 videos, with extensive English and Chinese annotations. Compared with MSR-VTT, it offers broader coverage, longer and lexically richer captions, and greater diversity.
- Data Collection: VATEX reuses Kinetics-600 videos covering 600 human activities and collects descriptions for 41,269 valid clips.The collection cost approximately $51,000 and used English and Chinese annotations.
- Data Collection: Each VATEX video receives 10 English captions and 10 Chinese descriptions from human annotators.The Chinese annotations include direct descriptions and English-Chinese translation pairs.
- Dataset Analysis: VATEX contains 413k English and 413k Chinese captions for 41.3k videos, compared with MSR-VTT’s 200k captions for 7k videos.VATEX covers 600 activities, whereas MSR-VTT covers 257.
- Dataset Analysis: VATEX-en, VATEX-zh, and MSR-VTT have average caption lengths of 15.23, 13.95, and 9.28, respectively.The VATEX captions are described as longer and more detailed than MSR-VTT captions.
- Dataset Analysis: VATEX is lexically richer and more diverse than MSR-VTT, with broader caption styles and coverage of actions, objects, and visual scenes.MSR-VTT has exact-caption duplication in 66.0% of videos, while VATEX captions within each video are unique and more diverse across the corpus.
4. VATEX Tasks
VATEX supports multilingual video captioning through shared and separate encoder-decoder designs. The experiments show that compact multilingual models improve efficiency while maintaining consistently improved captioning performance over monolingual models.
- Multilingual Video Captioning: Multilingual video captioning generates descriptions of a video in more than one language, such as English and Chinese.VATEX frames this as a task for multilingual understanding of video dynamics.
- Models: The baseline captioning pipeline uses a 3D ConvNet, a temporal video encoder, and an attention-based language decoder.The decoder uses the previous word and visual context to produce each next word.
- Models: The instantiated model uses pretrained I3D features, a bi-LSTM video encoder, an LSTM language decoder, and dot-product attention.The I3D model is used without fine-tuning for visual feature extraction.
- Models: VATEX compares separate monolingual Base models with Shared Enc and Shared Enc-Dec models that share multilingual components.Shared Enc shares the video encoder, while Shared Enc-Dec shares both encoder and decoder and uses language-specific word embeddings.
- Results and Analysis: Multilingual models consistently improve performance over the monolingual Base model while reducing parameters by 4.7M and 13.4M.The reported improvements are not significant, but the parameter reductions support compact unified captioning.
4.2. Video-guided Machine Translation
Video-guided machine translation translates source descriptions into target languages using corresponding video as spatiotemporal context. The model combines source and video encoders with a target decoder using source-word and temporal-video attention, and experiments show that dynamically attended video improves translation and noun/verb recovery.
- Task and Model: VMT translates a source-language sentence into a target language using the corresponding video as additional spatiotemporal context.
- Task and Model: The multimodal sequence-to-sequence model encodes source text and video, concatenates their embeddings, and feeds them into a target-language decoder.Video segments are converted into features with a 3D ConvNet and video encoder.
- Task and Model: Two attention mechanisms dynamically select important source words and relevant spatiotemporal video segments during decoding.The decoder uses language dynamics as a query for highlighting video features.
- Experimental Setup: The baseline suite compares text-only NMT, average video features, and LSTM video features without temporal attention.
- Results and Analysis: Temporal attention substantially outperforms passive video-feature integration, while average and non-attended LSTM video features provide only marginal improvements.
- Results and Analysis: VMT consistently outperforms NMT on BLEU-4 and noun/verb recovery across masking rates, with recovery gaps increasing as masking becomes heavier.The result indicates that video context helps recover subjects, objects, actions, and their relations when parallel text is scarce.
5. Discussion and Future Work
VATEX supports additional video-and-language directions beyond captioning and video-guided translation. The paper highlights retrieval, localization, zero-/few-shot learning, and multilingual analyses of attention across cultural and linguistic backgrounds.
- VATEX descriptions could support multilingual video retrieval and localization within untrimmed long videos.
- Its 600 fine-grained action labels enable held-out-class evaluations of captioning-model generalizability for zero-/few-shot learning.
- Multilingual captions of the same videos could support analyses of common and differing attention patterns across cultural and linguistic groups.
A. Implementation Details
The implementation uses pretrained or recurrent video encoders, bounded caption lengths, shared training settings across languages, and standard maximum-likelihood optimization with Adam.
- Videos are sampled at 25fps and represented with pretrained I3D features without fine-tuning; captions are truncated to 30 words for multilingual captioning.Chinese captions use segmented words rather than raw characters.
- Captioning models use a 512-unit bi-LSTM video encoder, a 1024-unit LSTM decoder, 512-dimensional word embeddings, dropout, and Adam optimization.
- VMT truncates captions to 40 words and uses a 2-layer bi-LSTM source encoder, 2-layer LSTM decoder, 512-dimensional embeddings, and a 512-unit bi-LSTM video encoder.
B. Data Collection Interfaces
The data collection process used separate interfaces for English captions, direct Chinese descriptions, and Chinese translations parallel to English captions.
- English caption collection used a dedicated AMT interface.
- Chinese data collection used separate interfaces for direct video descriptions and translations parallel to English captions.
- Chinese captions were segmented with the open-source Jieba tool.
C. More VATEX Samples
Figure 11 provides additional examples from the VATEX dataset, illustrating its caption structure across languages.
- Figure 11 presents more samples from the VATEX dataset.
- The additional examples extend the dataset illustration beyond the main paper’s example.
- These samples serve as supplementary qualitative evidence of VATEX data content.
D. Qualitative Results
The qualitative results describe VATEX caption collection and compare multilingual captioning with video-guided translation. Multilingual models produce more coherent and informative captions, while video context supports more precise translation and noun or verb recovery.
- Multilingual Video Captioning: Multilingual captioning compares monolingual Base, shared-encoder, and shared encoder-decoder models across English and Chinese outputs.
- Video-guided Machine Translation: Video-guided machine translation produces more precise translations than base NMT in examples where NMT misinterprets words or phrases, including masked nouns and verbs.
- Caption collection: English captions are collected through AMT with visible instructions and accepted and rejected examples to improve annotation quality.
- Caption collection: Chinese captions are collected both by direct video description and by post-editing translated reference sentences.
- VATEX samples: Each video has 10 English and 10 Chinese descriptions, with the last five Chinese-English descriptions paired as translations.
- Multilingual Video Captioning: Qualitative examples report that multilingual models generate more coherent and informative captions than the monolingual model.