PaperlayerView exact revision on arXiv

Source-linked AI summary

Just Ask: Learning to Answer Questions from Millions of Narrated Videos

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid

arXiv:2012.00451v3cs.CVcs.CLcs.LG

TL;DR

VideoQA datasets are costly to annotate and often constrain answers to limited vocabularies. The paper automatically creates HowToVQA69M from narrated videos, trains an open-vocabulary contrastive model, and reports strong zero-shot and benchmark results.

  • Problem

    Manual VideoQA annotation is costly and unscalable, while existing methods and datasets often rely on limited answer vocabularies and small training sets.

  • Method

    The paper uses text-trained transformers and narrated-video transcripts to generate HowToVQA69M, then aligns video-question and answer embeddings with contrastive learning.

  • Results

    The model achieves excellent zero-shot VideoQA results, especially for rare answers, and outperforms the state of the art on four existing benchmarks.

  • Takeaways & Limitations

    HowToVQA69M supports open-ended VideoQA training without manually annotated visual data and enables evaluation through the new zero-shot task and iVQA benchmark.

  • Takeaways & Limitations

    Manual evaluation found that only 30 of 100 sampled triplets were correctly generated and matched well to the video content.

Abstract

from arXiv · show

Recent methods for visual question answering rely on large-scale annotated datasets. Manual annotation of questions and answers for videos, however, is tedious, expensive and prevents scalability. In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision. We leverage a question generation transformer trained on text data and use it to generate question-answer pairs from transcribed video narrations. Given narrated videos, we then automatically generate the HowToVQA69M dataset with 69M video-question-answer triplets. To handle the open vocabulary of diverse answers in this dataset, we propose a training procedure based on a contrastive loss between a video-question multi-modal transformer and an answer transformer. We introduce the zero-shot VideoQA task and show excellent results, in particular for rare answers. Furthermore, we demonstrate our method to significantly outperform the state of the art on MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA. Finally, for a detailed evaluation we introduce iVQA, a new VideoQA dataset with reduced language biases and high-quality redundant manual annotations. Our code, datasets and trained models are available at https://antoyang.github.io/just-ask.html.

1. Introduction

The paper addresses VideoQA’s annotation and vocabulary-scaling problems by automatically generating HowToVQA69M from narrated videos, then training and evaluating open-ended models.

  • Manual VideoQA annotation is costly and unscalable, leaving existing datasets relatively small despite models’ need for substantial training data.
  • HowToVQA69M contains 69 million open-ended video-question-answer triplets generated from narrated videos using text-trained transformers and cross-modal supervision.
  • A contrastive objective aligns a video-question multi-modal transformer with an answer transformer, avoiding a fixed discrete answer-class vocabulary.
  • The model produces excellent zero-shot results, especially for rare answers, and outperforms the state of the art on four VideoQA benchmarks after finetuning.The benchmarks are MSRVTT-QA, MSVD-QA, ActivityNet-QA, and How2QA.
  • iVQA excludes questions answerable without watching the video and provides five independently collected answers per question to address language bias and answer multiplicity.

2. Related Work

Prior VideoQA work relies on fixed answer vocabularies, manually annotated data, or task-agnostic narrated-video pretraining. This paper instead uses narrated videos and neural question generation to scale open-ended VideoQA pretraining.

  • Earlier VQA and VideoQA methods commonly classify representations into predefined answer vocabularies, limiting application beyond specific datasets.
  • Zero-shot VQA has been defined for previously unseen questions, whereas this paper proposes a related but more challenging task without manual visual-data supervision.
  • Manual video-question-answer annotation is costly and difficult to scale, motivating the use of large-scale narrated videos without manual supervision.
  • The proposed generation pipeline uses punctuation inference, answer extraction, question generation, and narration timestamps to create aligned video-question-answer triplets.
  • Unlike task-agnostic narrated-video pretraining, the approach pretrains specifically for VideoQA using automatically generated question-answer pairs.

3. Large-scale generation of VideoQA data

The paper automatically generates large-scale VideoQA triplets from narrated videos using text-trained language models and temporal alignment. The resulting HowToVQA69M dataset contains 69,270,581 triplets but exhibits substantial annotation noise.

  • The pipeline punctuates ASR transcripts, aligns video clips to inferred sentences, extracts answers, and generates questions conditioned on each answer and sentence.It uses a punctuation model before extracting answers and generating questions with transformers.
  • HowToVQA69M contains 69,270,581 video clip-question-answer triplets generated from 1.2M HowTo100M videos.Each original video produces 43 clips on average, with 1.2 question-answer pairs per clip.
  • The dataset is two orders of magnitude larger than existing VideoQA datasets and contains over 16M unique answers.This scale supports diverse open-vocabulary VideoQA data.
  • 30% of 100 sampled triplets matched the video content correctly, while 31% were incorrect and 39% were correct but visually unrelated.The manual evaluation used five annotators per triplet and majority voting.
  • Punctuation improves generation quality, and the proposed method outperforms a rule-based question-answer generation approach in the manual comparison.The comparison includes variants with and without transcript sentence splitting.

4. VideoQA model and training procedure

The VideoQA model embeds video-question pairs and answers in a shared space, enabling open-vocabulary matching. Contrastive pretraining uses generated triplets and downstream finetuning adapts the negative-answer set to each task.

  • VQA-T uses a transformer video-question branch and a text answer encoder to produce comparable embeddings for multimodal inputs and free-form answers.The video-question branch models temporal interactions before projecting its CLS representation.
  • The contrastive objective brings each video-question embedding close to its correct answer and separates it from incorrect answers.Similarity is measured by the dot product f(v,q)^Tg(a).
  • Negative answers are drawn from the training batch, with duplicate negative answers counted only once.Removing duplicates helps mitigate differences between pretraining and downstream answer distributions.
  • During downstream finetuning, negatives come from all incorrect answers in the task vocabulary, making the objective equivalent to standard cross-entropy.For multiple-choice VideoQA, negatives are the incorrect options for each sample.
  • Masked language modeling is applied to question tokens during pretraining and finetuning as a regularizer for DistilBERT weights.The authors report a positive regularization effect during finetuning.

5. iVQA: new dataset for VideoQA evaluation

iVQA is introduced to evaluate VideoQA with manually collected answers and reduced language bias. It provides five answers per question and excludes questions answerable without watching the video.

  • iVQA is designed to evaluate visual information by excluding questions that can be answered without watching the video.The dataset focuses on objects and scenes in manually sampled video clips.
  • Each iVQA clip has one manually collected question and five independently collected answers.The authors manually verify annotation correctness.
  • iVQA contains 10,000 video clips split into 60% training, 20% validation, and 20% test subsets.Questions and answers contain 7.6 and 1.1 words on average, respectively.
  • The evaluation metric assigns 100% accuracy to answers confirmed by at least two annotators, 50% to answers confirmed by one, and 0% otherwise.This definition is specific to questions with multiple ground-truth answers.

6. Experiments

The experiments evaluate HowToVQA69M pretraining through zero-shot transfer, downstream finetuning, state-of-the-art comparisons, rare-answer performance, generation comparisons, ablations, and dataset scale.

  • Experimental setup: The evaluation covers zero-shot VideoQA, four downstream datasets, rare answers, generation quality, pretraining losses, and training-set scale.The datasets include MSRVTT-QA, MSVD-QA, ActivityNet-QA, iVQA, and multiple-choice How2QA.
  • Zero-shot VideoQA: VQA-T uses visual cues more effectively than QA-T in zero-shot evaluation when both are trained on HowToVQA69M.Training on HowToVQA69M also significantly outperforms training on HowTo100M and a random baseline.
  • Finetuning and benchmarks: HowToVQA69M pretraining consistently and significantly improves finetuned results over HowTo100M pretraining and no pretraining across the evaluated datasets.The largest improvement is reported for iVQA, which shares the HowToVQA69M domain.
  • Finetuning and benchmarks: VQA-T pretrained on HowToVQA69M outperforms previous methods on all tested datasets, including the HowTo100M-pretrained CoMVT approach.The comparisons cover MSRVTT-QA, MSVD-QA, ActivityNet-QA, and How2QA.
  • Rare answers: HowToVQA69M pretraining improves both common- and rare-answer performance, with zero-shot results on iVQA Q3 and Q4 exceeding some non-pretrained finetuned models.The authors conclude that additional large-scale, diverse VideoQA-specific data improves generalization.
  • Ablations: Removing duplicate negative answers and adding MLM loss improve pretraining outcomes, while larger HowToVQA69M subsets suggest further gains in zero-shot and finetuning settings.The loss ablations and scale study support the proposed contrastive pretraining strategy and the importance of dataset scale.

7. Conclusion

The paper proposes scalable VideoQA training without manually annotated visual data by automatically generating HowToVQA69M from narrated videos, and reports strong zero-shot and finetuned benchmark results.

  • Contribution: The approach automatically generates HowToVQA69M from narrated videos and speech transcripts, avoiding manually annotated visual data.The dataset is described as exceeding existing datasets in size and diversity.
  • Results: The model achieves zero-shot VideoQA without manually annotated images or videos during training.The conclusion identifies this as the first demonstration of that setting.
  • Results: Finetuning the HowToVQA69M-pretrained model outperforms the state of the art on MSRVTT-QA, MSVD-QA, ActivityNet-QA, and How2QA.The paper also validates the approach on the manually collected iVQA benchmark.

A. Analysis of HowToVQA69M dataset

HowToVQA69M exhibits broad vocabulary, answer, video-duration, and question-type diversity, while its automatically generated annotations remain noisy but generally supported by annotator agreement.

  • Dataset statistics: HowToVQA69M contains diverse questions and answers, longer answers than several downstream datasets, and clip durations peaking around seven seconds with a long tail.These statistics demonstrate diversity across videos and answers.
  • Dataset statistics: The dataset vocabulary includes speech-related words, and its examples illustrate both diversity and noise in automatically generated annotations.Figure 10 distinguishes relevant examples from generation failures and visually unrelated question-answer pairs.
  • Generation quality: Compared with Heilman et al., the proposed generation method produces higher-quality and more diverse question-answer pairs on uncurated narrated-video speech.The comparison is reported qualitatively and through downstream-performance evaluation.
  • Manual evaluation: Among 300 generated triplets, 94 reached agreement among all five annotators, 198 among at least four, and 299 among at least three.The authors use this agreement to support the reliability of their manual evaluation.
  • Question types: Action questions receive the best annotations, whereas counting questions most often produce visually unrelated answers and place questions most often produce generation errors.The analysis covers six common question categories and uses majority-vote evaluation.

B. VideoQA architecture

The VideoQA architecture combines a video-question multimodal transformer with a DistilBERT-based answer transformer to represent videos, questions, and free-form answers.

  • Architecture overview: The architecture has two main modules: a video-question multimodal transformer and an answer transformer based on DistilBERT.The fused video-question representation and answer representation are used as the model’s two learned branches.
  • Video-question module: The video-question branch concatenates DistilBERT question-token embeddings with fixed S3D video features sampled one per second.The multimodal transformer outputs contextualized question-token and temporal-video representations.
  • Answer module: The answer branch encodes answer tokens with DistilBERT and uses the contextualized [CLS] embedding to form an answer representation.A learned projection maps the [CLS] embedding into the answer embedding space.
  • iVQA design: The iVQA collection emphasizes visually grounded questions that cannot be easily guessed without watching the video and short answers that support annotator consensus.Five independently collected answers are used for each iVQA question.

C.2. Statistical Analysis

iVQA uses manually collected questions and answers designed to reduce language bias, with diverse clips and redundant annotations supporting consensus-based evaluation.

  • The most frequent answer, “spoon,” accounts for 2% of all iVQA answers.
  • Clip duration and relative start-time distributions are almost uniform because clips were randomly sampled from source videos.
  • Answers are mostly one or two words, and iVQA includes five independently collected answers per question.
  • 27.0% of questions reach perfect consensus among five annotators, while 77.3% reach consensus among at least three.
  • The annotation interfaces separately collect questions and answers through Amazon Mechanical Turk.

D. Additional experimental details

The dataset-generation pipeline truncates and pads text inputs, uses beam search for question decoding, and applies transformer-based objectives for video-text learning and masked-token prediction.

  • Question-generation and answer-extraction inputs are truncated and padded to 32 tokens.
  • Question decoding uses beam search that retains the four most probable states at each search-tree level.
  • The generation pipeline uses original HowTo100M captions, including stop words, and removes repetitions from adjacent clips.
  • Masked question outputs are classified over a 30,522-token vocabulary with cross-entropy loss.
  • Cross-modal matching samples one video negative and one text negative per positive pair and trains with binary cross-entropy.

E.1. Comparison to cross-dataset transfer

HowToVQA69M pretraining improves transfer across downstream VideoQA datasets, particularly for rare answers, while visual information appears especially important in iVQA.

  • E.1. Comparison to cross-dataset transfer: HowToVQA69M pretraining significantly outperforms transfer from MSRVTT-QA and ActivityNet-QA on all four downstream datasets in zero-shot and finetuning settings.
  • E.2. Results for rare answers and per question type: The pretrained model achieves the best finetuned results for rare-answer quartiles Q3 and Q4 on MSRVTT-QA, MSVD-QA, and ActivityNet-QA.
  • E.2. Results for rare answers and per question type: Pretraining produces consistent improvements across most question categories on MSRVTT-QA, MSVD-QA, and ActivityNet-QA.
  • E.3. Comparison between QA-T and VQA-T on different datasets.: On iVQA, VQA-T improves more over QA-T than on other datasets, suggesting that visual information is more important there.
Loading 2012.00451v3…