Source-linked AI summary
A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering
Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, Christopher Pal
TL;DR
Video understanding remains challenging, and sparse descriptive-video references make video-to-language evaluation difficult. MovieFIB turns descriptive annotations into fill-in-the-blank questions and compares five models with human performance. Combined 2D and 3D visual input performs best, but all models remain below human-level performance.
Problem
Video-to-language models must handle open-domain visual understanding, while sparse descriptions make common caption metrics unreliable against human judgments.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
Classification accuracy on MovieFIB corresponds well with human assessment, supporting its use to compare model responses.
Source: §1 · p. 2, §5 · p. 5Workers judged some ground-truth answers imperfect or nonsensical, often apparently because of language comprehension or style.
Source: §5.7 · p. 8
Abstract
from arXiv · showhide
While deep convolutional neural networks frequently approach or exceed human-level performance at benchmark tasks involving static images, extending this success to moving images is not straightforward. Having models which can learn to understand video is of interest for many applications, including content recommendation, prediction, summarization, event/object detection and understanding human visual perception, but many domains lack sufficient data to explore and perfect video models. In order to address the need for a simple, quantitative benchmark for developing and understanding video, we present MovieFIB, a fill-in-the-blank question-answering dataset with over 300,000 examples, based on descriptive video annotations for the visually impaired. In addition to presenting statistics and a description of the dataset, we perform a detailed analysis of 5 different models' predictions, and compare these with human performance. We investigate the relative importance of language, static (2D) visual features, and moving (3D) visual features; the effects of increasing dataset size, the number of frames sampled; and of vocabulary size. We illustrate that: this task is not solvable by a language model alone; our model combining 2D and 3D visual information indeed provides the best result; all models perform significantly worse than human-level. We provide human evaluations for responses given by different models and find that accuracy on the MovieFIB evaluation corresponds well with human judgement. We suggest avenues for improving video models, and hope that the proposed dataset can be useful for measuring and encouraging progress in this very interesting field.
1. Introduction
Video understanding remains difficult, and video-to-language evaluation is complicated by sparse descriptions and metrics that may not align with human judgments. MovieFIB reframes description understanding as scalable fill-in-the-blank question answering and evaluates models against human performance.
1. Introduction
- Video-to-language models must tackle open problems in recognizing entities and describing their attributes and relationships in rich, open-domain activities.The broader goal is complete understanding of visual scenes.Source: §1 · p. 1
- With only one description per video segment, BLEU, CIDEr, and ROUGE may not align with human assessment, complicating evaluation and model comparison.Multiple descriptions can be equally applicable to a scene.Source: §1 · pp. 1–2
- MovieFIB turns video descriptions into automatically generated fill-in-the-blank questions, yielding 300,000 video-question pairs about entities, actions, and attributes.Predicting a missing word requires models to use visual information about objects, people, appearance, activities, interactions, or scene context.Source: §1 · p. 2
- The combined 2D-and-3D visual model performs best, while every evaluated model remains significantly below human performance.The language-only model does not solve the task, and classification accuracy corresponds well with human judgment.Source: §1 · p. 2
2. Related work
Prior work developed video-captioning datasets from descriptive video and explored both image and video question answering. MovieFIB builds on these lines of work while using an open-vocabulary fill-in-the-blank format.
2.1. Video Captioning
- Video captioning moved from constrained, hand-crafted-feature systems toward open-domain neural encoder-decoder models [42] [10] [43] [4], supported by large descriptive-video datasets [39] [28].Caption evaluation remains difficult because many descriptions can fit a clip and common metrics correlate poorly with human judgments.Source: §2.1 · p. 2
2.2. Image and Video QA
- Image-QA datasets include VQA, COCO-QA [27], and DAQUAR [22], which use images and human- or algorithmically generated questions and answers.VQA combines MSCOCO images [20] with abstract scenes, while DAQUAR uses NYU-Depth V2 images.Source: §2.2 · p. 2
- Prior video-QA work reformulated caption datasets as multiple-choice fill-in-the-blank questions [45], whereas MovieFIB uses an open-vocabulary format.The earlier approach controlled question difficulty through the similarity of candidate answers; MovieQA instead targets story comprehension using video and associated text resources [38].Source: §2.2 · p. 3
3. MovieFIB: a fill-in-the-blank question-answering dataset
MovieFIB generates fill-in-the-blank questions from LSMDC2016 annotations and retains eligible content words as answers. Its examples follow the LSMDC split, and the answer-frequency distribution has a pronounced long tail.
3.1. Creating the dataset
- From LSMDC2016 annotations, MovieFIB blanks nouns, verbs, adjectives, and adverbs selected by part of speech, a curated stoplist, and a 50-occurrence training threshold.A pretrained maximum-entropy parser tags the annotation words.Source: §3.1 · p. 3
3.2. Dataset statistics and analysis
- The procedure yields 348,998 examples, averaging three per original LSMDC annotation; the 296,960/21,689/30,349 training/validation/test split uses validation and test movies disjoint from training.Most training-set blanked words occur 100–200 times; the most frequent word, her, occurs up to 12,541 times.Source: §3.2 · p. 3
4. Neural framework for video fill-in-the-blank question-answering
The framework encodes the question and video separately, then uses a classifier to predict the missing word. Its question encoder reads context on both sides of the blank, while its video encoder processes visual feature sequences.
4. Neural framework for video fill-in-the-blank question-answering
- A classifier predicts the missing answer from fixed-length question and video representations produced by separate encoders.The classifier outputs a probability distribution over possible answers.Source: §4 · p. 3
4.1. Question Encoder
- The question encoder runs recurrent networks forward and backward around the blank, then concatenates their neighboring hidden states for classification.The approach uses a Batch-Normalized LSTM variant to model sequences.Source: §4.1 · pp. 4–5
4.2. Video Encoder
- The video encoder applies 2D or 3D convolutional networks to frames or frame sequences, then a Batch-Normalized LSTM to obtain a fixed-length representation.The video encoder is described as extracting a fixed-length representation from a sequence of 2D frames.Source: §4.2 · p. 5
5. Experiments and Discussion
The experiments compare baseline models, human performance, and the effects of text and video preprocessing. Text is tokenized and lowercased; visual features are extracted from sampled video frames or temporal segments.
5. Experiments and Discussion
- The experiments compare baseline models, human performance, and the relationship between model accuracy and human assessment.They examine dataset statistics and baseline performance, and compare model results with human performance.Source: §5 · p. 5
5.1.1 Text Preprocessing
- 26,818 unique words remain after questions and answers are tokenized with NLTK’s wordpunct tokenizer and lowercased.The supplied passage identifies the tokenizer and vocabulary size.Source: §5.1.1 · p. 5
5.1.2 Video Preprocessing
- GoogLeNet pool5/7x7 features and C3D fc7 features are extracted without fine-tuning; 25 frames or segments are sampled by default, randomly in training and evenly at inference.C3D features use chunks of 16 consecutive frames.Source: §5.1.2 · p. 5
5.2. Language, static-visual, and moving-visual information
The experiments compare language-only, static-visual, and moving-visual inputs, alongside changes to initialization, frame sampling, and training-set size. Visual features improve accuracy, and the combined visual model performs better than either feature type alone.
5.2. Language, static-visual, and moving-visual information
- Comparable test performance is achieved with just two sampled frames for the finetuned GoogleNet-2D model.Source: Figure 6 · p. 6
- Qualitative examples show the combined GoogleNet-2D and C3D model correcting text-only errors, particularly for action recognition.Source: Figure 7 · p. 6
- Adding GoogleNet-2D or C3D visual input improves accuracy over the text-only baseline, and combining the visual features improves performance further.The authors describe the visual features as complementary.Source: §5.2 · p. 6
- Finetuned initialization from the text-only model yields better results than random initialization and tends to reduce overfitting.Source: §5.2 · p. 6
- Figure 8 reports fill-in-the-blank accuracy for Text-only and GoogleNet-2D + C3D (finetuned) models across 10%, 20%, 50%, and 100% training-data fractions, with video showing a larger test-versus-validation gain.Source: Figure 8 · p. 6
5.3. Human Performances on the test set
Human responses were collected on a representative subset of test examples under text-only and text-plus-video conditions. Video input improves human performance, but the best automated model still trails video-enabled humans.
5.3. Human Performances on the test set
- 569 representative test examples were answered by humans using a predefined searchable vocabulary, with three workers per question and majority responses retained.If all three answers differed, one candidate response was selected randomly.Source: §5.3 · p. 6
- Adding the video clip drastically improves human performance over the text-only condition.The two conditions provided workers with the query sentence alone or with both the sentence and video.Source: §5.3 · p. 7
- The best automated model remains significantly below video-enabled humans, although the text-only model outperforms humans in the text-only condition.The authors suggest that the model exploits statistical regularities in the professional descriptive-video writing style that workers may miss.Source: §5.3 · p. 7
5.4. Effect of Text and Video Preprocessing
Reducing input and output vocabulary sizes improves performance, while validation performance with the GoogLeNet-2D baseline nearly saturates after two sampled frames. Related work using MovieFIB reports higher accuracy for a ResNet-based model and improved performance from question-initialized video encoding.
5.4. Effect of Text and Video Preprocessing
- Restricting input words to those occurring more than 3 times and output words to those occurring more than 50 times improves performance.The resulting vocabularies contain 18,663 input words and 3,994 output words; all possible blanked words remain in the output vocabulary.Source: §5.4 · p. 7
- Validation performance for the GoogLeNet-2D baseline nearly reaches its best level with just 2 sampled video frames.The authors report that validation performance saturates quickly as the number of input frames increases.Source: §5.4 · p. 7
5.5. Related works using MovieFIB
- The ResNet-2D-biLSTM-attn model in [44] reports 38.0% accuracy as a single model and 40.7% as an ensemble on MovieFIB.The authors describe these as the highest reported accuracies on the dataset at that time.Source: §5.5 · p. 7
- The VGG-2D-MergingLSTMs model in [23] finds that initializing video encoding with question encoding improves performance.The work uses a separately trained Word2Vec embedding space for questions.Source: §5.5 · p. 7
5.6. Effects of increasing dataset size
As training data grows, video-based models benefit more than text-only models, suggesting that additional examples can help them generalize from visual input. At 10% of the training videos, however, their validation accuracies are nearly identical.
5.6. Effects of increasing dataset size
- Figure 11 shows that per-word true positive rate is highly correlated with answer prevalence.The plotted comparison is for GoogleNet-2D+C3D; the authors report similar correlations for all models.Source: Figure 11 · p. 8, §5.6 · p. 8
- At 10% of training data (9,511 videos), validation accuracy is 20.7% for Text-Only versus 21.0% for GoogleNet-2D+C3D.The authors suggest this amount of video data is insufficient for useful visual information to generalize to unseen examples.Source: §5.6 · p. 8
- As training data increases, the video-based model's performance rises more rapidly than the text-only model's.The authors conclude that video models can gain generalization from visual input given enough training examples.Source: §5.6 · p. 8
5.7. Human evaluation of results
Human evaluation indicates that MovieFIB accuracy tracks human judgments of model responses. Workers also sometimes judged ground-truth answers as imperfect or nonsensical, with language comprehension and style implicated in many of the latter cases.
5.7. Human evaluation of results
- MovieFIB accuracy tracks human evaluation of model responses well on the test set.Workers ranked responses while viewing the clip and question, including the ground truth.Source: §5.7 · p. 8
- Workers rated ground-truth answers “Perfect” in about 80% of cases and as sensible but imperfect in another 11%.They judged 3% of ground-truth answers nonsensical; the authors attribute many such cases to language comprehension or style.Source: §5.7 · p. 8
6. Conclusion
MovieFIB provides a large fill-in-the-blank video question-answering dataset built from descriptive video annotations. Across evaluated models, visual input matters and combining 2D and 3D features performs best, but models remain below human performance.
6. Conclusion
- MovieFIB contains over 300,000 fill-in-the-blank question-answer and video pairs based on descriptive video annotations for visually impaired viewers.Source: §6 · p. 8
- Among the five evaluated models, combining 2D and 3D visual information gives the best result, while all models perform significantly below human-level.The authors identify visual information as important to performance on this task.Source: §6 · p. 8
- Models leveraging visual input benefit more than text-only models as training samples increase.The authors suggest performance could improve by increasing the amount of training data.Source: §6 · p. 8
- Accuracy corresponds well with human assessment for the fill-in-the-blank task.Source: §6 · p. 8
- The authors hope MovieFIB will support development and evaluation of models that better understand video content and encourage further research.Source: §6 · p. 8