Source-linked AI summary
MovieQA: Understanding Stories in Movies through Question-Answering
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, Sanja Fidler
TL;DR
MovieQA addresses the challenge of evaluating story comprehension in long, semantically diverse movie data. It constructs a multimodal multiple-choice benchmark from movies and their associated sources, then analyzes QA methods and finds that open-ended story semantics remain difficult for simple approaches.
Problem
Static-image QA cannot capture much of the high-level action, intent, and motivation that emerge across temporal movie stories.
Method
The paper constructs a 14,944-question, five-choice benchmark over 408 movies using video clips, subtitles, scripts, plots, and DVS, then evaluates extended QA techniques.
Results
Question-answering with MovieQA’s open-ended semantics is hard: answer-length heuristics reach 25.3% versus 20% random, while question-answer similarity is insufficient.
Takeaways & Limitations
MovieQA provides a public benchmark for studying automatic story comprehension across long temporal data and multiple information sources.
Abstract
from arXiv · showhide
We introduce the MovieQA dataset which aims to evaluate automatic story comprehension from both video and text. The dataset consists of 14,944 questions about 408 movies with high semantic diversity. The questions range from simpler "Who" did "What" to "Whom", to "Why" and "How" certain events occurred. Each question comes with a set of five possible answers; a correct one and four deceiving answers provided by human annotators. Our dataset is unique in that it contains multiple sources of information -- video clips, plots, subtitles, scripts, and DVS. We analyze our data through various statistics and methods. We further extend existing QA techniques to show that question-answering with such open-ended semantics is hard. We make this data set public along with an evaluation benchmark to encourage inspiring work in this challenging domain.
1. Introduction
MovieQA targets machine comprehension of complex movie stories, where high-level actions, intent, and motivation require temporal visual and textual understanding. It introduces a diverse, multimodal benchmark spanning video and several accompanying text sources.
- Static images limit questions about high-level actions and intent because these semantics are typically inferred from temporal visual observations.
- MovieQA uses movies as linked life snapshots to evaluate comprehension of characters, actions, and motivations in complex stories.
- 14,944 multiple-choice questions cover 408 movies, each with one correct answer and four deceiving options.
- Questions range from vision-solvable Who–What–Whom questions to Why and How questions requiring both visual information and dialogs.
- The dataset combines video clips, subtitles, scripts, plots, and DVS to support automatic story comprehension from video and text.
2. Related work
Prior work integrates language with images and short videos, but existing QA datasets largely target static scenes or short textual forms. MovieQA extends this landscape to longer movie stories with multiple visual and textual information sources.
- Vision-language research has advanced through large captioned image datasets and deep neural models, while video-language work remains comparatively limited.
- Prior video-language studies include captioning, retrieval, and character or action modeling, while TV-series work often focuses on recognizing and tracking characters.
- QA research has developed neural and structured methods, but the cited literature reports no QA methods addressing the temporal domain.
- Existing visual QA datasets mainly ask about objects, attributes, relations, and counts in static RGB-D images or photographs.
- MovieQA differs from text QA datasets through significantly longer text and multiple sources including plots, subtitles, scripts, and DVS.
3. MovieQA dataset
MovieQA is a movie-story benchmark built from diverse textual and visual sources, with human-authored multiple-choice questions aligned to plot evidence and, for part of the dataset, video clips. Its statistics emphasize long answers and substantial reasoning content.
- QA Collection method: The benchmark presents each movie quiz as multiple-choice questions with five answers, only one correct, and access to textual and visual story sources.
- Movie sources: 408 subtitled movies were paired with Wikipedia plot synopses, scripts when available, and DVS transcriptions for 60 movies.
- Information sources: Plots emphasize story events and character interactions, while video supplies actions and dialogs may explain why events occurred; together they support fuller story understanding.
- QA Collection method: Annotators generated questions and correct answers from plot paragraphs, marked minimal supporting sentences, and created four deceiving alternatives in a second collection stage.
- Video alignment: Video alignment linked plot-supported QAs to timestamped clips, which are provided as benchmark inputs.
- Dataset statistics: 14,944 QAs from 408 movies average about 9 words per question and 5 words per answer, with a video split covering 140 movies and clips of approximately 200 seconds.
- Dataset statistics: Why questions have the longest average correct answers, whereas Who questions typically have short person-name answers.
- Dataset statistics: Reasoning-based Why, How, and Abstract questions form a large part of the dataset, although their scene contexts often still require visual information.
4. Multi-choice Question-Answering
The paper formulates MovieQA as selecting the best answer from five choices by scoring the story, question, and candidate answer. It compares story-agnostic baselines with story-searching similarity models and adapted memory networks for natural-language answers.
- The task selects one correct answer from M = 5 candidates by maximizing a scoring function over the story, question, and answer.The story may be a plot, subtitle, or video-shot sequence, and answering schemes differ through their scoring function.
- 4.1. The Hasty Student: The Hasty Student ignores the story and answers using biases or similarities in the answers and question-answer pairs.Baselines use answer length, within-answer similarity or difference, and question-answer similarity.
- 4.2. The Searching Student: The Searching Student locates story subsets similar to both the question and each answer, using either sliding-window cosine similarity or a learned neural similarity function.The cosine model searches for H sentences or shots maximizing story-question and story-answer similarity.
- 4.2. The Searching Student: SSCB represents question-answer similarities across story positions and candidates as a tensor, then uses 1 × 1 convolutions and max pooling to learn answer scores.The model combines question and answer similarities, pools across the storyline, and predicts with softmax; it is trained with cross-entropy and Adam.
- 4.3. Memory Network for Complex QA: The adapted MemN2N ranks natural-language answers by combining question-aware story attention with answer embeddings and shared projected fixed word embeddings.The modification replaces learned embedding matrices with pretrained Word2Vec embeddings and a shared linear projection, addressing the original model’s fixed-vocabulary output limitation.
5. QA Evaluation
The evaluation defines separate text- and video-based QA tasks and tests simple similarity, neural, and memory-network approaches. Results show that MovieQA remains difficult: plot-based text methods are strongest, while video-only answering stays near random.
- Evaluation protocol: The benchmark evaluates text-based stories from plots, subtitles, scripts, and DVS, alongside video-based stories with or without subtitles.Accuracy is the number of correctly answered questions divided by the total.
- Hasty Student: 25.3% accuracy from always choosing the longest answer exceeded the 20% random baseline, while TF-IDF question-answer similarity performed below random.Generic answer and similarity biases provide limited but measurable signal without consulting the story.
- Hasty Student: 27.6% accuracy was achieved by humans answering 200 questions without story access, falling to 24.7% after removing questions revealing the movie.The authors use this reduction to characterize the genuine difficulty of the questions.
- Searching Student: Plot-based cosine similarity outperformed subtitle, script, and DVS sources because questions were collected from plots and annotators often reproduced plot wording.SkipThoughts performed worse than TF-IDF and Word2Vec, while representation differences varied across question types.
- SSCB: The neural SSCB model outperformed cosine similarity on most tasks, with fusion achieving the highest performance when plot synopses were used.Excluding plots, accuracy was capped at about 30% for most modalities.
- Memory Network: The modified MemN2N handled long stories better than its original version, which overfit to near-random validation performance of ∼20%.Its attention mechanism performed well on DVS, subtitles, and scripts, with scripts performing best because they contain descriptions, dialogue, and speaker information.
- Video baselines: 22.3% accuracy from video-plus-subtitles under SSCB was below the 27.7% achieved by fusion of all text features.Video-only question answering remained close to random, showing that visual information alone was insufficient.
6. Conclusion
The paper introduces MovieQA as a benchmark for automatic story comprehension from both video and text. Its multiple information sources and extended QA methods are intended to expose the difficulty of understanding complex movie stories.
- 6. Conclusion: MovieQA contains video clips, subtitles, scripts, plots, and DVS, and provides baselines and extended QA techniques for analyzing task difficulty.The benchmark is accompanied by a public evaluation server.