Source-linked AI summary
DeepStory: Video Story QA by Deep Embedded Memory Networks
Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, Byoung-Tak Zhang
TL;DR
Video story QA requires models to integrate vision and language, but video-domain methods and datasets have lagged behind text and image QA. The paper introduces DEMN, which reconstructs stories from joint scene-dialogue streams and uses attention over memory, and reports state-of-the-art performance on PororoQA and MovieQA.
Problem
Video story QA remains comparatively underdeveloped because answering questions about videos requires integrating scene and dialogue information.
Method
DEMN reconstructs video stories from joint scene-dialogue streams using latent embeddings and uses attention to recall relevant question-story-answer information.
Results
DEMN achieves state-of-the-art performance on both PororoQA and MovieQA.
Takeaways & Limitations
PororoQA provides a coherent cartoon-video test bed with high-quality scene descriptions, and DEMN demonstrates the potential of this approach for video story QA.
Abstract
from arXiv · showhide
Question-answering (QA) on video contents is a significant challenge for achieving human-level intelligence as it involves both vision and language in real-world settings. Here we demonstrate the possibility of an AI agent performing video story QA by learning from a large amount of cartoon videos. We develop a video-story learning model, i.e. Deep Embedded Memory Networks (DEMN), to reconstruct stories from a joint scene-dialogue video stream using a latent embedding space of observed data. The video stories are stored in a long-term memory component. For a given question, an LSTM-based attention model uses the long-term memory to recall the best question-story-answer triplet by focusing on specific words containing key information. We trained the DEMN on a novel QA dataset of children's cartoon video series, Pororo. The dataset contains 16,066 scene-dialogue pairs of 20.5-hour videos, 27,328 fine-grained sentences for scene description, and 8,913 story-related QA pairs. Our experimental results show that the DEMN outperforms other QA models. This is mainly due to 1) the reconstruction of video stories in a scene-dialogue combined form that utilize the latent embedding and 2) attention. DEMN also achieved state-of-the-art results on the MovieQA benchmark.
1 Introduction
Video story QA remains less developed than text and image QA because understanding stories requires integrating scene and dialogue information. The paper introduces a large Pororo dataset and DEMN, which achieves state-of-the-art performance on both evaluated datasets.
- Video QA results have lagged behind text and image QA, with few methods and datasets addressing video story understanding.
- Late fusion of separate scene and dialogue models can degrade performance because video stories require both modalities together.
- PororoQA provides cartoon videos with simple, coherent story structures and high-quality scene descriptions for video story QA.
- DEMN achieves state-of-the-art performance on both datasets by reconstructing video stories from combined scene-dialogue information and using attention.
2 Video Story QA
The paper formulates video story QA as multiple-choice answering over videos composed of paired scenes and dialogues, supported by external scene descriptions. It develops PororoQA to provide coherent cartoon stories, annotated descriptions, and story-related questions for this task.
- 2.1 Task Statement: Each video is represented as paired scene sequences and dialogue sentences, with external scene descriptions available to support QA.
- 2.1 Task Statement: The QA model selects the correct answer sentence from five candidates using a score conditioned on the video and question.
- 2.2 Related Datasets for Video Story QA: MovieQA contains 14K QA pairs about 408 movie stories, but many questions rely on linguistic sources and its complex story structures make optimization difficult.
- 2.3 PororoQA Dataset: Pororo uses cartoon videos because they have simple story structures, repeated events, and relatively few characters and backgrounds.
- 2.3 PororoQA Dataset: 16,066 scene-dialogue pairs were extracted by segmenting videos according to the start and end times of character speech.
- 2.3 PororoQA Dataset: Annotators collected descriptive sentences by viewing videos and scene-dialogue pairs, while separate workers created questions with one correct and four incorrect answers.
- 2.3 PororoQA Dataset: PororoQA is presented as having the highest number of video QA examples among compared public video datasets and a coherent storyline throughout the dataset.
3 Deep Embedded Memory Networks
DEMN reconstructs joint scene-dialogue streams into video stories, retrieves question-relevant stories, and selects answers using word-level attention-based matching.
- Model overview: DEMN extends memory-network ideas to video by jointly modeling scene and dialogue streams rather than treating them separately.The model uses a video-story representation built from paired scenes and dialogues.
- Video story understanding: The video story understanding module combines embedded scene and dialogue representations, retrieves matching descriptions, and concatenates each description with its dialogue.Scene features and dialogue features are embedded, summed, and matched against description representations to reconstruct stories.
- QA modules: The story selection module reads reconstructed stories from long-term memory and retrieves the story with the highest question-story match score.The selected story is concatenated with the question before answer selection.
- QA modules: The answer selection module scores the question-story representation against candidate answers and chooses the highest-scoring sentence.Both story and answer selection are trained with supervised judgments and hinge-rank losses.
- Attention scoring: Word-level attention-based scoring uses bidirectional LSTMs and question-conditioned softmax weights to compare long stories, questions, and answer sentences.The scoring functions encode token sequences, attend to tokens on one side using the other sequence, and average updated token vectors.
4 Experimental Results
The experiments evaluate DEMN against human and computational baselines on PororoQA and report state-of-the-art performance on MovieQA. Results indicate that combining scene and dialogue stories, with attention, improves video story QA.
- Experimental Setup: PororoQA evaluation measures QA accuracy and MRR, with MRR assessing the rank of the correct story among candidate video stories.The dataset is split into 103 training, 34 validation, and 34 test episodes.
- Human Baselines: Six human evaluators answered every PororoQA test question, providing a human-performance baseline.The reported human accuracy was 96.9%.
- Human Baselines: Humans improved as more information was provided, and descriptions supplied more useful information than dialogues for answering questions.Videos were important for most questions, while visual-only QA models failed to solve the task reliably.
- PororoQA Model Comparisons: DEMN performed better than memory networks when linguistic and visual stories were used together, whereas performance was similar with a single story modality.The combined reconstruction improved MRR scores by 0.19 and 0.20, supporting higher QA accuracies.
- PororoQA Model Comparisons: Using attention, DEMN achieved the best performance among the compared QA models on PororoQA.The experiments included DEMN with and without attention and tested multiple question, dialogue, scene, and description input combinations.
- MovieQA Benchmark: On MovieQA, DEMN achieved 44.7% validation accuracy and 30.0% test accuracy in video QA mode, reported as state-of-the-art at submission.The benchmark contains 140 movies and 6,462 multiple-choice questions; the authors attribute relatively low performance to more varied story structures.
5 Concluding Remarks
The paper introduces DEMN and PororoQA for video story QA, demonstrating state-of-the-art performance on PororoQA and MovieQA. Future work targets optimization for more complex story structures.
- Contributions: The paper proposes DEMN together with the PororoQA dataset for video story question answering.PororoQA contains simple, coherent story-structured videos and high-quality scene descriptions.
- Contributions: DEMN achieves state-of-the-art performance on both PororoQA and MovieQA.The authors identify curriculum learning as a possible direction for handling more complex story structures.