Source-linked AI summary
End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features
Chiori Hori, Huda Alamri, Jue Wang, Gordon Wichern, Takaaki Hori, Anoop Cherian, Tim K. Marks, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, Irfan Essa, Dhruv Batra, Devi Parikh
TL;DR
The paper addresses dialog systems’ limited ability to discuss dynamic scenes using multimodal input. It introduces a video-dialog dataset and an end-to-end model incorporating multimodal attention-based video features; experiments report improved response quality with video and audio features.
Problem
Current dialog systems cannot understand dynamic scenes using vision and non-speech audio, limiting conversation about surrounding objects and events.
Source: §1 · p. 2Method
The paper introduces a video-dialog dataset and an end-to-end dialog model incorporating multimodal attention-based video-description technologies.
Source: §1 · p. 2Results
I3D features improved response quality over QA-only features, attentional fusion further improved all metrics, and VGGish contributed especially with attentional fusion.
Source: §5.2 · p. 8Takeaways & Limitations
The dataset, code, and models are intended to support a new Video Scene-Aware Dialog Challenge.
Source: §1 · p. 2Human-annotated captions improved performance, but such annotations are unavailable for real systems.
Source: §5.2 · p. 8
Abstract
from arXiv · showhide
Dialog systems need to understand dynamic visual scenes in order to have conversations with users about the objects and events around them. Scene-aware dialog systems for real-world applications could be developed by integrating state-of-the-art technologies from multiple research areas, including: end-to-end dialog technologies, which generate system responses using models trained from dialog data; visual question answering (VQA) technologies, which answer questions about images using learned image features; and video description technologies, in which descriptions/captions are generated from videos using multimodal information. We introduce a new dataset of dialogs about videos of human behaviors. Each dialog is a typed conversation that consists of a sequence of 10 question-and-answer(QA) pairs between two Amazon Mechanical Turk (AMT) workers. In total, we collected dialogs on roughly 9,000 videos. Using this new dataset for Audio Visual Scene-aware dialog (AVSD), we trained an end-to-end conversation model that generates responses in a dialog about a video. Our experiments demonstrate that using multimodal features that were developed for multimodal attention-based video description enhances the quality of generated dialog about dynamic scenes (videos). Our dataset, model code and pretrained models will be publicly available for a new Video Scene-Aware Dialog challenge.
1 Introduction
The introduction identifies a gap in dialog systems’ ability to discuss dynamic surroundings from multimodal input. It proposes combining end-to-end dialog with multimodal video-description features and introduces a video-dialog dataset.
1 Introduction
- Current dialog systems cannot understand dynamic scenes from vision and non-speech audio, limiting conversation about nearby objects and events.The paper argues that dynamic scene-aware dialog is needed for conversations about objects and events around machines or users.Source: §1 · p. 2
- A cascaded alternative would require extracting and training annotated semantic frames, whereas an end-to-end network can encode video features and user utterances together.The proposed encoder-decoder approach maps video features and user utterances to natural-language responses.Source: §1 · p. 2
- Existing visual question answering and visual dialog address static images, while video description uses multimodal attention to represent dynamic scenes.Video-description features include spatiotemporal motion and audio, alongside temporal attention.Source: §1 · p. 2
- The proposed approach incorporates multimodal attention-based video-description technologies into an end-to-end dialog system for discussing dynamic scenes.The paper places this research target at the intersection of natural language processing, computer vision, and audio processing.Source: §1 · p. 2
2 Audio Visual Scene-Aware Dialog Dataset
The dataset uses Charades videos for text-based conversations about short videos. Two AMT workers discuss video events through ten question-and-answer rounds, and the validation videos are split into validation and test sets.
2 Audio Visual Scene-Aware Dialog Dataset
- Charades contains 11,848 untrimmed, multi-action videos, with 157 action categories and 27,847 textual descriptions.The videos are split into 7,985 training, 1,863 validation, and 2,000 test videos; textual descriptions are available only for training and validation.Source: §2 · p. 2
- Two AMT workers conduct each video discussion: one answers after watching the video, while the questioner sees only its first, middle, and last frames.The questioner asks about events between those frames over 10 QA rounds, then summarizes the video.Source: §2 · p. 2
- Dialogs were collected for 7,043 Charades training videos and all 1,863 validation videos; the validation set was split into 732 validation and 733 test videos.The test set lacked scripts, motivating the split of the original validation set.Source: §2 · pp. 2–3
- Questions average 8 words and answers average 10 words.Table 1 presents statistics for the Video Scene-aware Dialog Dataset on Charades.Source: §2 · p. 3, Table 1 · p. 3
3 Video Scene-aware Dialog System
The system generates answers about video events by encoding questions, visual features, and dialog history, then decoding a response. It combines multimodal video features with question-conditioned attention so the attended features can vary during word generation.
3 Video Scene-aware Dialog System
- Question, visual features, and dialog history are encoded into context for an LSTM decoder that generates answers about video events.The history encodes question-answer pairs.Source: §3 · p. 3
3.1 End-to-end Conversation Modeling
- The conversation model represents prior turns and the response as word sequences, encoding input context and estimating response probabilities at each decoding step.Hierarchical LSTMs encode question-answer history, and contextual information is fed to the decoder at every prediction step.Source: §3.1 · pp. 3–4
3.2 Multimodal-attention based Video Features
- Multimodal attention uses the question encoding and modality context vectors to vary attention across modalities while predicting each word.Naïve Fusion instead combines modality features using projection matrices, with modality attention weights fixed at 1.Source: §3.2 · pp. 4–5
4 Experiments for Multimodal attention-based Video Features
The experiments compare multimodal video and audio features for video description, then report that I3D features outperform VGG-16 plus C3D and Audio Set VGGish performs best among the tested audio features.
4 Experiments for Multimodal attention-based Video Features
- The experiments first evaluate multimodal-attention video-description features to select features for the scene-aware dialog system.The feature evaluations use MSVD, MSR-VTT, and Charades.Source: §4 · p. 5, §4.1 · p. 5
4.2 Video Processing
- Video inputs include VGG-16 image features, C3D spatiotemporal features, and I3D features, with I3D RGB and optical-flow streams treated as separate modalities.I3D features are taken from the Mixed_5c layer; the RGB and flow streams each use stacks of 16 frames.Source: §4.2 · p. 6
4.1 Datasets
- The datasets differ in availability: MSVD audio was collected for 1,649 clips, while Charades descriptions support evaluation on its validation set.The MSVD count represents 84% of that dataset; Charades descriptions are available for training and validation only.Source: §4.1 · p. 5, §4.3 · p. 6
4.3 Audio Processing
- Audio inputs comprise MFCC, SoundNet, and Audio Set VGGish features, extracted at different temporal resolutions for the model.VGGish outputs a 128-dimensional vector every 0.48 seconds, SoundNet a 1024-dimensional vector every 0.67 seconds, and MFCC inputs are 260-dimensional.Source: §4.3 · p. 6
4.4 Experimental Setup
- The caption decoder is a 512-cell LSTM; image and deep-audio features use a 512-unit projection, while MFCC features pass through a BLSTM encoder.The decoder is trained with cross-entropy, L2 regularization, and RMSprop; input words use 256-dimensional embeddings.Source: §4.4 · p. 6
4.5 Evaluation
- Generated sentences are evaluated against ground-truth sentences using BLEU, METEOR, ROUGE_L, and CIDEr metrics.The evaluation uses the publicly available MS COCO caption-generation code.Source: §4.5 · p. 6
4.6 Results and Discussion
- I3D features outperformed VGG-16 plus C3D across the reported video-description evaluations, while Audio Set VGGish provided the best audio-feature performance.The evaluations covered MSVD, the available MSR-VTT subset, and Charades.Source: Table 3 · p. 7, Table 4 · p. 7, Table 5 · p. 7
- Response-generation evaluation reports objective measures, while candidate output sequences are limited to n-best hypotheses and their distribution is estimated by an encoder-decoder model.The paper says enumerating every vocabulary sequence is intractable.Source: §4.6 · p. 7, Table 6 · p. 8
5 Experiments for Video-scene-aware Dialog
The experiments evaluate an end-to-end scene-aware dialog system on collected Charades dialogs, comparing text, video, and audio inputs. I3D features and multimodal attention improve generated responses, and VGGish contributes especially with attentional fusion.
5 Experiments for Video-scene-aware Dialog
- The system extends end-to-end dialog to scene-aware dialog by embedding video and audio features selected earlier in the paper.The experiments compare multimodal feature combinations and test attentional fusion for response generation.Source: §5 · p. 8, §5.1 · p. 8
5.1 Conditions
- The experiments use collected Charades dialog data and compare combinations of QA text, visual, and audio features, selecting models by development-set perplexity.Training used ADAM with cross-entropy for up to 20 epochs.Source: §5.1 · p. 8
5.2 Evaluation Results
- The evaluation compares response-generation models using QA history and the last question, captions, VGG16 or I3D video features, and VGGish audio features.Reported measures are BLEU1-4, METEOR, ROUGE_L, and CIDEr.Source: §5.2 · p. 8
- Human-annotated captions improve response quality over QA-only features by supplying scene information, but such annotations are unavailable for real systems.QA-only features served as the baseline without video scene features.Source: §5.2 · p. 8
- I3D features improve over QA-only responses, and attentional fusion of I3D RGB and flow features further improves all metrics; VGGish contributes especially with attentional fusion.The study also reports a qualitative example where adding VGGish changes an incorrect denial of audio to an answer consistent with the ground truth.Source: §5.2 · pp. 8–9
6 Conclusion
The paper presents a new task for dialog about dynamic scenes, together with a model and video-dialog dataset. Experiments show that multimodal features developed for video description enhance generated-dialog quality.
6 Conclusion
- The paper introduces a dialog task for discussing dynamic scenes, an end-to-end model using multimodal video-description features, and a dataset of human dialogues about videos.The authors report training a model to respond in dialogs about input videos and plan to make the dataset and model publicly available for a challenge.Source: §6 · p. 9