Source-linked AI summary
Audio-Visual Scene-Aware Dialog
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K. Marks, Chiori Hori, Peter Anderson, Stefan Lee, Devi Parikh
TL;DR
The paper addresses how conversational agents can answer questions about dynamic scenes while using audiovisual evidence and prior dialog context. It introduces the AVSD task and dataset, evaluates baseline systems, and finds that models using all available modalities perform best. Evaluation remains retrieval-based rather than directly testing cooperative interaction with human users.
Problem
Static-image dialog cannot represent important temporal dynamics and audio, while scene questions also require grounding sequential dialog references.
Method
The paper introduces scene-aware dialog and the AVSD Dataset, then evaluates baseline models using video, audio, questions, and dialog history.
Results
Models leveraging video, audio, and dialog history perform best, indicating that AVSD dialogs are grounded in multimodal observations.
Takeaways & Limitations
AVSD provides a benchmark for evaluating and promoting progress in audiovisual intelligent agents.
Takeaways & Limitations
The evaluation leaves direct testing with human users in cooperative tasks for future work.
Abstract
from arXiv · showhide
We introduce the task of scene-aware dialog. Our goal is to generate a complete and natural response to a question about a scene, given video and audio of the scene and the history of previous turns in the dialog. To answer successfully, agents must ground concepts from the question in the video while leveraging contextual cues from the dialog history. To benchmark this task, we introduce the Audio Visual Scene-Aware Dialog (AVSD) Dataset. For each of more than 11,000 videos of human actions from the Charades dataset, our dataset contains a dialog about the video, plus a final summary of the video by one of the dialog participants. We train several baseline systems for this task and evaluate the performance of the trained models using both qualitative and quantitative metrics. Our results indicate that models must utilize all the available inputs (video, audio, question, and dialog history) to perform best on this dataset.
1. Introduction
The paper frames scene-aware dialog as multimodal conversation about temporally varying scenes, extending visual dialog beyond static images. It introduces AVSD as a benchmark for generating natural responses grounded in video, audio, and dialog history.
- Static-image dialog cannot capture important temporal dynamics and audio, limiting scene understanding for applications such as assistants and accessibility technologies.
- Scene-aware dialog requires holistic grounding in visual and audio information, temporal dynamics, and sequential dialog references.The paper illustrates this requirement with references such as “her” and “it” across turns.
- The proposed task asks systems to converse about a temporally varying scene and answer questions about a short video through question-answer turns.
- The task uses a video, dialog history beginning with a scene script, and a follow-up question to generate a complete, natural response.
- The AVSD Dataset provides more than 11,000 conversations about human-centered videos, covering actions, interactions, sound, and temporal dynamics.
2. Related Work
The work builds on video understanding, video question answering, and visual dialog while targeting a more natural multi-round setting. Unlike common one-shot or static-image tasks, AVSD centers conversations on dynamic video scenes.
- Video Datasets: Existing video datasets largely benchmark action classification, while Charades provides crowd-sourced videos of people acting out scripts of daily activities.
- Video-based Question Answering: Video-based question answering commonly uses end-to-end models to answer multiple-choice questions from video frames and associated scripts or subtitles.
- Video-based Question Answering: The paper characterizes one-shot video question answering as less natural and focuses AVSD on settings involving multiple dialog rounds.
- Visual Dialog: AVSD extends image-based visual dialog, which conditions answers on an image, dialog history, and a question, to include dynamic audiovisual scenes.
3. Audio Visual Scene-Aware Dialog Dataset
AVSD is a benchmark for dialog about temporally varying everyday scenes, designed to elicit complete, action-focused, multi-round conversations grounded in video and audio. Its dataset pairs Charades videos with dialogs, summaries, and analyses showing that questions commonly address audio and event order.
- Video Content: Charades provides 11,816 naturalistic videos averaging 30 seconds, each containing at least two actions and a sequence of discussable small events.The videos are crowd-sourced and feature diverse indoor activities.
- Data Collection: The collection pairs an Answerer who sees the video and script with a Questioner who sees three frames and asks about the scene.Instructions emphasize investigating actions rather than focusing on appearance or setting.
- Dataset Design: AVSD contains multi-round question/answer dialogs about short videos of people performing everyday activities in home environments.The dataset targets scene-aware dialog about temporally varying scenes rather than one-shot question answering.
- Dataset Statistics: 11,816 conversations and 118,160 QA pairs are split into 7,985 training, 1,863 validation, and 1,968 testing instances, each with a video summary.The summaries are written by the Questioner after each dialog.
- Dataset Analysis: AVSD questions and answers average 7.9 and 9.4 words, compared with 5.1 and 2.9 words in VisDial, indicating more verbose conversations.The benchmark is designed for complete explanatory responses rather than brief answers.
- Dataset Analysis: More than 70% of conversations include questions about what happened next, while 57% include questions about audio such as music, noise, or speech.These distributions reflect the benchmark’s emphasis on temporal dynamics and multimodal scene understanding.
4. Model
The baseline model treats scene-aware dialog as multimodal candidate-answer ranking. It separately encodes dialog history, the question, video, and audio, fuses their representations, and selects the most probable response.
- Task and Architecture: The model receives video, audio, dialog history, and a follow-up question, then selects the most appropriate response from 100 candidate answers.Dialog history includes the Charades script and all preceding question-answer pairs.
- Encoder: Late fusion concatenates dialog-history, question, video, and audio embeddings into a joint fixed-size representation for answer ranking.The fused representation is produced through a linear layer with tanh activation.
- Input Representations: Video frames are encoded with pretrained I3D, while the audio track is encoded with pretrained AENet operating over long-time-span spectrograms.I3D converts video frames to a fixed-size feature and AENet provides the corresponding audio representation.
- Decoder: The decoder embeds each candidate answer with an LSTM and scores it by inner product with the holistic input embedding.Training maximizes the correct answer’s log-likelihood; testing selects the candidate with the highest probability.
- Candidate Answers: Candidate pools combine the ground-truth answer, hard negatives from similar questions in different contexts, popular answers, and random-question answers.Hard negatives are selected using question-embedding similarity, while popular answers test discrimination beyond answer frequency.
5. Experiments
Experiments evaluate AVSD models in a 100-candidate retrieval setting using ranking metrics. The paper reports this evaluation as useful but acknowledges that dialog evaluation remains an open problem.
- Data Splits: AVSDv1.0 uses 11,816 instances split across 7,985 training, 1,863 validation, and 1,968 testing examples, with results reported on the test set.The splits correspond to the source Charades video splits.
- Evaluation: Each response is evaluated by selecting one answer from 100 candidates in a retrieval or multiple-choice setting.This setup evaluates individual responses at each dialog round.
- Evaluation: The reported metrics are Recall@k and mean reciprocal rank, where higher is better, plus mean rank, where lower is better.Mean rank captures overall ground-truth ranking tendencies, while MRR emphasizes placing the ground truth near the top.
- Limitations: Direct evaluation with human users in cooperative dialog tasks is identified as an attractive alternative for future work.The authors note that retrieval-based dialog evaluation still has many open questions.
6. Results and Analysis
Ablation analyses show that dialog history, temporal video information, and audio each contribute to answering AVSD questions, with the strongest systems combining modalities. Question order and modality usefulness also vary across temporal and audio-focused questions.
- Multimodal ablations: The best-performing models leverage video, audio, and dialog history, indicating that AVSD dialogs are grounded in multimodal observations.The ablation benchmark evaluates the usefulness of different input modalities on the AVSDv1.0 test set.
- Language-only baselines: Adding dialog history produces the strongest language-only result, reaching a mean rank of 4.72.Question information improves mean rank from 28.54 for the answer-prior model to 7.63 with question alone, while dialog history further improves language-only performance.
- Temporal perception: Adding video features consistently improves models, whereas replacing video with a middle frame marginally hurts performance.The comparison suggests temporal perception is useful, although the authors note a confound from using VGG for images and I3D for videos.
- Audio information: Audio generally improves performance, and combining audio with dialog history and video can provide additional gains because the signals contain complementary information.The Q+V to Q+V+A comparison is the stated exception to the general audio improvement pattern.
- Question types: For audio-related questions, both Q+A and Q+V outperform Q, with visual features more useful; Q+V performs much better on temporal questions.These comparisons use mean rank on keyword-filtered subsets of audio-related and temporal questions.
- Dialog order: Shuffling dialog question-answer pairs changes performance by about 1.87 in averaged R@k and 0.33 in averaged mean rank, showing that QA order matters.The experiment compares the original ordered test set with randomly shuffled dialogs.
7. Conclusion
The paper introduces Audio Visual Scene-Aware Dialog as a natural-language task for dynamic scenes and releases AVSD as a benchmark built from more than 11,000 human-action videos. Baseline analyses indicate that all available modalities contribute to the task.
- Conclusion: AVSD defines scene-aware dialog as answering questions about dynamic scenes through natural-language conversation.The task extends scene understanding beyond static visual dialog to temporally varying scenes.
- Conclusion: The AVSD Dataset contains dialogs collected through a two-person chat protocol on more than 11,000 videos of human actions.The paper presents the dataset as a benchmark for audiovisual intelligent agents.
- Conclusion: A model and ablation studies show that video, audio, questions, and dialog history all play a role in addressing the task.The conclusion characterizes the dataset as rich in modalities and complexity.
I. Qualitative examples from our dataset.
The qualitative examples examine model responses to challenging questions across audio, visual, temporal, and general-scene categories using different input modalities.
- I. Qualitative examples from our dataset.: The qualitative analysis samples test-set examples covering audio, visual appearance, temporal information, and actions.The examples are randomly selected and compare responses under different input-modality settings.
I.1. Examples with Q+DH+V+A
The examples illustrate that model rankings can include several valid answers, especially for audio questions, while other questions require visual or temporal grounding and detailed responses.
- I.1. Examples with Q+DH+V+A: For one audio question, the ground-truth answer ranks third, but the two higher-ranked answers are also valid responses.Another example has three valid answers among the top four, all answering “no.”
- I.1. Examples with Q+DH+V+A: Visual questions can require estimating a person’s age from video-frame cues, while temporal questions concern when events occur.The examples also include challenging questions about the general scene.
- I.1. Examples with Q+DH+V+A: AVSD answers are expected to provide further details rather than stopping at one-word responses such as “yes” or “no.”Answerers were explicitly asked to elaborate on their responses.
I.2. Examples comparing setups Q, Q+V, Q+A and Q+DH+V+A
Figure 10 compares question-only, video-augmented, audio-augmented, and full multimodal dialog models across audio-, visual-, and temporal-related questions.
- The Q+A model performs better on the audio-related example.
- The Q+V model performs best on the visual-related example.
- The Q+V model also performs best on the temporal-related example, highlighting the value of modality-specific inputs.
J. Summaries Interface.
The AVSD collection included a summary-writing interface and a separate AMT study comparing questioner summaries with summaries written after watching the full video.
- Four AMT participants watched each video and wrote summaries describing all events in it.These summaries served as a reference for evaluating the questioner’s summary.
- The questioner’s summary was compared with the four summaries written by people who watched the entire video.
- In the examples, questioner summaries capture most events described in the four full-video summaries.
- Figure 11 presents the interface used for the summaries data-collection task.
- Figure 12 contrasts four full-video summaries in its first four rows with the questioner’s summary in the last row.