Source-linked AI summary
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, Di Hu
TL;DR
AVQA asks machines to answer questions about visual objects, sounds, and their associations in dynamic videos, requiring multimodal and spatio-temporal reasoning. The paper builds the MUSIC-AVQA benchmark and proposes a spatio-temporal grounding model; results show benefits from multisensory content and improved performance over recent QA approaches. The dataset is currently limited mainly to musical scenarios.
Problem
AVQA requires reasoning across visual objects, sounds, and their associations in dynamic audio-visual scenes.
Method
The paper constructs the MUSIC-AVQA dataset and proposes a model with spatial and temporal grounding for fine-grained audio-visual scene reasoning.
Results
The authors report that all modalities contribute to AVQA and that their model outperforms recent QA approaches, especially with the proposed modules.
Takeaways & Limitations
MUSIC-AVQA provides a testbed for evaluating fine-grained audio-visual scene understanding and spatio-temporal reasoning.
Takeaways & Limitations
The current dataset is mainly limited to musical scenarios, while audio-visual interaction also occurs in more general daily situations.
Abstract
from arXiv · showhide
In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos. The problem requires comprehensive multimodal understanding and spatio-temporal reasoning over audio-visual scenes. To benchmark this task and facilitate our study, we introduce a large-scale MUSIC-AVQA dataset, which contains more than 45K question-answer pairs covering 33 different question templates spanning over different modalities and question types. We develop several baselines and introduce a spatio-temporal grounded audio-visual network for the AVQA problem. Our results demonstrate that AVQA benefits from multisensory perception and our model outperforms recent A-, V-, and AVQA approaches. We believe that our built dataset has the potential to serve as testbed for evaluating and promoting progress in audio-visual scene understanding and spatio-temporal reasoning. Code and dataset: http://gewu-lab.github.io/MUSIC-AVQA/
1. Introduction
AVQA addresses questions about visual objects, sounds, and their associations in dynamic scenes, where single-modality methods struggle with cross-modal spatio-temporal reasoning. The paper introduces MUSIC-AVQA and a spatio-temporal grounding model to support this task.
- AVQA requires multimodal understanding and spatio-temporal reasoning to answer questions about visual objects, sounds, and their associations.
- Existing VQA and AQA methods focus on single modalities and cannot reason well in natural audio-visual scenarios.The motivating example requires identifying both which clarinet sounds first and its spatial position.
- 45,867 question-answer pairs span 9,288 musical-performance videos, 22 instruments, and 33 question templates.The videos contain over 150 hours of content and average about five QA pairs per video.
- The proposed model uses spatial grounding to connect sounds with visual sources and temporal grounding to capture changing audiovisual scenes.The spatial module uses attention-based sound-source localization to model cross-modal associations.
- The dataset covers audio, visual, and audio-visual scenes, while experiments report benefits from multisensory perception and superiority over recent QA approaches.The reported advantage is especially associated with questions measuring spatio-temporal reasoning.
2. Related Work
Prior audio-visual work has advanced perception tasks such as localization and separation, but remains limited in reasoning about spatio-temporal correlations. MUSIC-AVQA targets this gap with questions covering richer audio-visual associations and dynamic scenes.
- Audio-Visual Scene Understanding: Audio-visual research has addressed sound localization, separation, event localization, action recognition, parsing, captioning, and dialogue.
- Audio-Visual Scene Understanding: Existing localization and separation methods primarily perceive audiovisual objects rather than reason about their spatio-temporal correlations.
- Question Answering: VQA and AQA research typically addresses visual or audio questions separately, while complex scenes require reasoning across modalities.
- Question Answering: MUSIC-AVQA includes audio, visual, and audio-visual questions focused on interactions between visual objects and their produced sounds.
- Question Answering: Pano-AVQA covers relatively simple existential or location associations, whereas MUSIC-AVQA supports spatio-temporal reasoning in dynamic, long-term scenes.
3. The MUSIC-AVQA Dataset
MUSIC-AVQA is a large-scale musical-performance dataset designed for audio-visual question answering and spatio-temporal reasoning across audio, visual, and audiovisual scenes.
- The dataset covers 22 instrument types and three scene scenarios: audio, visual, and audio-visual.
- Its 33 question templates span nine question types and include existential, location, counting, comparative, and temporal reasoning.
- Compared with prior audiovisual QA datasets, MUSIC-AVQA provides more comprehensive modality coverage and supports spatio-temporal reasoning in longer, richer scenes.
- MUSIC-AVQA combines 7,422 real musical-performance videos with synthetic videos to increase scene complexity and diversity.Real videos include solo, same-instrument ensemble, and different-instrument ensemble performances.
- MUSIC-AVQA contains 45,867 question-answer pairs across 9,288 videos spanning more than 150 hours.
4. Method
The proposed model answers AVQA questions by grounding audio-visual information spatially and temporally before fusing it with the question representation for prediction.
- Representations and fusion: The model extracts audio and visual features with pre-trained CNNs, encodes the question with an LSTM, and integrates the modalities for answer prediction.Audio uses VGGish, visual inputs use ResNet-18, and the question encoder is trained from scratch.
- Spatial Grounding Module: Spatial grounding localizes sound-related visual regions by matching audio features with visual feature maps through attention.Matched and randomly sampled non-matched audio-visual pairs train the spatial module with a cross-entropy matching loss.
- Temporal Grounding Module: Temporal grounding uses the question representation to assign larger weights to audio and visual segments that are more relevant to the question.The module aggregates question-aware audio and visual embeddings across the video sequence.
- Multimodal Fusion: The fusion network concatenates grounded audio and visual features, transforms them into an audiovisual embedding, and combines that embedding with the question by element-wise multiplication.
- Answer Prediction: A linear layer and softmax produce answer probabilities, with training optimized using cross-entropy loss and testing selecting the highest-probability answer.
5. Experiments
Experiments evaluate input modalities and grounding modules on MUSIC-AVQA, comparing the proposed model with recent QA approaches. Results show that multimodal inputs and spatio-temporal grounding improve performance, with the audio-visual model achieving the best overall result among compared methods.
- The experiments assess input modalities and proposed modules through ablations, alongside comparisons with recent QA approaches.The ablation study is reported in Table 3, while comparisons with recent methods appear in Table 2.
- 54.90 accuracy: the question-only model Q answers some questions from common sense but struggles with complex location and temporal questions.The passage identifies Q as using only questions and reports its accuracy as 54.90.
- Adding audio or visual inputs improves performance, while V+Q performs better overall than Q and A+Q, indicating a strong role for visual features.The comparison between V+Q and A+Q also holds for audio question answering in the reported analysis.
- The proposed spatial and temporal grounding modules support reasoning about sounding objects across timestamps and question-relevant moments.Spatial grounding localizes sounding sources, while temporal grounding aggregates timestamp information according to the question.
- The audio-visual model with temporal grounding achieves the overall best performance among the compared methods on MUSIC-AVQA.Table 2 reports AVQA results on the MUSIC-AVQA test set, whose top-two results are highlighted.
6. Discussion
The paper contributes MUSIC-AVQA and a spatio-temporal grounding model for fine-grained audio-visual question answering. Its dataset combines real and synthesized musical-performance videos with multimodal questions, while the discussion identifies scope and modeling limitations.
- MUSIC-AVQA contains 45,867 question-answer pairs spanning audio-visual modalities and different question types.
- The proposed spatio-temporal grounding model supports fine-grained scene understanding and reasoning, and the reported results show that multimodal inputs contribute to AVQA performance.The authors report that the model outperforms recent QA approaches, especially when equipped with the proposed modules.
- The dataset is limited mainly to musical scenarios, while irrelevant objects and sound sources can introduce learning noise and make question answering difficult.The authors also note that the temporal module lacks explicit temporal modeling between audio and visual modalities.
- Its collection covers solo, same-instrument ensemble, and different-instrument ensemble performances across diverse instrument combinations.The different-instrument ensemble design uses combinations across instrument categories, including duet, trio, and quartet cases.
- The dataset includes real musical performances and about 1,866 synthesized videos designed to create complex audio-visual associations.Synthetic videos use audio overlay, video stitching, and audio-video random matching operations.
- The labeling system generates questions from selected performance, instrument, scene, question-type, and template choices, then validates answers against video content.The system produces one question-answer pair after annotators judge question reasonableness and answer the question.
C.3. QA pairs samples
This section presents the MUSIC-AVQA dataset as a large-scale spatial-temporal audio-visual question-answering resource.
- MUSIC-AVQA is described as a large-scale spatial-temporal audio-visual dataset focused on question answering.
D.1. Temporal modeling with shuffled segments.
The shuffled-segment experiment evaluates whether temporal grounding uses temporal order explicitly. Correct temporal ordering performs better, but the modest drop after shuffling indicates that the module does not strongly encode temporal order.
- 65.17 on temporal questions drops to 63.71 after shuffling video segments in the AV+Q+TG model.
- The temporal grounding module does not explicitly encode temporal order information, so shuffling segments has only a limited effect.
- Temporal words such as “first” and “last” may help the model group inputs with corresponding temporal locations.
- The model with correct temporal information still performs better on temporal questions than the shuffled version.
- Adding motion information raises the method’s released MUSIC-AVQA performance from 71.53 to 71.75.Motion features are extracted with an R(2+1)D network and fused with visual features.
D.3. Experiments on existing video QA dataset
Experiments on TVQA test whether an existing video QA benchmark supports audio-visual question answering. The audio modality harms both compared methods there, while the proposed method remains more robust with less performance drop.
- On TVQA, Q+V and Q+V+A are not superior to the question-only method based on common sense.
- The proposed method outperforms the TVQA method with both visual-only and audio-visual inputs.
- Introducing audio harms the performance of both methods on TVQA.The authors attribute this to TVQA’s predominantly human-speech audio and difficult cross-modal interaction modeling.
- TVQA is considered unsuitable for AVQA because it does not adequately support interactions between audio and visual components.The proposed method shows better robustness with less performance drop in this setting.
E. Examples
The examples show that audio, visual, and joint multimodal inputs contribute complementary information to audio-visual question answering. Correct answers may require distinguishing instruments, separating sound sources, and reasoning over temporal order.
- The dataset includes real performances and synthetic scenes involving solo, same-instrument ensemble, different-instrument ensemble, audio-video mismatching, audio overlay, and video stitching.
- Audio helps identify which instrument is playing in audio-visual question answering.
- Visual information helps recognize multiple sources when same-category sounds are difficult to distinguish acoustically.
- Multisensory perception supports fine-grained understanding by combining sounding-scene recognition with temporal reasoning.The example identifies the trumpet after the accordion.
F. Personal data/Human subjects
The supplied material states that MUSIC-AVQA videos are public YouTube content annotated through crowdsourcing, without personally identifiable information or offensive content. It also references 33 question templates and illustrates answers involving temporal order, location, counting, and audio-visual correspondence.
- MUSIC-AVQA videos are public on YouTube and are annotated via crowdsourcing.The paper says crowdworkers were informed how the data would be used.
- The dataset does not contain personally identifiable information or offensive content.
- 33 question templates are listed for the audio-visual question-answering dataset.
- Examples cover temporal order, spatial location, source counting, and whether a sound comes from an instrument visible in the video.
- Figure 10 reports that the authors' model predicts correct answers and performs better than Pano-AVQA.