Source-linked AI summary
Pano-AVQA: Grounded Audio-Visual Question Answering on 360$^\circ$ Videos
Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, Gunhee Kim
TL;DR
Panoramic video QA has lacked benchmarks for jointly examining spherical spatial and audio-visual relationships. Pano-AVQA supplies grounded QA data from 360° clips and a transformer framework; its model outperforms two multimodal baselines in the reported comparison. The dataset’s scope and template-bias mitigation define important boundaries for interpreting its questions.
Problem
Panoramic video benchmarks have not covered linguistic queries for spherical spatial and audio-visual relationships, despite the distinct cues available beyond a fixed field of view.
Source: §1 · pp. 1–2Method
Results
The proposed model performs 5.85% better than SparseGraph and 2% better than LXMERT in the reported comparison.
Source: §5.2 · p. 7Takeaways & Limitations
Pano-AVQA provides a benchmark for evaluating spherical spatial and audio-visual reasoning in 360° surroundings, with 51.7K grounded QA pairs from 5.4K videos.
Source: §6 · p. 8Template-based QA generation may introduce answer-distribution bias, so the authors add pairs using unrelated descriptions or counterexample clips to mitigate it.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
360$^\circ$ videos convey holistic views for the surroundings of a scene. It provides audio-visual cues beyond pre-determined normal field of views and displays distinctive spatial relations on a sphere. However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings. We propose a novel benchmark named Pano-AVQA as a large-scale grounded audio-visual question answering dataset on panoramic videos. Using 5.4K 360$^\circ$ video clips harvested online, we collect two types of novel question-answer pairs with bounding-box grounding: spherical spatial relation QAs and audio-visual relation QAs. We train several transformer-based models from Pano-AVQA, where the results suggest that our proposed spherical spatial embeddings and multimodal training objectives fairly contribute to a better semantic understanding of the panoramic surroundings on the dataset.
1. Introduction
Panoramic videos expose spherical spatial relations and audio-visual cues beyond a fixed field of view, motivating question answering that jointly evaluates these forms of understanding. Pano-AVQA provides grounded QA data and a transformer-based framework for this setting.
1. Introduction
- Unlike conventional videos, 360° videos let viewers attend to any region of the original surroundings rather than a predetermined field of view.This wider view brings distinct spatial and audio-visual reasoning challenges.Source: §1 · p. 1
- Spherical spatial reasoning and audio-visual reasoning are underemphasized in normal-field-of-view video understanding, despite panoramic videos’ wider visual and sound context.The spherical ambient space requires recognizing relations among objects all around, while panoramic videos contain more diverse visual sources of sounds.Source: §1 · p. 1
- Pano-AVQA introduces spatial and audio-visual question answering for 360° videos, addressing the lack of linguistic-query tasks for panoramic understanding.The dataset contains 20K spatial and 31.7K audio-visual QA pairs with bounding-box grounding from 5.4K panoramic clips.Source: §1 · p. 2
- The proposed transformer-based framework fuses panoramic visual, audio, and language cues, using quaternion-based coordinates and auxiliary audio-skewness prediction.The authors describe these components as broadly applicable to multichannel audio inputs.Source: §1 · p. 2
2. Related Works
Prior panoramic-video work has mainly developed visual understanding, while related multimodal work has addressed audio-visual tasks under restricted viewing assumptions. Pano-AVQA instead uses language queries to examine fine-grained audio-visual relationships and spherical spatial reasoning.
Understanding of Panoramic Videos
- Earlier panoramic-video work spans cinematography, highlight detection, summarization, tracking, and saliency, with most work concentrating on visual cues.Other efforts address viewpoint grounding, spatial audio augmentation, or object removal, but do not use language queries to evaluate panoramic audio-visual understanding.Source: §2 · p. 2
- Pano-AVQA uses language queries to evaluate panoramic audio-visual signals, unlike prior work focused on visual cues or other individual modalities.The dataset may also support audio-visual grounding or scene-graph generation in panoramic videos.Source: §2 · p. 2
Multimodal Question Answering
- Video QA and sound-grounded QA address visual-linguistic or audio-related questions in varied settings, but the cited work does not describe their focus as panoramic spherical relations.The related examples include movie, TV, web-GIF, animation, spoken-question, and audio-visual dialog tasks.Source: §2 · p. 2
- AVSD uses audio and video for sequential question answering but mainly tests sound presence, whereas Pano-AVQA targets fine-grained grounding and spatial relations in 360° videos.An example Pano-AVQA question asks what is on the opposite side of a loud honking.Source: §2 · p. 2
Audio-Visual Scene Understanding
- Some audio-visual scene-understanding methods assume an attended salient viewpoint, while panoramic approaches may treat frames as normal and ignore distortion.Pano-AVQA aligns audio and video without a predetermined normal field of view, considering more surrounding context.Source: §2 · pp. 2–3
3. Pano-AVQA Dataset
Pano-AVQA is built for fine-grained spatio-temporal and audio-visual question answering on panoramic videos. It introduces open-ended spherical spatial and audio-visual tasks with answer grounding.
3.2. Data Collection
- The collection pipeline extracts roughly five-second clips from online 360° videos and gathers bounding-box and sound grounding, descriptions, and question-answer pairs.Source: §3 · p. 3, §3.2 · p. 3
3.1. Task Definition
- The two open-ended tasks cover relative object relations on an unoriented sphere and identifying sounds or sound sources from visual objects.Source: §3.1 · p. 3
3.3. Data Annotations
- Annotation is divided into bounding-box collection, visual and sound descriptions, and question-answer generation, with validation between stages.Source: §3.3 · pp. 3–4
- Template-based QA can introduce answer-distribution bias; unrelated descriptions and counterexample clips are added to mitigate it.Source: §3.3 · p. 4
3.4. Data Analysis
- Pano-AVQA contains 51.7K QA pairs from 5.4K clips across 2.9K videos, including 20K spatial and 31.7K audio-visual pairs.Source: §3.4 · pp. 4–5
- Human sound is the most frequent audio tag, but vehicles, animals, and musical instruments are also represented; grounded boxes extend from the equator toward the poles.Source: §3.4 · p. 5
4. Approach
LAViT addresses panoramic video, audio, and language representation and their reconciliation with spherical spatial embeddings and a transformer-based multimodal encoder. Its pipeline builds modality-specific inputs, fuses them across modalities, and trains with masked pretraining and grounded question answering.
4. Approach
- LAViT combines spherical spatial embeddings from visual objects and audio events with a transformer-based multimodal encoder for panoramic video, audio, and language.Source: §4 · p. 5
4.1. Input Representations
- Visual inputs combine object features with rotation-quaternion spatial coordinates, while stereo audio is divided into events and questions are represented as language tokens.Visual proposals use equirectangular and NFoV projections; stereo audio events are identified by shared top-3 labels.Source: §4.1 · pp. 5–6
4.2. Encoder
- Three unimodal encoders process language, audio, and visual inputs, then multimodal layers use one modality as the query and another as context.The model pools the multimodal outputs and feeds them to separate MLPs for answer prediction and grounding.Source: §4.2 · pp. 6–7
4.3. Training
- Masked pretraining predicts language tokens, spherical visual embeddings, and audio spatial skewness; grounded question answering trains answer classification and coordinate prediction.Audio skewness is derived from spherical-harmonic coefficients and predicted with the event timestamp.Source: §4.3 · p. 7
5. Experiments
Experiments compare multimodal QA baselines and spatial embeddings on Pano-AVQA, then test the contributions of input modalities and field-of-view choices. The proposed model performs better than the selected VQA baselines, while spherical embeddings perform better on spherical spatial reasoning.
5.1. Experimental Setup
- The setup compares AVSD, BERT, SparseGraph, and LXMERT using accuracy for answers and mean squared error for bounding-box grounding.Spatial embedding variants include Cartesian, spherical, normal 3D, and quaternion representations.Source: §5.1 · p. 7
5.2. Results and Analyses
- 5.85% and 2% better than SparseGraph and LXMERT, respectively, the proposed model’s reported comparisons on the Pano-AVQA test split.Source: §5.2 · p. 7, Table 2 · p. 8
- Removing unimodal encoders drops performance by 6.35%, while removing visual or audio input drops it by 2.5% or 1.76%, respectively.Source: §5.2 · p. 8
- Using both ER and NFoVs performs best; single-NFoV training is 2% lower, and ER-only performs below dense-NFoV training.The authors presume ER-only overlooks smaller objects.Source: §5.2 · p. 8
- Unit-sphere and quaternion embeddings perform better on spherical spatial reasoning than Cartesian coordinates, which perform worst.Grounding errors across embeddings are not directly comparable because their error scales differ; qualitative examples note quaternion localization ability, especially in the second column.Source: §5.2 · p. 8, Table 3 · p. 8, Figure 5 · p. 8
6. Conclusion
The paper introduces Pano-AVQA to evaluate spherical spatial and audio-visual reasoning in panoramic videos, and presents LAViT as a three-modality audio-visual QA framework. It also identifies audio-visual scene graphs and embodied tasks as possible future directions.
6. Conclusion
- Pano-AVQA contains 51.7K question-answer pairs with bounding boxes from 5.4K panoramic videos.Source: §6 · p. 8
- LAViT extends crossmodal attention to leverage visual, audio, and language modalities for audio-visual question answering.Source: §6 · p. 8
- Future directions include adding audio-visual scene-graph annotations and using 360° datasets for embodied question answering and language-guided navigation in simulated 3D environments.Source: §6 · p. 8