Source-linked AI summary
SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding
Yu Chen, Ruihang Liu, Yangguang Xu, Xinyue Jiang, Mohammed Bennamoun, Farid Boussaid, Xinyuan Qian, Qiuhong Ke
TL;DR
Existing spatial audio-visual benchmarks often rely on simulated scenes, evaluate isolated skills, and offer limited diagnosis of failures. SAVU-Bench and SAVU-Diag provide real-world, linked evaluation, and model results show audio-related spatial understanding and reasoning remain challenging even as explicit cues improve grounding and matching.
Problem
Existing benchmarks often use simulated scenes, assess spatial skills in isolation, and provide limited visibility into whether reasoning errors stem from prerequisite failures.
Source: §1 · p. 2Method
The paper introduces SAVU-Bench, a real-world benchmark spanning three capability levels and seven tasks, SAVU-Diag for same-scene prerequisite diagnosis, and SAVU-EA, a training-free baseline adding predicted spatial cues.
Source: §1 · p. 2, §4.4 · p. 9Results
Models perform better on visual than auditory spatial grounding, reasoning errors may persist despite correct prerequisites, and explicit cues improve grounding and matching without eliminating the reasoning gap.
Source: §5 · p. 9, §4.4 · p. 9Takeaways & Limitations
The findings identify robust auditory spatial perception and reliable composition of semantic and spatial relations as continuing challenges for current models.
Source: §4.4 · p. 9, §5 · p. 9
Abstract
from arXiv · showhide
Spatial audio-visual understanding requires models to recognize not only what is present, but also where events occur and how they relate across modalities. Existing benchmarks often rely on simulated scenes, evaluate isolated spatial skills, and provide limited diagnostic insight into failure modes. We introduce SAVU-Bench, a real-world benchmark that systematically evaluates spatial audio-visual understanding across three capability levels and seven evaluation tasks. We further introduce SAVU-Diag, a scene-linked diagnostic set that decomposes reasoning questions into their prerequisite grounding and alignment sub-tasks. Evaluation of 12 representative models on SAVU-Bench reveals that while visual spatial grounding is relatively mature, spatial perception involving audio remains a primary bottleneck. SAVU-Diag further demonstrates that most reasoning errors co-occur with failures on these prerequisite tasks, though reasoning gaps persist even when prerequisites are correctly resolved. Motivated by these findings, we introduce SAVU-EA, a training-free evidence-augmented baseline that makes spatial cues more explicit. While SAVU-EA substantially improves spatial grounding and joint matching, high-level spatial reasoning remains challenging. Our findings highlight the urgent need for both robust spatial audio perception and deeper integration of cross-modal spatial relations.
1 INTRODUCTION
Spatial audio-visual understanding requires more than recognizing content: models must localize visual and acoustic events and resolve cross-modal spatial relations. The paper introduces a real-world benchmark and scene-linked diagnostics, finding persistent audio and reasoning weaknesses while explicit spatial cues improve grounding and matching.
1 INTRODUCTION
- SAVU-Bench evaluates real-world recordings across Perception, Alignment, and Reasoning, with seven tasks built from six data sources using human annotation and expert verification.It provides a unified evaluation of how models perceive, align, and reason over cross-modal spatial information.Source: §1 · pp. 2–3
- SAVU-Diag links each reasoning instance to grounding and alignment tasks from the same scene, separating errors that co-occur with prerequisite failures from those that persist despite correct prerequisites.This provides diagnosis beyond aggregate benchmark accuracy.Source: §1 · pp. 2–3
- Across 12 models, visual grounding is stronger than audio-related spatial understanding, and reasoning failures can persist despite correct prerequisite grounding and alignment.SAVU-Diag also finds that similar reasoning accuracy can arise from different grounding and alignment error patterns.Source: §1 · p. 2
- SAVU-EA is a training-free baseline that adds explicit audio and visual spatial cues, improving auditory grounding and joint matching while leaving spatial reasoning challenging.The results indicate that stronger spatial perception alone does not resolve the reasoning gap.Source: §1 · pp. 2–3
2 RELATED WORK
Prior spatial audio-visual benchmarks vary in realism and scope, and often assess spatial capabilities separately. SAVU-Bench combines real-world recordings and unified capability evaluation, while SAVU-Diag links reasoning questions to prerequisite tasks.
2 RELATED WORK
- Many spatial benchmarks rely on simulated scenes, while existing real-world benchmarks often cover a narrower subset of spatial capabilities.Prior evaluations also commonly assess localization, matching, or reasoning separately, limiting unified assessment of how those capabilities relate.Source: §2 · p. 3
- SAVU-Bench combines diverse real-world spatial audio-visual recordings with unified evaluation of Perception, Alignment, and Reasoning.This complements benchmarks that focus on narrower capability subsets.Source: §2 · p. 3
- SAVU-Diag links reasoning questions to prerequisite grounding and alignment tasks from the same scenes, enabling scene-linked analysis of reasoning failures beyond task-level evaluation.Source: §2 · p. 3
3 SAVU-BENCH
SAVU-Bench evaluates spatial audio-visual understanding in real-world scenes across perception, alignment, and reasoning, and SAVU-Diag links reasoning to same-scene prerequisite tasks. Its four-stage curation process combines machine-generated candidates with human annotation and expert checks.
3 SAVU-BENCH
- SAVU-Bench covers three capability levels through seven tasks, using limited-field-of-view visual inputs and six real-world sources with human annotation and expert verification.Source: §3 · p. 3, §3.3 · p. 6
3.1 TASK TAXONOMY
- Perception tests modality-specific localization, relative position, and spatial dynamics; Alignment tests semantic and trajectory correspondence plus joint audio-to-video and video-to-audio matching.Joint matching includes a match, a semantic negative, and a spatial negative; reasoning tasks compose cross-modal relations, including relative direction, spatial ordering, and allocentric reasoning.Source: §3.1 · pp. 3–4
3.2 SAVU-DIAG: SCENE-LINKED DIAGNOSIS
- SAVU-Diag derives independent grounding and alignment tasks from each selected reasoning instance while preserving its recording, time window, target identities, and reference frame.This permits checking whether reasoning errors co-occur with prerequisite failures or persist when evaluated prerequisites are correct.Source: §3.2 · p. 4
3.3 DATA CURATION PIPELINE
- The four-stage curation process uses structured scenes and task templates to generate candidate questions, followed by human refinement and expert verification.Reasoning chains and diagnostic prerequisites are also reviewed by annotators; three experts independently answer each question, retaining those answered correctly by at least two.Source: §3.3 · p. 5
- 1,066 SAVU-Bench questions and 611 SAVU-Diag questions remain in the final datasets.Source: §3.3 · pp. 5–6
4 EXPERIMENTS
Across SAVU-Bench, auditory spatial grounding, joint cross-modal matching, and spatial reasoning remain difficult despite stronger visual grounding. SAVU-Diag exposes distinct prerequisite and relation-composition failures, while SAVU-EA improves several spatial tasks but does not resolve reasoning gaps.
4.1 EXPERIMENTAL SETUP
- Twelve models were evaluated with cyclic answer-option permutations; the 30-questions-per-task human subset used the same scenes and questions with audio adapted to each interface.Source: §4.1 · pp. 6–7
4.2 MAIN RESULT ANALYSIS
- Spatial-Omni reaches only 55.6% on Audio Grounding despite native FOA input, while models generally perform better on Visual Grounding.Source: §4.2 · p. 7
- Best A2V and V2A Joint Matching accuracies are 35.6% and 42.8% against 33.3% chance; Spatial Reasoning leaders reach 45.8%, 3.6 points above chance.Source: §4.2 · p. 7
4.3 DIAGNOSTIC ANALYSIS WITH SAVU-DIAG
- OmniVideo-7B’s reasoning errors are 61.4% grounding-only, whereas Spatial-Omni’s include 23.1% alignment-only and 46.2% failures in both prerequisites.Source: §4.3 · p. 8, Figure 3 · p. 8
- Gemini models sometimes reason incorrectly despite correct grounding and alignment, and a Gemini-3.6-Flash example composes the spatial relation incorrectly.Source: §4.3 · p. 8, Figure 3 · p. 8
4.4 DISCUSSION: CAN EXPLICIT SPATIAL CUES IMPROVE SPATIAL UNDERSTANDING?
- Against original Gemini-3.6-Flash, SAVU-EA improves Audio Grounding by 30.7 points, Joint A2V by 19.1 points, and Joint V2A by 21.7 points.Source: Figure 4 · p. 9, §4.4 · p. 9
- SAVU-EA approaches or exceeds humans on Visual Grounding and Joint V2A, but gaps remain in alignment, Joint A2V, and Spatial Reasoning.Source: Figure 4 · p. 9, §4.4 · p. 9
- SAVU-EA supplies Gemini-3.6-Flash with predicted audio and visual spatial cues alongside the original inputs and question, using modality-specific cues for perception tasks.Source: §4.4 · p. 9
5 CONCLUSION
SAVU-Bench reveals that spatial audio-visual understanding remains uneven: visual grounding is stronger than auditory grounding, joint matching, and spatial reasoning. SAVU-Diag and SAVU-EA expose distinct sources of these gaps, while showing that more explicit spatial cues do not eliminate the reasoning gap.
5 CONCLUSION
- Current models perform well on visual grounding but remain substantially weaker in auditory spatial grounding, joint matching, and spatial reasoning.Making spatial cues more explicit substantially improves grounding and downstream performance, but does not eliminate the reasoning gap.Source: §5 · p. 9
- Similar reasoning accuracy can arise from different prerequisite-error patterns, and reasoning failures can persist despite correct grounding and alignment answers.Source: §5 · p. 9
- Robust spatial audio-visual understanding requires progress in acquiring reliable spatial relations and consistently combining them for higher-level reasoning.Source: §5 · p. 9
AI USE STATEMENT
Generative AI assisted with candidate-question generation and language polishing. Human annotators verified generated questions, and the work describes participation and data-use conditions for benchmark construction and release.
AI USE STATEMENT
- Generative AI assisted in generating initial candidate questions, which human annotators subsequently reviewed and verified for validity.Source: AI USE STATEMENT · p. 10
- Generative AI also assisted with language polishing, while the authors take responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 10
- Human participants were recruited for annotation and evaluation, evaluators were disjoint from benchmark constructors, and derived data use will follow source licenses and terms.Source: ETHICS STATEMENT · p. 10
- The authors plan to release the benchmark, evaluation code, and construction scripts upon acceptance.Construction, task, diagnostic-set, and human-evaluation procedures are described in the paper and appendix.Source: REPRODUCIBILITY STATEMENT · p. 10
A.1.1 CHANCE-LEVEL CALCULATION
The chance-level calculation evaluates questions across cyclic answer-choice permutations. Chance levels differ by task according to the number of answer choices.
A.1.1 CHANCE-LEVEL CALCULATION
- 50.0% is the chance level for Audio Grounding, Visual Grounding, Semantic Alignment, and Spatial Alignment, each with two answer choices.Source: §A.1.1 · p. 13
- 33.3% is the chance level for Joint A2V and Joint V2A, which each have three answer choices.Source: §A.1.1 · p. 13
- Average Accuracy is computed over all question–ordering pairs after evaluating each question under all cyclic answer-choice permutations.Source: §A.1.1 · p. 13
- Spatial Reasoning is a special case because its questions have different numbers of answer choices.Source: §A.1.1 · p. 13
A.1.2 HUMAN SUBSET
The human-subset results distinguish strong isolated visual grounding from weaker cross-modal integration and reasoning. SAVU-EA narrows some gaps, but substantial differences from human performance remain, especially in reasoning.
A.1.2 HUMAN SUBSET
- 40.00%, 43.33%, and 46.67% are the best base-model accuracies on Joint A2V, Joint V2A, and Spatial Reasoning, versus human accuracies of 66.67%, 65.33%, and 85.00%, respectively.Table 4 reports the complete human-evaluation subset results, with 30 sampled questions per task.Source: Table 4 · p. 13, §A.1.2 · p. 13
- Several models reach 96.67–100.00% on Visual Grounding, comparable to or exceeding the human accuracy of 93.33%.Strong individual-task performance does not necessarily translate into reliable integration and reasoning over multiple spatial relations.Source: §A.1.2 · p. 13
- 83.33% on Audio Grounding, 70.00% on Joint V2A, and 66.67% on Spatial Reasoning are SAVU-EA results, but gaps remain in Semantic Alignment, Joint A2V, and especially Spatial Reasoning.The reported comparison suggests that explicit spatial evidence can improve downstream performance, while reliable reasoning still requires consistent integration and composition of cross-modal spatial relations.Source: §A.1.2 · pp. 13–14
A.1.3 SAVU-DIAG
SAVU-Diag shows that reasoning errors often coincide with failures in grounding or alignment, yet correct prerequisite answers do not guarantee correct reasoning. Supplying correct prerequisite facts improves accuracy, but does not eliminate errors.
A.1.3 SAVU-DIAG
- At 51.85% reasoning accuracy, Spatial-Omni has more alignment-only (23.1%) and joint failures (46.2%), while Nemotron-3-Nano-Omni has more grounding-only failures (41.0%).At 48.15% accuracy, 50.0% of MiniCPM-o-4.5 errors are grounding-only, while 52.4% of Qwen3.5-Omni-Plus errors involve both prerequisites.Source: §A.1.3 · p. 14
- 88.9% of video-SALMONN2+ reasoning errors coincide with failures in both grounding and alignment, while most models also fail with both prerequisites correct.The results distinguish prerequisite-acquisition failures from reasoning failures that persist despite correct prerequisite predictions.Source: §A.1.3 · p. 14
- 28.40 percentage points: Gemini-3.6-Flash accuracy rises from 49.38% to 77.78% when correct grounding and alignment facts are supplied.The oracle evaluation covers all 81 Spatial Reasoning questions; 18 errors remain.Source: §A.1.3 · p. 14
- Table 5 groups reasoning instances by whether all associated grounding questions and all associated semantic and spatial alignment questions are answered correctly.G1/A1 denote all-correct prerequisite answers; G0/A0 indicate at least one incorrect answer in the respective group.Source: Table 5 · p. 14, §A.1.3 · p. 14
A.1.4 ABLATION STUDY: SHORT-CUT ANALYSIS
This section examines the complementary contributions of audio and vision to Spatial Reasoning across 12 models and describes the construction and validation procedures for the benchmark, including an example of the human evaluation platform.
A.1.4 ABLATION STUDY: SHORT-CUT ANALYSIS
- Across 12 models, removing vision and audio reduces Spatial Reasoning accuracy by 4.32 and 3.50 percentage points on average, respectively.All 12 models perform better with both modalities than with either alone, although the relative contribution varies by model.Source: Table 6 · p. 15, §A.1.4 · p. 15
A.2.1 SPATIAL REFERENCE CONSTRUCTION
- Spatial references use target-centered limited-FoV views aligned with audio when needed, while preserving the complete acoustic scene, including off-screen sources.MRSAudio uses its supplied limited-FoV views directly; panoramic candidate views are refined through human annotation.Source: §A.2.1 · pp. 15–16
A.2.2 DETAILED TASK CONSTRUCTION
- Tasks cover coarse location, relative position, and motion grounding; semantic and spatial alignment; joint matching; and reasoning that composes cross-modal spatial relations.Reasoning uses a silent visual anchor, an in-view sounding source, and out-of-view sources so the in-view source bridges visual and acoustic relations.Source: §A.2.2 · pp. 16, 18, 19
A.2.3 DETAILED SAVU-DIAG CONSTRUCTION
- SAVU-Diag derives independent Grounding and Alignment questions from verified reasoning chains to test whether reasoning errors co-occur with prerequisite failures or persist despite correct prerequisites.GPT-5.4 proposes prerequisite questions, which human annotators inspect, complete, and verify against the scene evidence.Source: §A.2.3 · pp. 21–22
A.2.4 HUMAN ANNOTATION AND EVALUATION
- The final benchmark contains 1,066 SAVU-Bench questions and 611 SAVU-Diag questions, retained only when at least two of three experts select the intended answer.Construction includes coarse candidate screening and detailed annotation; human evaluation samples 30 questions per task, answered by ten participants using the same visual inputs and binaural-rendered FOA audio.Source: §A.2.4 · pp. 22–23
- Figure 13 presents an example of the human evaluation platform.Source: Figure 13 · p. 23