Source-linked AI summary
JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments
Zhan Liu, Changli Tang, Yuxin Wang, Zhiyuan Zhu, Youjun Chen, Yiwen Shao, Tianzi Wang, Lei Ke, Zengrui Jin, Chao Zhang
TL;DR
Existing AV-LLMs are largely limited to RGB video and monaural audio, leaving reliable 3D spatial grounding and reasoning under-supported. JAEGER integrates RGB-D perception, FOA audio, Neural IV, and the 61k-sample SpatialSceneQA benchmark, and reports strong performance across spatial tasks, while evaluation remains primarily simulation-based.
Problem
Existing AV-LLMs mainly use RGB video and monaural audio, while real-world datasets with dense spatial annotations and aligned visual depth remain scarce.
Method
JAEGER adapts an AV-LLM end to end with RGB-D observations, FOA multi-channel audio, Neural IV spatial representation, and SpatialSceneQA instruction-tuning data.
Results
JAEGER consistently surpasses 2D-centric baselines across spatial perception and reasoning tasks, achieving 99.2% accuracy on joint audio-visual reasoning.
Takeaways & Limitations
Explicitly modeling visual depth and spatial audio is indispensable for complex physical reasoning tasks in the evaluated simulated environments.
Takeaways & Limitations
Evaluation is primarily simulation-based, so transfer to real acoustics, sensors, synchronization, and calibration requires additional validation.
Abstract
from arXiv · showhide
Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a benchmark of 61k instruction-tuning samples curated from simulated physical environments. Extensive experiments demonstrate that our approach consistently surpasses 2D-centric baselines across diverse spatial perception and reasoning tasks, underscoring the necessity of explicit 3D modelling for advancing AI in physical environments. Our source code, pre-trained model checkpoints, and datasets are available at https://github.com/liuzhan22/JAEGER.
1. Introduction
JAEGER extends AV-LLMs from 2D RGB video and monaural audio to joint 3D audio-visual grounding and reasoning. It combines RGB-D geometry, spatial audio, a learned Neural IV representation, and the 61k-sample SpatialSceneQA benchmark.
- Existing AV-LLMs rely mainly on RGB video and monaural audio, constraining perception and reasoning about the 3D physical world.
- JAEGER jointly models visual depth and multi-channel spatial audio for end-to-end 3D direction-of-arrival estimation, box grounding, and multi-speaker matching.The framework uses RGB-D inputs with depth-aware positional encodings and FOA-based audio cues.
- Neural IV is a learnable FOA-based representation that encodes directional information for improved localization and cross-modal alignment.It is designed for azimuth perception in reverberant environments and under overlapping sound sources.
- SpatialSceneQA contains 61k samples pairing 3D RGB-D images, 4-channel FOA audio, and dense object-level spatial annotations.The benchmark supports precise localization, 3D grounding, and multi-speaker matching under source overlap.
- JAEGER achieves 2.21° median angular error for single sources, 4.11° under overlap, 0.32 3D IoU, 0.16 m median localization error, and 99.2% joint reasoning accuracy.
2. Related Work
Prior work develops spatial audio understanding and 3D visual grounding largely through separate modalities, while data with aligned spatial audio, RGB, and depth remains limited. JAEGER addresses this gap with end-to-end RGB-D and FOA reasoning supported by SpatialSceneQA.
- 2.1. Spatial Audio Understanding: Spatial audio understanding uses multi-channel recordings to infer acoustic-event configuration and has increasingly been integrated with language-model reasoning.
- 2.1. Spatial Audio Understanding: FOA offers a hardware-agnostic spatial representation, whereas binaural cues remain coupled to recording geometry and HRTFs, hindering cross-device generalization.
- 2.1. Spatial Audio Understanding: Real-world multi-channel datasets with dense spatial annotations are scarce; STARSS23 provides multi-channel audio and panoramic RGB video but lacks aligned visual depth.Simulation with SoundSpaces 2.0 enables scalable reverberant or anechoic data generation with controllable geometry and precise spatial annotations.
- 2.2. Visual 3D Object Grounding: 3D visual grounding research varies in how it incorporates geometry and whether models directly predict 3D outputs, using point clouds, 3D tokens, multi-view video, or RGB-D streams.
- 2.2. Visual 3D Object Grounding: Figure 1 depicts SpatialSceneQA examples on the left and a Habitat-Sim/SoundSpaces 2.0 synthesis pipeline on the right, producing synchronized RGB-D, FOA, and spatial metadata.
- 2.3. AV-LLMs for 3D Understanding: JAEGER differs from modular prior systems by adapting an AV-LLM to jointly perceive and reason over RGB-D observations and FOA audio end to end.It adds Neural IV and SpatialSceneQA to support unified spatial perception, grounding, and language-based reasoning.
3. SpatialSceneQA
SpatialSceneQA is a synthetically generated benchmark pairing synchronized RGB-D observations, 4-channel FOA audio, and precise 3D annotations for perception and cross-modal reasoning tasks.
- Dataset construction: 61k samples pair high-fidelity RGB-D scenes with spatial audio and fine-grained 3D annotations.The dataset is designed for 3D audio-visual instruction tuning and systematic evaluation.
- Data simulation: SoundSpaces 2.0 generates synchronized FOA audio and RGB-D observations in scanned 3D environments with geometry-aware acoustic rendering.The simulation models direct sound and higher-order effects through room impulse responses, reflections, transmission, diffraction, and air absorption.
- Data simulation: Source–receiver pairs are sampled within the same room at 1–4 m, with obstacle clearance and a geodesic-distance constraint enforcing navigability.Pairs are retained only when geodesic distance is less than twice Euclidean distance.
- Task design: Perception tasks estimate source direction or ground sound-emitting objects, while reasoning tasks identify visual loudspeakers corresponding to target speech sources.Tasks A–B estimate azimuth and elevation, Task C predicts 3D boxes, and Tasks D–E match audio sources to visual candidates.
- Task design: Task C represents each 3D bounding box as bbox(c, x, y, z, sx, sy, sz) and requires outputs for all visible inserted loudspeakers.Randomly inserting 1–3 loudspeakers per scene enforces explicit geometric grounding.
4. Method
JAEGER combines depth-aware visual encoding with spatial audio representations to support joint 3D grounding and reasoning. Its audio pathway contrasts fixed Classical IV features with Neural IV, which learns geometric cues from raw FOA waveforms.
- Classical Intensity Vector: Classical IV derives directional features from FOA spectrograms using cross-spectra, real-part extraction, and concatenation across X, Y, and Z axes.The omnidirectional W channel is combined with each directional channel to construct active-intensity features.
- Neural Intensity Vector: Neural IV replaces fixed STFT processing with a learnable encoder that extracts geometric cues directly from raw FOA waveforms.A CNN encodes omnidirectional and directional channels, whose latent features are combined element-wise, concatenated, and projected by an MLP into a spatial embedding.
- 3D-aware Visual Encoding: The visual stream injects sinusoidal encodings of downsampled metric coordinates reconstructed from aligned RGB and depth inputs.Depth values are back-projected into 3D points, aligned to visual-feature resolution, and fused with visual embeddings.
- Architecture: JAEGER processes RGB-D and FOA inputs through visual and dual-path audio streams aligned for joint 3D grounding and reasoning.The visual stream adds geometric priors, while audio combines omnidirectional semantic features with Classical IV or Neural IV spatial cues.
- Architecture: The architecture feeds adapted multimodal features into an LLM, using MLP adapters and LoRA for efficient fine-tuning.This design connects explicit visual geometry and spatial audio with the model’s joint grounding and reasoning process.
5. Experiments
Experiments evaluate JAEGER across audio localization, 3D visual grounding, and joint reasoning, using task-specific metrics and comparisons against specialized and 2D baselines. Ablations show benefits from Neural IV and depth encoding, while removing FOA causes joint-reasoning performance to approach random.
- Evaluation setup: JAEGER is evaluated on audio DoA, 3D visual grounding, and joint audio-visual reasoning with task-specific angular-error, IoU, offset, and accuracy metrics.The baseline suite includes specialized models, Qwen2.5-Omni, and a random-guess lower bound.
- Main results: 2.21° MAE matches BAT's 2.16° on single-source localization, while overlapping-source MAE falls from BAT's 19.09° to 4.11°.The Neural IV variant is comparable in single-source precision and substantially better under overlapping sources.
- Main results: 0.32 3D IoU and 0.16m visual offset demonstrate competitive 3D visual-grounding performance.These are the reported JAEGER-3D results for volumetric overlap and center-location error.
- Main results: Explicit 3D modeling separates JAEGER from 2D AV-LLMs, which fail to reliably associate visual objects with acoustic sources in 3D space after fine-tuning.JAEGER achieves near-perfect accuracy in both single- and overlapping-source joint-reasoning scenarios.
- Ablation studies: Neural IV consistently lowers angular error and remains more stable than Classical IV when training and testing source scenarios differ.The cross-evaluation compares matched and mismatched single- and overlapping-source conditions.
- Ablation studies: Depth encoding improves Mean 3D IoU from 0.29 to 0.32 and reduces Median Visual Offset from 0.18m to 0.16m.The study attributes these gains to explicit geometric priors that map 2D features to metric 3D coordinates.
- Ablation studies: Removing the FOA encoder reduces joint-reasoning accuracy to approximately 43–47%, while Neural IV reaches 99.2% versus Classical IV's 98.6% in the 2-speaker scenario.Removing depth encoding also lowers accuracy, supporting contributions from both spatial audio and depth.
6. Conclusion
JAEGER bridges 2D-centric AV-LLMs and simulated 3D environments by combining RGB-D perception with multi-channel FOA audio. Neural IV and SpatialSceneQA support robust spatial reasoning, while experiments identify explicit visual depth and spatial audio as indispensable for complex physical reasoning.
- Conclusion: JAEGER bridges 2D-centric AV-LLMs and simulated 3D environments through RGB-D visual perception and multi-channel FOA audio.The framework targets joint localization and reasoning beyond monaural, RGB-only systems.
- Conclusion: Neural IV is a data-driven acoustic representation designed to improve robustness in challenging, reverberant scenarios.It is presented as a core component of the framework's spatial-audio processing.
- Conclusion: SpatialSceneQA is introduced as a large-scale, high-fidelity dataset for 3D audio-visual instruction tuning.The dataset supports training and evaluation of the proposed 3D audio-visual framework.
- Conclusion: Experiments show that explicit modeling of visual depth and spatial audio is indispensable for complex physical reasoning tasks.The conclusion frames these modalities as necessary within the evaluated setting.
Impact Statement
The paper targets machine-learning systems that perceive and reason about 3D environments using synchronized visual geometry and spatial audio. Its evaluation is primarily simulation-based, and real-world deployment raises transfer, privacy, and governance considerations.
- Impact: The paper aims to advance 3D perception and reasoning from synchronized visual geometry and spatial audio for embodied AI and related applications.Potential application areas include assistive perception and human–machine interaction in spatially complex environments.
- Limitations: Simulation-based evaluation may not transfer cleanly to real acoustics, microphones, RGB-D sensors, synchronization pipelines, or calibration settings without additional validation.The stated boundary concerns real-world sensing and deployment conditions.
- Responsible deployment: Invasive localization or surveillance could create privacy risks, motivating consent, data minimization, limited retention, access control, and documentation of sensor limitations.These safeguards are recommended for real-world deployment.
A. Real-World Transfer on STARSS23
A small STARSS23 experiment tests whether SpatialSceneQA-pretrained spatial-audio representations transfer beyond simulation. On a limited single-speaker subset, pretraining improves real-world FOA localization, especially for elevation.
- Transfer setup: The STARSS23 transfer experiment uses 753 clips and 3.84 hours of real FOA audio from approximately stationary single-speaker clips.The subset was constructed to match the source-localization evaluation protocol more closely.
- Transfer results: SpatialSceneQA pretraining improves both Neural IV and Classical IV on real FOA localization.The reported transfer comparison evaluates models with SpatialSceneQA pretraining against real-data localization performance.
- Transfer results: 4.8° versus 7.3° elevation error shows the clearest transfer gain for Neural IV after SpatialSceneQA pretraining.Azimuth remains more challenging under real acoustics, although pretraining reduces median azimuth error for both variants.
B. Matched-Input Architectural Ablation
The matched-input ablation tests whether JAEGER’s gains arise from its architectural treatment of 3D geometry and spatial audio rather than richer modalities alone. SimpleFuse uses the same RGB-D and FOA inputs but performs worse across localization, grounding, and joint reasoning.
- Ablation setup: SimpleFuse preserves RGB-D and FOA inputs while removing depth-aware visual encoding and structured FOA spatial modeling.Depth is copied and concatenated with RGB, while FOA channels are flattened into a monaural waveform.
- Results: SimpleFuse is consistently worse than both JAEGER variants across audio localization, visual grounding, and joint reasoning.
- Results: 67.79% and 46.59% accuracy were achieved by SimpleFuse on audio-visual reasoning, versus 99.5% / 98.6% for Classical IV and 99.5% / 99.2% for Neural IV.
- Interpretation: The ablation indicates that performance gains depend on how JAEGER represents and aligns 3D geometry and spatial audio, not merely on exposing richer inputs.
C. Comparison between FOA and Binaural Spatial Audio
This comparison evaluates FOA and binaural spatial audio within matched JAEGER-style pipelines, while a harder distractor setting tests reasoning beyond the original near-ceiling benchmark. FOA performs better in the matched comparison, but accuracy declines as visible candidates increase.
- Matched setup: The binaural JAEGER variant replaces the FOA spatial branch with Spatial-AST binaural embeddings while retaining the same Qwen audio embeddings.
- FOA versus binaural: 98.6% and 99.2% two-speaker reasoning accuracy were achieved by Classical IV and Neural IV, compared with 85.0% for JAEGER (BAT).
- FOA versus binaural: The matched results support FOA as a more effective spatial-audio representation than binaural audio in this benchmark setting, without implying universal superiority.
- Harder reasoning setting: The harder split increases visible speaker candidates from two to three toward four to six, using only 160 training samples per candidate-number setting.
- Harder reasoning setting: 87.5% to 80.0% accuracy was observed for Classical IV as candidates increased from four to six, while Neural IV fell from 95.0% to 72.5%.The results show that larger distractor sets make the reasoning task substantially more challenging.
- Coordinate convention: Azimuth is measured around the frontal view from −180° to 180°, with 0° forward and positive values rotating left.
- Coordinate convention: Elevation is measured from the horizontal plane over [−90°, 90°], with positive values indicating upward direction.
F. Diversity Analysis of Speaker Point Clouds
The diversity analysis generates 120 speaker point-cloud instances from a shared text prompt while varying random seeds to produce structural and morphological variation. A visualization of 32 samples qualitatively demonstrates this diversity.
- Dataset construction: 120 unique speaker instances were synthesized with Hunyuan3D-1 and divided into training, validation, and testing sets using an 8:1:1 ratio.
- Generation procedure: All instances used the text prompt “floor standing speaker,” while random seeds were iteratively changed to encourage morphological diversity.
- Qualitative evidence: Figure 4 displays 32 randomly selected samples from the 120 generated instances, showing distinct structural and morphological variations despite the shared prompt.