Source-linked AI summary
Bin2Ambi: Learning Ambisonic Soundfield Reconstruction from Head-Tracked Binaural Audio
Gavin Milner, Nils Peters
TL;DR
Binaural recordings are intended for headphone playback, while Ambisonics supports scene-based audio, and no method was known for converting between them. The paper defines Bin2Ambi and estimates DirAC spatial parameters from binaural audio and head-tracking data; results report reduced extreme directional errors and perceived spatial quality comparable to a DirAC oracle.
Problem
Binaural audio is intended for headphone playback, limiting its direct use in scene-based systems, and no method was known to convert it into First-Order Ambisonics.
Source: §1 · p. 1Method
The system estimates directional and diffuseness parameters of the DirAC model from binaural features and head-tracking data, then synthesizes Ambisonics signals.
Source: §2 · p. 1Results
Head rotation reduced extreme directional errors, and a perceptual test found spatial quality comparable to the DirAC oracle; the AW-MAE model reached a lowest WMAE of 11.82°.
Source: §4 · p. 4, §3.3.2 · p. 4Takeaways & Limitations
Bin2Ambi could make modern earphones low-barrier spatial recording devices for user-generated content.
Source: §1 · p. 1Future work will better estimate the omnidirectional component from binaural input and optimize the system for more varied acoustic conditions.
Source: §4 · p. 4
Abstract
from arXiv · showhide
User-generated content has become one of the most-consumed content types. However, capturing spatial audio with consumer hardware is still challenging. Given the widespread success of smart earbuds, binaural audio could be a promising option to capture spatial audio on consumer devices, but its inherent signal characteristic limits its usability as a recording format. In this paper, we propose and define a new task, Binaural to Ambisonics conversion (Bin2Ambi). In our proposed system, we exploit simultaneously captured head-tracking data provided from the motion sensors in smart earbuds. We show that this motion data help resolve the inherent directional uncertainty of two-channel binaural audio due to front-back localization ambiguities and lateral errors in the cone of confusion. Our results show that our system learns directional and diffuse-field information and that head-tracking especially reduces extreme localization errors. Objective metrics and a subjective listening test suggest that the converted Ambisonics soundfield achieves an average directional error of up to $11.8^\circ$ and a perceived spatial quality similar to a DirAC ground-truth model. The proposed algorithm can serve as a baseline for future improvements to this novel Bin2Ambi task.
1. INTRODUCTION
The paper introduces Binaural-to-Ambisonics conversion to address the lack of a method for turning binaural recordings into a playback-agnostic scene-based format. It situates the task among work on head-motion cues and neural spatial-audio conversion, then proposes estimating DirAC parameters from binaural features and head-tracking data.
1. INTRODUCTION
- No processing method was known for transforming two-channel binaural audio into four-channel First-Order Ambisonics, a playback-agnostic scene-based representation.Binaural recordings are intended for headphone playback, limiting their direct use in object-based and scene-based audio systems.Source: §1 · p. 1
- Bin2Ambi converts two-channel binaural audio into a periphonic Ambisonics soundfield, unlike fixed-channel upmixing because the input is spatial and head motion requires compensation.The task targets scene-based Ambisonics rather than a fixed channel layout.Source: §1 · p. 1
- The proposed conversion could make modern earphones low-barrier spatial recording devices for user-generated content.Source: §1 · p. 1
1.1. State of the Art
- Prior work used head movement to improve binaural direction-of-arrival estimation, while neural spatial-audio conversion had primarily addressed Ambisonics-to-binaural rendering.The paper cites head-rotation studies of front–back confusion and prior learning-based spatial audio methods.Source: §1.1 · p. 1
- The authors estimate DirAC spatial parameters from binaural features and head-tracking data, evaluating the model across scene complexity and noise with objective metrics and a listening test.Source: §1.1 · p. 1
2. PROPOSED ARCHITECTURE
The proposed system converts binaural audio and head-tracking data into first-order Ambisonics by estimating DirAC directional and diffuseness parameters. Its pipeline combines binaural features, a neural network, and DirAC-based synthesis.
2. PROPOSED ARCHITECTURE
- The system estimates DirAC direction and diffuseness from binaural audio features plus head movement, then synthesizes first-order Ambisonics.Head tracking also supports correction of soundfield rotations caused by head movement.Source: §2 · pp. 1–2
2.1. Binaural Feature Extraction
- Five STFT features—mean magnitude, ILD, sine and cosine IPD, and IC—are Mel-compressed alongside synchronized head orientation represented as unit quaternions.The features use recursive temporal smoothing where indicated.Source: §2.1 · p. 2
2.2. Neural Network Architecture
- A CRNN processes spectral and rotation features in parallel, fuses them before a GRU, then uses skip-connected decoding to regress DirAC direction and diffuseness.The decoder restores frequency resolution to 64 Mel bands.Source: §2.2 · p. 2
2.2.1. Training
- Training scenes spatialize speech, music, and sound events using source recordings partitioned before scene generation; per-TF-bin ground truth includes DirAC direction and diffuseness.The direction is derived from active intensity, and diffuseness describes the proportion of diffuse to total sound energy.Source: §2.2.1 · p. 2
2.2.2. Loss Function
- The loss weights direction targets by intensity and (1−diffuseness), jointly estimates diffuseness, and MAE improved resynthesis performance over MSE.The diffuseness term is weighted by β=0.2.Source: §2.2.2 · p. 3
2.3. Ambisonics Synthesis
- The decoder approximates omnidirectional pressure from binaural signals, splits it into direct and diffuse streams, synthesizes FOA components, then reconstructs audio by inverse STFT.Mutually orthogonal decorrelators generate the diffuse B-format components.Source: §2.3 · p. 3
3. EVALUATION
The evaluation tests Bin2Ambi on generated scenes under varied noise and complexity conditions, using objective metrics and a listening test. Results show reduced extreme errors with head rotation and spatial quality comparable to the DirAC ground truth.
3.1. Dataset Generation
- The generated dataset contains 12,000 eight-second scenes totaling over 26.5 hours, split evenly between scenes with and without background noise.Scenes used foreground speech, music, and sound events, with binaural audio simulated using KU-100 HRTFs and time-varying head rotations.Source: §3.1 · p. 3
3.2. Tested Conditions
- Because no directly comparable Bin2Ambi baseline existed, four trained models were assessed against FOA ground truth, alongside a DirAC oracle; evaluations used parameter and resynthesized-signal metrics plus listening tests.The listening-test FOA signals were rendered to a 9.1.4 loudspeaker setup.Source: §3.2 · p. 3, Table 1 · p. 3
3.3.1. Objective Metrics
- WMAE worsened with more simultaneous sources and background noise; rotation did not substantially improve mean WMAE but reduced Dataset A’s 99th percentile from 49.1° to 35.4°.The reported reduction suggests head rotation mainly reduces large localization errors rather than uniformly improving directional accuracy.Source: §3.3.1 · pp. 3–4, Figure 4 · p. 4
- AW-MAE tracked resynthesized DirAC ground truth more closely than AW-MSE, especially in sparse scenes; localization accuracy fell as complexity increased and diffuseness decorrelated FOA energy.AMBIQUAL Localization Accuracy is sensitive to phase differences.Source: Figure 5 · p. 4, §3.3.1 · p. 4
3.3.2. Ablation Study
- Adding recurrence reduced WMAE from 40.25° to 13.87°, the largest architectural gain; later changes were smaller, with AW-MAE reaching 11.82°.The ablation used Dataset A’s 600-item test set.Source: §3.3.2 · p. 4, Table 2 · p. 4
3.3.3. Subjective Test
- DirAC GT scored 86.5 and AW-MAE 85.4 MUSHRA points; corrected paired tests found no significant difference, with seven valid listeners.Scores were pooled across test items.Source: Figure 6 · p. 4, §3.3.3 · p. 4
4. CONCLUSION
The proposed system demonstrates the feasibility of Bin2Ambi, with directional accuracy depending on scene complexity and ambient noise. Head rotation reduces extreme directional errors, and perceived spatial quality is comparable to the DirAC oracle.
4. CONCLUSION
- The proposed system demonstrates the feasibility of Bin2Ambi, with directional accuracy dependent on scene complexity and ambient background noise.It predicts DirAC parameters from binaural audio features using a CRNN with a U-Net-style decoder.Source: §4 · p. 4
- Head rotation reduces extreme directional errors, while the proposed system achieves perceived spatial quality comparable to the DirAC oracle.Source: §4 · p. 4
- Future work will focus on better estimating the omnidirectional component and optimizing the system for more varied acoustic conditions.Source: §4 · p. 4