PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Field-of-View for Binaural Signal Matching with Wearable Microphone Arrays

Matan Yifrach, Boaz Rafaely

arXiv:2609.28343v1eess.AS

TL;DR

Wearable binaural reproduction must work with limited microphone arrays, while signal-dependent approaches can rely on explicit scene estimates. FoV-BSM-Net learns FoV weighting from microphone signals and integrates it with BSM; it improves on BSM across objective binaural measures and maintains a perceptual advantage over both baselines at high DRR.

  • Problem

    Binaural reproduction for wearable arrays must work with few microphones, while existing signal-dependent methods rely on explicit acoustic-scene parameter estimation.

    Source: §I · pp. 1–2

  • Method

    FoV-BSM-Net learns directional FoV parameters from microphone signals and integrates the resulting weighting into BSM without directly estimating acoustic-scene parameters.

    Source: §I · p. 2

  • Results

    FoV-BSM-Net consistently improves over BSM in binaural NMSE and interaural-cue errors as DRR rises, matching the reference perceptually at low DRR and outperforming both baselines at high DRR.

    Source: C. Results · p. 10, §VIII · p. 11

  • Takeaways & Limitations

    Learned FoV parameters tend toward narrower azimuth and wider elevation apertures than the fixed baseline, suggesting less need for broad azimuthal coverage.

    Source: §VIII · p. 11

    Evaluation was limited to simulated single-source scenarios; real-world recordings and multi-source conditions remain future work.

    Source: §VIII · p. 11

Abstract

from arXiv · show

The growing use of spatial audio in applications such as augmented and virtual reality has driven the development of binaural reproduction methods for wearable arrays with a limited number of microphones. Binaural signal matching (BSM) is one such method, producing high-quality binaural signals under a diffuse-field assumption, but degrading at high direct-to-reverberant ratios (DRR) where the direct sound dominates. Previous extensions incorporate Field-of-View (FoV) weighting, either with fixed apertures or based on explicit source localization, but these approaches are limited by coarse spatial coverage or reliance on localization estimation accuracy. This paper introduces FoV-BSM-Net, a signal-dependent FoV-BSM formulation that avoids explicit source estimation by learning the FoV parameters end-to-end from the microphone signals using a Convolutional Recurrent Neural Network. The method is evaluated in simulated rooms across varying reverberation conditions, and compared against BSM and a fixed FoV-BSM baseline. Results show that FoV-BSM-Net consistently improves over BSM, with gains that grow with DRR in both binaural NMSE and interaural cue errors, and are further supported by perceptual evaluation showing a substantial advantage over both baselines across low and high DRR conditions.

I. INTRODUCTION

The paper addresses binaural reproduction with few-microphone wearable arrays, proposing a signal-dependent FoV enhancement to BSM without explicit acoustic-scene estimation.

I. INTRODUCTION

  • HOA binaural reproduction can provide high-quality signals, but typically requires spherical arrays with many microphones, making capture with few-microphone arrays challenging –.Source: §I · p. 1
  • Conventional BSM is signal-independent and assumes a diffuse sound field, limiting its performance in non-diffuse conditions –.Source: §I · p. 1
  • Parametric approaches reproduce direct sound –, while signal-dependent BSM incorporates ambient BSM –; these approaches rely on DOA and source-signal estimates.Such dependence may limit performance with multiple sources and reverberation.Source: §I · pp. 1–2
  • FoV-BSM-Net uses a DNN to predict directional FoV weights from signals, integrating them into BSM without directly estimating acoustic-scene parameters.The paper evaluates the approach using simulations and a listening test.Source: §I · p. 2

A. Microphone Array Signal Model

  • The array signal model represents microphone measurements as a vector generated by directional source signals through array steering vectors, with additive noise.Source: A. Microphone Array Signal Model · p. 2

B. Binaural Signal Model

  • The binaural model represents ear signals as source signals convolved with direction-specific HRTFs for the listener’s left and right ears.Source: B. Binaural Signal Model · p. 2

C. Binaural Signal Matching (BSM)

BSM filters microphone signals to reproduce target binaural signals by minimizing mean-squared error, with a closed-form solution under diffuse-field-like covariance assumptions.

C. Binaural Signal Matching (BSM)

D. Field of View (FoV) BSM

FoV-BSM weights directions within a defined angular region; the proposed method makes its FoV signal-dependent by predicting its parameters from microphone signals.

D. Field of View (FoV) BSM

  • FoV-BSM weights steering directions and HRTFs according to a region around a front-looking axis, emphasizing directions within the FoV.The weighted steering matrix and HRTF vector are used to form the FoV-BSM filter.Source: D. Field of View (FoV) BSM · p. 3

A. The proposed approach

  • Unlike the earlier fixed FoV, the proposed signal-dependent FoV is tailored to the target speaker’s position.Source: A. The proposed approach · p. 3
  • Directly estimating source direction can be error-prone under reverberation or noise, and classical DOA methods do not provide the FoV’s width and height.Source: A. The proposed approach · p. 3

III. PROPOSED METHOD

  • A DNN regresses the FoV centre and angular apertures from microphone-array segments, and the resulting weighting matrix is integrated into BSM filters.The pipeline maps a spatial representation of microphone signals to FoV parameters and then to BSM filters.Source: A. The proposed approach · p. 3, §III · p. 3

B. Input representation

The FoV-BSM-Net front end represents microphone signals with magnitude and inter-channel phase features, maps them to FoV parameters, and uses those parameters in BSM filter design.

B. Input representation

  • The input tensor stacks normalized log-magnitude spectrograms from all microphones with normalized inter-channel phase differences for the non-reference microphones.Source: B. Input representation · p. 4

C. Network architecture

  • The centre is predicted as a unit-norm vector and converted to spherical angles, while bounded aperture outputs ensure valid FoV regions.Source: C. Network architecture · p. 4

D. Integration with BSM framework

  • A differentiable soft mask replaces the hard FoV indicator, allowing predicted weights to be integrated into BSM filters and trained end-to-end with a binaural-domain objective.The predicted FoV parameters define a diagonal weighting matrix used in the BSM filter design.Source: D. Integration with BSM framework · p. 4

C. Network architecture

  • A ResNet-style convolutional encoder and bidirectional GRU produce a temporal embedding that feeds three regression heads for FoV parameters.The front end maps input tensor S to FoV parameters Θ.Source: C. Network architecture · p. 4, Figure 2 · p. 5

IV. ERROR MEASURES

The section evaluates binaural reproduction with complementary signal and interaural-cue measures. Binaural NMSE uses different error criteria below and above 1.5 kHz, while ITD is measured after low-pass filtering the binaural signals.

A. Evaluation metrics

  • The evaluation uses binaural NMSE alongside ITD, ILD, ICLD, and IPD errors to assess reproduction quality.The measures provide complementary signal and interaural-cue evaluations.Source: A. Evaluation metrics · p. 5

1) Binaural NMSE:

  • Binaural NMSE preserves full complex error below 1.5 kHz and evaluates only magnitude error above that cutoff.The frequency split is motivated by duplex theory and is consistent with the MagLS criterion used in BSM filter design.Source: 1) Binaural NMSE · p. 5

2) ITD:

  • ITD is obtained from low-pass-filtered binaural signals, using a 1.5 kHz cutoff.The cutoff matches the frequency range relevant for binaural time-difference perception.Source: 2) ITD · p. 5

3) ILD:

ILD is evaluated over ERB filter bands, while ICLD and IPD errors compare estimated and reference interaural cues. IPD errors are wrapped to the principal phase interval and reported in degrees.

3) ILD:

  • ILD is computed over equivalent rectangular bandwidth (ERB) filter bands, with error averaged across the bands of interest.The band-level ILD uses an ERB filter centered at fc and bounded by its upper frequency limit.Source: 3) ILD · pp. 5–6

4) ICLD error:

  • ICLD error is the absolute deviation between estimated and reference interaural channel level differences.Per-direction ICLD errors are averaged across frequency over [1.5, 8] kHz.Source: 4) ICLD error · p. 6, 5) IPD error · p. 6

5) IPD error:

  • IPD error is the absolute deviation between estimated and reference phases, wrapped to (−π, π] and reported in degrees.Per-direction IPD errors are averaged over frequency in [0, 1.5] kHz.Source: 5) IPD error · p. 6

B. Training loss

FoV-BSM-Net is trained with a composite loss targeting spatial accuracy, perceptual fidelity, and spectral distortion. Its terms supervise the FoV centre, interaural level and time differences, and binaural NMSE.

B. Training loss

  • The three-term composite training loss targets spatial accuracy, perceptual fidelity, and spectral distortion.Its components are built from evaluation metrics with adjustments for end-to-end training.Source: B. Training loss · p. 6
  • The DOA loss supervises predicted FoV centre angles against the dominant source's ground-truth direction, anchoring the FoV centre.Source: B. Training loss · p. 6
  • The ILD/ITD loss uses a differentiable soft-argmax instead of the discrete arg max and compares estimated cues with reference cues.The reference is evaluated at the ground-truth source direction using the system's binaural transfer function.Source: B. Training loss · p. 6
  • The NMSE loss follows the evaluation measure, while IPD and ICLD are reserved for per-direction evaluation rather than training.Computing the cue loss from HRTFs rather than binaural signals decouples it from source content and room response.Source: B. Training loss · p. 6

V. SIMULATION STUDY

The simulation study uses image-source room simulations with a five-microphone wearable array and measured HRTFs for binaural references. Its Monte Carlo dataset samples indoor room conditions, speech, and source configurations.

A. Setup

  • Room simulations use the image-source method, and binaural references use measured KU100 HRTFs from the Cologne database.The HRTFs are co-located with the array and applied using the same propagation paths as the array simulation.Source: A. Setup · pp. 6–7
  • The study simulates a single speech source with a five-microphone Aria-reduced wearable array.The layout uses two lower-lens, two front-temple, and one nose-bridge microphone, consistently modeled in simulation and steering vectors.Source: A. Setup · p. 7

B. Dataset

  • The dataset samples rooms with T60 values of 0.30, 0.61, and 1.40 seconds and uses randomized room configurations.Each scenario contains a fixed 3-second multichannel segment sampled at 16 kHz.Source: B. Dataset · p. 7
  • The Monte Carlo dataset contains 12,000 scenarios, split into 9,600 training, 1,200 validation, and 1,200 test samples.The split is designed to balance spatial and acoustic conditions.Source: B. Dataset · p. 7

C. Methodology

FoV-BSM-Net predicts spatial focus from microphone signals and integrates it into BSM filters; simulations compare it with signal-independent BSM and a fixed FoV-BSM baseline. Across tested DRR conditions, it improves binaural NMSE over both baselines, with larger gains at high DRR.

C. Methodology

  • FoV-BSM-Net predicts FoV parameters from microphone signals and uses them in BSM filters; all three methods share array geometry, HRTFs, MagLS regime, regularization, and STFT settings.The network is trained with a composite loss and early stopping on validation loss.Source: C. Methodology · pp. 7–8

A. FoV baseline parameter study

  • The fixed FoV-BSM baseline uses a front-facing (60° × 20°) aperture; an ad-hoc search selected β = 0.2, with performance locally stable around the operating point.The center is fixed at the front-facing prior, and the reported sensitivity is within a fraction of a dB of the best value for β and azimuth aperture on both ears.Source: C. Methodology · p. 7, A. FoV baseline parameter study · p. 8

B. Binaural NMSE evaluation

  • From approximately 200 Hz onward, FoV-BSM-Net’s frequency-dependent binaural NMSE is at or below both baselines, with a growing margin over BSM above the 1.5 kHz crossover.FoV-BSM closely follows BSM across the frequency range.Source: Figure 3 · p. 8, B. Binaural NMSE evaluation · p. 8
  • At high DRR, FoV-BSM-Net improves binaural NMSE by 1.7 dB over BSM, versus 0.8 dB for FoV-BSM; across all tested DRR conditions, gains are 1.0 dB and 0.5 dB, respectively.The evaluation covers the full test dataset.Source: B. Binaural NMSE evaluation · p. 8

C. FoV-BSM-Net learned parameters

The learned FoV centers are mostly close to source directions, while the network favors narrower azimuth and wider elevation apertures. In an azimuth sweep, FoV-BSM-Net has lower mean IPD and ICLD errors than both baselines; the listening experiment compares subjective quality in two simulated conditions.

C. FoV-BSM-Net learned parameters

  • Learned center-direction deviations are mostly below the 5° × 3° training-grid resolution, and the network favors wider elevation and narrower azimuth apertures than fixed FoV-BSM.The authors suggest the wider elevation compensates for residual uncertainty.Source: C. FoV-BSM-Net learned parameters · pp. 8–9

D. IPD and ICLD error analysis

  • Cue errors are measured over 121 source azimuths from −60° to +60° in a room with T60 = 0.61 s, using direct-sound transfer functions derived from filters designed for reverberant scenes.Estimated transfer functions are compared with anechoic KU100 HRTFs at matching directions.Source: D. IPD and ICLD error analysis · p. 9
  • FoV-BSM-Net reduces mean ICLD error by 5.8 dB and IPD error by 34.1° relative to BSM; FoV-BSM reductions are 3.8 dB and 26.3°, respectively.FoV-BSM-Net also has smaller cue errors across wider angles, while fixed-aperture FoV-BSM ICLD error rises toward the edges of its 60° azimuth region.Source: Table V · p. 9, D. IPD and ICLD error analysis · p. 9, Figure 4 · p. 9

VII. LISTENING EXPERIMENT

  • The MUSHRA listening experiment compares the methods in two simulated scenarios: high DRR (12.1 dB) and low DRR (4.4 dB).Each scenario forms one test screen.Source: A. Setup · p. 10, §VII · p. 10

B. Methodology

The listening test found that FoV-BSM-Net matched the reference at low DRR and retained the strongest perceptual quality at high DRR. The evaluation was limited to simulated single-source scenarios.

B. Methodology

  • Twelve normal-hearing subjects rated the hidden reference and three methods on overall spatial and timbral quality in randomized MUSHRA screens.Scores ranged from 0 to 100; Fig. 5 plots score distributions for the low- and high-DRR scenarios.Source: B. Methodology · p. 10, Figure 5 · p. 11

C. Results

  • Method, Scenario, and their interaction had significant effects on listening scores; method rankings depended on the acoustic scenario.A repeated-measures ANOVA was used, with Greenhouse–Geisser correction for the Method factor's sphericity violation.Source: C. Results · p. 10
  • At low DRR (4.4 dB), FoV-BSM-Net was statistically indistinguishable from the reference, while BSM was clearly worse.FoV-BSM was rated close to the reference with a small but significant gap; FoV-BSM-Net versus FoV-BSM was not significant.Source: C. Results · p. 10
  • At high DRR (12.1 dB), FoV-BSM-Net was rated significantly higher than FoV-BSM, while all three methods differed from the reference.The FoV-BSM-Net versus FoV-BSM comparison was significant (t(11) = 3.23, pcorr = .016).Source: C. Results · p. 10

VIII. CONCLUSIONS

  • FoV-BSM-Net matched the reference at low DRR and retained the highest perceptual quality at high DRR.The evaluation was limited to simulated single-source scenarios.Source: C. Results · p. 10, §VIII · p. 11
Loading 2609.28343v1…