PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Deep Neural Networks and Head Movements for Robust Binaural Localisation of Multiple Sources in Reverberant Environments

Ning Ma, Tobias May, Guy J. Brown

arXiv:1904.03001v1eess.AScs.SD

TL;DR

Machine localisation is unreliable with multiple sources and reverberation, and full-azimuth systems face front–back confusions because binaural cues are similar across hemifields. This paper combines DNNs using the complete CCF and ILDs with head movements; it reports 96% average localisation accuracy with up to three competing talkers and room reverberation.

  • Problem

    Machine localisation is unreliable with interfering sources and reverberation, while localisation across the full 360° range is complicated by similar frontal and rear binaural cues.

    Source: §I · p. 1

  • Method

    The system maps the complete CCF and ILDs to source azimuths with DNNs and uses head movements to help resolve front–back ambiguity.

    Source: §I · pp. 1–2

  • Results

    96% average localisation accuracy under acoustic scenarios with up to three competing talkers and room reverberation; compared with GMM, the DNN substantially reduced localisation errors by exploiting the entire CCF.

    Source: §V · pp. 8–9

  • Takeaways & Limitations

    Multi-conditional training generalised to unseen reverberant conditions and mismatched artificial heads, while ILDs were necessary to reduce front–back confusions.

    Source: §V · p. 8

    The number of active sources is assumed known a priori, and the study considers only static sound sources.

    Source: §V · p. 9

Abstract

from arXiv · show

This paper presents a novel machine-hearing system that exploits deep neural networks (DNNs) and head movements for robust binaural localisation of multiple sources in reverberant environments. DNNs are used to learn the relationship between the source azimuth and binaural cues, consisting of the complete cross-correlation function (CCF) and interaural level differences (ILDs). In contrast to many previous binaural hearing systems, the proposed approach is not restricted to localisation of sound sources in the frontal hemifield. Due to the similarity of binaural cues in the frontal and rear hemifields, front-back confusions often occur. To address this, a head movement strategy is incorporated in the localisation model to help reduce the front-back errors. The proposed DNN system is compared to a Gaussian mixture model (GMM) based system that employs interaural time differences (ITDs) and ILDs as localisation features. Our experiments show that the DNN is able to exploit information in the CCF that is not available in the ITD cue, which together with head movements substantially improves localisation accuracies under challenging acoustic scenarios in which multiple talkers and room reverberation are present.

I. INTRODUCTION

Machine localisation remains unreliable with interfering sources and reverberation, despite humans localising such mixtures with ease. The paper proposes full-azimuth binaural localisation using DNNs and head movements to address this gap and front–back ambiguity.

I. INTRODUCTION

  • Earlier binaural systems commonly estimated ITDs and ILDs in frequency bands and used statistical models such as GMMs; multi-conditional training improved robustness to reverberation and interfering sources –.Prior work also explored head movement strategies, including a strategy that rotated the head toward the target,.Source: §I · p. 1
  • ITD/ILD cues resemble one another in the frontal and rear hemifields, producing front–back confusions that listeners rarely make.Listeners use head movements and pinna-related spectral cues to help resolve these confusions,,.Source: §I · p. 1
  • The proposed system combines DNN classifiers and head movements to localise multiple talkers across the full 360° azimuth range in reverberant environments.Unlike many prior machine systems, it is not restricted to the frontal hemifield.Source: §I · p. 1
  • The DNN uses the complete cross-correlation function (CCF) and ILD in each frequency band, rather than reducing the CCF to a single ITD estimate.The full CCF provides information for azimuth classification; earlier CCF approaches and assumed a target fixed at 0°.Source: §I · p. 2

II. SYSTEM

The system trains frequency-specific DNNs on binaural features and combines their azimuth outputs across frequency and time. It uses the complete CCF with ILD, then integrates the DNN posteriors to estimate source activity by azimuth.

II. SYSTEM

  • During training, clean speech is spatialised with HRIRs and diffuse noise is added; during testing, a moving binaural receiver processes mixtures of several talkers.The test-time DNN output is combined with a head-movement strategy to localise sources in reverberant environments.Source: §II · p. 2

A. Binaural feature extraction

  • The 34-dimensional feature vector in each frequency band comprises the full CCF and ILD, avoiding peak-picking a single ITD estimate.The CCF can show azimuth-related changes in its main and side peaks, which a suitable classifier can exploit even with multiple sources.Source: A. Binaural feature extraction · p. 2

B. DNN localisation

  • A separate DNN maps each frequency band’s 34-dimensional features to azimuth posteriors, which are integrated across frequency and averaged across time to estimate source activity.Frequency-specific modelling exploits the premise that each local time-frequency region is mostly dominated by one source, allowing training on single-source data.Source: B. DNN localisation · pp. 2–3

C. Localisation with head movements

The head-movement strategy addresses ambiguous azimuth posteriors by comparing predictions before and after a rotation. Consistent, rotation-shifted peaks are retained as likely source locations, while phantom peaks are removed.

C. Localisation with head movements

  • When a posterior has ambiguous peaks, the system triggers a random head movement between −30° and 30° to help resolve front–back confusion.The initial posterior may include phantom peaks because binaural features are similar in the front and rear hemifields.Source: C. Localisation with head movements · p. 3
  • A true-source peak should reappear after head rotation shifted by the rotation angle, whereas a phantom peak should not; unmatched peaks are removed before the posteriors are averaged.The resulting prominent peaks are treated as active source positions, with the number of active sources assumed known a priori.Source: C. Localisation with head movements · p. 3
  • The strategy uses late fusion of model predictions, avoiding head rotation during training and preserving information that early fusion of cross-correlation patterns may lose.The paper contrasts this with, which was tested in anechoic conditions with a single source, while the proposed system handles overlapping sources in reverberant conditions.Source: C. Localisation with head movements · p. 3

III. EVALUATION

The evaluation used anechoic KEMAR HRIRs for training and reverberant BRIRs from Surrey and TU Berlin for testing. It covered multiple room conditions, source configurations, and a cross-head training–testing mismatch.

A. Binaural simulation

  • Training used anechoic KEMAR HRIRs for 72 azimuths spanning 0°–355° in 5° steps, while models were not retrained for room conditions.Surrey BRIR testing included four reverberant rooms and an anechoic condition; TU Berlin Auditorium3 recordings added a mid-size lecture room with T60 of 0.7 s.Source: A. Binaural simulation · p. 4
  • Auditorium3 evaluation selected five source positions, including one in the rear hemifield, from recordings made across head orientations.The room's estimated T60 was 0.7 s; Figure 4 identifies source distances, azimuths, and respective T60 values.Source: A. Binaural simulation · p. 4, Figure 4 · p. 4
  • Training and Surrey testing used different dummy heads, but the authors report that MCT systems generalized well across this HRIR mismatch.Training used KEMAR; Surrey BRIRs used Cortex HATS.Source: A. Binaural simulation · p. 4
  • Binaural mixtures were formed by spatializing sources separately and adding them in both channels; Auditorium3 source levels were scaled before mixing to approximately equalize direct-sound levels.The level adjustment addressed differences in impulse-response amplitude across source positions; it was not applied to Surrey BRIRs.Source: A. Binaural simulation · p. 4
  • Surrey source positions covered only ±90°, although the system could report 72 azimuths around the head; testing head movements were limited to ±30°.Figure 3 also illustrates a typical three-talker mixture with sources at −50°, −30°, and 15°.Source: Figure 3 · p. 4

B. Head movement simulation

Head movements were simulated by changing source azimuths relative to head orientation and selecting corresponding BRIRs. The simulation differed between Surrey and Auditorium3 recordings, and movement was limited to ±30°.

B. Head movement simulation

  • For Surrey, head movement was approximated by computing source azimuths relative to head orientation and loading BRIRs at those relative azimuths.The approximation reflects that Surrey BRIRs were measured by moving loudspeakers around a fixed dummy head.Source: B. Head movement simulation · pp. 4–5
  • Auditorium3 movements were simulated more realistically by loading the BRIR corresponding to the desired head orientation.Source: B. Head movement simulation · p. 5
  • Head movements were limited to ±30° in all experiments.Source: B. Head movement simulation · p. 5

C. Multi-conditional training

The models assumed no prior room knowledge and were trained with anechoic HRIRs plus diffuse noise, rather than reverberant BRIRs. Multi-conditional training mixed target signals with diffuse noise across signal-to-noise ratios.

C. Multi-conditional training

  • The localisation models assumed no prior knowledge of room conditions and used no reverberant BRIRs during training.Training used anechoic HRIRs with added diffuse noise.Source: C. Multi-conditional training · p. 5
  • Binaural MCT features mixed a target at a specified azimuth with diffuse noise at various SNRs.The diffuse noise summed 72 uncorrelated white Gaussian noise sources spatialized across 360° in 5° steps.Source: C. Multi-conditional training · p. 5
  • Diffuse-noise training was preferred to adding reverberation because prior studies suggested it would generalize better across varied reverberant test conditions.The target and diffuse noise used the same anechoic KEMAR HRIR.Source: C. Multi-conditional training · p. 5
  • The training used TIMIT sentences spatialized at 72 azimuths and corrupted with diffuse noise at 20, 10, and 0 dB SNR; only features above −5 dB a priori SNR were retained.The criterion was intended to capture high-frequency multimodal clusters caused by periodic cross-correlation ambiguities.Source: C. Multi-conditional training · p. 5

D. Experimental setup

The experiments compared GMM and DNN localisation systems across training strategies, feature sets, and head-movement conditions. Evaluation used speech mixtures and a gross accuracy criterion based on azimuth error.

D. Experimental setup

  • The GRID evaluation used separate sets of 50 mixtures with one, two, or three simultaneous talkers, holding mixtures constant across azimuth and room conditions.The central 1 s of each sentence was evaluated, and sources in multi-talker mixtures were separated by at least 10°.Source: D. Experimental setup · p. 5
  • The baseline was a GMM using ITD and ILD features, compared with a DNN using CCF and ILD features.Each DNN had four layers, including two hidden layers of 128 nodes each.Source: D. Experimental setup · p. 5
  • Systems were tested with clean training or MCT, different feature sets, and with or without head movements, using the same 1 s signal duration in both movement conditions.With movement, the signal was split into two 0.5 s blocks, with the second provided after the movement.Source: D. Experimental setup · p. 5
  • Gross localisation accuracy counted estimates within 5° of the true azimuth, with the number of active sources assumed known in advance.The authors preferred this measure to RMS error because full 360° localisation can produce large errors from front-back confusions.Source: D. Experimental setup · p. 5

A. Influence of MCT

MCT was evaluated in frontal-only and full-360° localisation, where front-back confusions could occur. Its benefits were clearest in the full-range task and included fewer front-back errors.

A. Influence of MCT

  • The experiment compared frontal-only estimates in [−90°, 90°] with unrestricted estimates across 360°, where front-back errors could occur.Table II covers one, two, and three competing talkers across BRIR sets in both localisation ranges.Source: A. Influence of MCT · pp. 5–6, Table II · p. 6
  • In frontal-only localisation, clean-trained DNNs averaged 97.8% accuracy, while MCT-trained GMMs averaged 97.4% versus 75.6% with clean training.MCT particularly helped GMMs in multi-talker and reverberant conditions; DNN gains appeared only in multi-talker scenarios.Source: A. Influence of MCT · p. 6, Table II · p. 6
  • In full 360° localisation, MCT raised average accuracy from 62.9% to 92.6% for GMMs and from 87% to 95% for DNNs.The largest benefits occurred in Room B, with the lowest direct-to-reverberant ratio, and Room D, with the longest reverberation time.Source: A. Influence of MCT · p. 6, Table II · p. 6
  • MCT reduced both front-back confusions and other errors from reverberation and overlapping talkers in full-range localisation.The authors attribute improved discrimination of front-back pairs to better regularisation of binaural features during training.Source: A. Influence of MCT · p. 6
  • With MCT, anechoic localisation of one or two sources reached 100% despite using KEMAR heads for training and HATS heads for testing.The authors suggest MCT also reduced sensitivity to receiver mismatch.Source: A. Influence of MCT · p. 6

B. Contribution of the ILD cue

This experiment compares localisation features, including the contribution of ILDs. The DNN performed better than the GMM with CCF-ILD features, while ILDs reduced front-back errors.

B. Contribution of the ILD cue

  • Without ILDs, average accuracy fell from 92.6% to 84.8% for GMMs and from 95% to 92.7% for DNNs.The GMM decline was particularly pronounced in the strongly reverberant Rooms B and D; the DNN’s CCF feature was more robust than the GMM’s ITD feature in reverberant multi-talker conditions.Source: B. Contribution of the ILD cue · p. 6, Table III · p. 7
  • The DNN’s full CCF supplied richer azimuth information than a single ITD estimate, which relies on peak picking that may be unreliable in noise and reverberation.The comparison was made without active head movement, using MCT-trained systems.Source: B. Contribution of the ILD cue · pp. 6–7
  • Without ILDs, errors increased largely through front-back confusions; adding ILDs greatly reduced those errors across conditions.The authors attribute the learned front-back asymmetry to the combination of ILDs with ITDs or ITD-related CCF features.Source: B. Contribution of the ILD cue · p. 7, Table III · p. 7
  • With CCF-ILD features, DNN accuracy averaged 95%, compared with 88.5% for GMMs, which struggled to model correlated CCF features.The GMM also performed worst in the more reverberant Rooms B and D.Source: B. Contribution of the ILD cue · p. 7, Table III · p. 7

C. Benefit of the head movement strategy

The head-movement strategy reduced front-back errors in one-, two-, and three-talker localisation. The DNN also performed strongly at rear azimuths in the Auditorium3 evaluation.

C. Benefit of the head movement strategy

  • Head movement reduced front-back errors and brought single-talker localisation close to 100% accuracy; it also improved overlapping-talker localisation by removing many such errors.With overlapping talkers, errors other than front-back confusions remained because only partial evidence was available for each source.Source: C. Benefit of the head movement strategy · p. 7, Figure 7 · p. 8
  • Without head movement, GMM errors at lateral azimuths exceeded 20% even in Room A, while central-azimuth errors were mostly front-back confusions and rose in Rooms B and D.The authors suggest central front-back pairs are less distinguishable by binaural time and level differences than more lateral pairs.Source: C. Benefit of the head movement strategy · p. 7
  • In Auditorium3, the DNN achieved near 100% accuracy for single-source localisation, including the rear 131° source; GMM error at that rear location was about 5%.For two-source localisation at 131°, head movement reduced DNN error to 8%, while GMM error remained 20%.Source: C. Benefit of the head movement strategy · p. 8
  • Across azimuths, DNNs significantly outperformed GMMs in Auditorium3, and both systems generally benefited from head movement in two- and three-source tasks.The GMM benefit was especially pronounced at 51°: its error fell from 14% to 4% in two-source localisation and from 36% to 14% in the passage’s stated two-source localisation comparison.Source: C. Benefit of the head movement strategy · p. 8, Table IV · p. 8
  • The 51° and 131° locations were harder to localise with multiple sources, likely because of their room acoustics and greater distance from the listener.Overlapping talkers and stronger reverberation made these locations more difficult.Source: C. Benefit of the head movement strategy · p. 8

V. CONCLUSION

The DNN framework combines head movements with binaural cues to localise multiple sources across 360° in reverberant conditions. With head rotation, it achieved 96% average localisation accuracy with up to three competing talkers.

V. CONCLUSION

  • Multi-conditional training (MCT) improved robustness to reverberation and generalised to unseen reverberant conditions and mismatched artificial heads.The model was trained with anechoic signals.Source: §V · p. 8
  • Interaural level differences (ILDs) were necessary to reduce front-back confusions in reverberant rooms, and head rotation further increased system robustness.Source: §V · p. 8
  • 96% average localisation accuracy was achieved with up to three competing talkers and room reverberation when head rotation was used.The DNN exploited the full cross-correlation function (CCF), substantially reducing localisation errors compared with a GMM-based system.Source: §V · pp. 8–9
  • Possible extensions include adding interaural coherence, monaural localisation cues, or visual features to the localisation system.Source: §V · p. 9
  • In the robotic interaction implementation, audio acquired during head rotation was not processed because rotation can produce self-noise that makes audio unusable.Source: §V · p. 9
  • The current systems assume the active-source count is known in advance and address static sources; estimating source count and tracking moving sources are future directions.Source: §V · p. 9
Loading 1904.03001v1…