Source-linked AI summary
Deep Neural Networks for Multiple Speaker Detection and Localization
Weipeng He, Petr Motlicek, Jean-Marc Odobez
TL;DR
The paper addresses simultaneous detection and localization of multiple sound sources in real human-robot interaction, where prior neural-network methods largely focused on single-source localization. It proposes likelihood-based output coding, sub-band cross-correlation features, and three neural-network architectures, achieving approximately 90% precision and recall for unknown-source-count detection and localization while outperforming traditional spatial spectrum methods. The study is limited by training data that may not cover all source-position combinations.
Problem
Prior neural-network SSL methods focused mainly on single-source localization and did not adequately address simultaneous detection and localization of multiple voices in real HRI.
Method
The paper uses likelihood-based output coding, investigates sub-band cross-correlation features, and evaluates three neural-network architectures on real robot-recorded HRI data.
Results
The proposed methods achieve approximately 90% precision and recall for simultaneous detection and localization with an unknown number of sources on both datasets, outperforming baseline methods.
Takeaways & Limitations
Likelihood-based coding enables neural networks to detect an arbitrary number of overlapping sound sources in the evaluated HRI setting.
Takeaways & Limitations
Training data may not cover all possible source-position combinations because their number grows exponentially with the number of sources.
Abstract
from arXiv · showhide
We propose to use neural networks for simultaneous detection and localization of multiple sound sources in human-robot interaction. In contrast to conventional signal processing techniques, neural network-based sound source localization methods require fewer strong assumptions about the environment. Previous neural network-based methods have been focusing on localizing a single sound source, which do not extend to multiple sources in terms of detection and localization. In this paper, we thus propose a likelihood-based encoding of the network output, which naturally allows the detection of an arbitrary number of sources. In addition, we investigate the use of sub-band cross-correlation information as features for better localization in sound mixtures, as well as three different network architectures based on different motivations. Experiments on real data recorded from a robot show that our proposed methods significantly outperform the popular spatial spectrum-based approaches.
I. INTRODUCTION
Sound source localization and speaker detection support multi-party human-robot interaction, but real HRI involves noisy environments, multiple speakers, short utterances, and obstacles. Learning-based methods address localization without strong assumptions, yet prior neural-network approaches largely omit simultaneous detection and localization of multiple voices.
- SSL and speaker detection help robots determine where and who a speaker is for appropriate responses in multi-party HRI.
- Real HRI challenges include noisy environments, robot ego-noise, multiple simultaneous speakers, short utterances, and obstacles blocking direct sound paths.
- Traditional SSL methods rely on analytical assumptions about signals, noise, and environments that may not hold under these HRI conditions.
- Most learning-based SSL methods address direction-of-arrival estimation without strong assumptions but do not handle simultaneous detection and localization of multiple voices in real multi-party HRI.
B. Existing Neural Network-based SSL Methods
Existing neural-network SSL methods mainly localize a single source and emphasize localization accuracy. Extensions to multiple sources either require temporal pooling with a known source count or use an ambiguous two-source output assignment.
- Recent neural-network SSL methods achieve promising single-source localization but generally classify audio into one location label.
- Pooling posterior probabilities across multiple time frames can extend single-source networks to multiple sources, but requires a known source count and long input periods.
- A two-source approach encodes two marginal posterior vectors but uses ad-hoc location ordering, making source-to-vector assignments ambiguous.
C. Contributions
The paper develops neural-network methods for multiple sound-source localization in real human-robot interaction, combining likelihood-based modeling, cross-correlation features, and evaluation on robot recordings.
- The methods target short-input, overlapping-speech scenarios with an unknown number of sources and strong robot ego-noise.Experiments use real recordings from the robot Pepper.
- Three neural-network architectures use likelihood-based output encoding to handle an arbitrary number of sources.The architectures are based on different motivations.
- The paper releases a benchmark dataset of real recordings for developing and evaluating learning-based sound-source localization in human-robot interaction.
- The proposed input features include GCC-PHAT information, including full delay functions rather than only peak-based TDOA estimates.The experiments use the center 51 delays, τ ∈[-25, 25].
- GCC-PHAT on a mel-scale filter bank preserves delay information within frequency bands for sub-band analysis of speech mixtures.The implementation uses 40 mel-scale filters covering 100 to 8000 Hz; separate sources can produce high cross-correlation values in different bands.
B. Likelihood-based Output Coding
The paper encodes multiple-source localization as directional likelihoods rather than normalized posterior probabilities. This representation supports arbitrary source counts and peak-based decoding.
- Likelihood encoding: The output is a 360-value vector assigning each azimuth direction a sound-source likelihood.Values are defined from Gaussian-like functions centered on the true directions of arrival.
- Likelihood encoding: Likelihood coding can represent no source, one source, or N sources through zero or multiple directional peaks.Unlike posterior coding, it is not constrained to sum to one.
- Likelihood encoding: Soft assignment models correlations between adjacent directions, unlike binary posterior coding, and supports better neural-network generalization.The output layer is not normalized with a softmax function.
- Peak decoding: During testing, detected sources are decoded as output peaks above a threshold ξ while suppressing nearby peaks.The experiments use σ = σn = 8° for the coding and neighborhood distance.
C. Neural Network Architectures
The paper evaluates three neural-network architectures for multiple sound-source localization, using GCC-PHAT or sub-band GCCFB features. The two-stage model explicitly extracts frequency-wise directional features before aggregation.
- Architectures: Three architectures are investigated: MLP-GCC, CNN-GCCFB, and TSNN-GCCFB.They reflect different choices about input representation and feature sharing.
- MLP-GCC: MLP-GCC uses GCC-PHAT input with three fully connected ReLU-and-batch-normalization hidden layers and a sigmoid output layer.The sigmoid constrains outputs to the desired 0-to-1 range.
- CNN-GCCFB: CNN-GCCFB applies convolutional layers to high-dimensional sub-band cross-correlation features, using weight sharing to reduce parameters.Pooling is omitted because delay and frequency positions are important localization cues; stride-based downsampling is used instead.
- TSNN-GCCFB: TSNN-GCCFB first extracts latent direction-of-arrival features within each filter bank, then aggregates them across frequencies in neighboring DOA regions.Its design uses the structure of GCCFB rather than treating the feature map simply as an image.
- TSNN-GCCFB: TSNN-GCCFB is trained first through subnet-level latent-feature supervision and then end-to-end across both stages.Subnet 1 has two hidden layers, while Subnet 2 has one; all hidden layers have size 500.
III. EXPERIMENT
The experiment implements the proposed neural methods and compares them with traditional sound-source-localization approaches using data collected from a robot.
- Experimental comparison: The proposed methods are compared with traditional SSL approaches on robot-collected data.The experiment evaluates the implemented neural-network methods against conventional approaches.
A. Datasets
The study collects two real-data sets with Pepper: one using loudspeakers and one involving human subjects. Together they support evaluation across varied positions, rooms, speech overlap, and human-robot interaction.
- Datasets: Two real-data sets are collected for development and evaluation: a loudspeaker set and a human-subject set.Both sets are recorded with the Pepper robot.
- Recording platform: Pepper records four directional microphones arranged in a 5.8 × 6.9 cm rectangle at a 48 kHz sample rate.The microphone signals are strongly affected by fan noise inside the robot’s head.
- Loudspeaker dataset: The loudspeaker recordings use clean AMI-corpus speech placed at random robot and loudspeaker locations in rooms of different sizes.Automatic head movement increases the diversity of loudspeaker-to-robot positions.
- Human-subject dataset: The human-subject recordings contain interaction phrases, including both single utterances and overlapping speech.Voice-activity labels are manually annotated, while mouth positions are obtained automatically from person tracking and pose detection.
B. Evaluation Protocol
The evaluation measures localization when the source count is known and measures joint detection and localization when it is unknown.
- With a known source count, the highest output peaks are selected as predicted DOAs and matched one by one to ground truth DOAs.
- Known-source evaluation reports mean absolute error (MAE) and accuracy (ACC).A prediction is correct when its error is below the admissible error Ea.
- With an unknown source count, evaluation jointly considers detection and localization using predictions based on Eq. 4.
- Precision–recall curves are computed by varying the prediction threshold ξ.Precision is the proportion of correct predictions among all predictions, while recall is the proportion of correctly detected ground-truth sources.
C. Network Training
The networks are trained on loudspeaker data with MSE optimization, while several spatial spectrum-based methods provide comparison baselines under a fixed covariance-processing setup.
- Network Training: Training uses 506k frames containing no source, one source, or two sources from the loudspeaker dataset.
- Network Training: The neural networks use Adam optimization, mean squared error loss, and mini-batches of 256.
- Network Training: MLP-GCC and CNN-GCCFB are trained for ten epochs, while TSNN-GCCFB uses four first-stage epochs followed by ten end-to-end epochs.
- Baseline Methods: The comparison includes SRP-PHAT, SRP-NONLIN, MVDR-SNR, SEVD-MUSIC, and GEVD-MUSIC spatial spectrum-based methods.The baselines span steered-response, beamforming, and MUSIC-based approaches with the stated score or decomposition variants.
- Baseline Methods: All comparison methods compute empirical spatial covariance matrices from seven-frame, 2048-sample blocks with 50% overlap.Each block is 170 ms long.
E. Results
The proposed neural-network methods outperform the baseline spatial spectrum approaches for known-source localization and unknown-source detection-localization, while training-data coverage remains a scope limitation.
- Known Number of Sources: Less than 5° average error and more than 90% accuracy are achieved by all three proposed models on the loudspeaker dataset with known source counts.The best baseline, SRP-PHAT, reaches 21.5° error and 78% accuracy.
- Known Number of Sources: On human-subject data, baselines have slightly better MAE for single-source frames, but proposed methods achieve higher accuracy, especially with overlapping sources.
- Unknown Number of Sources: Approximately 90% precision and recall are achieved on both datasets for simultaneous detection and localization with an unknown number of sources.
- Unknown Number of Sources: TSNN-GCCFB performs best among the proposed models, particularly on overlapping frames; the authors attribute this to sub-band features and a two-stage structure.
- Overall Findings: The comprehensive evaluation reports that the proposed methods significantly outperform traditional spatial spectrum-based methods.
- Limitations: Training samples may not cover all source-position combinations because their number grows exponentially with the number of sources.The authors identify this as a potential limitation and propose future work on generalization with limited training data.