PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Speaker DOA Estimation Using Deep Convolutional Networks Trained with Noise Signals

Soumitro Chakrabarty, Emanuël A. P. Habets

arXiv:1807.11722v1eess.AScs.LGcs.SD

TL;DR

The paper addresses multi-speaker DOA estimation under adverse and changing acoustic conditions. It proposes a CNN that treats DOA estimation as multi-label classification, learns from microphone STFT phase maps, and uses synthesized-noise training. Experiments demonstrate robustness to unseen conditions and noise types, with M −1 convolution layers yielding the best localization performance for an M-microphone array.

  • Problem

    Accurate DOA estimation for multiple speakers is difficult in noise and reverberation, motivating supervised methods that can adapt through training.

  • Method

    A CNN performs multi-label DOA classification from microphone STFT phase components, with synthesized-noise training exploiting disjoint speaker activity in the STFT domain.

  • Results

    The method adapts to unseen acoustic conditions and noise types, while M −1 convolution layers provide the best localization performance for an M-microphone array.

  • Takeaways & Limitations

    The framework localizes speakers in unseen acoustic environments and noise types and supports dynamic scenarios with varying numbers of sources.

Abstract

from arXiv · show

Supervised learning based methods for source localization, being data driven, can be adapted to different acoustic conditions via training and have been shown to be robust to adverse acoustic environments. In this paper, a convolutional neural network (CNN) based supervised learning method for estimating the direction-of-arrival (DOA) of multiple speakers is proposed. Multi-speaker DOA estimation is formulated as a multi-class multi-label classification problem, where the assignment of each DOA label to the input feature is treated as a separate binary classification problem. The phase component of the short-time Fourier transform (STFT) coefficients of the received microphone signals are directly fed into the CNN, and the features for DOA estimation are learnt during training. Utilizing the assumption of disjoint speaker activity in the STFT domain, a novel method is proposed to train the CNN with synthesized noise signals. Through experimental evaluation with both simulated and measured acoustic impulse responses, the ability of the proposed DOA estimation approach to adapt to unseen acoustic conditions and its robustness to unseen noise type is demonstrated. Through additional empirical investigation, it is also shown that with an array of M microphones our proposed framework yields the best localization performance with M-1 convolution layers. The ability of the proposed method to accurately localize speakers in a dynamic acoustic scenario with varying number of sources is also shown.

I. INTRODUCTION

The paper develops a CNN-based supervised approach for estimating multiple speakers’ DOAs, motivated by the difficulty of localization under noise, reverberation, and multiple simultaneous sources. It extends prior work with synthesized-noise training and investigates architectural and robustness properties.

  • DOA information supports enhancement, camera steering, navigation, and distant speech applications, but accurate estimation is difficult under noise, reverberation, and multiple sources.
  • Supervised learning methods can adapt to different acoustic conditions through training and become robust to noise and reverberation when trained on diverse scenarios.
  • The method uses synthesized noise signals to generate multi-speaker training data, addressing the difficulty of nonlinear phase combinations from overlapping noise signals.
  • The proposed multi-speaker formulation estimates frame-level posterior probabilities for DOA classes, averages them across a block, and selects the L highest-probability classes when the speaker count is known.
  • The CNN design investigates whether M −1 convolution layers are needed to aggregate phase correlations across all microphone pairs and how this affects localization and parameter count.
  • The study evaluates generalization to unseen acoustic conditions and noise types, including scenarios with more than two speakers at the block level.

II. PROBLEM FORMULATION

The framework formulates multi-source DOA estimation as frame-level multi-label classification using phase-based STFT features, then aggregates posterior probabilities over configurable signal blocks. A CNN learns localization features directly from microphone phase information, while the source count is assumed known for final DOA selection.

  • II. PROBLEM FORMULATION: The framework targets mapping recorded microphone signals to multiple simultaneously active speakers’ DOAs using a CNN trained on labeled data.
  • II. PROBLEM FORMULATION: Multi-source DOA estimation is formulated as an I-class multi-label classification problem with each DOA class handled as a separate binary classification task.
  • II. PROBLEM FORMULATION: Frame-level posterior probabilities are averaged over a chosen block length, allowing shorter blocks for dynamic scenes and longer blocks for static sources.
  • II. PROBLEM FORMULATION: The final estimates select the L DOA classes with the highest probabilities, while the current formulation assumes that the number of sources L is known.
  • III. INPUT REPRESENTATION: The input representation uses the phase component of received microphone STFT coefficients so the CNN can learn DOA-relevant features during training.
  • III. INPUT REPRESENTATION: For each STFT time frame, phase values across microphones and frequency bins are arranged into a phase-map feature matrix.
  • III. INPUT REPRESENTATION: The CNN receives each phase map and produces posterior probabilities for the I DOA classes.

IV. DOA ESTIMATION WITH CNNS

The CNN learns DOA-discriminative features from phase-map inputs using local microphone correlations, then estimates multiple speakers through multi-label posterior classification and block-level aggregation.

  • Small 2 × 1 filters learn phase correlations between neighboring microphones separately at each frequency sub-band.The design avoids neighboring-frequency mixing, which may contain dominant activity from different speakers.
  • With M microphones, M −1 convolution layers aggregate phase-correlation information across all microphone pairs.The growing receptive field combines correlations from progressively broader microphone neighborhoods.
  • The CNN uses I sigmoid output units, assigning each DOA class a separate binary classification objective trained with binary cross-entropy.
  • For each time block, frame-level posterior probabilities are averaged across N frames for every DOA class.
  • The L DOA classes with the highest averaged posterior probabilities are selected as the final estimates.The paper uses this simple selection procedure rather than advanced peak-detection post-processing.

V. TRAINING DATA GENERATION

The paper generates multi-speaker CNN training data from synthesized noise and room impulse responses by exploiting approximately disjoint speech activity in the time-frequency domain.

  • Speech training signals are replaced with synthesized noise because frame-level training with speech would require highly accurate voice activity detection.For overlapping noise sources, nonlinear phase mixing makes CNN learning difficult.
  • The method assumes W-disjoint orthogonality, meaning two simultaneously active speech sources approximately do not overlap in their time-frequency representations.
  • Single-speaker training signals are formed by convolving direction-specific room impulse responses with spectrally white noise, then combining signals from different DOAs.
  • Time-frequency-bin randomization is performed separately within each frequency sub-band so spectral structure and frequency-dependent phase correlations are preserved.
  • The randomized representation yields approximately balanced activity from two DOAs at each time frame, so each phase-map sub-band corresponds to one source.
  • Training data are generated across angular combinations and acoustic conditions, including different rooms, positions, and spatially white-noise levels.
  • The CNN is trained on two-speaker frame-level DOA posteriors, while the same procedure can be extended to more than two speakers per frame and can estimate more than two speakers per block.

VI. EXPERIMENTAL EVALUATION

The evaluation uses simulated and measured acoustic data with controlled microphone-array, STFT, angular-resolution, and room configurations to test the proposed localization system.

  • Experiments generally use a four-microphone ULA with 8 cm spacing, 512-point DFTs, 50% overlap, 16 kHz sampling, and 37 DOA classes at 5° resolution.
  • Evaluation mixtures contain randomly selected LIBRI speech utterances forming five two-speaker mixtures, each lasting 2 s.
  • The central 0.8 s of each mixture is evaluated, producing N = 50 STFT time frames whose posteriors are averaged for final DOA estimation.
  • Training-data rooms are specified as 2.7 m high, while the test-data configuration for Sections VI-B1 and VI-B2 uses rooms 3 m high.

A. Baselines and objective measures

The evaluation compares the proposed method with SRP-PHAT and broadband MUSIC using mean absolute error and localization accuracy. Accuracy counts a speech mixture as correctly localized only when every speaker’s DOA error is at most 5°.

  • Baselines: The proposed method is compared with SRP-PHAT and broadband MUSIC.Broadband MUSIC averages its pseudo-spectrum across frequency sub-bands and time frames using 5° angular resolution.
  • Objective measures: Mean Absolute Error (MAE) and localization accuracy (Acc.) are the two objective evaluation measures.MAE measures the difference between true and estimated DOAs for each evaluated acoustic condition.
  • Objective measures: Localization accuracy requires every speaker in a speech segment to have an estimated DOA within 5° of its true DOA.The accuracy measure uses the number of speech mixtures satisfying this all-speaker criterion.

B. Experiments with simulated RIRs

Simulated-room experiments test generalization across unseen acoustic conditions and spatially white noise levels. The proposed method remains accurate and outperforms MUSIC and SRP-PHAT across both rooms and noise levels.

  • 1) Generalization to unseen acoustic conditions:: The experiments evaluate two rooms with different reverberation times, four array positions per room, and varying spatially white noise.For each room and array position, evaluation includes 3150 speech mixtures; SNRs are 10, 20, and 30 dB.
  • 1) Generalization to unseen acoustic conditions:: At 30 dB input SNR, the proposed method accurately localizes both sources in 98% of speech mixtures and has very low MAE.Performance worsens as noise increases but remains better than both MUSIC and SRP-PHAT.
  • 1) Generalization to unseen acoustic conditions:: The proposed method provides much higher localization accuracy and lower error than MUSIC and SRP-PHAT at all tested spatially white-noise levels in both rooms.Its performance is relatively similar across the two rooms, while the signal-processing baselines perform better in the less reverberant room.

2) Generalization to unseen noise type:

The method is evaluated with diffuse babble noise, which differs spatially and spectrally from the white noise used during training. It remains accurate at very low SNR and outperforms signal-processing baselines, especially under difficult babble-noise conditions.

  • 2) Generalization to unseen noise type:: Diffuse babble noise is unseen during training and has different spatial and spectral characteristics from the trained-on spatially white noise.The experiment adds babble noise at input SNRs of -5 dB, 0 dB, and 5 dB, with spatially white noise fixed at 40 dB SNR.
  • 2) Generalization to unseen noise type:: Approximately 90% of speech mixtures achieve accurate localization of both speakers at -5 dB babble-noise SNR in both rooms.The babble field is isotropic and spherically diffuse, so its spatial coherence varies with frequency.
  • 2) Generalization to unseen noise type:: The proposed method has much better localization accuracy and lower error than signal-processing methods, with the largest difference at low babble-noise SNRs.The authors attribute the relative robustness to extracting localization features from higher-SNR frequency regions when babble noise dominates low frequencies.

3) Influence of source-array distance:

The source-array-distance experiment evaluates localization from 0.4 m to 3 m under a far-field-trained CNN. Performance is best near the two training distances and degrades mainly when sources are very close to the array.

  • 3) Influence of source-array distance:: Source-array distances from 0.4 m to 3 m are evaluated in a simulated 10 × 11 × 3 m3 room with reverberation time 0.38 s.The evaluation uses three array positions, one two-speaker mixture, and spatially white noise at 20 dB input SNR.
  • 3) Influence of source-array distance:: The minimum error and maximum localization accuracy occur at the 1 m and 2 m distances used during training.Measures at each plotted distance are averaged over all angular combinations and array positions.
  • 3) Influence of source-array distance:: Very short source-array distances produce higher localization error because the CNN was trained for far-field conditions while these sources are in the near field.Distances between or beyond the training values cause only slight deterioration, whereas performance degrades more clearly when sources are very close.

4) Influence of number of convolution layers:

The study varies convolutional depth across microphone arrays and finds that M −1 layers provide the best localization performance while aggregating phase correlations across microphones.

  • Influence of number of convolution layers:: M −1 convolution layers are required for the best localization performance with an array of M microphones.Networks with fewer layers degraded in both MAE and localization accuracy.
  • Influence of number of convolution layers:: For each tested array, decreasing the depth below M −1 degraded both MAE and localization accuracy.With 2 × 1 filters, fewer layers fail to incorporate phase-correlation information from all microphone pairs.
  • Influence of number of convolution layers:: The array with more microphones achieved better localization performance, attributed to its larger aperture.The comparison covered arrays with M = 4, M = 6, and M = 8.
  • Influence of number of convolution layers:: Using fewer than M −1 layers increased trainable parameters because the final feature maps remained larger.The authors note that larger parameter counts can increase overfitting risk when training data are insufficient.
  • Influence of number of convolution layers:: More aggressive receptive-field expansion could potentially replace the M −1-layer requirement, but this alternative is beyond the paper’s scope.The stated requirement is tied to the slowly growing receptive field of the small filters.

C. Experiments with measured RIRs

Measured-room experiments evaluate the method under reverberation, source-distance changes, and dynamic source counts. The proposed approach maintains strong localization and produces clearer DOA peaks than the signal-processing baselines.

  • C. Experiments with measured RIRs: Even when trained only with simulated data, the proposed method provided very good localization in measured, reverberant conditions and at distant source positions.The measured RIR database included RT60 values of 0.160, 0.360, and 0.610 s; mixtures used 30 dB input SNR.
  • C. Experiments with measured RIRs: The proposed method performed significantly better than MUSIC and SRP-PHAT, with the difference becoming more prominent as reverberation increased.The comparison used measured room impulse responses.
  • C. Experiments with measured RIRs: 36?
  • 1) Dynamic acoustic scenario:: The method handled a dynamic scenario in which the number of active speakers changed over time.The evaluation used a 6 s mixture with sequential one-, two-, three-, and one-speaker segments.
  • 1) Dynamic acoustic scenario:: The proposed method produced clearer peaks at the true DOAs than MUSIC, including all three active DOAs in the three-speaker segment.For one-speaker segments, the highest peaks corresponded to the true DOA.

VII. CONCLUSION

The paper presents a CNN method trained with synthesized noise for multi-speaker DOA estimation and reports strong performance in unseen acoustic conditions and noise types. It also identifies M −1 convolution layers as the best depth for an M-microphone array.

  • VII. CONCLUSION: The proposed CNN estimates multiple-speaker DOAs and is trained using synthesized noise signals.The approach was evaluated for adaptation to unseen acoustic environments and robustness to unseen noise types.
  • VII. CONCLUSION: The method provided excellent localization performance in unseen acoustic environments and with unseen noise types, outperforming SRP-PHAT and MUSIC.
  • VII. CONCLUSION: For an array with M microphones, M −1 convolution layers gave the best localization performance and fewer trainable parameters.This depth aggregates phase-correlation information from all microphone pairs when using contiguous small filters.
Loading 1807.11722v1…